SYSTEM AND METHOD FOR DETECTING OBJECTS - Patent application

Abstract

The technology described herein relates to computerized methods and apparatus for detecting objects in an image. The technology described herein further relates to computerized methods and apparatus for detecting one or more objects using a pre-trained machine learning model and one or more other machine learning models that may be trained in a field training process. The pre-trained machine learning model may be a deep machine learning model.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims priority under 35 U.S.C. §119(e) to U.S. Provisional Application No. 63 / 253,496, entitled “SYSTEMS AND METHODS FOR DETECTING OBJECTS,” filed on October 7, 2021, which is incorporated by reference herein in its entirety.

[0002] The present technology relates to machine vision systems and methods, and more particularly to systems and methods for detecting objects. [Background technology]

[0003] It may be desirable to detect objects, such as characters, in an image. Various techniques may be used to detect objects in an image. Optical character recognition (OCR) technology is often used in many machine vision systems to detect text associated with various manufacturing processes, such as, for example, text printed on and / or affixed to machine parts. However, setting parameters for a given OCR application may be difficult, especially for a new user. For example, for character recognition, a user may select an area around an OCR string to indicate to the machine vision system that the character is for the OCR process. If the system does not correctly recognize the character, it may be difficult for a user to manually troubleshoot the problem. Furthermore, unlike some applications where such problems can be solved in advance (e.g., by a system integrator), adjusting parameters as part of the manufacturing process often requires a technician or engineer to troubleshoot such problems (e.g., train or modify run-time parameters to provide better OCR) in the actual production field. Such training or adjustments may be required, for example, when manufacturing new parts or using new print or labels. Summary of the Invention

[0004] The present disclosure relates to techniques for detecting objects, such as characters, in an image. Some aspects of the described techniques provide a computerized method, a non-transitory computer-readable medium, and / or a system for recognizing one or more objects in an image. The method includes determining a feature map of the image. The method may determine the feature map using a pre-trained machine learning model. The pre-trained machine learning model may be a deep machine learning model. The method further includes processing the feature map of the image using a first machine learning model to generate an object-centered heat map for the image, the object-centered heat map including a plurality of samples each having a value indicating a likelihood that a corresponding sample in the image is a center of an object. The method further includes determining a location of one or more objects in the image based on the object-centered heat map. In some examples, the method may further include processing the location of the one or more objects in the image using a second machine learning model and the feature map to recognize at least one of the one or more objects. The first and second machine learning models may be trained in a field training process.

[0005] In some embodiments, recognizing at least one of the one or more objects includes generating an object feature vector using a portion of a feature map of the image, where the portion of the feature map is based on a region surrounding a location of the object; processing the object feature vector using a second machine learning model to generate a class vector, where the class vector includes a plurality of values, each corresponding to one of a plurality of known labels; and using the class vector to classify the object into a label of the plurality of known labels.

[0006] In some embodiments, each value in the class vector indicates a predicted score associated with a corresponding one of a plurality of known labels, and classifying the object includes selecting a maximum value from among a plurality of values ​​in the class vector, the selected value corresponding to the label.

[0007] In some embodiments, the plurality of known labels includes a plurality of text character labels.

[0008] In some embodiments, the plurality of known labels further includes a background label.

[0009] In some embodiments, the method further includes training each of the first machine learning model and the second machine learning model using the respective machine learning method and using the respective set of field training data.

[0010] In some embodiments, the one or more objects include one or more text characters printed on a part included in the image, and the method further includes tracking the part using at least one text character recognized from the one or more text characters.

[0011] In some embodiments, determining the locations of the one or more objects includes smoothing the object centered heatmap to generate a smoothed object centered heatmap and selecting the locations of the one or more objects, wherein the value of each location in the smoothed object centered heatmap is higher than the values ​​of areas adjacent to the location.

[0012] In some embodiments, smoothing the object-centered heatmap comprises applying a Gaussian filter with a standard deviation proportional to the object size.

[0013] In some embodiments, selecting the locations further comprises filtering one or more locations where the value of the smoothed object center heatmap is below a threshold.

[0014] In some embodiments, the first machine learning model includes a weight vector, a feature map of the image includes a number of samples each associated with a respective feature vector, and a value of each sample in the object-centered heatmap is a dot product of the feature vector of a corresponding sample in the feature map and the weight vector.

[0015] In some embodiments, determining the feature map of the image includes processing the image using a pre-trained neural network model to generate the feature map of the image.

[0016] In some embodiments, the method further comprises capturing the image using a 1D barcode scanner or a 2D barcode scanner.

[0017] In some embodiments, a non-transitory computer-readable medium is provided that includes instructions operable, when executed by one or more processors on a computing device, to cause the one or more processors to perform any of the methods described above.

[0018] In some embodiments, a system is provided that includes a scanner including an image capture device configured to capture an image of a part on an inspection station, and a processor configured to execute programming instructions to perform operations including any of the methods described above.

[0019] Some aspects of the described techniques provide computerized methods, non-transitory computer-readable media, and / or systems for recognizing one or more characters in an image. In some embodiments, the method includes processing the image using a pre-trained machine learning model to generate a feature map of the image, processing the feature map of the image using a first machine learning model to generate a character centered heatmap for the image, the character centered heatmap including a plurality of samples, each having a value indicating a likelihood that a corresponding sample in the image is a center of a character, and processing the feature map of the image and the character centered heatmap for the image using a second machine learning model to recognize one or more characters in the image.

[0020] In some embodiments, recognizing at least one character of the one or more characters includes generating a character feature vector using the image feature map and the character center heatmap of the image; processing the character feature vector using a second machine learning model to generate a class vector, the class vector including a plurality of values, each corresponding to one of a plurality of known labels; and classifying the character into a label of the plurality of known labels using the class vector.

[0021] In some embodiments, each value in the class vector indicates a predicted score associated with a corresponding one of a plurality of known labels, and classifying the character includes selecting a maximum value from among a plurality of values ​​in the class vector, the selected value corresponding to the label.

[0022] In some embodiments, the plurality of known labels further includes a background label.

[0023] In some embodiments, the method further includes determining a location of one or more characters in the image based on the character center heatmap, and generating a character feature vector for the at least one character includes generating the character feature vector using a portion of the feature map of the image, the portion of the feature map being based on a region surrounding the location of the character.

[0024] In some embodiments, determining the locations of the one or more characters includes smoothing the character center heatmap to generate a smoothed character center heatmap and selecting locations of the one or more characters, where each location's value in the smoothed character center heatmap is higher than values ​​in areas adjacent to the location.

[0025] In some embodiments, smoothing the character center heatmap includes applying a Gaussian filter with a standard deviation proportional to the character size.

[0026] In some embodiments, selecting the locations further comprises filtering one or more locations where the value of the smoothed character center heatmap is below a threshold.

[0027] In some embodiments, the first machine learning model includes a weight vector, a feature map of the image includes a number of samples each associated with a respective feature vector, and a value of each sample in the character-centric heatmap is a dot product of the feature vector of a corresponding sample in the feature map and the weight vector.

[0028] In some embodiments, the method includes training a second machine learning model by obtaining a plurality of training images and a plurality of training labels, each of the plurality of training labels associated with a corresponding one of the plurality of training images, determining a plurality of training feature maps each using the plurality of training images and one of the pre-trained machine learning models, and determining weights for the second machine learning model using the plurality of training feature maps and the plurality of training labels.

[0029] In some embodiments, determining the weights of the second machine learning model includes: (1) for each training label of the plurality of training labels, determining a corresponding character feature vector using a portion of a feature map of a corresponding training image with which the training label is associated, and determining a target vector; and (2) determining weights of the second machine learning model using the character feature vectors and the target vector of the plurality of training labels.

[0030] In some embodiments, determining the weights of the second machine learning model includes using a machine learning method on the character feature vector and the target vector.

[0031] In some embodiments, the method further includes, for each training label of the plurality of training labels, (1) obtaining a location of the training label in a corresponding training image, and (2) determining a corresponding character feature vector based on the location of the training label using a portion of the feature map of the corresponding training image.

[0032] In some embodiments, a portion of a feature map is represented by a bounding box centered on the location of the training label in the corresponding training image.

[0033] In some embodiments, determining the corresponding character feature vector includes concatenating multiple sub-feature vectors, each formed based on a respective position within a portion of the feature map.

[0034] In some embodiments, the method further includes training a first machine learning model by determining a plurality of ground truth character centered heatmaps of the plurality of training feature maps and determining weights for the first machine learning model using the plurality of training feature maps and the plurality of ground truth character centered heatmaps.

[0035] In some embodiments, determining the weights of the first machine learning model includes determining the weights of the first machine learning model using a machine learning method.

[0036] In some embodiments, determining a plurality of ground truth character center heatmaps includes a plurality of samples, each having a value based on a distance to a nearest ground truth location of the training label in the training image.

[0037] In some embodiments, the method further includes receiving, via the graphical user interface, a one-to-one mapping between characters in the plurality of training images and character labels for each of the one or more characters in the plurality of training images, and generating a plurality of training labels based on the received one-to-one mapping for the one or more characters in the plurality of training images.

[0038] In some embodiments, the method further includes receiving, via the graphical user interface, a correction to the one or more recognized characters, updating the plurality of training labels based on the correction, and retraining the first machine learning model and the second machine learning model further based on the updated plurality of training labels.

[0039] In some embodiments, the one or more characters include one or more text characters printed on the part included in the image, and the method further includes tracking the part using at least one text character recognized from the one or more text characters.

[0040] In some embodiments, a non-transitory computer-readable medium is provided that includes instructions operable, when executed by one or more processors on a computing device, to cause the one or more processors to perform any of the methods described above.

[0041] In some embodiments, a system is provided that includes a scanner including an image capture device configured to capture an image of a part on an inspection station, and a processor configured to execute programming instructions to perform any of the methods described above.

[0042] It will be appreciated that all combinations of the foregoing concepts, and additional concepts described in more detail below, provided such concepts are not mutually inconsistent, are contemplated as part of the inventive subject matter disclosed herein, and in particular, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as part of the inventive subject matter disclosed herein.

[0043] Additional embodiments of the present disclosure, as well as its features and advantages, will become more apparent by reference to the following description taken in conjunction with the accompanying drawings, in which components are not necessarily drawn to scale. Moreover, in the drawings, like reference numerals designate corresponding parts throughout the different views. [Brief description of the drawings]

[0044] [Figure 1] FIG. 1 is a diagram of a system for detecting objects in accordance with some embodiments of the techniques described herein.

[0045] [Diagram 2] FIG. 2 is another diagram of a system for detecting objects in accordance with some embodiments of the techniques described herein.

[0046] [Diagram 3] FIG. 3 is a flowchart of an example process for detecting an object in accordance with some embodiments of the techniques described herein.

[0047] [Figure 4] FIG. 4 is a flowchart of an exemplary process for detecting characters in accordance with some embodiments of the techniques described herein.

[0048] [Diagram 5] FIG. 5A is a flowchart of an example process for training a machine learning model used for character detection according to some embodiments of the techniques described herein.

[0049] FIG. 5B is a flowchart of an example process for training another machine learning model for use in character detection according to some embodiments of the techniques described herein.

[0050] [Figure 6] FIG. 6 is an example part having detectable characters printed thereon in accordance with some embodiments of the techniques described herein.

[0051] [Figure 7] FIG. 7 shows an example graphical user interface for correcting labels in the training process according to some embodiments of the techniques described herein.

[0052] [Figure 8] FIG. 8 illustrates an exemplary implementation of a computer system that may be used to implement any of the aspects of the techniques and embodiments disclosed herein, in accordance with some embodiments of the techniques described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0053] For the purposes of promoting an understanding of the principles of the present disclosure, reference will now be made to the embodiments illustrated in the drawings and specific language will be used to describe the same, it being understood nevertheless that no limitation of the scope of the invention is intended thereby.

[0054] For various machine vision applications, it may be desirable to detect objects, such as characters, in an image. For example, it may be desirable to detect text printed on and / or affixed to a machine part as part of a manufacturing or inspection process. Various techniques may be used to detect objects for machine vision applications. As an example, an OCR application in a manufacturing process may include recognizing a printed label having one or more text characters depicting a part number. In the present application, an image may be captured that includes a part having a printed label. OCR techniques may be used to determine the location of one or more characters in the printed label and recognize the characters. The inventors have recognized and appreciated that applying conventional object detection techniques to the above-mentioned manufacturing processes is particularly challenging and that such conventional techniques may be improved upon. For example, images captured during a manufacturing process may be of poor quality (e.g., compared to scanned documents) in conventional object detection systems due to the various conditions under which the images are captured. Examples of these various conditions may include, for example, the movement of the part being captured at the site, the distance between the camera and the part, the unknown or uncontrollable pose of the part at the site, and / or the lighting conditions of the part at the site. Such conditions may affect, for example, the contrast of the image. For text, for example, the distance between the camera and the part may affect the size of the text in the image. As another example, variations in the position of a label printed on a part may affect the position of the text in the captured image. These and other conditions during the manufacturing process have presented challenges in applying such conventional techniques.

[0055] Some object detection systems are designed to fit a particular application or the environment in which the application will run by training specific system parameters. The training data used to train the system parameters is collected from the target application domain. For example, the training data may be collected or generated in a lab environment that simulates the conditions in which the system will be deployed. However, this approach has several drawbacks. First, the environment in which the object detection system is trained and developed (e.g., in a lab environment) may be quite different from the environment in which the object detection system will be deployed in the field. Second, collecting training data for a particular application or environment may be expensive.

[0056] Some object detection systems, such as OCR systems, are designed to be adaptable to specific applications to provide improved object detection performance by allowing users to adjust certain parameters. However, these systems have drawbacks because adjusting the parameters typically requires a system integrator, engineer, or even personnel with knowledge of the mechanics of the object detection system. For example, in a typical system, there may be 20-30 parameters that may require adjustment. Adjusting these parameters would be beyond the skill level of most users.

[0057] Some object detection systems use machine learning techniques. For example, deep machine learning techniques have been used in OCR systems to detect characters from images. However, the inventors have recognized and understood that conventional systems for object recognition using machine learning techniques typically provide little or no flexibility to be able to adjust the machine learning model used during deployment. For example, machine learning (e.g., deep machine learning) techniques inherit the same drawbacks as mentioned above because machine learning techniques generally require huge training datasets, such as hundreds, thousands, or tens of thousands of training images and associated ground truth data (e.g., providing information about objects contained in the images). Such huge training datasets may be difficult and / or expensive to collect. Furthermore, when using deep machine learning techniques, the training process may even be very time-consuming considering the large number of images and the large number of weights in the deep machine learning model. For example, in a convolutional neural network (CNN), there may be tens of thousands of weights. Thus, it is generally not feasible to deploy these machine learning-based techniques in a manner that matches the machine learning models in the field. Once in the field, there may not be sufficient training data and / or computational resources to train and / or retrain these systems in a reasonable period of time (e.g., the training or tuning process is fast enough so as not to result in a degradation of the user experience).

[0058] Further, the inventors have recognized and appreciated that conventional machine learning based systems often use a black box approach to process images from end to end. For example, conventional systems receive an image as an input and generate object detection results as output, such as OCR results, but do not provide visibility into other parts of the machine learning model and / or processing pipeline used to generate the object detection results. Additionally or alternatively, such systems do not provide for recognizing individual objects when multiple objects are present in the scene. For example, OCR systems do not provide for recognizing individual characters of a string of characters or for generating intermediate results or data of the processing pipeline, such as providing data indicating the location of the characters within an image that is processed by OCR (e.g., which is then used to configure the OCR process). Instead, these systems typically only provide recognition for blocks of text, such as a line or lines of text within an image.

[0059] Therefore, the inventors have developed the techniques described herein to provide an object detection system that can be easily and quickly trained in the field. In some embodiments, the system can be provided to include one or more machine learning models that can be trained in the field. In contrast to using complex machine learning models in a black box approach, using multiple machine learning models may allow the system to use smaller and simpler machine learning models, each focused on extracting specific features of an image.

[0060] Object detection may include operations for detecting one or more objects within an image, video, and / or any other suitable media asset. Examples of object detection may include determining one or more objects and / or recognizing objects within an image, video, and / or any other suitable media asset. For example, in a manufacturing process where an image may include one or more machine parts moving on an assembly line, object detection may include locating one or more parts within an image. Object detection may also include recognizing one or more parts and respective known classes corresponding to each recognized part.

[0061] It is understood that the one or more objects in an image may include various types of items. Further examples of objects detected from an image may include text (e.g., characters in a printed label), graphics, symbols, icons, bar codes, machines, human faces, animals, pieces of furniture, labels, landmarks, or any other suitable textual / graphical representations, or combinations thereof. Objects included in an image may include any color or combination of colors. Text characters may be in English, or any other suitable language. Thus, the techniques described herein may be applied to detecting various types of items, including characters, in an image. Characters may be either textual or graphical, and may include any symbol. Examples of characters may include any character in the ASCII set, English characters of any suitable language, or non-textual symbols, such as mathematical or other symbols.

[0062] Thus, the described techniques include systems, computerized methods, and non-transitory instructions for processing an image to detect one or more objects in the image. In some embodiments, the system may obtain an image including one or more objects and process the image to determine a feature map of the image using a pre-trained machine learning model. In some examples, the pre-trained machine learning model may be a deep machine learning model, such as a convolutional neural network (CNN) having one or more hidden layers, or any other suitable model. The pre-trained machine learning model may be configured to code features of a portion of the image for subsequent processing to output a feature map. The feature map may include a plurality of samples, each comprising a vector (e.g., a feature vector) having a plurality of values ​​that code semantic information about the image at each location. The size of the feature map may differ from the size of the image. For example, the feature map may be smaller than the image, and its size is 1 / 4 or 1 / 8 of the size of the image.

[0063] In some embodiments, the system may further process the feature map of the image using a first machine learning model to generate an object-centered heatmap for the image. The first machine learning model may be trained in a field training process. The object-centered heatmap may be a representation of the image depicting the likelihood that one or more pixels in the image are the center of an object. For example, the object-centered heatmap may include a plurality of samples, each having a value that indicates the likelihood that the corresponding sample in the image is the center of the object. Thus, samples near the center of an object in the image may have a higher value in the object-centered heatmap than samples in other regions in the image. In some examples, the first machine learning model may include a weight vector that includes a plurality of weights in the vector. The size of the object-centered heatmap may have the same size as the feature map. In some embodiments, the value of each sample in the object-centered heatmap may be a dot product of the feature vector of the corresponding sample in the feature map and the weight vector. In some examples, the weight vector may have a dimension value of K, i.e., the weight vector may have a value of K, where K is the number of known labels for one or more objects in the image.

[0064] In some embodiments, the system may further determine the location of one or more objects based on the values ​​of corresponding samples in the object center heatmap. For example, a list of locations in the image may be selected as object centers by filtering out locations where the values ​​of corresponding samples in the object center heatmap are below a threshold. In some embodiments, the system may apply a filter to the object center heatmap to generate a smoothed object center heatmap and use the smoothed object center heatmap to select the location of one or more objects. In some examples, the filter may be a Gaussian filter, such as an anisotropic Gaussian filter. The size of the Gaussian filter may be determined based on the size of the objects in the image. For example, to detect text character labels, the size (e.g., width and height) of the Gaussian filter may be determined based on character sizes, which may be trained and stored in the system.

[0065] In some embodiments, the system may recognize one or more objects in an image by processing the one or more object locations in the image using a second machine learning model and a feature map. The second machine learning model may also be trained during a field training process. When recognizing an object, at each location from the one or more object locations, the system may generate an object feature vector. The object feature vector may be a representation of the object that describes certain features of the object, such as features extracted from the image using a pre-trained machine learning model. For example, the object feature vector may be generated using a portion of the image's feature map, where the portion of the feature map is based on an area surrounding each object location. The second machine learning model may include a weight matrix that includes multiple weights in the matrix. The weights in the weight matrix may be trained. In some examples, the system may recognize one or more objects by processing the object feature vector using the second machine learning model to generate a class vector and classifying each object into a label of multiple known labels using the class vector. The class vector may be a representation of known classes for one or more objects (e.g., characters, machine parts, or other objects). The class vector may include multiple values, each corresponding to one of multiple known labels for the object to be detected.

[0066] In some aspects, the above-described techniques may be applied to problems of character detection (e.g., OCR). For example, a first machine learning model may be configured to extract certain features of an image that allow the system to detect character locations within the image. A second machine learning model may be configured to extract certain other features of an image that allow the system to recognize characters within the image. Each of the first and second machine learning models may include much fewer weights than complex machine learning models and may therefore be quicker to train. Furthermore, training these smaller machine learning models may not require large training sets, making it more feasible to train these models in the field.

[0067] In some embodiments, the system may use the pre-trained machine learning model to extract a feature map of the image for further processing steps. The feature map may be a representation of the image and may include specific features within the image. For example, the feature map may include a plurality of samples, each including a vector having a plurality of values ​​that code semantic information at a corresponding location in the image. In some examples, the pre-trained machine learning model may be a deep machine learning model, such as a CNN or other neural network model. The pre-trained machine learning model may be trained offline using a large collection of training data. Once trained, the pre-trained machine learning model does not need to be tuned and can be deployed in field applications. Using the pre-trained machine learning model to generate the feature map of the image takes advantage of the deep machine model's advantages in its robustness and immunity to noise.

[0068] In some embodiments, the system may combine different types of machine learning models. For example, the inventors have recognized that it may be desirable to (1) use a pre-trained machine learning model to extract certain semantic features in an image, and (2) use other machine learning models to extract other features of the image, where the other machine learning models can be quickly trained when deployed in the field. For example, the pre-trained machine learning model may be a deep machine learning model, e.g., a CNN, or other neural network model. The other machine learning models may include the first machine learning model described above, and a second machine learning model. In some embodiments, the system may use the feature map obtained from the first machine learning model and the pre-trained machine learning model to detect the location of characters. Furthermore, the system may use the feature map obtained from the second machine learning model and the pre-trained machine learning model to recognize characters in an image. This combination of deep machine learning models (e.g., which may be pre-trained, taking into account the time and / or computational power required to train such models) with other types of models that may be trained at the time of deployment thus provides the flexibility to tailor the system to specific field applications, while gaining the benefits of deep machine learning models in system performance and robustness. This may overcome some or all of the shortcomings in conventional systems, as discussed above.

[0069] In some aspects, some variations of the above-described embodiments may include systems, computerized methods, and non-transitory instructions that may be provided for processing an image including one or more characters and detecting the one or more characters. Detecting the one or more characters may include operations for detecting the location of the one or more characters. Furthermore, detecting the characters in the image may also include recognizing the one or more characters for each known label (class). In some embodiments, the system may obtain an image including one or more characters and process the image using a pre-trained machine learning model to generate a feature map of the image. The pre-trained machine learning model may be a deep machine learning model in some examples. The system may generate a character-centric heatmap by processing the feature map of the image using a first machine learning model. Similar to an object-centric heatmap, a character-centric heatmap may be a representation of an image that depicts the likelihood that one or more pixels in the image are the center of a character. For example, a character-centric heatmap may include a plurality of samples, each having a value that indicates the likelihood that the corresponding sample in the image is the center of a character. In some examples, if the image includes text characters, the value of each sample in the character-centric heatmap may indicate the likelihood that the corresponding sample in the image is the center of a text character. Thus, the character centrality heatmap may represent a "character centrality" score for each corresponding location of a character in the image. For example, the values ​​of samples in the character centrality heatmap may be high at locations proximal to the centers of the characters in the image and low at locations distal to the centers of the characters or in areas of the image that do not have characters. This may facilitate locating individual characters within the image.

[0070] In some examples, the system may determine the locations of one or more characters based on the values ​​of corresponding samples in the character center heatmap. For example, the system may select a list of locations in the image as character centers by filtering out locations where the values ​​of corresponding samples in the character center heatmap are below a threshold. In some embodiments, the system may apply a filter to smooth the character center heatmap and use the smoothed character center heatmap to select the locations of one or more characters. In some examples, the filter may be a Gaussian filter, e.g., an anisotropic Gaussian filter. The size of the Gaussian filter may be determined based on the size of the characters, e.g., pre-stored character sizes, which may be trained and / or stored. The resulting list of locations may include predicted centers of the characters in the image.

[0071] In some embodiments, the system may recognize one or more characters in an image by processing the positions of the characters in the image using the second machine learning model and the feature map. In recognizing a character, at each position from the positions of the one or more characters, the system may generate a character feature vector. Similar to the object feature vector, the character feature vector may be a representation of the character that depicts certain characteristics of the character, such as features represented in a feature map of the image corresponding to the character (obtained using the pre-trained machine learning model). For example, the character feature vector may be generated using a portion of the feature map of the image, the portion of the feature map based on an area surrounding each character position. The second machine learning model may include a weight matrix including multiple weights in the matrix. The weights of the weight matrix may be trained in situ. In some examples, the system may recognize one or more characters by processing the character feature vector using the second machine learning model to generate a class vector and classifying each character into a label of a plurality of known labels using the class vector. The class vector may be a representation of known classes for one or more characters. For example, the class vector may include multiple values, each corresponding to one of a plurality of known labels. In a non-limiting example, for an industrial part, the plurality of known labels may include the entire set of characters or symbols used on printed labels of the industrial part.

[0072] In some embodiments, the above-described system may include applying the results of detecting objects (e.g., characters) in one or more applications. For example, one or more objects in an image may include one or more printed text characters on a mechanical part included in the image. Thus, the system may be configured to recognize one or more text characters on the part and track the part using at least one recognized text character. In some embodiments, the system may be implemented in a barcode scanner, where the image capture device may be configured to capture an image including the part having a label printed thereon. The system may be configured to process the captured image by detecting the location of the barcode and decoding the located barcode.

[0073] As described above, various embodiments use a first machine learning model and a second machine learning model. In some embodiments, various techniques are described herein, including systems, computerized methods, and non-transient instructions, for training the first and / or second machine learning models in a field training process. In some embodiments, the training process may include an initial training process using an initial set of training data, and a retraining process using an updated set of training data. In the retraining process, the updated set of training data may include a portion of the initial set of training data and / or new training data, which may be obtained by a user, for example, via a user interface.

[0074] The field training process may include a training process in which the machine learning model being trained may be adapted to specific conditions associated with a specific field application at a specific customer site or production line. Examples of specific conditions include a specific font of characters or printing process used, specific lighting conditions, or a specific background, a specific type of image being processed, such as an image containing machine parts on an assembly line. Field training may also include training data that is different from the training data used to train the pre-trained machine learning model, with or without overlap. The field training process may include, for example, a user adding custom symbols (labels) and providing training images containing new symbols for training the machine learning model. Thus, field training may enable the system to achieve improved performance over a dedicated pre-configured system.

[0075] As an example, a training system that can be used to train the first and second machine learning models in object detection is described in detail. In some embodiments, the training system may train a first machine learning model used in object detection, and the first machine learning model may include a weight vector. The system may obtain a plurality of training images and training labels corresponding to objects in the training images. The training labels may include a list of ground truth object locations, each associated with an object in a corresponding one of the plurality of training images, indicating where the object appears in the image.

[0076] In some embodiments, the training system may determine a plurality of training feature maps each using a plurality of training images and one of the pre-trained machine learning models in a manner similar to how feature maps are determined from the aforementioned images in the system to detect characters / objects. For example, the pre-trained machine learning model may be a deep machine learning model. The system for training the machine learning model may further include determining a ground truth object-centered heatmap and using the training feature map and the ground truth object-centered heatmap to determine weights for the first machine learning model. In some embodiments, the value of each sample in the ground truth object-centered heatmap may be generated based on the distance to the nearest ground truth object location in the training image.

[0077] In some embodiments, for each object in the training image, the system may further determine a corresponding object feature vector using a portion of the training feature map of the corresponding training image. The object feature vector may be a representation of the object that describes certain characteristics of the object, and may be obtained in a similar manner as described above for character / object detection.

[0078] In some examples, the system may obtain the object location from the ground truth object location in the training image, and determine an object feature vector using a portion of the feature map of the corresponding training image based on the object location. The operation of determining the portion of the feature map may be similar to that described above in the system for detecting characters / objects. In determining the object feature vector, the system for training the machine learning model may use an alignment method to determine multiple sub-feature vectors, each based on a respective position of the portion of the feature map, and concatenate the multiple sub-feature vectors to form the object feature vector.

[0079] Thus, for every training image, the system may determine a number of object feature vectors and a number of ground truth object-centered heatmap values ​​according to the techniques described above. When determining the weights in the first machine learning model, the system may use a suitable machine learning method such that the dot product between the first machine learning model (e.g., the weight vector) and each object feature vector is as close as possible to the corresponding ground truth object-centered heatmap value. The method for training the weight vector may include linear regression, SVM, or other suitable machine learning method.

[0080] In some examples, a training system may be provided to train a second machine learning model used to detect objects (e.g., characters), and the second machine learning model may include a weight matrix. In some examples, the system may obtain a plurality of training images and training labels. Each of the plurality of training labels may be associated with a character in a corresponding one of the plurality of training images. For example, the training labels may include a list of ground truth character positions, each associated with a character in a corresponding one of the plurality of training images to indicate a position in the image where the character appears. The system may also include a ground truth character class list including a list of classes representing the identity of the character at each ground truth character position. For text characters, the character classes may correspond to text characters, such as "A" through "Z", or other labels that may be used in field applications. In some examples, the system may obtain the character class list by using a label encoding method. For example, the system may collect all character classes that appear in the training labels and discard repeated ones to generate a list of distinct character classes.

[0081] Further, the system may determine a training feature map using a pre-trained machine learning model in a manner similar to that described above in training the first machine learning model. For example, the pre-trained machine learning model may be a deep machine learning model. The system may further determine weights for the second machine learning model using the training feature maps and the training labels. When determining the weights for the second machine learning model, for each object in the training image, the system may determine a corresponding object feature vector using a portion of the feature map of the corresponding training image in which the object is located in a manner similar to that described above in training the first machine learning model.

[0082] Further, the system may also determine a target vector for each of the objects in the training images. The target vector may be a representation of an object (e.g., a character) that describes the association of the object with one or more known classes. In some embodiments, the target vector may be determined using an encoding method. For example, a one-hot encoding method may be used. Thus, after determining the object feature vector and the target vector for each of the ground truth object locations in the training images, the system may determine a second machine learning model (e.g., a weight matrix) such that when the second machine learning model (e.g., a weight matrix) is multiplied with each object feature vector, the product reproduces the corresponding target vector as closely as possible. Similar to the training of the first machine learning model, the weight matrix may be trained in the second machine learning model using a machine learning method such as linear regression, SVM, or other suitable machine learning method.

[0083] Although the techniques for training machine learning models are described using objects, such training techniques may also be applied to train machine learning models for types of objects (e.g., character detection). Although some examples described herein are described in conjunction with OCR techniques, it should be understood that such examples are for illustrative purposes only and are not intended to be limiting, as the techniques are not limited to OCR applications.

[0084] The techniques described herein may provide advantages over conventional systems in improving the performance of object / character detection. By using a pre-trained deep neural network as a feature extractor, the system may use a higher level of semantic information for object location and classification. Compared to systems that do not use deep neural networks, the systems described herein are more robust and less susceptible to image noise, object appearance variations, print quality variations, lighting and background differences, etc.

[0085] The techniques described herein may also provide advantages over conventional systems in further improving the performance of object / character detection by additionally using one or more other machine learning models that may be trained in a field training process. These machine learning models may be trained in a field training process and adapted to specific conditions that may occur in field applications. Thus, the system may allow users to adapt the system with custom-provided training images and / or updated labels. By training only a small portion of the system, other machine learning models may be trained efficiently without using expensive computational resources or large sets of training images, such as when training deep machine learning models using simple machine learning methods such as linear regression or other suitable methods.

[0086] While various embodiments have been described, it will be apparent to those skilled in the art that many more embodiments and implementations are possible. Thus, the embodiments described herein are examples, rather than the only possible embodiments and implementations. Moreover, the advantages described above are not necessarily the only advantages, and it is not necessarily expected that all described advantages will be achieved in all embodiments.

[0087] FIG. 1 is a diagram of a system for detecting objects in accordance with some embodiments of the techniques described herein.

[0088] In some embodiments, the system 100 may include an image capture device 112, such as a camera, configured to capture a video or image of the part 104 or any component on the inspection platform 103. The captured video may include a sequence of frame images, each of which may include one or more parts 104 on the inspection platform 103. The system 100 may further include a server 110 having at least a processor, memory, storage medium, and / or other components. The at least one processor may be configured to execute programming instructions stored in the memory to analyze the images and perform object detection on each of the images to detect one or more objects in the images. FIG. 6 is an exemplary part having detectable characters printed thereon, in accordance with some embodiments of the techniques described herein. In FIG. 6, one or more objects 602 may include text character labels printed on the part 604. Of course, the objects in the images may not be limited to text character labels.

[0089] Returning to FIG. 1 , the server 110 may be configured to process captured images of one or more parts in real time during the inspection process. In a non-limiting configuration, the part 104 may be placed on the inspection platform 103. For example, the inspection platform 103 may be a conveyor belt of an assembly line or a turntable on which multiple parts to be inspected are placed. The inspection platform 103 may be controlled to move the parts. Each time the inspection platform 103 moves a part (e.g., part 104) to a location for inspection, the part on the inspection platform 103 is captured by the image capture device 112 such that the system 100 may process the captured image to detect one or more objects (e.g., text) on the part. Once detection is complete, the inspection platform 103 moves and the next part is moved into position for inspection, a new image is captured including the part just moved into position, and the new image is processed to detect one or more objects in the image. This inspection process is repeated for each part to be inspected. The server 110 may be configured to process the captured image of each part during the inspection process. Although the embodiments of Figures 1 and 6 show only one part 104 per captured image, it should be understood that multiple parts may be included in a captured image which may be processed to detect objects in each of the multiple parts, if desired.

[0090] The system 100 may be configured to detect one or more objects in an image. In some embodiments, detecting the one or more objects may include determining the location of the one or more objects in the image using one or more machine learning models. For example, for an input image of a part (e.g., an image captured by the image capture device 112), the system 100 may be configured to process the input image using a pre-trained machine learning model to extract a feature map of the image. The pre-trained machine learning model may include a previously trained and stored deep machine learning model. For example, the feature map may include a plurality of examples, which are vectors that encode semantic information at corresponding locations in the image.

[0091] In some embodiments, the system 100 may generate an object center heatmap from the feature map using a machine learning model (e.g., a first machine learning model). The object center heatmap may include a plurality of samples, each having a value that indicates the likelihood that a corresponding pixel in the image is the center of an object. Thus, in the object center heatmap, a sample close to the center of the object may have a value that is different from the value of another sample that is distal to the center of the object, and the difference in values ​​between the samples in the object center heatmap may be used to distinguish samples close to the center of the object from samples that are distal to the center of the object in the image. Thus, the server 110 may be configured to determine a center location of an object in the image based on the object center heatmap.

[0092] In some embodiments, detecting the one or more objects may further include recognizing one or more objects in the image by processing the location of the one or more objects in the image using another machine learning model (e.g., a second machine learning model) and a feature map obtained from the pre-trained machine learning model. In some examples, for each of the one or more located objects in the image, the system 100 may determine a portion of the feature map of the image that corresponds to an area surrounding the location of the object. The system 100 may use the portion of the feature map to classify the object into a label of a plurality of known labels. The label may include a representation of the object in the image. For example, the label may be an ASCII representation for a text character or any suitable value that may represent one of a plurality of classes. The plurality of known labels may vary depending on the application. For example, to detect a label on a part, the plurality of known labels may be limited to characters. In that case, the known label of the part may include both numeric and alphanumeric characters, and the known label may be converted into a character class list. Thus, the second machine learning model may be trained such that the second machine learning model may be configured to output a class of a plurality of known labels for the one or more objects in the image. The generation of the class list is further described in the field training process, such as 500 (in FIG. 5A).

[0093] In some embodiments, the first and second machine learning models described above in the system 100 may differ from the pre-trained machine learning model in that the pre-trained machine learning model is a deep machine learning mode, while the first and second machine learning models are not deep machine learning models. For example, the first and second machine learning models may each include a respective vector or matrix having multiple weights that can be trained quickly. This may be particularly useful for deploying the system 100 in domains where training a deep machine learning model (e.g., a pre-trained machine learning model) may be infeasible due to lack of computing resources and lack of training data at the user's site. Instead, the first and second machine learning models may be trained on-site to extract specific features specific to the application. For example, when recognizing a label on a particular part, a set of training datasets may be obtained by a user to train the first and second machine learning models, the training datasets including training images that include the part with a label thereon. Details of methods that may be implemented in the system 100 for detecting objects in images using one or more machine learning models and training the machine learning models are further described with reference to FIGS. 2-4 and 5A-5B.

[0094] 1, upon detection of one or more objects in the image, object data may be transmitted to user device 114. In some embodiments, the object data may include the location of one or more objects on a part included in the image. In other embodiments, the object data may include labels recognized from one or more objects in the image, such as text character labels on the part. User device 114 may be configured to use the object data obtained from server 110 to perform various actions associated with the portions in the image.

[0095] In a non-limiting example, one or more objects in an image may include one or more printed text characters on a part included in the image. Thus, system 100 may be configured to recognize one or more text characters on the part and track the part using at least one recognized text character. In some embodiments, system 100 may be implemented in a barcode scanner, where an image capture device (e.g., 112) may be configured to capture an image of a part having a label printed thereon. System 100 may be configured to process the captured image by detecting the location of the barcode and decoding the located barcode. Various arrangements of components in system 100 may be configured to enable specific applications. For example, user equipment 114 may be configured to be a barcode (e.g., 1D or 2D) scanner. User equipment 114 may include an image capture device 112 (e.g., a phone camera) and may capture an image of a part having a barcode thereon. The user equipment may be configured as a barcode scanner to detect the location of a barcode (e.g., a 1D or 2D barcode) in a captured image and decode the barcode in the image based on the detected location of the barcode. Various other applications may also be implemented in system 100.

[0096] In some embodiments, the user equipment 114 may be provided with a graphical user interface for displaying the object data. For example, the object data may include a recognized label on a part, and the user equipment 114 may display the recognized label along with an image including the part so that the user may determine whether the label is correctly recognized. Additionally, the user equipment 114 may be configured to allow the user to perform corrections if the label is incorrectly recognized. FIG. 7 illustrates an exemplary graphical user interface for correcting labels in a training / retraining process according to some embodiments of the techniques described herein. As shown in FIG. 7, a graphical user interface 700 that may be implemented in the user equipment 114 (FIG. 1) may include a portion 702 that displays each of the objects (e.g., characters) in the image in a first region 702-1 and displays the corresponding recognized label in a second region 702-2. As shown, the characters in the first region 702-1 and the recognized label in the second region 702-2 are aligned so that the user may determine whether the corresponding label is correctly recognized. In some embodiments, the user interface 700 may allow a user to correct an incorrectly recognized label. For example, in region 702-2, the recognized label may be displayed in an editable text box that allows the user to modify the label.

[0097] In some embodiments, object data transmitted from server 110 (FIG. 1) to a user equipment (e.g., 114 in FIG. 1) may include the location of each of the characters along with the recognized labels. The user equipment may use the location information received for each of the characters to segment the characters from the captured image and display them in a user interface 700, such as display area 702-1 in FIG. 7. Additionally, user interface 700 may also include area 704 configured to display a portion of the image that includes the characters.

[0098] Returning to FIG. 1, in some embodiments, user equipment 114 may be configured to transmit any user-modified information, such as modified labels entered by a user via graphical user interface 700 (FIG. 7), to server 110. This modified information may be used by server 110 to train one or more machine learning models used in system 100. Details of training methods that may be implemented in system 100 to detect objects in images are described further in this disclosure with reference to FIGS. 4 and 5A-5B.

[0099] 1, server 110 and user equipment 114 are shown performing object detection using a machine learning model and training the machine learning model, it is understood that a single computing device (e.g., a server, or a user equipment) may be configured to perform the detection / training operations. In other variations, the embodiments described herein may also be implemented on multiple computing devices in a distributed system.

[0100] FIG. 2 is a diagram of a system 200 for detecting objects using a machine learning model and training the machine learning model, according to some embodiments of the techniques described herein. The system 200 may include one or more blocks that may be implemented in the system 100 (FIG. 1). In some embodiments, the system 200 may include one or more blocks configured to perform object detection from an image using a machine learning model. The system 200 may also include one or more blocks configured to perform training of one or more machine learning models used in the detection operations. As shown in FIG. 2, the system 200 may be configured to operate in a detection mode and a field training mode. For example, in the detection mode, the system 200 may perform one or more detection operations using the pre-trained machine learning model 220 and other machine learning models (e.g., 222, 226). Furthermore, in the field training mode, the system 200 may be configured to train other machine learning models (e.g., 222, 226) in one or more field training processes.

[0101] A pre-trained machine learning model may be a machine learning model that is trained once before the machine learning model is deployed and used in the field (e.g., in a customer application). A pre-trained machine learning model may be trained at a different field than where the machine learning model is deployed in a field application. In some embodiments, once deployed, the pre-trained machine learning model may not be modified. In some embodiments, some types of machine learning models (e.g., 222, 226) may be trained in a field training process. These machine learning models may be trained at a different field than where the pre-trained machine learning model is trained, at a later time than the pre-trained machine learning model is trained, and / or at a different stage than the stage at which the pre-trained machine learning model is trained. In a non-limiting example, a pre-trained machine learning model (e.g., 220) may be trained in a development / design phase before the system 200 is deployed. Other machine learning models (e.g., 222, 226) may be trained in a deployment phase while the system 200 is deployed. For example, the deployment phase may include a field training mode in which machine learning models may be trained in a field training process, as described above.

[0102] The pre-trained machine learning model 220 may be a deep machine learning model and may be trained at a site or location different from where the system 200 is deployed. In the deployed system 200, the pre-trained machine learning model 220 may be used in combination with other machine learning models (e.g., 222, 226) that may be traditionally trained by a user. For example, a user may collect application-specific training data and use the training system 212 to train the other machine learning models (e.g., 222, 226). The system 200 will now be described in further detail.

[0103] In some embodiments of the techniques described herein, the system 200 may include a feature extractor 202 configured to process an image and generate a feature map of the image using a pre-trained machine learning model 220. The image as input to the feature extractor may include an image captured in the system 100 (e.g., via the image capture device 112 of FIG. 1). The pre-trained machine learning model may be a deep machine learning model that is trained to process the input image and output a feature map of the image. In some examples, the pre-trained machine learning model may be trained from a training dataset that includes a plurality of images that include one or more objects, e.g., labels that include text characters. Other machine learning models (e.g., 222, 226) may be trained using a different training dataset, such as a custom dataset that includes images captured from an application in a particular domain, such as the images shown in FIG. 6.

[0104] In some embodiments, the feature map may include multiple samples, each including a vector with multiple values ​​(feature vector) that encodes semantic information at each location in the image, as an output of the pre-trained machine learning model. For example, the feature map may have a tensor of shape h×w×D, where h is the height of the feature map, w is the width of the feature map, and D is the number of channels of the feature map. The pre-trained machine learning model 220 may be configured to code several features of the image for subsequent processing. In some examples, the pre-trained machine learning network may include a feature pyramid network, as well as an upsampling and concatenation network. The feature pyramid network may be a neural network that takes an image as an input and outputs one or more feature maps that are generated independently, for example, at different scales. In a non-limiting example, the feature pyramid network may generate multiple feature maps of different sizes, for example, h1×w1×D1, h2×w2×D2, and h3×w3×D3. Various implementations of the feature pyramid network may be available. For example, one may use the feature pyramid networks described in Lin et al., “Feature Pyramid Networks for Object Detection,” December, 2016 (arxiv.org / abs / 1612.03144), the entire contents of which are incorporated herein by reference.

[0105] In some embodiments, the upsampling and concatenation network in the pre-trained machine learning network may be configured to take each feature map and upsample it (e.g., using bilinear interpolation) to a common size h×w. In this way, the multiple feature maps may be converted to feature maps of size h×w×D1, h×w×D2, and h×w×D3, respectively. These feature maps may be concatenated to generate an output feature map of size h×w×D. In some examples, the number of channels D in the output feature map may have a value D=D1+D2+D3. In non-limiting examples, the feature maps may have a size different from the size of the image. For example, h and w may take values ​​of H / 8 and W / 8, respectively, or other suitable values, where H and W are the size of the image. The width and height of the image may not be equal, nor may the width and height of the feature map. For example, h may have a value of 16 and w may have a value of 24. Any other suitable values ​​may be possible for h and w. In a non-limiting example, the number of channels D may have a value of 128, or other suitable values. If the pre-trained machine learning network is a deep CNN, the CNN may be configured to include a feature pyramid network by selecting some layers in the CNN and using their activations as outputs. Such pre-trained machine learning networks may be deployed in applications in various domains without the need to be retrained.

[0106] With further reference to FIG. 2, the system 200 may include an object-centered heatmap generator 204 configured to generate an object-centered heatmap by processing a feature map of the image using a first machine learning model (e.g., 222 in FIG. 2). The object-centered heatmap may include a plurality of samples, each having a value indicating the likelihood that a corresponding sample in the image is a center of an object. In a non-limiting example, the first machine learning model (e.g., 222 in FIG. 2) may include a weight vector. The weight vector may have a dimension value of K, i.e., the weight vector may have K values, where K is the number of known labels for one or more objects in the image. In some examples, each value of the weight vector may be a floating point number. Of course, the value of the weight vector may be an integer. In some embodiments, the value of each sample in the object-centered heatmap may be a dot product of the feature vector of the corresponding sample in the feature map and the weight vector.

[0107] 2, the system 200 may further include an object localizer 206 configured to determine the location of one or more objects in the image. In some embodiments, samples near the center of an object in the image may have higher values ​​of the object center heatmap (provided by the object center heatmap generator 204) than samples in other regions in the image. In this manner, the object localizer 206 may determine the location of one or more objects in the image based on the values ​​of the corresponding samples in the object center heatmap. For example, a list of locations in the image may be selected as object centers by filtering out locations where the value of the corresponding sample in the object center heatmap is below a threshold.

[0108] In some embodiments, the object localizer 206 may perform a smoothing operation on the object-centered heatmap to generate a smoothed object-centered heatmap and use the smoothed object-centered heatmap to select the location of one or more objects. In a non-limiting example, a Gaussian filter, e.g., an anisotropic Gaussian filter, may be used for the smoothing operation. In some embodiments, the size of the Gaussian filter may be determined based on the size of the object in the image. For example, to detect character labels, the size (e.g., width and height) of the Gaussian filter may be determined based on the object size 224, which may be trained and stored in the system 200.

[0109] In some embodiments, the system 200 may include an object recognizer 208 configured to recognize one or more objects in an image by processing one or more object locations in the image (determined by the object localizer 206) using a second machine learning model (e.g., 226 in FIG. 2) and a feature map, where the second machine learning model 226 may include a weight matrix. In recognizing an object, the object recognizer 208 may generate an object feature vector at each location from a list of one or more object locations using a portion of the feature map of the image, where the portion of the feature map is based on a region surrounding each object location. The object recognizer 208 may process the object feature vector using the second machine learning model 226 to generate a class vector, where the class vector includes a plurality of values ​​each corresponding to one of a plurality of known labels. Thus, the object recognizer 208 may use the class vector to classify an object into a label of a plurality of known labels.

[0110] With further reference to FIG. 2, the system 200 may include a training system 212 configured to train various machine learning models used in the detection process, such as a first machine learning model 222 and a second machine learning model 226. The first and second machine learning models 222, 226 may each be non-deep machine learning models and may be efficiently trained in the training system in a field training process. The system 200 may further include a graphical user interface 210 that may be configured to provide user interface tools that allow a user to modify detection results from the object-centric heatmap generator 204 or the object recognizer 208. User modifications received at the user interface 210 may be provided to the training system 212. A method for training the first and second machine learning models is described in detail with reference to FIGS. 5A and 5B.

[0111] FIG. 3 is a flow chart illustrating an example computerized method 300 for detecting an object according to some embodiments of the techniques described herein. The method 300 may be implemented in, for example, system 100 (FIG. 1) or 200 (FIG. 2). In some techniques described herein, the method 300 may begin with operation 302 to acquire an image. The image may include an image captured in the system (e.g., via image capture device 112 of system 100 of FIG. 1). The method 300 may proceed to operation 304, where a pre-trained machine learning model may be used to determine a feature map of the image. Operation 304 may be performed in a feature extractor, for example, 202 (FIG. 2). In a non-limiting example, the feature extractor may use a pre-trained machine learning model, for example, a deep CNN, having an input layer with a size (e.g., height, weight, channel). For example, an input image as an input to a pre-trained machine learning model may have a size of H×W×C, where H is the height of the image, W is the width of the image, and C is the number of channels of the image. Of course, the number of channels may be any suitable number. For example, C may have a value of 3 in an RGB color image, while C may have a value of 1 in a grayscale image.

[0112] In some embodiments, the feature map may have a tensor of shape h×w×D as an output of a pre-trained machine learning model, where h is the height of the feature map, w is the width of the feature map, and D is the number of channels of the feature map. In other words, each sample of the h×w grid of the feature map may be associated with a respective vector (feature vector). An example of a pre-trained machine learning model is described above in FIG. 2 with reference to the pre-trained machine learning model 220. As described above, the feature map may include multiple samples, each of which includes a vector that encodes semantic information at a respective location in the image.

[0113] With further reference to FIG. 3, the method 300 may include generating an object-centered heatmap at operation 306. In some implementations, operation 306 may be performed in an object-centered heatmap generator (e.g., 204 of FIG. 2). Operation 306 may process the feature map of the image using a first machine learning model (e.g., 222 of FIG. 2) to generate an object-centered heatmap for the image, where the first machine learning model may be trained in a field training process. The object-centered heatmap may include a number of samples, each having a value indicating the likelihood of the corresponding sample in the image being a center of an object. The size of the object-centered heatmap may be the same as the size of the feature map, e.g., h×w. For a sample in the object-centered heatmap, the corresponding sample in the image may be a pixel in a grid H×W. In the example above, for a sample (x,y) in the object-centered heatmap, h=H / 8, w=W / 8, and the corresponding sample in the image is the pixel at location (x*8,y*8). In some examples, if an image contains text characters, the value of each sample in the object center heatmap may indicate the likelihood that the corresponding sample in the image is the center of a character.

[0114] In a non-limiting example, the first machine learning model (e.g., 222 in FIG. 2) may include a weight vector. In the above example, the feature map may have h×w samples arranged in an array corresponding to the pixel array in the image, with each sample represented by a D-dimensional vector. The object-centered heatmap may also include a plurality of samples arranged in a manner similar to how samples in the feature map are arranged, such that samples in the object-centered heatmap may have a one-to-one correspondence with samples in the feature map that may correspond to pixels in the image. In some embodiments, the value of each sample in the object-centered heatmap may be a dot product of the feature vector of the corresponding sample in the feature map and the weight vector.

[0115] In some embodiments, the weight vector may have a dimension value of K, i.e., the weight vector may have K values, where K is the number of known labels for one or more objects in the image. The weight vector may be trained such that a value in the object center heatmap may indicate the likelihood that a corresponding sample in the image is a center of an object. In some examples, each value in the weight vector may be a floating point number. Of course, the values ​​in the weight vector may be integers. The training process for training the weight vector is described in further detail with reference to FIG. 5A.

[0116] Continuing to refer to FIG. 3, method 300 may further include determining a location of one or more objects at operation 308. In some embodiments, operation 308 may be performed by an object localizer (e.g., 206 of FIG. 2) of system 200. As previously discussed, samples near a center of an object in an image may have higher values ​​in the object center heatmap than samples in other regions of the image. In this manner, operation 308 may determine a location of one or more objects in the image based on values ​​of corresponding samples in the object center heatmap. For example, a list of locations in the image may be selected as object centers by filtering out locations where the value of the corresponding sample in the object center heatmap is below a threshold.

[0117] In some embodiments, operation 308 may perform a smoothing operation on the object-centered heatmap to generate a smoothed object-centered heatmap and use the smoothed object-centered heatmap to select one or more object locations. In some examples, operation 308 may perform the smoothing operation using a smoothing filter, such as a Gaussian filter. The Gaussian filter may be anisotropic. The size of the Gaussian filter may be determined based on the size of the objects in the image. For example, to detect character labels, the size (e.g., width and height) of the Gaussian filter may be determined based on the character size. In some examples, the object sizes may be trained and stored (e.g., at 224 in FIG. 2) for subsequent use. As described above, the resulting list of locations obtained in operation 308 may include predicted centers of objects in the image. In some examples, if the corresponding locations of samples in the object-centered heatmap are not on the grid of the image (e.g., H×W), interpolation may be used to determine the location on the grid.

[0118] In some embodiments, the method 300 may optionally include recognizing one or more objects in the image at operation 310. Operation 310 may be performed in an object recognizer (e.g., 208 of FIG. 2) in some embodiments. Operation 310 may process the one or more object locations in the image using a second machine learning model (e.g., 226 of FIG. 2) and the feature map to recognize the one or more objects. The second machine learning model may also be trained during a field training process. When recognizing the objects, operation 310 may generate an object feature vector at each location from a list of one or more object locations using a portion of the feature map of the image, the portion of the feature map being based on a region surrounding each object location. Operation 310 may process the object feature vector using the second machine learning model to generate a class vector, the class vector including a plurality of values ​​each corresponding to one of a plurality of known labels. Thus, operation 310 may use the class vector to classify the object into a label of a plurality of known labels.

[0119] In some embodiments, operation 310 may use an alignment method to extract feature tensors from a portion of the feature map. For example, operation 310 may define a region around each of the list of locations (predicted locations of one or more objects) obtained in operation 308. The size of the region around the location may be equal to a pre-stored object size (e.g., object size 224 in FIG. 2). In some embodiments, the area of ​​each object may be represented by a bounding box. The system may use an alignment method to extract feature tensors from a portion of the feature map of the image using the bounding box. For example, the system may extract the feature tensors by sampling the feature map at each of the locations within the bounding box. Examples of alignment methods that may be used, such as the RoIAlign method, are described in He et al., “Mask R-CNN” (arxiv.org / abs / 1703.06870), which is incorporated by reference herein in its entirety. In a non-limiting example, the alignment methods described above, such as RoIAlign, may extract a feature tensor from a portion of the feature map. For example, the extracted feature tensor may be a tensor having a size of 3×3×D, where D is the number of channels in the feature map. Other sizes may be possible, such as 4×4, 5×5, 8×8, or any other suitable size.

[0120] Thus, operation 310 may determine an object feature vector for each object by converting the extracted feature tensor into a one-dimensional vector. In the example above, if the feature tensor is of size 3×3×D, operation 310 may determine the object feature vector by concatenating the individual vectors resulting from the alignment method (e.g., the RoIAlign method described above) into a one-dimensional vector having all of the values ​​of the portion of the feature map arranged in one dimension. In this case, the object feature vector has f×D values, where f=9 and D is the number of channels in the feature map.

[0121] In some embodiments, the multiple known labels used to recognize characters in an image may include characters that may appear in the label. For example, for English labels, the number of known labels may be 26 for English capital letters "A" to "Z". The second machine learning model (e.g., 226 in FIG. 2) may include a weight matrix. For example, the size of the weight matrix may be K×(f*D), where D is the number of channels in the feature map, K is the number of known labels (e.g., 26 for English capital letters), and f is an index according to a particular method of feature vector pooling. In the above example, f=3×3=9.

[0122] In some embodiments, the system may determine a class score vector for each object by multiplying a weight matrix (of shape K×(f*D)) by the object feature vector (of size f*D), such that the class score vector has a size of K, where K is the number of known labels. Thus, each value in the class score vector corresponds to a predicted score for one of a number of known classes, such as known labels. In some examples, the system may select a maximum value from among a number of values ​​in the class vector, with the selected value corresponding to the label to be recognized. In the above examples, the known labels for object detection may include a number of text characters. In some embodiments, the known labels may also include a class of non-text, which may be referred to as a background label. In some embodiments, the weight matrix may be trained such that the resulting value of the object class vector may indicate the likelihood of the object being the corresponding known label. The training process for training the weight matrix is ​​further described with reference to FIG. 5A.

[0123] Operation 310 may be repeated for each of the locations in the list of locations determined from operation 308, where each location in the list may correspond to a potential object (or character in the above example). Upon completion of operation 310, the system may output a list of predicted characters at each location obtained from operation 308, where the predicted characters may be one of the known labels, including a background label (e.g., non-text). In such a case, if the predicted character is a background label, it means that the object to be recognized is non-text or an unknown label.

[0124] As described above with reference to method 300, the first machine learning model may include a weight vector, and the second machine learning model may include a weight matrix. These machine learning models may be trained by respective machine learning methods, such as linear regression, support vector machine (SVM), gradient booted decision tree, or other suitable methods. Because the first and second machine learning models are not deep machine learning models, training may be fast, making them suitable for field training in the deployment of the system. The training process for training the first and second machine learning models is further described with reference to Figures 5A and 5B.

[0125] FIG. 4 is a flowchart of an example process 400 for detecting characters using a machine learning model and training the machine learning model, according to some embodiments of the techniques described herein. In some examples, the method 400 may be performed in the system 100 (of FIG. 1 ) or the system 200 (of FIG. 2 ). The method 400 may be similar to the method 300 (of FIG. 3 ), except that the method 400 is provided to recognize one or more characters in an image. In various embodiments described herein, the method 400 may include acquiring an image at operation 402. As with operation 302, the image may include an image captured from the system 100 (FIG. 1 ), such as the image capture device 112. The method 400 may proceed to processing the image using a pre-trained machine learning model to generate a feature map for the image at operation 404. In some examples, the operation 404 may be performed in a feature extractor (e.g., 202 of FIG. 2 ). As with operation 304, the pre-trained machine learning model may be a deep machine learning model (e.g., 220 of FIG. 2 ). An example of a deep machine learning model may include a CNN having multiple layers. Configurations of pre-trained machine learning models may also be used in method 400, as described with reference to Figure 3. The feature map obtained from operation 404 may include multiple samples, each of which includes a vector having multiple values ​​at corresponding positions that code semantic information about the image.

[0126] With further reference to FIG. 4, method 400 may include generating a character-centric heatmap at operation 406. In some examples, operation 406 may be performed in an object-centric heatmap generator (204 of FIG. 2). Operation 406 may process the feature map of the image using a first machine learning model (e.g., 222 of FIG. 2) to generate a character-centric heatmap for the image. The character-centric heatmap may include a plurality of samples, each having a value that indicates the likelihood that the corresponding sample in the image is the center of a character. In some examples, if the image includes text characters, the value of each sample in the character-centric heatmap may indicate the likelihood that the corresponding sample in the image is the center of a text character. Similar to method 300, samples in an h×w grid of the character-centric heatmap may correspond to pixels in an H×W grid of the image.

[0127] As discussed above, the character centered heatmap may represent a "character centrality" score for each corresponding location of a character in the image. For example, the values ​​of samples in the character centered heatmap may be high at locations near the center of the character in the image and low at locations distal to the center of the character or in areas of the image that do not have characters. This facilitates locating individual characters in the image, which may be further described. With continued reference to FIG. 4, method 400 may further include determining the location of one or more characters at operation 408. In some embodiments, operation 408 may be performed by an object localizer (e.g., 206 of FIG. 2) of system 200. As previously discussed, samples near the center of a character in the image may have higher values ​​in the character centered heatmap than values ​​of samples in other areas of the image. In this manner, operation 408 may determine the location of one or more characters in the image based on the values ​​of the corresponding samples in the character centered heatmap. For example, a list of locations in the image may be selected as character centers by filtering out locations where the value of the corresponding sample in the character centered heatmap is below a threshold.

[0128] In some embodiments, operation 408 may include performing a smoothing operation on the character-centric heatmap in a manner similar to operation 308. For example, operation 408 may generate a smoothed character-centric heatmap by applying a smoothing filter to the character-centric heatmap and using the smoothed character-centric heatmap to select the locations of one or more characters. The smoothing filter may be a Gaussian filter, e.g., an anisotropic Gaussian filter. The size of the Gaussian filter may be determined based on the size of the characters, e.g., pre-stored character sizes (e.g., 224 in FIG. 2), that may be trained. The resulting list of locations obtained in operation 408 may include predicted centers of the characters in the image. The character size may be any suitable size. For example, the character size may be 60 pixels by 40 pixels in the original input image (obtained from operation 402). As an example, if the feature map size is related to the image size (e.g., h=H / 8 and w=W / 8), then the character size, in units of feature vector samples, is 60 / 8×40 / 8. The size of the Gaussian filter is then determined as a fixed multiple of the character size.

[0129] In some embodiments, method 400 may include recognizing one or more characters in the image at operation 410. Operation 410 may be performed in an object recognizer (208 of FIG. 2) in some embodiments and may be performed in a manner similar to that described for operation 310. For example, operation 410 may process the location of one or more characters in the image using a second machine learning model (e.g., 226 of FIG. 2) and a feature map to recognize the one or more characters from known labels stored in a character class list (e.g., object class list 228 of FIG. 2). Similar to the second machine learning model of method 300, the second machine learning model may also be trained during a field training process.

[0130] Similar to operation 310, the output of operation 410 may include a list of predicted characters for each position obtained from operation 408, where the predicted character may be one of the known labels. In some embodiments, the known label may also include a background label (e.g., non-text). Similar to method 300, the first machine learning model used in method 400 may include a weight vector, and the second machine learning model used in method 400 may include a weight matrix. As can be appreciated, the first and second machine learning models used in operations 406, 408 are not deep machine learning models, and therefore may be efficiently trained in a field training process. For example, these machine learning models may be trained by respective machine learning methods, such as linear regression, support vector machine (SVM), gradient booted decision tree, or other suitable methods. The training process for training the first and second machine learning models is further described with reference to FIGS. 5A and 5B.

[0131] With further reference to FIG. 4, method 400 may further include receiving user modifications at 412 via a user interface. The user interface may be implemented in a GUI (e.g., 210 in FIG. 2) or in any device of system 100 (e.g., server 110 or user equipment 114 in FIG. 1). In some embodiments, the system may display the recognition results from operation 410 in a user interface, such as the embodiment shown in FIG. 7 (see portion 702). The system may receive the user modifications via the user interface. The user modifications may include updated (e.g., corrected) labels or new labels. Thus, new training data may be generated that includes a new one-to-one mapping between the characters in the image and one of the known labels. In some embodiments, for a character in the image, the user may provide a new label that was not present in the known labels. In such a case, new training data may be generated that includes a new one-to-one mapping between the characters in the image and the new label. The new label may also be saved in a character class list, e.g., object class list 228 in FIG. 2. In some examples, the character size may be updated based on new training data. For example, the character size may be the average of the character sizes of all characters in the training images. As new training images are generated, the character size is updated and stored in a character size database, such as, for example, 224 of FIG. 2. The new training data may then be used to train the first and second machine learning models (e.g., 222, 226 of FIG. 2) for the new labels. In some embodiments, the method 400 may use the new training labels to retrain the first machine learning model at operation 416 or retrain the second machine learning model at operation 418. The training process for training the first and second machine learning models is further described in FIGS. 5A and 5B.

[0132] Further described herein are training methods for training a machine learning model in various embodiments described in this disclosure, such as a method for retraining the system 100 (FIG. 1), the system 200 (FIG. 2), the training system (212 in FIG. 2), the first machine learning model (416 in FIG. 4), or the second machine learning model (418 in FIG. 4). These methods may be performed in a field training process.

[0133] In some examples, in a field training process, a user may provide or supplement training data including training images and corresponding training labels, where each training image may include a known object. The training labels may include ground truth data for each of the training images, where the ground truth data may include, for example, labels of objects appearing in the images, the location of each object appearing in the images, and the size of the objects. In some examples, this information may be provided by a user via a user interface, such as user interface 700 shown in FIG. 7.

[0134] Methods for providing training data and training the machine learning model in a field training process are further described in the example of character labels. In a non-limiting example, the training system may include a user interface (e.g., 700 in FIG. 7) configured to display each of the training images in a display area (e.g., 704) and receive a user selection of a location (e.g., annotation) for each of the characters appearing in the image. The user interface may further receive user input, which may include character location and label correspondence, to indicate that each character in the image should be recognized. In some examples, the field training data may be generated in a batch process. In other examples, the field training data may be updated incrementally while a user is performing a recognition operation (e.g., 300 in FIG. 3 or 400 in FIG. 4), in which process the user may input corrections for characters that are incorrectly recognized in the image.

[0135] FIG. 5A is a flowchart of an example process 500 for training a machine learning model according to some embodiments of the techniques described herein. In some embodiments, the method 500 may train a first machine learning model (e.g., 222 of FIG. 2) used to detect characters and may be implemented in the system 100 (FIG. 1), the system 200 (FIG. 2), such as the training system (e.g., 212 of FIG. 2), or an operation to retrain the first machine learning model (e.g., 416 of FIG. 4). The method 500 may include obtaining a plurality of training images at operation 502 and obtaining training labels corresponding to characters in the training images at operation 504. As an example, in a system for detecting characters in an image, the training labels may include a list of ground truth character positions, each associated with a character in a corresponding one of the plurality of training images to indicate a location where the character appears in the image.

[0136] Method 500 may further include determining a plurality of training feature maps, each using the plurality of training images and one of the pre-trained machine learning models, at operation 506. For example, the training feature maps may be determined by using corresponding training images and a pre-trained machine learning model (e.g., 220 of FIG. 2). This operation may be performed in a manner similar to the detection process described in operations 404 (FIG. 4), 304 (FIG. 3), or methods performed by feature extractor 202.

[0137] The method 500 may further include determining a ground truth character center heatmap at operation 508 and using the training feature map and the ground truth character center heatmap to determine weights for the first machine learning model at operation 510. The individual values ​​of the samples in the ground truth character center heatmap may be referred to as ground truth character centrality values. The ground truth character center heatmap may be generated based on the distance to the nearest ground truth character location in the training label. For example, the ground truth character center heatmap includes samples that have high values ​​at locations close to the center of the ground truth character location of the training label and low values ​​at locations far away from the ground truth character location of the training label. In an example implementation, the ground truth character center heatmap values ​​may be calculated as a Gaussian function applied to the distance to the nearest ground truth character location in the training label. The standard deviation or width of the Gaussian function may be calculated as a fixed multiple of the ground truth character size in some examples.

[0138] In some embodiments, if multiple ground truth characters in the training labels are relatively close to each other, the ground truth character centroid heatmap value may instead be calculated as the difference between two Gaussian functions applied to the distance to the two closest ground truth character positions. In some examples, the ground truth character centroid heatmap may have the same size as the training feature map. Various methods for generating character centrality values ​​using Gaussian functions may be used. For example, methods that may be used are described in Baek et al., “Character Region Awareness for Text Detection,” 3 Apr 2019, arxiv.org / abs / 1904.01941, which is incorporated by reference in its entirety.

[0139] In some embodiments, for each character in the training image, operation 510 may further determine a corresponding character feature vector using a portion of the feature map of the corresponding training image. For example, method 500 may obtain a character location from a ground truth character location of a training label associated with the corresponding training image in which the character is located, and determine a character feature vector using a portion of the feature map of the corresponding training image based on the character location. The operation of determining the portion of the feature map may be similar to the method described in the detection process, such as the method performed in recognizer 208 (FIG. 2), operation 310 (FIG. 3), or operation 410 (FIG. 4). For example, the character feature vector may be determined based on the ground truth character location, the character size (e.g., stored in 224 of FIG. 2), and the feature map determined from the training image obtained from operation 506.

[0140] The method for determining the portion of the feature map corresponding to each of the characters in the image is performed in a manner similar to that described in the detection process, such as operations 310, 410 (of FIGS. 3 and 4, respectively). In some embodiments, a portion of the feature map may be represented by a bounding box centered on the location of the training label in the corresponding training image. A corresponding character feature vector for each located character may be determined by concatenating multiple sub-feature vectors, each formed based on the location of each of the portions of the feature map.

[0141] According to the above-described embodiment, for all of the training images, the system determines N character feature vectors and the sum of N ground truth character centered heatmap values ​​(feature centrality value), where N is the total number of grid cells in the training feature map (e.g., the sum of h×w across all training images, where h, w are the size of each training feature map). In some embodiments, the first machine learning model (e.g., 222 in FIG. 2) may include values ​​for a weight vector. The training objective is to find a weight vector wt such that the dot product of the weight vector wt with each ground truth character feature vector v is as close as possible to the corresponding ground truth character centered heatmap value c (wt·v≈c). In some embodiments, finding the weights in the weight vector may use any suitable machine learning method, such as linear regression, SVM, or other suitable machine learning method.

[0142] 5B is a flowchart of an example process for training another machine learning model according to some embodiments of the techniques described herein. In some embodiments of the techniques described herein, the training method 520 may train a second machine learning model (e.g., 226 in FIG. 2), may be implemented in the system 100 (FIG. 1), the system 200 (FIG. 2), for example, a training system (e.g., 212 in FIG. 2), or may operate to retrain the second machine learning model (e.g., 418 in FIG. 4).

[0143] Method 520 may include obtaining a plurality of training images at operation 522 and obtaining a plurality of training labels at operation 524 in a manner similar to operations 502, 504 of method 500 (FIG. 5A). Each of the plurality of training labels may be associated with a character in a corresponding one of the plurality of training images. For example, the training labels may include a list of ground truth character locations, each associated with a character in a corresponding one of the plurality of training images to indicate where the character appears in the image. The training labels may also include a ground truth class list including a list of character classes that represent the identity of the character at each ground truth character location. In the above example for text characters, the character classes may correspond to text characters, such as "A" to "Z", or other labels that may be specific to the domain in which the system is deployed. In some embodiments, character sizes of all characters may be assumed to be approximately the same. In such a case, the character size stored in the database (e.g., 222 of FIG. 2) may include two values ​​for width and height. It is understood that other suitable representations of character size may also be possible.

[0144] In some embodiments, the system may obtain the character class list by using a label encoding method. For example, the system may collect all character classes that appear in the training labels, discarding the repeated ones, and generate a list of distinct character classes. The system may sort these distinct character classes in order. In some examples, the system may sort the distinct character classes in any order to obtain the character class list. In some other examples, the system may sort the distinct character classes in another suitable order, such as alphabetical order of English characters. Once the character class list is generated, it may be stored in the system (e.g., 228 in FIG. 2).

[0145] Method 520 may further include determining a training feature map at operation 526, which may be performed in a manner similar to operation 506 of FIG. 5A. For example, the training feature map may be determined using a pre-trained machine learning model having a plurality of training images as input, where the pre-trained machine (e.g., 220 of FIG. 2) may be a deep machine learning model, such as a CNN having a plurality of hidden layers. This operation may be performed in a manner similar to the detection process described in operations 404 (FIG. 4), 304 (FIG. 3), or the method implemented in feature extractor 202.

[0146] Method 520 may further determine weights for a second machine learning model at operation 528 using the plurality of training feature maps and the plurality of training labels. In some embodiments, the second machine learning model may be a weight matrix, the size of which is K×(f*D), where D is the number of channels in the feature map, K is the number of known labels (e.g., 26 for English uppercase letters), and f is a factor that depends on the particular method of feature vector pooling, as described above in the detection process described in this disclosure. Thus, each row of the weight matrix corresponds to one of a plurality of known classes corresponding to a known label.

[0147] In some examples, when determining the weights of the second machine learning model, method 520 may determine, for each of the characters in the training image, a corresponding character feature vector using a portion of the feature map of the corresponding training image in which the character is located, in a manner similar to how the character feature vector is determined in method 500 (e.g., operation 510). Method 520 may also determine a target vector for each of the characters. In some embodiments, the target vector may be determined using an encoding method. For example, each labeled character of the training label may be associated with a one-hot target vector of length K, where K is the number of distinct character classes. In some embodiments, a one-hot encoding method may be used. For example, if the character class list is ["A", "B", "C", "D"] and a particular labeled character is of class "C", the one-hot target vector is the vector (0,0,1,0). Other encoding methods may be used.

[0148] Thus, after determining the character feature vector and the target vector for each of the ground truth character positions in the training images, the training process 520 may determine a second machine learning model (e.g., a weight matrix) such that when the weight matrix is ​​multiplied by each character feature vector, the product reproduces the results of the corresponding target vector as closely as possible. Similar to training the weight vectors in method 500, the weight matrix in the second machine learning model may be trained using a machine learning method, such as linear regression, SVM, or other suitable machine learning method.

[0149] An exemplary implementation of a computer system 800 that may be used to implement any of the aspects of the techniques and embodiments disclosed herein is shown in FIG. 8. For example, computer system 800 may be installed in system 100 of FIG. 1, such as by server 110. Computer system 800 may be configured to perform various methods and operations as described with respect to FIGS. 1-7. Computer system 800 may include one or more processors 810, one or more non-transitory computer-readable storage media (e.g., memory 820 and one or more non-volatile storage media 830), and a display 840. Processor 810 may control the writing and reading of data to and from memory 820 and non-volatile storage 830 in any suitable manner, as aspects of the invention described herein are not limited in this respect. To perform the functions and / or techniques described herein, the processor 810 may execute one or more instructions stored on one or more computer-readable storage media (e.g., memory 820, storage media, etc.), which may function as non-transitory computer-readable storage media that store instructions for execution by the processor 810.

[0150] Code used in connection with the techniques described herein, for example, to detect objects in an image / video, may be stored on one or more computer-readable storage media of computer system 800. Processor 810 may execute any such code to provide any technique for detecting objects, as described herein. Any other software, programs, or instructions described herein may also be stored and executed by computer system 800. Of course, computer code may be applied to any aspect of the methods and techniques described herein. For example, computer code may be applied to interact with an operating system to detect objects through conventional operating system processes.

[0151] The various methods or processes outlined herein may be coded as software executable on one or more processors employing any one of a variety of operating systems or platforms. Further, such software may be written using any of a number of suitable programming languages ​​and / or programming or scripting tools, and may be compiled as executable machine language code or intermediate code that runs on a virtual machine or suitable framework.

[0152] In this regard, various inventive concepts may be embodied as at least one non-transitory computer-readable storage medium (e.g., a computer memory, one or more floppy disks, compact disks, optical disks, magnetic tapes, flash memory, circuitry in a field programmable gate array or other semiconductor device, etc.) encoded with one or more programs that, when executed on one or more computers or other processors, implement various embodiments of the invention. The non-transitory computer-readable medium may be portable, as discussed above, such that the programs stored thereon may be loaded into any computer resource to implement various aspects of the invention.

[0153] As used herein, the terms "program," "software," and / or "application" are used in a general sense to refer to any type of computer code or set of computer-executable instructions that may be employed to program a computer or other processor to perform various aspects of the embodiments discussed above. It will further be appreciated that, according to one aspect, one or more computer programs that, when executed, perform the methods of the present invention need not reside on a single computer or processor, but may be distributed modularly among different computers or processors to perform various aspects of the present invention.

[0154] Computer-executable instructions may be in many forms, such as program modules, executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Typically the functionality of the program modules may be combined or distributed as desired in various embodiments.

[0155] Also, the data structures may be stored in the non-transitory computer-readable storage medium in any suitable form. The data structures may have fields that are related through their location within the data structure. Such relationships may be achieved by allocating storage for the fields with locations in the non-transitory computer-readable medium that also convey the relationship between the fields. However, any suitable mechanism may be used to establish relationships between information in fields of the data structures, including the use of pointers, tags, or other mechanisms that establish relationships between data elements.

[0156] Various inventive concepts may be embodied as one or more methods, examples of which are provided. The acts performed as part of a method may be ordered in any suitable manner. Thus, embodiments may be constructed in which acts are performed in an order different from that shown, and may include performing some acts simultaneously despite being shown as sequential acts in an example embodiment.

[0157] As used herein in the specification and claims, the indefinite articles "a" and "an" should be understood to mean "at least one" unless expressly indicated to the contrary. As used herein in the specification and claims, the phrase "at least one" in connection with a list of one or more elements should be understood to mean at least one element selected from any one or more elements in the list of elements, but does not necessarily include at least one of each and every element specifically listed in the list of elements, and does not exclude any combination of elements in the list of elements. This allows for the optional presence of elements other than those specifically identified in the list of elements to which the phrase "at least one" refers, whether or not related to those specifically identified elements.

[0158] As used herein in the specification and claims, the term "and / or" should be understood to mean "either or both" of the elements so conjoined, i.e., elements that are conjointly present in some cases and disjointly present in other cases. Multiple elements listed with "and / or" should be construed in the same manner, i.e., "one or more" of the elements so conjoined. Other elements, whether related or unrelated to those elements specifically identified, may optionally be present other than the elements specifically identified by the "and / or" clause. Thus, as a non-limiting example, a reference to "A and / or B," when used in conjunction with an open-ended phrase such as "comprising," may refer in one embodiment to only A (optionally including elements other than B), in another embodiment to only B (optionally including elements other than A), in yet another embodiment to both A and B (optionally including other elements), etc.

[0159] As used herein in the specification and claims, "or" should be understood to have the same meaning as "and / or" defined above. For example, when separating items in a list, "or" or "and / or" should be interpreted as being inclusive, i.e., including at least one of a number or list of elements, but including two or more, and optionally, elements of additional unlisted items. Only terms clearly indicated to the contrary, such as "only one of" or "exactly one of," or, when used in the claims, "consisting of," refer to the inclusion of exactly one element of a number or list of elements. In general, as used herein, the term "or" should be interpreted as indicating exclusive alternatives (i.e., "one or the other, but not both") only when preceded by an exclusive term, such as "only one of," "one of," "only one of," or "exactly one of." When used in the claims, "consisting essentially of" shall have its ordinary meaning as used in the field of patent law.

[0160] The use of ordinal terms such as "first," "second," "third," etc. in the claims to modify claim elements does not, by itself, imply a priority, precedence, or order of one claim element relative to another claim element, or the temporal order in which method actions are performed. Such terms are used solely as labels to distinguish one claim element having a particular name from other elements having the same name (but for common language usage).

[0161] The words and terms used herein are for purposes of description and should not be regarded as limiting. The use of "including," "having," "containing," "involving," and variations thereof are meant to encompass the items listed thereafter as well as additional items.

[0162] Although several embodiments of the present invention have been described in detail, various modifications and improvements will readily occur to those skilled in the art. Such modifications and improvements are intended to be within the spirit and scope of the present invention. Accordingly, the foregoing description is by way of example only and is not intended to be limiting.

Claims

1. 1. A computerized method for detecting one or more objects in an image, the method comprising: determining a feature map of the image; processing the feature map of the image using a first machine learning model to generate an object-centered heatmap for the image, the object-centered heatmap comprising a plurality of samples, each sample having a value indicative of a likelihood that the corresponding sample in the image is a center of an object; and determining a location of one or more objects within the image based on the object-centered heatmap.

2. 10. The method of claim 1, further comprising: processing the locations of the one or more objects in the image using a second machine learning model and the feature map to recognize the objects of the one or more objects.

3. recognizing the object of the one or more objects, generating an object feature vector using a portion of the feature map of the image, the portion of the feature map being based on a region surrounding the location of the object; processing the object feature vector using the second machine learning model to generate a class vector, the class vector including a plurality of values ​​each corresponding to one of a plurality of known labels; and using the class vector to classify the object into a label of the plurality of known labels.

4. each value of the class vector indicates a predicted score associated with a corresponding one of the plurality of known labels; The method of claim 3 , wherein classifying the object comprises selecting a maximum value from among the plurality of values ​​in the class vector, the selected value corresponding to the label.

5. The method of claim 3 or 4, wherein the plurality of known labels comprises a plurality of text character labels.

6. The method of claim 3 or 4, wherein the plurality of known labels further comprises a background label.

7. 7. The method of claim 2, further comprising training each of the first machine learning model and the second machine learning model using a respective machine learning method and using a respective set of field training data.

8. the one or more objects include one or more printed text characters on parts included in the image; The method of any of claims 2 to 6, wherein the method further comprises tracking the part using at least one text character recognized from the one or more text characters.

9. determining the position of the one or more objects smoothing the object-centered heatmap to generate a smoothed object-centered heatmap; and selecting the locations of the one or more objects, wherein the value of each location in the smoothed object-centered heatmap is higher than the value of an adjacent region of the location.

10. The method of claim 9 , wherein smoothing the object-centered heatmap comprises applying a Gaussian filter with a standard deviation proportional to object size.

11. 11. The method of claim 9, wherein selecting the locations further comprises filtering one or more locations in the smoothed object-centered heatmap where the value is below a threshold.

12. the first machine learning model includes a weight vector; the feature map of the image includes a plurality of samples each associated with a respective feature vector; The method of any of claims 1 to 11, wherein the value of each sample in the object-centered heatmap is the dot product of the feature vector and weight vector of the corresponding sample in the feature map.

13. Determining the feature map of the image A method according to any preceding claim, comprising processing the image using a pre-trained neural network model to generate the feature map of the image.

14. The method of any of claims 1 to 13, further comprising capturing the image using a 1D barcode scanner or a 2D barcode scanner.

15. A non-transitory computer-readable medium comprising instructions operable, when executed by one or more processors on a computing device, to cause the one or more processors to perform the method of any of claims 1 to 14.

16. 1. A system comprising: a scanner including an image capture device configured to capture an image of the part on the inspection station; a processor configured to execute programming instructions to perform the operations of any of claims 1 to 14.

17. 1. A computerized method for recognizing one or more characters in an image, the method comprising: processing the image using a pre-trained machine learning model to generate a feature map for the image; processing the feature map of the image using a first machine learning model to generate a character center heatmap for the image, the character center heatmap including a plurality of samples, each sample having a value indicating a likelihood that a corresponding sample in the image is a character center; and processing the feature map of the image and the character centered heatmap for the image using a second machine learning model to recognize one or more characters in the image.

18. recognizing at least one character of the one or more characters generating a character feature vector using the feature map of the image and the character center heatmap of the image; processing the character feature vector using the second machine learning model to generate a class vector, the class vector including a plurality of values ​​each corresponding to one of a plurality of known labels; and using the class vector to classify the character into a label of the plurality of known labels.

19. each value of the class vector indicates a predicted score associated with a corresponding one of the plurality of known labels; 20. The method of claim 18, wherein classifying the character comprises selecting a maximum value from among the plurality of values ​​in the class vector, the selected value corresponding to the label.

20. 20. The method of claim 18, wherein the plurality of known labels further comprises a background label.

Images