System and method for detecting an object

Through the combination of pre-trained models and on-site training machine learning models, the object and character heat maps are generated, which solves the problems of difficulty in adjusting parameter of object detection system and high cost of training data collection in the existing technology, and achieves fast and flexible object detection and Identify.

CN119032384APending Publication Date: 2024-11-26COGNEX CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202280081511.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2021-10-07
Filing Date
2022-10-07
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

In terms of detecting objects in images, especially character recognition, existing machine vision systems have problems such as difficulty in setting parameters, requiring technical personnel to adjust, high cost of training data collection, and insufficient flexibility in model adjustment.

Method used

A pre-trained machine learning model is used to generate feature maps, and the object-center heatmap and character-center heatmap are generated through the first and second machine learning models. The machine learning model is trained on the spot to quickly identify object locations and categories in the image. , using smaller weight vectors and matrices for feature extraction and classification.

Benefits of technology

It realizes rapid training and deployment of object detection systems on site, improves detection performance, reduces the demand for resources and data, and enhances the robustness and adaptability of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119032384A_ABST
    Figure CN119032384A_ABST
Patent Text Reader

Abstract

The technology described herein relates to a computerized method and apparatus for detecting an object in an image. The techniques described herein also relate to computerized methods and apparatus for detecting one or more objects using a pre-trained machine learning model and one or more other machine learning models that may be trained in a live training process. The pre-trained machine learning model may be a deep machine learning model.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Related Applications

[0002] This application claims priority under 35 U.S.C. 119(e) to U.S. Provisional Application No. 63 / 253,496, filed on October 7, 2021, entitled “Systems and Methods for Detecting Objects,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present technology relates to machine vision systems and methods, and more particularly, to systems and methods for detecting objects. Background Art

[0004] It may be necessary to detect objects, such as characters, in an image. Various techniques can be used to detect objects in an image. For example, optical character recognition (OCR) technology is often used in many machine vision systems to detect text associated with various manufacturing processes, such as text printed and / or affixed to machine parts. However, setting parameters for a given OCR application can be difficult, especially for new users. For example, for character recognition, a user can select an area around an OCR string to indicate to the machine vision system where the characters are for OCR processing. If the system does not correctly recognize the characters, it is difficult for the user to manually resolve the issue. In addition, unlike some applications where these problems can be resolved in advance (e.g., by a system integrator), parameter adjustments as part of the manufacturing process often require technicians or engineers to resolve these issues on the actual production site (e.g., training or modifying runtime parameters to provide better OCR). For example, such training or adjustments may be required when new parts are manufactured or new printing or labels are used. Summary of the invention

[0005] The present disclosure relates to techniques for detecting objects such as characters in an image. Some aspects of the described techniques provide a computerized method, a non-transitory computer-readable medium, and / or a system for identifying one or more objects in an image. The method includes determining a feature map of an image. The method may determine the feature map using a pre-trained machine learning model. The pre-trained machine learning model may be a deep machine learning model. The method also includes processing the feature map of the image using a first machine learning model to generate an object center heat map of the image, wherein the object center heat map includes a plurality of samples, each sample having a value indicating the likelihood that the corresponding sample in the image is the center of the object. The method also includes determining the location of one or more objects in the image based on the object center heat map. In some examples, the method may also include processing the location of one or more objects in the image using a second machine learning model and the feature map to identify at least one of the one or more objects. The first and second machine learning models may be trained during a field training process.

[0006] In some embodiments, identifying at least one of the one or more objects includes: generating an object feature vector using a portion of the feature map of the image, wherein the portion of the feature map is based on an area surrounding the location of the object; processing the object feature vector using the second machine learning model to generate a category vector, wherein the category vector includes multiple values, each value corresponding to one of a plurality of known labels; and classifying the object into one of the plurality of known labels using the category vector.

[0007] In some embodiments, each value of the category vector indicates a prediction score associated with a corresponding one of a plurality of known labels, and classifying the object includes selecting a maximum value among the plurality of values ​​of the category vector, wherein the selected value corresponds to the label.

[0008] In some embodiments, the plurality of known tags includes a plurality of text character tags.

[0009] In some embodiments, the plurality of known tags also includes background tags.

[0010] In some embodiments, the method also includes training each of the first machine learning model and the second machine learning model using a corresponding machine learning method and using a corresponding set of field training data.

[0011] In some embodiments, the one or more objects include one or more printed text characters on a part contained in the image; and the method further includes tracking the part using at least one recognized text character from the one or more text characters.

[0012] In some embodiments, determining the positions of the one or more objects includes: smoothing the object center heat map to generate a smoothed object center heat map; and selecting the positions of the one or more objects, wherein the value at each corresponding position in the smoothed object center heat map is higher than the value in the neighboring area of ​​the position.

[0013] In some embodiments, smoothing the object center heat map includes applying a Gaussian filter with a standard deviation proportional to the object size.

[0014] In some embodiments, selecting locations further comprises filtering one or more locations whose values ​​in the smoothed object center heat map are below a threshold.

[0015] In some embodiments, the first machine learning model includes a weight vector; the feature map of the image includes multiple samples, each sample is associated with a corresponding feature vector; and the value of each sample in the object center heat map is the dot product of the feature vector of the corresponding sample in the feature map and the weight vector.

[0016] In some embodiments, determining the feature map of the image includes: processing the image using a pre-trained neural network model to generate the feature map of the image.

[0017] In some embodiments, the method further comprises capturing the image using a 1D barcode scanner or a 2D barcode scanner.

[0018] In some embodiments, a non-transitory computer-readable medium is provided that includes instructions that, when executed by one or more processors on a computing device, are operable to cause the one or more processors to perform any of the methods described above.

[0019] In some embodiments, a system is provided that includes: a scanner including an image capture device configured to capture an image of a part on an inspection station; and a processor configured to execute programming instructions to perform operations including any of the methods described above.

[0020] Some aspects of the described technology provide a computerized method, non-transitory computer-readable medium, and / or system for recognizing one or more characters in an image. In some embodiments, a method includes processing an image using a pre-trained machine learning model to generate a feature map of the image; processing the feature map of the image using a first machine learning model to generate a character center heat map of the image, wherein the character center heat map includes a plurality of samples, each sample having a value indicating a likelihood that the corresponding sample in the image is a character center; and processing the feature map of the image and the character center heat map of the image using a second machine learning model to recognize one or more characters in the image.

[0021] In some embodiments, identifying at least one of the one or more characters includes: generating a character feature vector using the feature map of the image and the character center heat map of the image; processing the character feature vector using the second machine learning model to generate a category vector, wherein the category vector includes multiple values, each value corresponding to one of multiple known labels; and classifying the character into one of the multiple known labels using the category vector.

[0022] In some embodiments, each value of the category vector indicates a prediction score associated with a corresponding one of the multiple known labels, and classifying the character includes selecting a maximum value among the multiple values ​​of the category vector, wherein the selected value corresponds to the label.

[0023] In some embodiments, the plurality of known tags also includes background tags.

[0024] In some embodiments, the method also includes determining the location of one or more characters in the image based on the character center heat map, wherein generating a character feature vector for at least one character includes generating the character feature vector using a portion of a feature map of the image, wherein the portion of the feature map is based on an area surrounding the character location.

[0025] In some embodiments, determining the positions of the one or more characters includes: smoothing the character center heat map to generate a smoothed character center heat map; and selecting the positions of the one or more characters, wherein the value at each corresponding position in the smoothed character center heat map is higher than the value in the neighboring area of ​​the position.

[0026] In some embodiments, smoothing the character center heat map includes applying a Gaussian filter with a standard deviation proportional to the character size.

[0027] In some embodiments, selecting positions further comprises filtering one or more positions whose values ​​in the smoothed character center heat map are below a threshold.

[0028] In some embodiments, the first machine learning model includes a weight vector; the feature map of the image includes multiple samples, each sample is associated with a corresponding feature vector; and the value of each sample in the character center heat map is the dot product of the feature vector of the corresponding sample in the feature map and the weight vector.

[0029] In some embodiments, the method includes training the second machine learning model by the following steps: obtaining multiple training images and multiple training labels, each of the multiple training labels is associated with a character in a corresponding one of the multiple training images; determining multiple training feature maps using one of the multiple training images and the pre-trained machine learning model, respectively; and determining the weights of the second machine learning model using the multiple training feature maps and the multiple training labels.

[0030] In some embodiments, determining the weight of the second machine learning model includes: (1) for each training label in the multiple training labels: using a portion of the feature map of the corresponding training image associated with the training label to determine a corresponding character feature vector; and determining a target vector; and (2) using the character feature vectors and target vectors of the multiple training labels to determine the weight of the second machine learning model.

[0031] In some embodiments, determining the weights of the second machine learning model includes applying a machine learning method to the character feature vector and the target vector.

[0032] In some embodiments, the method further includes: for each training label among the multiple training labels: (1) obtaining a position of the training label in the corresponding training image; and (2) based on the position of the training label, using the portion of the feature map of the corresponding training image to determine a corresponding character feature vector.

[0033] In some embodiments, the portion of the feature map is represented by a bounding box centered at the location of the training label in the corresponding training image.

[0034] In some embodiments, determining the corresponding character feature vector includes concatenating a plurality of sub-feature vectors, each sub-feature vector formed based on a corresponding position in the portion of the feature map.

[0035] In some embodiments, the method also includes training the first machine learning model by the following steps: determining multiple real character center heat maps of multiple training feature maps; and determining the weight of the first machine learning model using the multiple training feature maps and the multiple real character center heat maps.

[0036] In some embodiments, determining the weights of the first machine learning model includes using a machine learning method to determine the weights of the first machine learning model.

[0037] In some embodiments, determining the plurality of true character center heat maps comprises a plurality of samples, each sample having a value based on a distance to a nearest true location of a training label in the training image.

[0038] In some embodiments, the method further includes: receiving, through a graphical user interface, a one-to-one mapping between characters in the multiple training images and character labels for each of one or more characters in the multiple training images; and generating the multiple training labels based on the received one-to-one mapping of the one or more characters in the multiple training images.

[0039] In some embodiments, the method further includes: receiving a correction to one or more recognized characters through a graphical user interface; updating the multiple training labels based on the correction; and further retraining the first machine learning model and the second machine learning model based on the updated multiple training labels.

[0040] In some embodiments, the one or more characters include one or more printed text characters on a part contained in the image; and the method further includes tracking the part using at least one text character recognized from the one or more text characters.

[0041] In some embodiments, a non-transitory computer-readable medium is provided that includes instructions that, when executed by one or more processors on a computing device, are operable to cause the one or more processors to perform any of the methods described above.

[0042] In some embodiments, a system is provided that includes: a scanner including an image capture device configured to capture an image of a part on an inspection station; and a processor configured to execute programming instructions to perform any of the methods described above.

[0043] It should be understood that all combinations of the aforementioned concepts and the additional concepts described in more detail below (assuming that these concepts are not mutually contradictory) are considered to be part of the inventive subject matter disclosed herein. In particular, all combinations of the claimed subject matter appearing at the end of this disclosure are considered to be part of the inventive subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Additional embodiments of the present disclosure and their features and advantages will become more apparent by referring to the description herein in conjunction with the accompanying drawings. The components in the accompanying drawings are not necessarily drawn to scale. In addition, in the accompanying drawings, the same reference numerals represent corresponding parts throughout the different views.

[0045] Figure 1 is a schematic diagram of a system for detecting an object according to some embodiments of the technology described herein.

[0046] Figure 2 is another schematic diagram of a system for detecting an object according to some embodiments of the technology described herein.

[0047] Figure 3 is a flow chart of an illustrative method for detecting an object in accordance with some embodiments of the techniques described herein.

[0048] Figure 4 is a flow chart of an illustrative method for detecting characters according to some embodiments of the techniques described herein.

[0049] Figure 5A is a flowchart of an illustrative method for training a machine learning model to be used in detecting characters according to some embodiments of the techniques described herein.

[0050] Figure 5B is a flowchart of an illustrative method for training another machine learning model to be used in detecting characters according to some embodiments of the techniques described herein.

[0051] Figure 6is an example part including detectable characters printed thereon according to some embodiments of the technology described herein.

[0052] Figure 7 An example graphical user interface for correcting labels during training is shown in accordance with some embodiments of the techniques described herein.

[0053] Figure 8 An illustrative implementation of a computer system according to some embodiments of the techniques described herein is shown, which computer system may be used to perform any aspects of the techniques and embodiments disclosed herein. DETAILED DESCRIPTION

[0054] For the purposes of promoting an understanding of the principles of the present disclosure, reference will now be made to the embodiments illustrated in the drawings and specific language will be used to describe these embodiments. It will be understood, however, that this is not intended to limit the scope of the invention.

[0055] For various machine vision applications, it may be desirable to detect objects, such as characters, in an image. For example, as part of a manufacturing or inspection process, it may be desirable to detect text printed and / or affixed to a machine part. Various techniques may be used to detect objects in machine vision applications. For example, an OCR application in a manufacturing process may include recognizing a printed label with one or more text characters describing a part number. In this application, an image includes a portion with a printed label, which may be captured. OCR techniques may be used to determine the location of one or more characters in the printed label and recognize the characters. The inventors have recognized and appreciated that applying conventional object detection techniques to the above-mentioned manufacturing processes is particularly challenging and that improvements may be made to such conventional techniques. For example, in conventional object detection systems, images captured during a manufacturing process may be of low quality (e.g., compared to a scanned document) due to various conditions under which the images are captured. Examples of these various conditions may include, for example, the motion of the part to be captured in the scene, the distance between the camera and the part, the unknown or uncontrollable posture of the part in the scene, and / or the lighting conditions of the part in the scene. For example, these conditions may affect the contrast of the image. For example, for characters, the distance between the camera and the part may affect the size of the characters in the image. As another example, variations in the location of a printed label on a part may affect the location of characters in a captured image. These varying conditions in the manufacturing process and others present difficulties in applying these conventional techniques.

[0056] Some object detection systems are designed to adapt to a specific application or the environment in which the application is running by training certain system parameters. The training data used to train the system parameters is collected from the target application domain. For example, the training data can be collected or generated in a laboratory environment that simulates the conditions under which the system is deployed. However, this approach has several disadvantages. First, the environment in which the object detection system is trained and developed (for example, in a laboratory environment) may be very different from the environment in which the object detection system is deployed in the field. Second, collecting training data for a specific application or environment can be costly.

[0057] Some object detection systems, such as OCR systems, are designed to work for certain applications to provide improved object detection performance by allowing the user to adjust certain parameters. However, these systems have disadvantages because adjusting the parameters usually requires a system integrator, a technician, or even someone with knowledge of how the object detection system works. For example, in a typical system, there may be 20-30 parameters that need to be adjusted. Adjusting these parameters may be beyond the skill level of most users.

[0058] Some object detection systems use machine learning techniques. For example, deep machine learning techniques have been used in OCR systems to detect characters from images. However, the inventors have recognized and appreciated that conventional systems for object recognition using machine learning techniques generally provide little or no flexibility in adjusting the machine learning models used during deployment. For example, machine learning (e.g., deep machine learning) techniques inherit the same shortcomings discussed above because machine learning techniques generally require large training data sets, such as hundreds, thousands or tens of thousands of training images and associated real data (e.g., providing information about the objects contained in the images). Collecting such large training data sets may be difficult and / or expensive. In addition, when using deep machine learning techniques, the training process may even be extremely time-consuming, considering the large number of images and large number of weights in the deep machine learning model. For example, in a convolutional neural network (CNN), there may be tens of thousands of weights. Therefore, it is generally not feasible to deploy these machine learning-based techniques in a way that allows machine learning models to be adjusted in the field. Once in the field, there may not be enough training data and / or computing resources to train and / or retrain these systems within a reasonable period of time (e.g., the training or adjustment process is fast enough so as not to result in a poor user experience).

[0059] In addition, the inventors have recognized and appreciated that traditional machine learning-based systems typically use a black box approach to process images end-to-end. For example, a traditional system may provide for receiving an image as input and generating an object detection result as output, such as an OCR result, but does not provide any visibility into the machine learning model and / or other parts of the processing pipeline used to generate the object detection result. Additionally or alternatively, such a system does not provide recognition of a single object when there are multiple objects in the scene. For example, an OCR system does not provide for identifying individual characters of a string or generating intermediate results or data of a processing pipeline, such as providing data indicating the position of characters in an image being OCRed (e.g., which data can then be used to configure the OCR process). Instead, these systems typically only provide for recognition of blocks of text, such as one or more lines of text in an image.

[0060] Therefore, the inventors have developed the technology described herein to provide an object detection system that can be easily and quickly trained in the field. In some embodiments, a system can be provided that includes one or more machine learning models that can be trained in the field. As opposed to using complex machine learning models in a black box approach, using multiple machine learning models can allow the system to use smaller and simpler machine learning models, each focusing on extracting certain features of an image.

[0061] Object detection may include operations for detecting one or more objects in an image, video, and / or any other suitable media asset. Examples of object detection may include determining one or more objects and / or identifying objects in an image, video, and / or any other suitable media asset. For example, in a manufacturing process where an image may include one or more machine parts traveling on an assembly line, object detection may include locating the one or more parts in the image. Object detection may also include identifying one or more parts and a corresponding known category corresponding to each identified part.

[0062] It is understood that one or more objects in the image may include various types of items. Other examples of objects to be detected from the image may include text (e.g., characters in a printed label), graphics, symbols, icons, bar codes, machines, faces, animals, devices, logos, landmarks, or any other suitable text / graphic representations or combinations thereof. The objects contained in the image may include any color or a combination of multiple colors. The text characters may be in English or any other suitable language. Therefore, the technology described herein may be applied to detect various types of items, including characters in the image. The characters may include any symbol, whether a text symbol or a graphic symbol. Examples of characters may include any character in the ASCII set, letters in any suitable language, or non-text symbols, such as mathematical symbols or other symbols.

[0063] Thus, the described techniques include systems, computerized methods, and non-transitory instructions for processing an image to detect one or more objects in the image. In some embodiments, the system may obtain an image containing one or more objects and process the image using a pre-trained machine learning model to determine a feature map of the image. In some examples, the pre-trained machine learning model may be a deep machine learning model, such as a convolutional neural network (CNN) having one or more hidden layers or any other suitable model. The pre-trained machine learning model may be configured to encode some features of the image for subsequent processing to output a feature map. The feature map may include multiple samples, each sample including a vector having multiple values ​​(e.g., a feature vector) that encode semantic information about the image at each corresponding position. The size of the feature map may be different from the size of the image. For example, the feature map may be smaller than the image, with a size of 1 / 4 or 1 / 8 the size of the image.

[0064] In some embodiments, the system may further process the feature map of the image using the first machine learning model to generate an object center heat map of the image. The first machine learning model may be trained in a field training process. The object center heat map may be an image representation depicting the likelihood that one or more pixels in the image are the center of the object. For example, the object center heat map may include multiple samples, each sample having a value indicating the likelihood that the corresponding sample in the image is the center of the object. Therefore, in the object center heat map, samples near the center of the object in the image may have higher values ​​than samples in other areas of the image. In some examples, the first machine learning model may include a weight vector that includes multiple weights in a vector. The size of the object center heat map may be the same as the size of the feature map. In some embodiments, the value of each sample in the object center heat map may be the dot product of a feature vector of the corresponding sample in the feature map and the weight vector. In some examples, the weight vector may have K dimensional values, that is, the weight vector may have K values, where K is the number of known labels of one or more objects in the image.

[0065] In some embodiments, the system can also determine the location of one or more objects based on the values ​​of the corresponding samples of the object center heat map. For example, a list of multiple locations in the image can be selected as the object center by filtering out the locations where the values ​​of the corresponding samples in the object center heat map are lower than a threshold. In some embodiments, the system can apply a filter to the object center heat map to generate a smoothed object center heat map, and use the smoothed object center heat map to select the location of the one or more objects. In some examples, the filter can be a Gaussian filter, such as an anisotropic Gaussian filter. The size of the Gaussian filter can be determined based on the size of the object in the image. For example, in order to detect text character labels, the size (e.g., width and height) of the Gaussian filter can be determined based on the character size, which can be trained and stored in the system.

[0066] In some embodiments, the system may process the positions of the one or more objects in the image using a second machine learning model and the feature map to identify the one or more objects in the image. The second machine learning model may also be trained during field training. When identifying an object from each of the positions of one or more objects, the system may generate an object feature vector. The object feature vector may be a representation of an object describing certain features of an object, such as features extracted from the image using the pre-trained machine learning model. For example, the object feature vector may be generated using a portion of the feature map of the image, wherein a portion of the feature map is based on an area around the position of the corresponding object. The second machine learning model may include a weight matrix including a plurality of weights in a matrix. The weights in the weight matrix may be trained. In some examples, the system may process the object feature vector using the second machine learning model to generate a category vector, and identify the one or more objects by using the category vector to classify each object into a label in the plurality of known labels. The category vector may be a representation of a known category of one or more objects (e.g., a character, a machine part, or other object). The category vector may include a plurality of values, each corresponding to one of a plurality of known labels of an object to be detected.

[0067] In some aspects, the above techniques can be applied to the problem of character detection (e.g., OCR). For example, the first machine learning model can be configured to extract certain features of the image that allow the detection of character locations in the image. The second machine learning model can be configured to extract certain other features of the image that allow the system to recognize characters in the image. Each of the first and second machine learning models can include far fewer weights than a complex machine learning model and can therefore be trained more quickly. In addition, training these smaller machine learning models may not require a large training set, which makes it more feasible to train these models in the field.

[0068] In some embodiments, the system may use a pre-trained machine learning model to extract a feature map of the image for other processing steps. The feature map may be a representation of an image and contain certain features in the image. For example, the feature map may include multiple samples, each sample including a vector with multiple values ​​that encode semantic information at corresponding positions in the image. In some examples, the pre-trained machine learning model may be a deep machine learning model, such as a CNN or other neural network model. The pre-trained machine learning model may be trained using a large amount of offline training data. Once trained, the pre-trained machine learning model will not require adjustment, making it feasible to deploy in field applications. Using the pre-trained machine learning model to generate the feature map of the image takes advantage of the deep machine model in terms of its robustness and noise resistance.

[0069] In some embodiments, the system can combine various types of machine learning models. For example, the inventors have realized that it may be necessary to (1) use a pre-trained machine learning model to extract certain semantic features in the image; and (2) use other machine learning models to extract other features of the image, wherein the other machine learning models can be quickly trained when deployed in the field. For example, the pre-trained machine learning model can be a deep machine learning model, such as a CNN or other neural network model. The other machine learning model can include the first machine learning model and the second machine learning model described above. In some embodiments, the system can use the first machine learning model and the feature map obtained from the pre-trained machine learning model to detect the location of the characters. Additionally, the system can use the second machine learning model and the feature map obtained from the pre-trained machine learning model to identify the characters in the image. This combination of a deep machine learning model (e.g., the deep machine learning model can be pre-trained given the amount of time and / or computing power required to train such a model) and other types of models that can be trained at the time of deployment can therefore obtain the benefits of a deep machine learning model in terms of system performance and robustness, while providing the flexibility to adapt the system to specific field applications. This can overcome some or all of the shortcomings of conventional systems as described above.

[0070] In some aspects, some variations of the above embodiments may include systems, computerized methods, and non-transitory instructions that may be provided to process an image containing one or more characters and detect the one or more characters. Detecting one or more characters may include an operation for detecting the position of one or more characters. In addition, detecting characters in an image may also include identifying one or more characters as corresponding known labels (categories). In some embodiments, the system may obtain an image containing one or more characters and process the image using a pre-trained machine learning model to generate a feature map of the image. In some examples, the pre-trained machine learning model may be a deep machine learning model. The system may generate a character center heat map by processing the feature map of the image using a first machine learning model. Similar to an object center heat map, a character center heat map may be an image representation depicting the likelihood that one or more pixels in the image are the center of a character. For example, the character center heat map may include a plurality of samples, each sample having a value indicating the likelihood that the corresponding sample in the image is the center of the character. In some examples, when the image contains text characters, the value of each sample in the character center heat map may indicate the likelihood that the corresponding sample in the image is the center of a text character. Therefore, the character center heat map may represent a "character center" score for each corresponding position of a character in the image. For example, the values ​​of samples in the character center heat map may be high near the center of the character in the image, and low far from the center of the character or in areas of the image where there are no characters. This may help locate multiple independent characters in the image.

[0071] In some examples, the system can determine the position of one or more characters based on the values ​​of the corresponding samples of the character center heat map. For example, by filtering out the positions in the character center heat map whose corresponding sample values ​​are below a threshold, a list of multiple positions in the image can be selected as multiple character centers. In some embodiments, the system can apply a filter to smooth the character center heat map, and use the smoothed character center heat map to select the position of the one or more characters. In some examples, the filter can be a Gaussian filter, such as an anisotropic Gaussian filter. The size of the Gaussian filter can be determined based on the size of the character, such as a pre-stored character size that can be trained and / or stored. The result list consisting of multiple obtained positions can include multiple predicted centers of multiple characters in the image.

[0072] In some embodiments, the system can recognize one or more characters in the image by processing the positions of multiple characters in the image using a second machine learning model and the feature map. When recognizing characters, at each position of one or more characters, the system can generate a character feature vector. Similar to the object feature vector, the character feature vector can be a character representation that depicts certain features of the character, such as multiple features corresponding to the character represented in the feature map of the image (obtained using a pre-trained machine learning model). For example, the character feature vector can be generated using a portion of the feature map of the image, wherein a portion of the feature map is based on an area around the corresponding character position. The second machine learning model may include a weight matrix that includes multiple weights in a matrix. The weights in the weight matrix can be trained in the field. In some examples, the system can process the character feature vector using the second machine learning model to generate a category vector, and identify the one or more characters by classifying each character into one of the multiple known labels using the category vector. The category vector can be a representation of a known category of one or more characters. For example, the category vector may include multiple values, each corresponding to one of the multiple known labels. In one non-limiting example, for an industrial part, the plurality of known labels may include the entire set of characters or symbols used in printed labels for the industrial part.

[0073] In some embodiments, the above system may include the results of applying the detection object (e.g., character) in one or more applications. For example, the one or more objects in the image may include one or more printed text characters on a machine part contained in the image. Therefore, the system can be configured to recognize the one or more text characters on the part and track the part using at least one recognized text character. In some embodiments, the system can be implemented in a barcode scanner, and the image capture device of the system can be configured to capture an image containing a part with a printed label on the part. The system can be configured to process the captured image by detecting the location of a barcode and decoding the located barcode.

[0074] As described above, various embodiments use a first machine learning model and a second machine learning model. In some embodiments, various techniques are described herein for training the first and / or second machine learning models in a field training process, including systems, computerized methods, and non-transitory instructions. In some embodiments, the training process may include an initial training process using an initial training data set, and a retraining process using an updated training data set. During the retraining process, the updated training data set may include a portion of the initial training data set, and / or new training data that may be acquired by a user, such as through a user interface.

[0075] The field training process can include a training process in which the trained machine learning model can adapt itself to specific conditions associated with a specific field application at a specific customer site or production line. Examples of specific conditions include specific fonts or printing processes used, specific lighting conditions or specific backgrounds, specific types of images to be processed, such as images containing machine parts on an assembly line. Field training can also include training data that is different from the training data used to train the pre-trained machine learning model, with or without overlap. The field training process can include, for example, a user adding custom symbols (labels) and providing training images containing new symbols for training the machine learning model. Therefore, field training can enable the system to achieve better performance than a specifically preconfigured system.

[0076] As an example, a training system that can be used to train first and second machine learning models when detecting an object will be described in detail. In some embodiments, the training system can train a first machine learning model for detecting an object, wherein the first machine learning model can include a weight vector. The system can obtain a plurality of training images and training labels corresponding to objects in the training images. The training labels can include a list of a plurality of real object locations, each real object location being associated with an object in a corresponding one of the plurality of training images to indicate where the object appears in the image.

[0077] In some embodiments, the training system may determine multiple training feature maps using one of the multiple training images and the pre-trained machine learning model, respectively, in a manner similar to determining a feature map from a previously described image in a system for detecting characters / objects. For example, the pre-trained machine may be a deep machine learning model. The system for training a machine learning model may also include determining a real object center heat map, and using the training feature map and the real object center heat map to determine a weight of a first machine learning model. In some embodiments, the value of each sample in the real object center heat map may be generated based on the distance to the nearest real object location in the training image.

[0078] In some examples, for each object in the training image, the system can also determine a corresponding object feature vector using a portion of the training feature map of the corresponding training image. The object feature vector can be a representation of the object that describes certain features of the object and can be obtained in a manner similar to that previously described for detecting characters / objects.

[0079] In some examples, the system can obtain the position of the object from the real object position in the training image; and determine the object feature vector based on the position of the object using the portion of the feature map of the corresponding training image. The operation of determining a portion of the feature map can be similar to the operation previously described in the system for detecting characters / objects. When determining the object feature vector, the system for training the machine learning model can use an alignment method to determine multiple sub-feature vectors, each sub-feature vector is based on a corresponding position in the portion of the feature map, and connect the multiple sub-feature vectors to form the object feature vector.

[0080] Therefore, for all training images, the system may have determined multiple object feature vectors and multiple real object center heat map values ​​according to the above techniques. When determining the weights in the first machine learning model, the system can use a suitable machine learning method so that the dot product between the first machine learning model (e.g., the weight vector) and each object feature vector is as close as possible to the corresponding real object center heat map value. The method for training the weight vector may include linear regression, SVM, or other suitable machine learning methods.

[0081] In some examples, a training system may be provided to train the second machine learning model for detecting objects (e.g., characters), wherein the second machine learning model may include a weight matrix. In some examples, the system may obtain multiple training images and training labels. Each of the multiple training labels may be associated with a character in a corresponding one of the multiple training images. For example, the training label may include a list of multiple real character positions, each position associated with a character in a corresponding one of the multiple training images to indicate where the character appears in the image. The system may also include a real character category list, which contains a list of multiple categories that represent the character identity at each real character position. In the case of text characters, the character category may correspond to the text character, such as "A" to "Z" or other labels that can be used in field applications. In some examples, the system may obtain a list of character categories by using a label encoding method. For example, the system may collect all character categories that appear in the training label and discard duplicate character categories to generate a list of different character categories.

[0082] In addition, the system can use a pre-trained machine learning model to determine the training feature map in a manner similar to that previously described in training the first machine learning model. For example, the pre-trained machine can be a deep machine learning model. The system can also use multiple training feature maps and multiple training labels to determine the weights of the second machine learning model. When determining the weights of the second machine learning model, for each object in the training image, the system can use a portion of the feature map of the corresponding training image in which the object is located to determine the corresponding object feature vector in a manner similar to that previously described in training the first machine learning model.

[0083] Additionally, the system can also determine a target vector for each object in the training image. The target vector can be a representation of an object (e.g., a character) that describes the association of the object with one or more known categories. In some examples, an encoding method can be used to determine the target vector. For example, a one-hot encoding method can be used. Therefore, after determining the object feature vector and the target vector for each real object position in the training image, the system can determine a second machine learning model (e.g., a weight matrix) so that when the second machine learning model (e.g., a weight matrix) is multiplied by each object feature vector, the product reproduces the corresponding target vector as closely as possible. Similar to the training of the first machine learning, a machine learning method such as linear regression, SVM or other suitable machine learning method can be used to train the weight matrix in the second machine learning model.

[0084] Although techniques for training machine learning models using objects are described, such training techniques can also be applied to training machine learning models for multiple types of objects (e.g., detecting characters). Although some examples described herein are described in conjunction with OCR techniques, it should be understood that these examples are for illustrative purposes only and not for limitation, as these techniques are not limited to OCR applications.

[0085] The techniques described in this paper can provide advantages over traditional systems in improving the performance of object / character detection. By using a pre-trained deep neural network as a feature extractor, the system can use higher-level semantic information for object localization and classification. Compared with systems that do not use deep neural networks, the system described in this paper is more robust and less affected by image noise, changes in object appearance, changes in printing quality, lighting and background differences, etc.

[0086] The techniques described herein may also provide advantages over conventional systems in further improving the performance of object / character detection by additionally using one or more other machine learning models that may be trained during the field training process. These machine learning models may be trained during the field training process and adapted to specific conditions that may arise in field applications. Thus, the system may enable users to adapt the system with custom provided training images and / or updated labels. By training only a small portion of the system, other machine learning models may be effectively trained using simple machine learning methods, such as linear regression or other suitable methods, without using expensive computing resources or large amounts of training images as is required for training deep machine learning models.

[0087] Although various embodiments have been described, it will be apparent to those of ordinary skill in the art that more embodiments and implementations are possible. Therefore, the embodiments described herein are examples, not the only possible embodiments and implementations. Furthermore, the advantages described above are not necessarily the only advantages, nor is it necessarily expected that every embodiment will achieve all of the advantages described.

[0088] Figure 1 is a schematic diagram of a system for detecting an object according to some embodiments of the technology described herein.

[0089] In some embodiments, the system 100 may include an image capture device 112, such as a camera, configured to capture a video or image of the part 104 or any component on the inspection platform 103. The captured video may include a sequence of multiple frame images, and each image may contain one or more parts 104 on the inspection platform 103. The system 100 may also include a server 110, which has at least a processor, a memory, a storage medium, and / or other components. The at least one processor may be configured to execute programming instructions stored in the memory to analyze the image and perform object detection on each image to detect one or more objects in the image. Figure 6 is an example part according to some embodiments of the technology described herein, the part including characters printed thereon that can be detected. Figure 6 In the embodiment of the present invention, one or more objects 602 may include text character labels printed on part 604. It is understood that the objects in the image may not be limited to text character labels.

[0090] Back to Figure 1, the server 110 can be configured to process captured images of one or more parts in real time during the inspection process. In a non-limiting configuration, the part 104 can be placed on the inspection platform 103. For example, the inspection platform 103 can be a conveyor belt of an assembly line or a turntable on which multiple parts to be inspected are placed. The inspection platform 103 can be controlled to move the parts. Whenever the inspection platform 103 moves a part (e.g., part 104) to a position for inspection, the part on the inspection platform 103 is captured by the image capture device 112 so that the system 100 can process the captured image to detect one or more objects (e.g., characters) on the part. When the inspection is completed, the inspection platform 103 moves so that the next part moves to the position for inspection, and a new image containing the part that has just been moved to the position is captured, and the new image is processed to detect one or more objects in the image. The inspection process is repeated for each part to be inspected. The server 110 can be configured to process the captured image of each part during the inspection process. Although Figure 1 and Figure 6 The example in FIG. 1 shows only one part 104 per captured image, but it should be understood that a captured image may contain multiple parts, which can be processed as needed to detect objects in each of the multiple parts.

[0091] The system 100 can be configured to detect one or more objects in an image. In some embodiments, detecting one or more objects can include using one or more machine learning models to determine the location of the one or more objects in the image. For example, for an input image of a part (e.g., an image captured by the image capture device 112), the system 100 can be configured to process the input image using a pre-trained machine learning model to extract a feature map of the image. The pre-trained machine model can include a previously trained and stored deep machine learning model. For example, the feature map can include multiple samples, each sample being a vector that encodes semantic information at a corresponding position in the image.

[0092] In some embodiments, the system 100 may generate an object center heat map from the feature map using a machine learning model (e.g., a first machine learning model). The object center heat map may include multiple samples, each sample having a value indicating the likelihood that a corresponding pixel in the image is the center of an object. Therefore, in the object center heat map, a sample close to the center of an object may have a different value from another sample far from the center of the object, wherein the difference in values ​​between the samples in the object center heat map can be used to distinguish samples close to the center of the object in the image from samples far from the center of the object. Therefore, the server 110 may be configured to determine the center position of the object in the image based on the object center heat map.

[0093] In some embodiments, detecting the one or more objects may also include identifying the one or more objects in the image by processing the location of the one or more objects in the image using another machine learning model (e.g., a second machine learning model) and the feature map obtained from the pre-trained machine learning model. In some examples, for each of the one or more located objects in the image, the system 100 may determine a portion of the feature map of the image that corresponds to an area around the location of the object. The system 100 may use the portion of the feature map to classify the object into one of a plurality of known labels. The label may include a representation of an object in the image. For example, the label may be an ASCII representation of a text character or may represent any suitable value of one of a plurality of categories. The plurality of known labels may vary depending on the application. For example, in order to detect labels on a part, the plurality of known labels may be limited to text characters. In this case, the known labels for a part may include numbers and letters, and the known labels may be converted into a list of character categories. Thus, the second machine learning model may be trained so that the second machine learning model may be configured to output a category of the plurality of known labels for one or more objects in the image. In a method such as ( Figure 5A Generating a category list is further described during the on-site training process of 500 in FIG.

[0094] In some embodiments, the first machine learning model and the second machine learning model in the above-mentioned system 100 may be different from the pre-trained machine learning model because the pre-trained machine learning model is a deep machine learning model, while the first and second machine learning models are not deep machine learning models. For example, the first and second machine learning models may each include a corresponding vector or matrix having multiple weights that can be quickly trained. This may be particularly useful for deploying the system 100 in areas where training deep machine learning models (e.g., pre-trained machine learning models) may not be feasible due to lack of computing resources and lack of training data at the user. Instead, the first and second machine learning models can be trained on-site to extract certain features specific to the application. For example, when identifying labels in certain parts, the user may obtain a set of training data sets to train the first and second machine learning models, wherein the training data set includes training images that contain parts with labels thereon. Reference Figure 2-Figure 4 and Figure 5A-Figure 5B , further describing details of a method that can be implemented in system 100 for detecting objects in an image using one or more machine learning models and training the machine learning models.

[0095] Further references Figure 1Once one or more objects in the image are detected, object data may be transmitted to the user device 114. In some embodiments, the object data may include the location of one or more objects on a part contained in the image. In other embodiments, the object data may include recognized labels from the one or more objects in the image, such as text character labels on the part. The user device 114 may be configured to use the object data obtained from the server 110 to perform various operations associated with the part in the image.

[0096] In a non-limiting example, one or more objects in the image may include one or more printed text characters on a part contained in the image. Thus, the system 100 may be configured to recognize the one or more text characters on the part and track the part using at least one recognized text character. In some embodiments, the system 100 may be implemented in a barcode scanner, wherein an image capture device (e.g., 112) may be configured to capture an image containing a part with a label printed thereon. The system 100 may be configured to process the captured image by detecting the location of a barcode and decode the located barcode. Various arrangements of components in the system 100 may be configured to activate a specific application. For example, the user device 114 may be configured as a barcode (e.g., 1D or 2D) scanner. The user device 114 may include an image capture device 112 (e.g., a cell phone camera) to capture an image of a part having a barcode thereon. The user device may be configured as a barcode scanner to detect the location of a barcode (e.g., 1D or 2D barcode) in a captured image and decode the barcode in the image based on the detected barcode location. Various other applications may also be implemented in system 100 .

[0097] In some embodiments, the user device 114 may be provided with a graphical user interface to display the object data. For example, the object data may include an identified label on a part, and the user device 114 may display the identified label and an image containing the part so that the user can determine whether the label is correctly identified. In addition, the user device 114 may be configured to allow the user to make corrections if the label is incorrectly identified. Figure 7 An example graphical user interface for correcting labels during training / retraining is shown according to some embodiments of the techniques described herein. Figure 7 As shown, the graphical user interface 700 may be displayed on the user device 114 ( Figure 1), may include a portion 702 that displays each object (e.g., character) in an image in a first region 702-1 and displays a corresponding identified label in a second region 702-2. As shown, the characters in the first region 702-1 and the identified labels in the second region 702-2 are aligned so that a user can determine whether the corresponding label is correctly identified. In some embodiments, the user interface 700 may allow a user to correct an incorrectly identified label. For example, in region 702-2, the identified label may be displayed in an editable text box that allows the user to modify the label.

[0098] In some embodiments, from server 110 ( Figure 1 ) is transmitted to the user device (for example, Figure 1 The object data of 114 may include the position of each character and the identified label. The user device may use the received position information about each character to segment the characters from the captured image and display them in the user interface 700, for example Figure 7 In addition, the user interface 700 may further include an area 704 configured to display a portion of an image including characters.

[0099] Back to Figure 1 In some embodiments, the user device 114 may be configured to send any information modified by the user to the server 110, such as when the user modifies the information through the graphical user interface 700 ( Figure 7 ) The modified label input by the server 110 can use the modified information to train one or more machine learning models used in the system 100. Figure 4 and Figure 5A-Figure 5B Details of a training method for detecting objects in images that may be implemented in system 100 are further described.

[0100] exist Figure 1 In the embodiment, although the server 110 and the user device 114 are shown as using a machine learning model to perform object detection and train the machine learning model, it should be understood that a single computing device (e.g., at a server or a user device) can be configured to perform the detection / training operation. In other variations, the embodiments described herein can also be implemented in multiple computing devices in a distributed system.

[0101] Figure 2 2 is a schematic diagram of a system 200 that detects objects using a machine learning model and trains the machine learning model according to some embodiments of the technology described herein. The system 200 may include one or more blocks that may be implemented in the system 100 ( Figure 1). In some embodiments, the system 200 may include one or more blocks configured to perform object detection from an image using a machine learning model. The system 200 may also include one or more blocks configured to perform training of one or more machine learning models used in the detection operation. Figure 2 As shown, the system 200 can be configured to operate in a detection mode and a field training mode. For example, in the detection mode, the system 200 can use the pre-trained machine learning model 220 and other machine learning models (e.g., 222, 226) to perform one or more detection operations. Additionally, in the field training mode, the system 200 can be configured to train other machine learning models (e.g., 222, 226) in one or more field training processes.

[0102] A pretrained machine learning model may be a machine learning model that is trained before the machine learning model is deployed and used in the field (e.g., in a customer application). The pretrained machine learning model may be trained at a location different from where the machine learning model is deployed in the field application. In some embodiments, once deployed, the pretrained machine learning model may not be modified. In some embodiments, some types of machine learning models (e.g., 222, 226) may be trained during field training. These machine learning models may be trained at a location different from the location where the pretrained machine learning model is trained, at a time after the pretrained machine learning model is trained, and / or at a stage different from the stage where the pretrained machine learning model is trained. In a non-limiting example, a pretrained machine learning model (e.g., 220) may be trained in the development / design phase before the system 200 is deployed. When the system 200 is deployed, other machine learning models (e.g., 222, 226) may be trained during the deployment phase. For example, the deployment phase may include a field training mode as described above, in which the machine learning model may be trained during the field training process.

[0103] The pre-trained machine learning model 220 may be a deep machine learning model and is trained at a location or position different from where the system 200 is deployed. When the system 200 is deployed, the pre-trained machine learning model 220 may be used in conjunction with other machine learning models (e.g., 222, 226) that are trained by the user. For example, the user may collect application-specific training data and use the training system 212 to train other machine learning models (e.g., 222, 226). The system 200 is now described in further detail.

[0104] In some embodiments of the techniques described herein, system 200 may include a feature extractor 202 configured to process an image and generate a feature map of the image using a pre-trained machine learning model 220. An image as input to the feature extractor may include an image captured in system 100 (e.g., Figure 1 The pre-trained machine learning model can be a deep machine learning model that is trained to process an input image and output a feature map of the image. In some examples, the pre-trained machine learning model can be trained with a training dataset that includes multiple images that contain one or more objects, such as labels that include text characters. The other machine learning models (e.g., 222, 226) can be trained using different training datasets, such as a customized dataset that includes images captured from an application in a specific domain, such as Figure 6 The image shown in .

[0105] In some embodiments, as an output of the pre-trained machine learning model, the feature map may include multiple samples, each sample including a vector (feature vector) having multiple values, which encode semantic information at each corresponding position in the image. For example, the feature map may have a tensor of shape h×w×D, where h is the height of the feature map, w is the width of the feature map, and D is the number of channels of the feature map. The pre-trained machine learning model 220 can be configured to encode some features of the image for subsequent processing. In some examples, the pre-trained machine learning network may include a feature pyramid network and an upsampling-and-concatenation network. A feature pyramid network may be a neural network that takes an image as input and outputs one or more feature maps that are independently generated, for example, at different scales. In a non-limiting example, a feature pyramid network can generate multiple feature maps of different sizes, for example, h1×w1×D1, h2×w2×D2, and h3×w3×D3. Various embodiments of feature pyramid networks are available. For example, the feature pyramid network described in (Linetal., “Feature Pyramid Networks for Object Detection,” December, 2016, arxiv.org / abs / 1612.03144) can be used, and the entire content of which is incorporated herein by reference.

[0106] In some embodiments, the upsampling and cascading networks in the pre-trained machine learning network can be configured to take each feature map and upsample it (e.g., using bilinear interpolation) to a common size h×w. In this way, multiple feature maps can be converted into feature maps of size: h×w×D1, h×w×D2, and h×w×D3, respectively. These feature maps can be connected to generate an output feature map of size h×w×D. In some examples, the number of channels D in the output feature map can have a value of D=D1+D2+D3. In a non-limiting example, the feature map can have a size different from that of the image. For example, h and W can take values ​​of H / 8 and W / 8, respectively, or other suitable values, where H and W are the size of the image. The width and height of the image may not be equal, and the width and height of the feature map are also not equal. For example, h can have a value of 16, and w can have a value of 24. Any other suitable values ​​are possible for h and w. In a non-limiting example, the number of channels D can have a value of 128, or other suitable values. When the pre-trained machine learning network is a deep CNN, by selecting some layers in the CNN and using their activations as output, the CNN can be configured to include a feature pyramid network. This pre-trained machine learning network can be deployed in applications in various fields without the need for retraining.

[0107] Further references Figure 2 , the system 200 may include an object center heat map generator 204 configured to generate an object center heat map by using a first machine learning model (e.g., Figure 2 222) processes the feature map of the image to generate an object center heat map. The object center heat map may include multiple samples, each sample having a value indicating the likelihood that the corresponding sample in the image is the object center. In a non-limiting example, the first machine learning model (e.g., Figure 2 222 in the image) may include a weight vector. The weight vector may have a dimension value of K, that is, the weight vector may have K values, where K is the number of known labels of the one or more objects in the image. In some examples, each value in the weight vector may be a floating point number. It should be understood that the values ​​in the weight vector may also be integers. In some embodiments, the value of each sample in the object center heat map may be the dot product of the feature vector of the corresponding sample in the feature map and the weight vector.

[0108] Continue to refer Figure 2, the system 200 may also include an object locator 206 configured to determine the location of one or more objects in the image. In some embodiments, in the object center heat map (provided by the object center heat map generator 204), samples near the center of the object in the image may have higher values ​​than samples in other areas of the image. In this way, the object locator 206 can determine the location of one or more objects in the image based on the values ​​of the corresponding samples of the object center heat map. For example, a list of multiple locations in the image can be selected as the object center by filtering out locations where the values ​​of the corresponding samples in the object center heat map are lower than a threshold.

[0109] In some embodiments, the object locator 206 can perform a smoothing operation on the object center heat map to generate a smoothed object center heat map, and use the smoothed object center heat map to select the position of the one or more objects. In a non-limiting example, a Gaussian filter, such as an anisotropic Gaussian filter, can be used for the smoothing operation. In some embodiments, the size of the Gaussian filter can be determined based on the size of the object in the image. For example, to detect text character tags, the size (e.g., width and height) of the Gaussian filter can be determined based on the object size 224, which can be trained and stored in the system 200.

[0110] In some embodiments, system 200 may include object identifier 208 configured to identify the object by using a second machine learning model (e.g., Figure 2 226) and the feature map to process the location of the one or more objects in the image (determined by the object locator 206) to identify the one or more objects in the image, wherein the second machine learning model 226 may include a weight matrix. When identifying an object, the object identifier 208 may use a portion of the feature map of the image to generate an object feature vector at each location in the list consisting of the locations of the one or more objects, wherein a portion of the feature map is based on the surrounding area of ​​the location of the corresponding object. The object identifier 208 may use the second machine learning model 226 to process the object feature vector to generate a category vector, wherein the category vector includes multiple values, each value corresponding to one of multiple known labels. Therefore, the object identifier 208 can use the category vector to classify an object into one of the multiple known labels.

[0111] Further references Figure 2, the system 200 may include a training system 212 configured to train various machine learning models used in the detection process, such as a first machine learning model 222 and a second machine learning model 226. The first and second machine learning models 222, 226 can each be a non-deep machine learning model and can be effectively trained in the training system during a field training process. The system 200 may also include a graphical user interface 210, which may be configured to provide a user interface tool to allow a user to modify the detection results from the object center heat map generator 204 or the object identifier 208. User modifications received in the user interface 210 can be provided to the training system 212. Reference Figure 5A and Figure 5B Methods for training the first and second machine learning models are described in detail.

[0112] Figure 3 is a flow chart illustrating an exemplary computerized method 300 for detecting an object according to some embodiments of the technology described herein. For example, the method 300 may be implemented in the system 100 ( Figure 1 ) or 200( Figure 2 In some techniques described herein, method 300 may begin with act 302 of obtaining an image. The image may include an image captured in the system (e.g., via Figure 1 The method 300 may use a pre-trained machine learning model to determine a feature map of the image in step 304. Step 304 may be performed in a feature extractor such as 202 ( Figure 2 ). In a non-limiting example, the feature extractor can use a pre-trained machine learning model, such as a deep CNN, in which the input layer has a size (e.g., height, weight, channels). For example, an input image as input to the pre-trained machine learning model can have a size of H×W×C, where H is the height of the image, W is the width of the image, and C is the number of channels of the image. It should be understood that the number of channels can be any suitable number. For example, the value of C in an RGB color image can be 3, while the value of C in a grayscale image can be 1.

[0113] In some embodiments, the feature map output by the pre-trained machine learning model may have a tensor of shape h×w×D, where h is the height of the feature map, w is the width of the feature map, and D is the number of channels of the feature map. In other words, each sample in the h×w grid of the feature map may be associated with a corresponding vector (feature vector). Figure 2 An example of a pre-trained machine learning model is described in reference to the pre-trained machine learning model 220. As described above, the feature map may include multiple samples, each sample including a vector that encodes semantic information at each corresponding position in the image.

[0114] Further references Figure 3 , method 300 may include generating an object center heat map at act 306. In some examples, act 306 may be performed in an object center heat map generator (e.g., Figure 2 Action 306 may be implemented using a first machine learning model (e.g., Figure 2 222) to process the feature map of the image to generate an object center heat map of the image, wherein the first machine learning model can be trained in a field training process. The object center heat map may include multiple samples, each sample having a value indicating the possibility that the corresponding sample in the image is the center of the object. The size of the object center heat map may be the same as the size of the feature map, for example h×w. For the samples in the object center heat map, the corresponding samples in the image may be pixels in a grid H×W. In the above example, where h=H / 8, w=W / 8, for the sample (x, y) in the object center heat map, the corresponding sample in the image is a pixel at position (x*8, y*8). In some examples, when the image contains text characters, the value of each sample in the object center heat map may indicate the possibility that the corresponding sample in the image is the center of the character.

[0115] In a non-limiting example, the first machine learning model (e.g., Figure 2 222) may include a weight vector. In the above example, the feature map may have h×w samples, which are arranged in an array corresponding to the pixel array in the image, where each sample is represented by a D-dimensional vector. The object center heat map may also include multiple samples arranged in a manner similar to the arrangement of the samples in the feature map, so that a sample in the object center heat map can correspond one-to-one to a sample in the feature map, and the sample can correspond to a pixel in the image. In some embodiments, the value of each sample in the object center heat map can be the dot product of the feature vector and the weight vector of the corresponding sample in the feature map.

[0116] In some embodiments, the weight vector may have K dimensional values, i.e., the weight vector may have K values, where K is the number of known labels of the one or more objects in the image. The weight vector may be trained so that the values ​​in the object center heat map may indicate the likelihood that the corresponding sample in the image is the object center. In some examples, each value in the weight vector may be a floating point number. It should be understood that the values ​​in the weight vector may also be integers. Reference Figure 5A The training process for training the weight vector is further described.

[0117] Continue to refer Figure 3, method 300 may also include determining the location of one or more objects at act 308. In some embodiments, act 308 may be performed by an object locator (e.g., Figure 2 206). As previously described, in the object center heat map, samples near the object center in the image may have higher values ​​than samples in other areas of the image. Thus, action 308 can determine the location of one or more objects in the image based on the values ​​of the corresponding samples of the object center heat map. For example, a list of multiple locations in the image can be selected as the object center by filtering out locations where the values ​​of the corresponding samples in the object center heat map are below a threshold.

[0118] In some embodiments, action 308 may perform a smoothing operation on the object center heat map to generate a smoothed object center heat map, and use the smoothed object center heat map to select the location of the one or more objects. In some examples, action 308 may perform a smoothing operation using a smoothing filter such as a Gaussian filter. The Gaussian filter may be anisotropic. The size of the Gaussian filter may be determined based on the size of the object in the image. For example, to detect text character labels, the size (e.g., width and height) of the Gaussian filter may be determined based on the character size. In some examples, object sizes (e.g., in Figure 2 224) for subsequent use. As described above, the result list of multiple locations obtained in action 308 may include the predicted center of the object in the image. In some examples, if the corresponding position of the sample in the object center heat map is not on the grid (e.g., H×W) of the image, interpolation can be used to determine the position on the grid.

[0119] In some embodiments, method 300 may optionally include identifying the one or more objects in the image at act 310. In some embodiments, act 310 may be performed in an object identifier (e.g., Figure 2 Action 310 may be implemented using a second machine learning model (e.g., Figure 2226) and the feature map to process the location of the one or more objects in the image to identify the one or more objects. The second machine learning model can also be trained in the field training process. When identifying an object, action 310 can use a portion of the feature map of the image to generate an object feature vector at each location in the list consisting of the locations of one or more objects, wherein a portion of the feature map is based on the area around the location of the corresponding object. Action 310 can use the second machine learning model to process the object feature vector to generate a category vector, wherein the category vector includes multiple values, each value corresponding to one of multiple known labels. Therefore, action 310 can use the category vector to classify an object into one of multiple known labels.

[0120] In some embodiments, act 310 may extract a feature tensor from a portion of the feature map using an alignment method. For example, act 310 may define an area around each location of a list of multiple locations (predicted locations of one or more objects) obtained in act 308. The size of the area around the location may be equal to a pre-stored object size (e.g., Figure 2 The object size 224 in the image. In some embodiments, the area of ​​each object can be represented in a bounding box. The system can use an alignment method to extract a feature tensor from a portion of the feature map of the image using the bounding box. For example, the system can extract multiple feature tensors by sampling the feature map at each of multiple positions within the bounding box. An example of an alignment method that can be used, such as the RoIAlign method, is described in (He et al., "MaskR-CNN", arxiv.org / abs / 1703.06870), the entire contents of which are incorporated herein by reference. In a non-limiting example, the above-mentioned alignment method (e.g., RoIAlign) can extract a feature tensor from a portion of the feature map. For example, the extracted feature tensor can be a tensor of size 3×3×D, where D is the number of channels in the feature map. Other sizes, such as 4×4, 5×5, 8×8, or any other suitable size are also possible.

[0121] Therefore, action 310 can determine the object feature vector of each object by converting the extracted feature tensor into a one-dimensional vector. In the above example, where the size of the feature tensor is 3x3xD, action 310 can determine the object feature vector by connecting multiple independent vectors obtained from an alignment method (e.g., the above-mentioned RoIAlign method) into a one-dimensional vector having all values ​​of the portion of the feature map arranged in one dimension. In this case, the object feature vector has fxD values, where f=9 and D is the number of channels in the feature map.

[0122] In some embodiments, the plurality of known tags for identifying characters in an image may include a plurality of characters that may appear in one tag. For example, for English tags, the number of known tags for English capital letters "A" to "z" may be 26. The second machine learning model (e.g., Figure 2 226) may include a weight matrix. For example, the size of the weight matrix may be Kx(f*D), where D is the number of channels of the feature map, K is the number of known labels (e.g., 26 in the case of uppercase English letters), and f is a factor that depends on the specific method of the feature vector pool. In the above example, f=3x3=9.

[0123] In some embodiments, the system can determine a classification score vector for each object by multiplying a weight matrix of shape K×(f*D) by the object feature vector (of size f*D), thereby obtaining a classification score vector of size K, where K is the number of known labels. Therefore, each value in the category score vector corresponds to a prediction score for one of multiple known categories (e.g., known labels). In some examples, the system can select a maximum value from multiple values ​​in the category vector, where the selected value corresponds to the label to be identified. In the above example, the known label for object detection may include multiple text characters. In some embodiments, the known label may also include non-text categories, which may be referred to as background labels. In some embodiments, the weight matrix can be trained so that the resulting value in the object category vector may indicate the likelihood that the object is the corresponding known label. The training process for training the weight matrix will refer to Figure 5A Further description.

[0124] Action 310 may be repeated for each position in the list of multiple positions determined from action 308, wherein each position in the list may correspond to a potential object (or character in the above example). Once action 310 is completed, the system may output a list of multiple predicted characters at the corresponding position obtained from action 308, wherein the predicted character may be one of the known labels including a background label (e.g., non-text). In this case, if a predicted character is a background label, this means that the object being identified is a non-text or unknown label.

[0125] As described above with reference to method 300, the first machine learning model may include a weight vector and the second machine learning model may include a weight matrix. These machine learning models may be trained by corresponding machine learning methods, such as linear regression methods, support vector machines (SVMs), gradient guided decision trees, or other suitable methods. Because the first and second machine learning models are not deep machine learning models, training can be fast, making them suitable for field training of deployed systems. Figure 5A and Figure 5B The training process for training the first and second machine learning models is described in detail.

[0126] Figure 4 is a flow chart of an illustrative process 400 for detecting characters using a machine learning model and training the machine learning model according to some embodiments of the technology described herein. In some examples, the method 400 can be performed in ( Figure 1 ) system 100, or ( Figure 2 The method 400 may be implemented in the system 200 of ( Figure 3 ) method 300, except that method 400 is used to recognize one or more characters in an image. In various embodiments described herein, method 400 may include obtaining an image at act 402. Similar to act 302, the image may include from system 100 ( Figure 1 ), such as from image capture device 112. Method 400 may continue at act 404 to process the image using a pre-trained machine learning model to generate a feature map for the image. In some examples, act 404 may be performed in a feature extractor (e.g., Figure 2 Similar to action 304, the pre-trained machine learning model may be a deep machine learning model (e.g., Figure 2 220 in ). Examples of deep machine learning models may include CNNs with multiple layers. Figure 3 The configuration of the described pre-trained machine learning model can also be used in method 400. The feature map obtained from action 404 can include multiple samples, each sample including a vector with multiple values, which encode semantic information about the image at the corresponding position.

[0127] Further references Figure 4 , method 400 may include generating a character center heat map at action 406. In some examples, action 406 may be performed in an object center heat map generator ( Figure 2 Action 406 may be implemented using a first machine learning model (e.g., Figure 2222) processes the feature portion of the image to generate a character center heat map of the image. The character center heat map may include multiple samples, each sample having a value indicating the likelihood that the corresponding sample in the image is the center of the character. In some examples, when the image contains text characters, the value of each sample in the character center heat map may indicate the likelihood that the corresponding sample in the image is the center of the text character. As in method 300, the samples in the h×w grid of the character center heat map may correspond to pixels in the H×W grid of the image.

[0128] As described above, the character center heat map can represent the "character center" score for each corresponding position of the character in the image. For example, the value of the sample in the character center heat map may be high near the center of the character in the image, and may be low at a position far from the center of the character or in an image area where there is no character. This helps to locate multiple independent characters in the image, which will be explained further. Figure 4 , method 400 may also include determining the location of one or more characters at act 408. In some embodiments, act 408 may be performed at an object locator (e.g., Figure 2 206). As previously described, in the character center heat map, samples near the character center in the image may have higher values ​​than samples in other areas of the image. Thus, action 408 can determine the position of the one or more characters in the image based on the values ​​of the corresponding samples of the character center heat map. For example, by filtering out positions in the character center heat map whose corresponding sample values ​​are below a threshold, a list of multiple positions in the image can be selected as multiple character centers.

[0129] In some embodiments, action 408 may include performing a smoothing operation on the character center heat map in a manner similar to action 308. For example, action 408 may generate a smoothed character center heat map by applying a smoothing filter to the character center heat map, and use the smoothed character center heat map to select the position of the one or more characters. The smoothing filter may be a Gaussian filter, such as an anisotropic Gaussian filter. The size of the Gaussian filter may be determined based on the size of the character, such as a pre-stored character size (e.g., Figure 2 224), which can be trained. The result list consisting of multiple obtained positions in action 408 may include the predicted centers of characters in the image. The character size can be any suitable size. For example, in the original input image (obtained from action 402), the character size can be 60 pixels by 40 pixels. For example, if the feature map size is related to the image size (e.g., h=H / 8 and w=W / 8), then the character size will be 60 / 8 by 40 / 8 in units of feature vector samples. The size of the Gaussian filter is then determined to be a fixed multiple of the character size.

[0130] In some embodiments, method 400 may include recognizing the one or more characters in the image at act 410. In some embodiments, act 410 may be performed in an object identifier ( Figure 2 208) and can be implemented in a similar manner as described in action 310. For example, action 410 can use a second machine learning model (e.g., Figure 2 226) and the feature map to process the position of the one or more characters in the image to select from a character category list stored in a character category list (e.g., Figure 2 The one or more characters are identified from known labels in the object category list 228 in the method 300. Similar to the second machine learning model in the method 300, the second machine learning model can also be trained in the field training process.

[0131] Similar to action 310, the output of action 410 may include a list of predicted characters at corresponding positions obtained from action 408, where the predicted character may be one of the known labels. In some embodiments, the known labels may also include a background label (e.g., non-text). Similar to method 300, the first machine learning model used in method 400 may include a weight vector, and the second machine learning model used in method 400 may include a weight matrix. It will be understood that the first and second machine learning models used in actions 406 and 408, respectively, are not deep machine learning models and, therefore, can be effectively trained in a field training process. For example, these machine learning models can be trained by corresponding machine learning methods, such as linear regression methods, support vector machines (SVMs), gradient-guided decision tree methods, or other suitable methods. Reference Figure 5A and Figure 5B The training process for training the first and second machine learning models is described in detail.

[0132] Further references Figure 4 , method 400 may also include receiving user modifications via a user interface at 412. The user interface may be in a GUI (e.g., Figure 2 210) or any device in system 100 (e.g., Figure 1 In some examples, the system can display the recognition results from action 410 in a user interface, such as Figure 7, see section 702. The system may receive user modifications via a user interface. User modifications may include an updated (e.g., corrected) label or a new label. Thus, new training data may be created that includes a new one-to-one mapping between a character in the image and one of the known labels. In some examples, for a character in an image, a user may provide a new label that does not exist in any of the known labels. In this case, new training data may be created that includes a new one-to-one mapping between a character in the image and the new label. The new label may also be stored in a character category list, such as Figure 2 In some examples, the character size may be updated based on new training data. For example, the character size may be the average of the character sizes of all characters in the training image. When a new training image is generated, the character size is updated and stored in the character size database, for example Figure 2 224 in. The new training data can then be used to train the first and second machine learning models for the new label (e.g., Figure 2 222, 226). In some embodiments, method 400 may use the new training labels to retrain the first machine learning model in action 416, or retrain the second machine learning model in action 418. Figure 5A and Figure 5B The training process for training the first and second machine learning models is further described.

[0133] Further described herein are training methods for training machine learning models in various embodiments described in the present disclosure, such as system 100 ( Figure 1 ), system 200( Figure 2 ), training system( Figure 2 212), retrain the first machine learning model ( Figure 4 416) or the second machine learning model ( Figure 4 These methods can be implemented during field training.

[0134] In some examples, during the on-site training process, a user may provide or supplement training data including multiple training images and corresponding multiple training labels, wherein the multiple training images may each contain known objects. The multiple training labels may include real data for each training image, wherein the real data may include, for example, multiple labels for multiple objects appearing in the image, the position (s) of each object appearing in the image, and the size of the objects. In some examples, this information may be provided by the user through a user interface, such as Figure 7 User interface 700 is shown.

[0135] A method of providing training data and training a machine learning model during field training is also described by way of an example of character labels. In a non-limiting example, the training system may include a user interface (e.g., Figure 7 700 in the example), the user interface is configured to display each of the plurality of training images in a display area (e.g., 704) and receive a user selection of a position (e.g., an annotation) for each of the plurality of characters appearing in the image. The user interface may also receive user input, which may include a character position and label correspondence to indicate what each character in the image should be recognized as. In some examples, live training data may be generated in batches. In other examples, while the user is performing a recognition operation (e.g., Figure 3 300 or Figure 4 400), the live training data can be incrementally updated during which the user can enter corrections for characters that were not correctly recognized in the image.

[0136] Figure 5A 5 is a flow chart of an illustrative process 500 for training a machine learning model according to some embodiments of the technology described herein. In some embodiments, the method 500 can train a first machine learning model (e.g., Figure 2 222) for detecting characters, and can be used in system 100 ( Figure 1 ), system 200( Figure 2 ) such as training systems (e.g., Figure 2 212), or retraining the first machine learning model (e.g., Figure 4 416). Method 500 may include obtaining a plurality of training images at act 502, and obtaining training labels corresponding to characters in the training images at act 504. For example, in a system for detecting characters in an image, the training labels may include a list of true character positions, each position being associated with a character in a corresponding one of the plurality of training images to indicate where the character appears in the image.

[0137] Method 500 may also include determining a plurality of training feature maps using one of the plurality of training images and the pre-trained machine learning model, respectively, at step 506. For example, the training feature maps may be determined by using the corresponding training image and the pre-trained machine learning model (e.g., Figure 2 220 in step 404) to determine the training feature map. This operation can be performed in the same manner as in step 404 ( Figure 4 )、304( Figure 3 ) or is performed in a manner similar to the detection process described in ) or the method implemented in the feature extractor 202.

[0138] Method 500 may also include determining a real character center heat map in action 508, and using the training feature map and the real character center heat map to determine the weight of the first machine learning model in action 510. The individual values ​​of the samples in the real feature center heat map can be referred to as real feature center values. The real character center heat map can be generated based on the distance to the nearest real character position in the training label. For example, the real character center heat map includes a plurality of samples having high values ​​at positions close to the center of the real character position in the training label and low values ​​at positions far from the real character position in the training label. In an example embodiment, the real character center heat map value can be calculated as a Gaussian function applied to the distance to the nearest real character position in the training label. In some examples, the standard deviation or width of the Gaussian function can be calculated as a fixed multiple of the real character size.

[0139] In some embodiments, when multiple real characters in the training label are relatively close to each other, the real character center heat map value can be calculated as the difference between two Gaussian functions, which are applied to the distances to the two nearest real character positions. In some examples, the ground truth character center heat map can have the same size as the training feature map. Various methods of generating character center values ​​using Gaussian functions can be used. For example, a method that can be used is described in (Baek et al., "Character Region Awareness for Text Detection," 3Apr 2019, arxiv.org / abs / 1904.01941), which is incorporated herein by reference in its entirety.

[0140] In some embodiments, for each character in the training image, action 510 may also use a portion of the feature map of the corresponding training image to determine a corresponding character feature vector. For example, method 500 may obtain a position of a character from the actual character position in the training label associated with the corresponding training image where the character is located; and determine the character feature vector based on the position of the character using the portion of the feature map of the corresponding training image. The operation of determining a portion of the feature map may be similar to the method described in the detection process, such as in ( Figure 2 ) identifier 208, action 310 ( Figure 3 ) or Action 410( Figure 4 ). For example, the method can be implemented in Figure 2 224) and determine the character feature vector based on the feature map determined from the training image obtained from action 506.

[0141] The method for determining the portion of the feature map corresponding to each character in the image is implemented in a similar manner to the method described in the detection process, for example (respectively Figure 3 , Figure 4 ) actions 310, 410. In some examples, the portion of the feature map can be represented by a bounding box centered at the location of the training label in the corresponding training image. The corresponding character feature vector for each located character can be determined by concatenating multiple sub-feature vectors, each of which is formed based on the corresponding position in the portion of the feature map.

[0142] According to the above embodiment, for all training images, the system has determined a total of N character feature vectors and N true character center heat map values ​​(character center value), where N is the total number of grid cells in the training feature map (e.g., the sum of h×w for all training images, where h and w are the size of each training feature map). In some embodiments, the first machine learning model (e.g., Figure 2 222 in may include a weight vector. The training goal is to find a weight vector wt so that its dot product with each real character feature vector v is as close as possible to the corresponding real character center heat map value c, so that wt·v≈c. In some embodiments, any suitable machine learning method can be used to find the weights in the weight vector, such as linear regression, SVM or other suitable machine learning methods.

[0143] Figure 5B is a flowchart of an illustrative process for training another machine learning model according to some embodiments of the technology described herein. In some embodiments, the training method 520 can train a second machine learning model (e.g., Figure 2 226), and can be in system 100 ( Figure 1 ), system 200( Figure 2 ) such as training systems (e.g., Figure 2 212), or for retraining a second machine learning model (e.g., Figure 4 of 418).

[0144] Method 520 may include: Figure 5AIn a manner similar to actions 502 and 504 of method 500, a plurality of training images are obtained in action 522, and a plurality of training labels are obtained in action 524. Each of the plurality of training labels can be associated with a character in a corresponding one of the plurality of training images. For example, the training labels may include a list of a plurality of real character positions, each position being associated with a character in a corresponding one of the plurality of training images to indicate where the character appears in the image. The training labels may also include a list of real character categories, the list of real character categories containing a list of a plurality of character categories representing the identity of the character at each real character position. In the above example for text characters, the character class may correspond to a text character, such as "A" to "Z" or other labels that may be specific to the domain in which the system is deployed. In some examples, it may be assumed that the character size of all characters is approximately the same. In this case, the characters stored in the database (e.g., Figure 2 The character size in 222) may include two values ​​of width and height. It should be understood that other suitable representations of character size are also possible.

[0145] In some embodiments, the system can obtain a list of character categories by using a label encoding method. For example, the system can collect all character categories that appear in the training labels and discard duplicate character categories to generate a list of different character categories. The system can classify these different character categories in order. In some examples, the system can sort the different character categories in an arbitrary order to obtain the character category list. In some other examples, the system can sort the different character categories in other suitable orders, such as the alphabetical order of English letters. Once the character category list is generated, it can be stored in the system (for example, Figure 2 228 in ).

[0146] Method 520 may also include determining a training feature map at step 526, which may be performed in the same manner as Figure 5A The method of step 506 is performed in a similar manner. For example, the training feature map can be determined using a pre-trained machine learning model having a plurality of training images as input, wherein the pre-trained machine (e.g., Figure 2 220) may be a deep machine learning model, such as a CNN with multiple hidden layers. This operation may be performed in the same manner as in action 404 ( Figure 4 )、304( Figure 3 ) or is performed in a manner similar to the detection process described in ) or the method implemented in the feature extractor 202.

[0147] Method 520 may also use the multiple training feature maps and the multiple training labels to determine the weights of the second machine learning model in action 528. In some examples, as explained above in the detection process described in the present disclosure, the second machine learning model may be a weight matrix, where the size of the matrix is ​​K×(f*D), where D is the number of channels of the feature map, K is the number of known labels (e.g., 26 in the case of English capital letters), and f is a factor that depends on a specific method of feature vector pooling. Thus, each row of the weight matrix corresponds to one of multiple known categories of known labels.

[0148] In some examples, when determining the weights of the second machine learning model, method 520 may determine a corresponding character feature vector for each character in the training image using a portion of the feature map of the corresponding training image in which the character is located, in a manner similar to determining the character feature vector in method 500 (e.g., action 510). Method 520 may also determine a target vector for each of the multiple characters. In some embodiments, an encoding method may be used to determine the target vector. For example, each labeled character in the training label may be associated with a one-hot target vector of length K, where K is the number of different character categories. In some embodiments, a one-hot encoding method may be used. For example, if the character category list is ["A", "B", "C", "D"], and the particular labeled character belongs to category "C", the one-hot target vector is the vector (0, 0, 1, 0). Other encoding methods may be used.

[0149] Therefore, after determining the character feature vector and target vector of each real character position in the plurality of training images, the training process 520 can determine the second machine learning model (e.g., weight matrix) so that when the weight matrix is ​​multiplied by each character feature vector, the result of the recalculation of the product is as close as possible to the corresponding target vector. Similar to the training of the weight vector in method 500, machine learning methods such as linear regression, SVM or other suitable machine learning methods can be used to train the weight matrix in the second machine learning model.

[0150] Figure 8 An illustrative implementation of a computer system 800 that can be used to perform any aspect of the techniques and embodiments disclosed herein is shown. For example, the computer system 800 can be installed in Figure 1 The computer system 800 may be configured to execute the system 100 as described in reference Figure 1-7The various methods and actions described herein. The computer system 800 may include one or more processors 810 and one or more non-transitory computer-readable storage media (e.g., memory 820 and one or more non-volatile storage media 830) and a display 840. The processor 810 may control writing data to the memory 820 and the non-volatile storage device 830 and reading data from the memory 820 and the non-volatile storage device 830 in any suitable manner, as the various aspects of the invention described herein are not limited in this regard. In order to perform the functions and / or techniques described herein, the processor 810 may execute one or more instructions stored in one or more computer-readable storage media (e.g., memory 820, storage media, etc.), which may serve as a non-transitory computer-readable storage medium for storing instructions executed by the processor 810.

[0151] In conjunction with the techniques described herein, code for, for example, detecting an object in an image / video may be stored on one or more computer-readable storage media of the computer system 800. The processor 810 may execute any such code to provide any technique for detecting an object as described herein. Any other software, program, or instruction described herein may also be stored and executed by the computer system 800. It should be understood that the computer code may be applied to any aspect of the methods and techniques described herein. For example, the computer code may be applied to interact with an operating system to detect an object through conventional operating system processes.

[0152] The various methods or processes outlined herein may be encoded as software that can be executed on one or more processors using any of a variety of operating systems or platforms. In addition, such software may be written using any of a variety of suitable programming languages ​​and / or programming or scripting tools, and may also be compiled into executable machine language code or intermediate code that is executed on a virtual machine or suitable framework.

[0153] In this regard, various inventive concepts may be embodied as at least one non-transitory computer-readable storage medium (e.g., computer memory, one or more floppy disks, optical disks, tapes, flash memory, circuit configurations in field programmable gate arrays or other semiconductor devices, etc.), encoded with one or more programs that, when executed on one or more computers or other processors, implement various embodiments of the present invention. One or more non-transitory computer-readable media may be transportable, such that one or more programs stored thereon may be loaded onto any computer resource to implement various aspects of the present invention as described above.

[0154] The terms "program", "software" and / or "application" are used in a general sense herein to refer to any type of computer code or computer executable instruction set that can be used to program a computer or other processor to implement various aspects of the above-described embodiments. In addition, it should be understood that, according to one aspect, one or more computer programs that perform the methods of the present invention when executed do not need to reside on a single computer or processor, but can be distributed in a modular manner among different computers or processors to implement various aspects of the present invention.

[0155] Computer executable instructions can be in various forms, such as program modules, that are executed by one or more computers or other devices. Typically, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. Typically, the functions of program modules can be combined or distributed as needed in various embodiments.

[0156] In addition, the data structure can be stored in a non-transitory computer-readable storage medium in any suitable form. The data structure can have fields that are related by location in the data structure. This relationship can also be achieved by allocating storage space for the fields, which has locations in the non-transitory computer-readable medium that convey the relationship between the fields. However, any suitable mechanism can be used to establish the relationship between the information in the fields of the data structure, including through the use of pointers, tags, or other mechanisms that establish relationships between data elements.

[0157] Various inventive concepts may be embodied as one or more methods, examples of which have been provided. The actions performed as part of a method may be ordered in any suitable manner. Thus, embodiments may be constructed in which the actions are performed in a different order than shown, which may include performing some actions simultaneously, even though shown as sequential actions in an illustrative embodiment.

[0158] The indefinite articles "a" and "an" used in the specification and claims, unless clearly indicated to the contrary, should be understood to mean "at least one". As used in the specification and claims herein, the phrase "at least one" in accordance with a list of one or more elements should be understood to mean at least one element selected from any one or more elements of the list consisting of multiple elements, but not necessarily including at least one of each element specifically listed in the list consisting of multiple elements, and not excluding any combination of multiple elements in the list consisting of multiple elements. This allows elements to optionally be present in addition to the specific element in the list consisting of multiple elements to which the phrase "at least one" refers, whether or not related to that specific element.

[0159] The phrase "and / or" as used herein in the specification and claims should be understood to mean "one or both" of the elements so combined, i.e., elements that exist in combination in some cases and separately in other cases. Multiple elements listed with "and / or" should be interpreted in the same manner, i.e., "one or more" elements are so connected. In addition to the elements specifically identified by the "and / or" clause, other elements may optionally be present, whether or not related to the specifically identified elements. Thus, as a non-limiting example, in one embodiment, when used in conjunction with open language such as "comprises", a reference to "A and / or B" may refer to only A (optionally including elements other than B); in another embodiment, only B (optionally including elements other than A); in yet another embodiment, applies to both A and B (optionally including other elements); and so on.

[0160] As used in the specification and claims, "or" should be understood to have the same meaning as "and / or" defined above. For example, when separating items in a list, "or" or "and / or" should be interpreted as inclusive, i.e., including at least one but also including more than one multiple or series of elements, and optionally including additional unlisted items. Only terms that clearly indicate the contrary, such as "only one" or "exactly one", or when used in the claims, "comprising" will refer to including exactly one element of a plurality or series of elements. In general, the term "or" as used herein should only be interpreted as indicating an exclusive choice (i.e., "one or the other but not two") when preceded by an exclusive term, such as "either", "one", "only one" or "exactly one". When used in the claims, "essentially consisting of..." should have the ordinary meaning used in the field of patent law.

[0161] The use of ordinal terms such as "first," "second," "third," etc. in the claims to modify a claim element by itself does not imply any priority, precedence, or order of one claim element relative to another claim element, or the temporal order in which the actions of a method are performed. These terms are merely used as labels to distinguish one claim element with a particular name from another element with the same name (but with the use of an ordinal term).

[0162] The phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of "including," "comprising," "having," "containing," "involving," and variations thereof, is meant to encompass the items listed thereafter and additional items.

[0163] Having described several embodiments of the present invention in detail, those skilled in the art will readily appreciate various modifications and improvements. Such modifications and improvements are intended to fall within the scope and range of the present invention. Therefore, the foregoing description is merely exemplary and not restrictive.

Claims

1. A computerized method for detecting one or more objects in an image, the method comprising: include: determining a feature map of the image; processing the feature map of the image using a first machine learning model to generate an object center heat map of the image, wherein the object center heat map comprises a plurality of samples, each sample having a value indicating a likelihood that a corresponding sample in the image is an object center; and The locations of one or more objects in the image are determined based on the object center heat map.

2. The method according to claim 1, further comprising: include: The locations of one or more objects in the image are processed using a second machine learning model and the feature map to identify an object among the one or more objects.

3. The method of claim 2, wherein identifying an object among the one or more objects include: generating an object feature vector using a portion of the feature map of the image, wherein the portion of the feature map is based on a region surrounding a location of the object; Processing the object feature vector using the second machine learning model to generate a category vector, wherein the category vector includes a plurality of values, each value corresponding to one of a plurality of known labels; and The object is classified into one of the plurality of known labels using the class vector.

4. The method according to claim 3, in: Each value of the category vector represents a predicted score associated with a corresponding one of the plurality of known labels; and Classifying the object includes selecting a maximum value among the plurality of values ​​in the category vector, wherein the selected value corresponds to the label.

5. The method according to any one of claims 3 and 4, wherein the plurality of known tags comprises a plurality of text character tags.

6. The method according to any one of claims 3 and 4, wherein the plurality of known tags further comprises background tags.

7. The method according to any one of claims 2-6 also includes using a corresponding machine learning method and using a corresponding set of field training data to train each of the first machine learning model and the second machine learning model.

8. The method according to any one of claims 2 to 6, in: The one or more objects include one or more printed text characters on a part contained in the image; and The method also includes tracking the part using at least one recognized text character from the one or more text characters.

9. The method according to any one of claims 1 to 8, wherein determining the position of the one or more objects include: smoothing the object center heat map to generate a smoothed object center heat map; and Positions of the one or more objects are selected, wherein a value at each corresponding position in the smoothed object center heat map is higher than a value in a neighborhood of the position.

10. The method of claim 9, wherein smoothing the object center heat map comprises applying a Gaussian filter with a standard deviation proportional to the object size.

11. The method according to any one of claims 9 and 10, wherein selecting the locations further comprises filtering one or more locations whose values ​​in the smoothed object center heat map are below a threshold.

12. The method according to any one of claims 1 to 11, in: The first machine learning model includes a weight vector; The feature map of the image comprises a plurality of samples, each sample being associated with a corresponding feature vector; and The value of each sample in the object center heat map is the dot product of the feature vector and the weight vector of the corresponding sample in the feature map.

13. The method according to any one of claims 1 to 12, wherein determining the feature map of the image include: The image is processed using a pre-trained neural network model to generate the feature map of the image.

14. The method of any one of claims 1-13, further comprising capturing the image using a 1D barcode scanner or a 2D barcode scanner.

15. A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors on a computing device, are operable to cause the one or more processors to perform the method of any one of claims 1-14.

16. A system, include: a scanner including an image capture device configured to capture an image of a part at an inspection station; and A processor configured to execute programming instructions to perform operations including any one of claims 1-14.

17. A computerized method for recognizing one or more characters in an image, the method include: Processing the image using a pre-trained machine learning model to generate a feature map of the image; processing the feature map of the image using a first machine learning model to generate a character center heat map of the image, wherein the character center heat map includes a plurality of samples, each sample having a value indicating a likelihood that a corresponding sample in the image is a center of a character; and The feature map of the image and the character center heat map of the image are processed using a second machine learning model to recognize one or more characters in the image.

18. The method of claim 17, wherein identifying at least one of the one or more characters include: generating a character feature vector using the feature map of the image and the character center heat map of the image; Processing the character feature vector using the second machine learning model to generate a category vector, wherein the category vector includes a plurality of values, each value corresponding to one of a plurality of known labels; and The character is classified into one of the plurality of known labels using the category vector.

19. The method according to claim 18, in: Each value of the category vector represents a predicted score associated with a corresponding one of the plurality of known labels; and Classifying the character includes selecting a maximum value among a plurality of values ​​of the category vector, wherein the selected value corresponds to the label.

20. The method of any one of claims 18 and 19, wherein the plurality of known tags further comprises background tags.

21. The method according to any one of claims 18-20 further includes determining multiple positions of the one or more characters in the image based on the character center heat map, wherein generating the character feature vector of the at least one character includes using a portion of the feature map of the image to generate the character feature vector, wherein the portion of the feature map is based on an area surrounding the position of the character.

22. The method of claim 21, wherein determining the position of the one or more characters include: smoothing the character center heat map to generate a smoothed character center heat map; and Positions of the one or more characters are selected, wherein a value at each corresponding position in the smoothed character center heat map is higher than a value in a neighborhood of the position.

23. The method of claim 22, wherein smoothing the character center heat map comprises applying a Gaussian filter with a standard deviation proportional to the character size.

24. The method according to any one of claims 22 and 23, wherein selecting the positions further comprises filtering one or more positions whose values ​​in the smoothed character center heat map are below a threshold.

25. The method according to any one of claims 17 to 24, in: The first machine learning model includes a weight vector; The feature map of the image comprises a plurality of samples, each sample being associated with a corresponding feature vector; and The value of each sample in the character center heat map is the dot product of the feature vector and the weight vector of the corresponding sample in the feature map.

26. The method of any one of claims 17-25, further comprising training the second machine learning model by: obtaining a plurality of training images and a plurality of training labels, each of the plurality of training labels being associated with a character in a corresponding one of the plurality of training images; Determine a plurality of training feature maps using one of the plurality of training images and the pre-trained machine learning model, respectively; and The weights of the second machine learning model are determined using the plurality of training feature maps and the plurality of training labels.

27. The method of claim 26, wherein determining the weights of the second machine learning model include: For each training label in the plurality of training labels: determining a corresponding character feature vector using a portion of the feature map of the corresponding training image associated with the training label; and determining a target vector; and The weights of the second machine learning model are determined using the character feature vectors and target vectors of the multiple training labels.

28. The method of claim 27, wherein determining the weights of the second machine learning model comprises using a machine learning method on the character feature vector and the target vector.

29. The method according to any one of claims 27 and 28, further comprising: include: For each training label in the plurality of training labels: Obtaining the position of the training label in the corresponding training image; and Based on the position of the training label, a corresponding character feature vector is determined using the portion of the feature map of the corresponding training image.

30. The method of claim 29, wherein the portion of the feature map is represented by a bounding box centered at the location of a training label in a corresponding training image.

31. The method according to any one of claims 29 and 30, in, Determining the corresponding character feature vector includes concatenating a plurality of sub-feature vectors, each sub-feature vector being formed based on a corresponding position in the portion of the feature map.

32. The method of any one of claims 17-31, further comprising training the second machine learning model by: determining a plurality of basic feature center heat maps of the plurality of training feature maps; and The weights of the first machine learning model are determined using the multiple training feature maps and the multiple real character center heat maps.

33. According to the method according to claim 32, determining the weight of the first machine learning model includes using a machine learning method to determine the weight of the first machine learning model.

34. A method according to any one of claims 32 and 33, wherein determining the multiple true character center heat maps includes multiple samples, each sample having a value based on the distance to the nearest true position of the training label in the training image.

35. The method according to any one of claims 17 to 34, further comprising: include: receiving, via a graphical user interface, a one-to-one mapping between characters in the plurality of training images and character labels for each of one or more characters in the plurality of training images; and The plurality of training labels are generated based on a one-to-one mapping of one or more characters in the received plurality of training images.

36. The method according to claim 35, further comprising: include: receiving, via the graphical user interface, a correction to one or more recognized characters; updating the plurality of training labels based on the correction; and The first machine learning model and the second machine learning model are also retrained based on the updated multiple training labels.

37. The method according to any one of claims 17 to 36, in: The one or more characters include one or more printed text characters on a part contained in the image; and The method also includes tracking the part using at least one recognized text character from the one or more text characters.

38. A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors on a computing device, are operable to cause the one or more processors to perform the method of any one of claims 17-37.

39. A system, include: a scanner including an image capture device configured to capture an image of a part at an inspection station; and A processor configured to execute programmed instructions to perform operations including any one of claims 17-37.