A text recognition method, system and storage medium

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating at least text and a particular type of prediction associated with an object are disclosed. A first model output is generated by processing input data including one or more objects using a first machine learning model. The first model output identifies a presence of a particular object in the input data and specifies a characteristic of the particular object. A type of the particular object is determined based on the specified characteristic. The type includes a single-line type and a multi-line type. A single-line representation of the particular object is generated. A second model output is generated by processing the single-line representation. The second model output includes a prediction of characters corresponding to a particular vehicle license plate.

Description

Technical Field

[0001] This application relates to the field of text recognition technology, and more specifically, to a text recognition method, system, and storage medium. Background Technology

[0002] This specification relates to the identification of text content, and more specifically, to the analysis of image data of objects in a captured scene and the use of machine learning models to identify characters in the text content presented in one or more lines of the captured object.

[0003] For example, a scene can be associated with one or more roads, and the objects captured by the image data can include pedestrians, vehicles, license plates, and road signs. The text content can include information associated with the objects, such as a sequence of characters associated with a vehicle license plate.

[0004] Humans can identify objects in image data, as well as the textual content associated with those objects. Alternatively, by way of example, a system for processing image data can be configured to identify objects within the image data and to identify associated textual content using traditional techniques such as Bayesian statistical classification and pattern recognition.

[0005] Artificial intelligence (AI) is a machine-demonstrated intelligence, representing the ability of computer programs or machines to think and learn. Multiple machine learning models can be trained using one or more computers based on training samples, and inference operations can be performed on each trained machine learning model to process model inputs and generate model outputs.

[0006] Neural networks are a subfield of machine learning models. A neural network can use one or more layers of nodes to represent various computations. One or more computers can be configured to perform the computations of a neural network to generate outputs, such as classification, prediction, or segmentation of received inputs. In addition to the output layer, some neural networks also include one or more hidden layers. The output of each hidden layer serves as the input to the next layer in the network, i.e., the next hidden layer or output layer. Each layer of the network generates its output from the received inputs based on the current values ​​of its corresponding set of network parameters. Summary of the Invention

[0007] This specification describes techniques for identifying text content (e.g., character sequences associated with text content) from objects captured from image data, particularly objects with more than one line of text.

[0008] The term "object" in this specification refers to any suitable object captured in image data and having one or more lines of text. For example, an object can be one or more road signs, billboards, or landmarks, each including one or more lines of text. In some cases, an object can be associated with one or more vehicles (e.g., vans, bicycles, and motor vehicles); for example, an object can be a sticker or decal affixed to a vehicle and having one or more lines of text. As another example, an object can be a license plate attached to a vehicle.

[0009] Recognizing the text content of vehicle license plates can be applied in various scenarios, such as monitoring vehicle traffic on local roads or highways, improving garage entrance management, monitoring toll fees for vehicles passing through specific highways or bridges, assisting criminal investigations, and enhancing public security. License plate recognition can be used to identify vehicle information as well as vehicle owner information.

[0010] Some traditional license plate recognition technologies are inefficient. For example, some technologies involve a significant amount of human inference, such as having one or more people view an image or video containing a vehicle's license plate and identify the corresponding characters. As another example, some technologies use manually extracted features to analyze the input image or video to obtain license plate information, such as the Local Binary Pattern (LBP) method, the Speeded-up Robust Feature (SURF) method, the Scale-invariant Feature Transform (SIFT) method, and the Histogram of Oriented Gradients (HOG) method. These technologies are less efficient compared to systems that incorporate one or more machine learning models trained to recognize license plates.

[0011] However, some other techniques using machine learning models are not accurate enough. For example, when the input image contains different license plate types (e.g., single-line or multi-line), traditional techniques fail to generate predictions of characters in different types of license plates with satisfactory accuracy. Other techniques address this issue by using different machine learning models to process different types of license plates, which can be very time-consuming and computationally expensive.

[0012] The techniques described in the following specification are designed to address the aforementioned problems faced by conventional techniques. The described techniques involve identifying different types of vehicle license plates from input data, such as single-line or multi-line types (i.e., license plate types including multi-line text), using one or more end-to-end trained machine learning models to identify the text (e.g., character sequences) associated with the vehicle license plate.

[0013] Typically, the method, system, or one or more computer-readable storage media performing the described techniques can at least identify the type and text associated with each object in the input data. The input data can be one or more frames of images, each frame capturing one or more objects. As previously stated, each object is associated with at least text and a specific type, such as a vehicle license plate.

[0014] The technology disclosed in this application can use a first machine learning model (e.g., a first deep neural network) to process multiple frames of images to generate a first set of model outputs. The first set of model outputs can indicate the presence of a specific object and the features associated with that specific object. Features may include, for example, the object's color, multiple keypoints associated with the object, and the confidence level of the object's presence. The color may be the object's background color. The technology disclosed in this application can also determine the type of a specific object, including single-line and multi-line types as described above.

[0015] Once the type is determined, the technology disclosed in this application can generate a single-line representation of the object as representation data. The technology disclosed in this application can use a second machine learning model (e.g., a second deep neural network) to process the representation data to generate a character sequence corresponding to the text associated with the object. Furthermore, the technology disclosed in this application can process the sequence by identifying and removing spaces and redundant occurrences.

[0016] The term "vehicle license plate" in this specification refers to a license plate affixed to a vehicle for official identification purposes. As previously stated, the term "vehicle" refers to various vehicles that travel on public roads, including motor vehicles such as cars, trucks, motorcycles, or tractors. Each vehicle's license plate includes a sequence of characters (i.e., text content) that uniquely identifies the vehicle within the jurisdiction where the license plate was issued. The character sequence can have different types and lengths depending on the jurisdiction. For example, the character sequence can be single-line, double-line, or multi-line. As another example, the character sequence can have a length of one or more characters, such as 2, 4, 7, 9, and 12. For simplicity, the term "vehicle license plate" is sometimes also referred to as "license plate" in this specification.

[0017] In the specification, the term "character" corresponding to the identified license plate refers to the text associated with the license plate. Characters may include numbers, letters, regional symbolic characters (e.g., Chinese characters, Hindi characters, Arabic characters, Japanese characters, or Korean characters), and optional symbols (e.g., dots, underscores, or graphic symbols between characters).

[0018] The term "machine learning model" in this specification refers to any suitable model used for machine learning. As an example, a machine learning model may include one or more neural networks trained to perform different inference tasks. Examples of neural networks and the tasks they perform are described in more detail at the end of this specification. For simplicity, the term "machine learning model" will sometimes also be referred to as a "neural network model" or "deep neural network" throughout this specification.

[0019] Specific embodiments of the subject matter described in this specification may be implemented to achieve one or more of the following advantages.

[0020] First, the technology described in this specification can automatically perform the process of recognizing license plates from input data (e.g., input images or videos) and identifying character sequences associated with the license plates. The described technology is based on one or more machine learning models (e.g., neural networks) and is faster and more accurate than other non-machine learning models in verifying and recognizing license plates.

[0021] Second, the technology described in this specification provides a robust and efficient distinction between different types of license plates. Specifically, the technology can determine the type of the license plate being identified. License plate types can include single-line and multi-line types. A system implementing the technology can process input images or videos capturing different types of license plates in various scenarios and accurately determine the corresponding type of the license plate. Thus, the system does not need to use different models (e.g., a first model trained on data containing only single-line license plates, and a second model trained on data containing only double-line license plates) to confirm and identify different types of license plates.

[0022] Furthermore, the techniques described in this specification are highly efficient in terms of training. Systems implementing these techniques may include one or more deep neural network models that are sequentially integrated and trained end-to-end based on the same training sample set (e.g., multiple training images or videos containing different types of license plates). Throughout this specification, the term "end-to-end training" means that all parameters (e.g., node weights in each network layer) are trained or updated simultaneously during backpropagation under the constraint of a common loss function. The common loss function takes into account the task or accuracy metric of each of the one or more deep neural networks integrated into the system. Because the one or more deep neural networks are trained end-to-end, the parameters of each neural network are trained (e.g., converge) substantially simultaneously. This is more advantageous than training two or more neural network models separately using their respective loss functions, as the training period is shorter and the output accuracy of the trained system is higher.

[0023] Furthermore, the techniques described in this specification can be readily applied to processing input data from different jurisdictions. In some cases, the system can be trained based on training samples collected for a first jurisdiction or geographic region (e.g., China) and used to perform inference operations specified in the trained system to identify license plates in input data associated with a second jurisdiction or geographic region (e.g., the United States). In other cases, the system can be trained or retrained based on training samples collected for a third region (e.g., India) and configured to perform inference operations to process input data associated with the third region.

[0024] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter of this application will become apparent from the description, drawings, and claims. Attached Figure Description

[0025] Figure 1 An example system is shown for generating output data corresponding to the identified license plate after processing the input data.

[0026] Figure 2A An example license plate feature engine is shown for generating aggregated data after processing the input data.

[0027] Figure 2B An example license plate with key points is shown.

[0028] Figure 3A An example license type engine is shown for determining license type.

[0029] Figure 3B An example license plate processing engine is shown for generating representation data after processing aggregated data.

[0030] Figure 4 An example of a single-line representation of a two-line license plate is shown.

[0031] Figure 5A An example license plate recognition engine is shown for processing representation data to generate output data corresponding to the identified license plate.

[0032] Figure 5B It shows the use of Figure 5A This is an example of the decoder's process for generating output data.

[0033] Figure 6 This is a flowchart illustrating an example of the process of generating output data corresponding to objects identified from the input data.

[0034] In the various figures, the same reference numerals and names indicate the same elements. Detailed Implementation

[0035] Figure 1 An example system 100 is shown, configured to generate output data 180 corresponding to the license plate being identified after processing input data 110. Although the following description outlines techniques for text recognition of different types of vehicle license plates, it should be understood that, as mentioned above, these techniques can be applied to identify other objects in image data and recognize text content associated with the identified objects. Exemplarily, other objects may be road signs, billboards, landmarks, and stickers affixed to one or more motor vehicles with one or more lines of text.

[0036] Generally, system 100 is configured to identify license plates from input data 110, which includes multiple license plates, and to identify one or more characters associated with the identified license plate. System 100 is an example of a system implemented on one or more computers at one or more locations, and the systems, components, and techniques described below can also be implemented in system 100. Some components of system 100 can be implemented as computer programs that run on one or more computers.

[0037] like Figure 1 As shown, system 100 may include a license plate processing subsystem 105 configured to receive input data 110 and generate output data 180. System 100 may also include a memory unit 190 communicatively coupled to the license plate processing subsystem 105. The memory unit 190 may provide input data 110 to the license plate processing subsystem 105 and may store the output data 180 generated by the license plate processing subsystem 105. For example, the memory unit 190 may receive and store images or videos captured by an image sensor and provide the stored images or videos as input data 110 to the license plate processing subsystem 105. In some embodiments, the memory unit 190 may be an on-chip memory located on a chip, with the license plate processing subsystem 105 compiled and deployed on the same chip. Alternatively, the memory unit 190 may be wirelessly coupled to the license plate processing subsystem 105 and located in a different location.

[0038] System 100 may further include a server 195 communicatively coupled to license plate processing subsystem 105. Server 195 may provide trained or updated neural network parameters to configure one or more deep neural networks in license plate processing subsystem 105. The neural network parameters may be stored in memory unit 190 and retrieved by license plate processing subsystem 105 each cycle, or server 195 may directly provide these neural network parameters to license plate processing subsystem 105. In some implementations, server 195 may retrieve output data 180 stored in memory unit 190 at specific time intervals. Alternatively, server 195 may directly retrieve output data 180 from the data stream output of license plate processing subsystem 105. Optionally, server 195 may retrieve input images and videos stored in memory unit 190 at specific time intervals for further processing.

[0039] Input data 110 may be sensor data, including one or more image or video frames captured from various angles of the scene, and each image may include one or more different objects in the captured scene. For example, when the scene is associated with a road (exemplarily, a local road, highway, and expressway), the different objects may be vehicles, pedestrians, traffic signs, or traffic lights. Each frame of these scene images may typically include one or more vehicle license plates. However, in some cases, one or more images or videos in input data 110 may not include any vehicle license plates.

[0040] The license plate processing subsystem 105 can use one or more engines to process the input data 110 to identify whether a license plate exists in the image of the input data, determine the features of the license plate to be identified, and generate output data 180, each output data 180 being associated with the license plate to be identified and represented as a sequence of characters identified in the license plate to be identified.

[0041] One or more engines in the license plate processing subsystem 105, such as Figure 1 As shown, the system may include a license plate feature engine 120, a license plate processing engine 140, and a license plate recognition engine 160. Among these engines 120, 140, and 160, the license plate feature engine 120 and the license plate recognition engine 160 may each include a separate machine learning model. For example, the license plate feature engine 120 may include a first machine learning model 125, while the license plate recognition engine 160 may include a second machine learning model 165. The first machine learning model 125 and the second machine learning model 165 can be any machine learning model suitable for image processing, classification, and recognition. Some examples of machine learning models will be described in more detail below.

[0042] License plate feature engine 120, such as Figure 1As shown. It can receive input data 110 and process the input data 110 using a first machine learning model 125 to generate a first model output. The first model output identifies the presence of a vehicle license plate and specifies the features of the identified license plate. The features include at least the type of the identified license plate, such as a single-line type, a double-line type, or a multi-line type. The term "single-line" type refers to a license plate type where the characters in the license plate are arranged in a single line. The term "double-line" type refers to a license plate type where the characters in the license plate are arranged in two lines. The term "multi-line" type refers to a license plate type where the characters in the license plate are arranged in multiple lines. The term "multi-line" is also referred to as "more than one line" in this specification. The license plate feature engine 120 can also aggregate the model output data to generate aggregated data 130. Further details of the license plate feature engine 120 will be discussed in conjunction with... Figure 2A Describe it.

[0043] License plate processing engine 140, such as Figure 1 As shown, the license plate feature engine 120 can receive and process aggregated data 130 associated with the identified license plate to generate representation data 150 of the identified license plate. Representation data 150 includes data that specifically describes the adjusted or transformed image pixels representing the identified license plate. For example, the license plate processing engine 140 can generate a single-line representation of a two-line license plate. Further details on generating representation data 150 will be provided in conjunction with... Figure 3B Describe it.

[0044] License plate recognition engine 160, such as Figure 1 As shown, the license plate processing engine 140 can receive representation data 150 and process the representation data 150 using a second machine learning model 165 to generate a second model output. The license plate recognition engine 160 can also process the second model output to determine and remove spaces and / or redundancy, and generate output data 180 including a sequence of recognized characters corresponding to the license plate being recognized. Details of generating and processing the second model output will be discussed later. Figure 5A and Figure 5B Describe it.

[0045] Figure 2A An example license plate feature engine 205 is shown, configured to generate aggregated data 260 after processing input data 210. For example, license plate feature engine 205 could be... Figure 1 The license plate feature engine 120 and aggregated data 260 shown can be... Figure 1 The aggregated data 130 shown, and the input data 210 can be... Figure 1 The input data is 110.

[0046] The first machine learning model 220 can be any trained machine learning model suitable for generating predictions of the presence of at least one object, and optionally, predictions of the location of the identified object. For example, the first machine learning model 220 can be trained to receive and process each input data 210 through operations specified in the first machine learning model 220 to generate a first model output 225. The input data 210 can be multiple frames of images or video captured by an image sensor. The first model output 225 may include at least data identifying whether a vehicle license plate was captured in a frame of the input image or video, and optionally, the pixel-by-pixel position of the identified license plate (e.g., pixels in the input image enclosed by a bounding box representing the identified license plate).

[0047] In some implementations, the identification of whether a license plate exists in an input image is based on the prediction of the probability of one or more pixels in the image frame representing a license plate.

[0048] The first machine learning model 220 can also crop out pixels including the license plate being identified and process the cropped image using computer vision techniques to correct the license plate. The corrected image can be used by the license plate feature engine 205 for further operations, such as generating predictions of the features of the license plate being identified.

[0049] The first machine learning model 220 can be trained using supervised learning methods to process training samples (e.g., training images or videos of one or more license plates in different scenarios) and generate model output. The training process can be end-to-end training with other machine learning models in system 100. Each training sample is labeled with reference standard information, such as the presence of a license plate, the color of the license plate, and key points and their positions within the license plate.

[0050] For a license plate, the term "color" can refer to different colors associated with the license plate. The color of a license plate can include one or more background colors, as well as one or more colors of the text content associated with the license plate (e.g., one or more lines of text or other symbolic characters). The background color and text color of the license plate can be the same or different.

[0051] For a license plate, the term "key point" used in the specification refers to one or more specific points on the license plate. One or more specific points may include vertices of the license plate, points defining different regions of the license plate, or both. For example, a first set of points may be vertices defining a first region of the license plate, and a second set of points may be vertices defining a second region of the license plate. Key points can also be used to determine the type of license plate (e.g., single-line or multi-line), which will be described in more detail below. The total number of key points varies depending on the recognition task. For example, key points may include 4, 6, 8, 10, or 20 points.

[0052] In some implementations, keypoints can also be represented by one or more pixels corresponding to license plate vertices, which are represented by multiple pixels in an image frame. One or more pixels in the training samples can be labeled as keypoints to train a first machine learning model 220 to predict keypoint information 230b of the input data 210.

[0053] The training process aims to minimize the training loss, which defines the difference between the reference standard result represented by the labeled training samples and the prediction generated by the first machine learning model 220. The parameters in the first machine learning model 220 are updated via backpropagation while minimizing the training loss. The training process may include one or more techniques such as batch normalization, dropout, linear correction units, or early stopping. The first machine learning model 220 may also be further trained using bootstrapping to obtain estimates of the mean and variance of each prediction (e.g., the pixel-wise location of a keypoint). This allows the first machine learning model 220 to introduce uncertainty into the prediction of keypoints, thereby enabling it to operate in a more stable manner.

[0054] Continue to refer to Figure 2A As shown, the first model output 225 may further include a prediction of license plate features in response to determining the presence of a captured license plate in an image frame. The features of the identified license plate may include a prediction of the color 230a of the identified license plate (e.g., background color). As previously described, exemplarily, the predicted background color may be red, yellow, blue, black, or white. For simplicity, the term "color" in the following specification refers to the license plate background color unless otherwise specified. The features of the identified license plate may also include license plate keypoint information 230b. Keypoint information 230b may include the total number of keypoints associated with the license plate and the pixel-wise positions of the keypoints relative to the input image.

[0055] The license plate type engine 240 can process the key point information 230b as input and generate type data 245 including a prediction of the type of license plate being identified. The details of how the license plate type engine 240 processes the key point information 230b will be combined with... Figure 3A Describe it.

[0056] In some implementations, the features associated with the identified license plate may also include a confidence level 230c to avoid false positives. Specifically, a first machine learning model 220 is configured to associate a prediction of the identified license plate with a confidence level 230c. The confidence level in the specification can be a real number representing the probability that the prediction is true. The higher the confidence level, the more accurate the prediction. For example, the presence of the identified license plate may be associated with a first confidence level, which indicates how likely the predicted probability of the identified license plate's presence is to be true and trustworthy. As another example, the color of the identified license plate may be associated with a second confidence level, indicating how likely the predicted color is to be true. Similarly, each key point of the identified license plate may be associated with a corresponding confidence level. The confidence level in the specification can range from 0% to 100%, for example, 80%, 90%, or 95%.

[0057] The machine learning model 220 can be provided with a preset confidence level threshold, such as 90%, and determine that predictions associated with confidence levels below this threshold are false positives and reject such predictions. For example, if the confidence level associated with a prediction that the license plate is in the input frame is 85%, which is lower than the preset confidence level threshold of 90%, the first machine learning model 220 rejects the prediction and determines that there is no license plate in the input frame.

[0058] In some implementations, the first machine learning model 220 may be a trained deep neural network comprising multiple network layers that specify operations for different tasks. For example, the first set of network layers of the deep neural network may be configured to output a cropped image based on the bounding box of the license plate after recognizing a license plate in an input image. The deep neural network may also process the cropped image representing the license plate to output features of the recognized license plate, such as color, key points, and the type of the recognized license plate.

[0059] After generating the features of the identified license plate, aggregation engine 250 can at least concatenate the aforementioned predictions (e.g., predictions of color 230a, key point information 230b, confidence level 230c, and type data 245) to generate aggregated data 260. Aggregated data 260 can include any suitable data structure to group the aforementioned predictions. For example, aggregated data 260 can be a tensor, a linked list, or a hash table. Aggregation engine 250 can determine the size of the data structure when concatenating the predictions, and optionally, determine the memory location for storing the aggregated data 260 for further operations.

[0060] Figure 2B License plates 290 and 295 are shown as examples with key points 270 and 280.

[0061] like Figure 2BAs shown, license plate 290 is a double-line license plate because it has characters arranged in two lines, while license plate 295 is a single-line license plate because the characters associated with the license plate are arranged in one line. Although Figure 2B The example license plates shown are single-line and double-line types. It should be understood that the described techniques can be applied to verify and identify other types of license plates in the issuing area (e.g., third-line or multi-line types with characters arranged in more than two lines).

[0062] Continuing with reference to the identified license plates 290 and 295 and in conjunction with the above description, the license plate feature engine 205 can apply the first machine learning model 220 to identify the license plate and crop an image containing pixels representing the license plate based on the predicted position of the license plate in the input image frame. In some cases, the cropped image may consist essentially only of pixels representing the identified license plate. For example, the identified license plate 290 is represented by pixels in the first cropped image from the first input frame, and the identified license plate 295 is represented by pixels in the second cropped image from the second input frame.

[0063] The first model output 225 generated by the first machine learning model 220 may include key point information 230b associated with the identified license plate. The first machine learning model 220 can be trained using training samples of image frames including multiple key point markers and can be configured to process input image frames to generate key point information 230b.

[0064] As previously described, key point information 230b may include a corresponding number of key points and the locations associated with the key points of the identified license plate. For example, the total number of key points 270 of the identified license plate 290 may include six different key points 270a-f. As another example, the total number of key points 280 of the identified license plate 295 may include six different key points 280a-f. Although... Figure 2B Each license plate 290 and 295 shown has six key points. It should be understood that other numbers of key points are possible, such as 8, 10, 12 and 20.

[0065] Each of the keypoints 270a-f and 280a-f is associated with a pixel-wise position or coordinates relative to the original input image frame or a cropped image of a specific license plate. In some cases, the pixel-wise position can be transformed to its actual size based at least on the sensor position, sensor characteristics, and pixel size. However, it should be understood that any suitable coordinate system can be used to represent the pixel-wise position or coordinates of the keypoint.

[0066] Figure 3A An example license plate type engine 305 is shown, configured to determine the type of license plate. Figure 3A The license plate type of the engine 305 can be equivalent to Figure 2ALicense type engine 240.

[0067] The license plate type engine 305 can receive key point information 310, which is equivalent to Figure 2A The keypoint information 230b is used to generate multiple pairs of keypoints via a pairing generator 320. These multiple pairs of keypoints are generated based on factors such as the relative positions of the keypoints. For example, a first pair of keypoints might include a first keypoint located at the top left vertex of the license plate and a second keypoint located substantially on the left edge and below the first keypoint. As another example, a second pair of keypoints might include a third keypoint located at the top right vertex of the license plate and a fourth keypoint on the right edge and below the third keypoint.

[0068] In some implementations, the pairing generator 320 can pair all the key points used for subsequent analysis. For example, continuing to refer to... Figure 2B For the identified license plate 290 shown, the pairing generator 320 can generate a first keypoint pair of keypoints 270a and 270e, and a second keypoint pair of keypoints 270b and 270f. As another example, continuing to refer to the identified license plate 295, the pairing generator 320 can generate a first keypoint pair of keypoints 280a and 280e, and a second keypoint pair of keypoints 280b and 280f. It should be noted that for the identified license plate, other possible keypoint pairs may exist, such as the keypoint pairs for license plate 290: 270a and 270d, 270a and 270b, 270e and 270d, and 270e and 270f.

[0069] Distance generator 330 can receive keypoint pairs 325 from pair generator 320 and generate a distance 335 for each keypoint pair. The distance between a pair of keypoints can be any suitable distance, such as Euclidean distance, Euclidean distance in a corrected image of the license plate being identified, or distance in a specific direction (e.g., along the direction of the first edge). In some implementations, the distance can be a pixel-per-pixel distance, such as the number of pixels between a pair of keypoints. For example, the distance can be 5 pixels, 10 pixels, 20 pixels, or more.

[0070] Combination Figure 2B For the identified license plate 290 shown, the distance generator 330 can determine a first distance between a first pair of key points (e.g., key points 270a and 270e) and a second distance between a second pair of key points (e.g., key points 270b and 270f). Similarly, as another example and in conjunction with the identified license plate 295 shown in Figure 2B, the distance generator 330 can determine a first distance between a first pair of key points (e.g., key points 280a and 280e) and a second distance between a second pair of key points (e.g., key points 280b and 280f).

[0071] The license plate type engine 305 may also include a threshold comparison unit 340 configured to determine the type of the identified license plate. To determine the type of the identified license plate, the threshold comparison unit 340 may compare the distance between one or more keypoint pairs with a preset threshold distance. For example, the threshold comparison unit 340 may compare a first distance and a second distance associated with the identified license plate 290 with a threshold distance. If both the first distance and the second distance are equal to or greater than the threshold distance, the threshold comparison unit 340 may determine that the identified license plate 290 is a two-line type license plate 345a; otherwise, it is a single-line type license plate. As another example, after comparing the first distance and the second distance associated with the identified license plate 295 with the threshold distance, if both the first distance and the second distance are less than the threshold distance, the threshold comparison unit 340 may determine that the identified license plate 295 is a single-line type license plate 345b. Alternatively, the threshold comparison unit 340 may determine that the license plate is a two-line type as long as there exists a pair of keypoints whose distance is greater than or equal to the threshold distance.

[0072] The threshold distance can vary depending on the requirements of the recognition task. As mentioned earlier, the threshold distance can be a pixel distance. For efficiency, the threshold distance can be determined based on the pixel size of the license plate. For example, assuming the license plate height captured in the image is 40 pixels, the threshold distance can be determined to be 10 pixels. As another example, the threshold distance can be a fraction of the pixel height or width of the license plate captured in the image, such as 10%, 25%, or 50% of the license plate's pixel height. In some implementations, the threshold distance can be determined based on the actual size of the license plate. For example, the threshold distance can range from 1 cm to 5 cm.

[0073] The license plate type engine 305 can generate type data 350 as output, which specifies the type of the license plate being identified. The system 100 can perform additional operations based on the type data 350.

[0074] Figure 3B An example license plate processing engine 395 is shown, configured to generate representation data 390 after processing aggregated data 360. Figure 3B The license plate processing engine 395 can be equivalent to Figure 1 The license plate processing engine is 140.

[0075] like Figure 3B As shown, the license plate processing engine 395 can process the input aggregated data 360 and generate representation data 390 for further operations.

[0076] Receiver 365 can receive aggregated data 360; for example, aggregated data 360 can be... Figure 2A Aggregated data 260 or Figure 1The aggregated data 130. The aggregated data 360 may include features predicted by the system 100 for the identified license plate. The receiver 365 may optionally receive data representing a cropped image including the identified license plate.

[0077] Image processing unit 370 can use image processing techniques to process the data received by receiver 365 to generate transformed data 380. Transformed data 380 is generally useful because the original image representing the license plate to be identified in the input image frame may be captured at a specific angle, not necessarily a frontal view, which may reduce the accuracy of recognizing the license plate characters. Transformed data 380 may be a cropped image transformed into a frontal view version.

[0078] To generate transformation data 380, image processing unit 370 can perform homography transformation on the cropped image representing the identified license plate using keypoint information. Homography transformation of an image typically involves a projection transformation between one or more related images. Although one or more license plates may be captured from different views in an image frame, the alignment of all license plates in the image frame (e.g., the alignment of keypoints in the license plates) may be related to or substantially similar to each other. Image processing unit 370 can transform each license plate image captured from different views into a common predetermined view (e.g., a front view) based on the alignment, for example, by aligning keypoints in the upper left region of the license plate.

[0079] For example, the image processing unit 370 can rescale the image representing the license plate to a different pixel size, such as scaling it from 20x20 pixels to 500x500 pixels. As another example, the image processing unit 370 can translate the image by one or more pixels or rotate the image around an axis by a certain angle, for example, translating all pixels of the image horizontally by 10 pixels and rotating all pixels clockwise by 30 degrees around an axis perpendicular to the center pixel of the image.

[0080] The license plate processing engine 395 may also include a type determination unit 375, configured to determine, based on aggregated data 360, whether the identified license plate is predicted to be a multi-line type license plate or a single-line type license plate.

[0081] In response to determining that the license plate being identified is a multi-line type license plate (e.g., a two-line type license plate), the license plate processing engine 395 can use the image processing unit 370 to process the image representing the license plate to generate a single-line representation 385 of the license plate and provide the single-line representation 385 as representation data 390 for character recognition by the system 100.

[0082] When the input data includes license plates of different types that need to be identified, generating a single-line representation of a two-line or multi-line license plate can improve the accuracy of recognizing characters associated with the license plate. The details of generating a single-line representation of a two-line license plate will be combined with... Figure 4 Describe it.

[0083] In response to determining that the identified license plate is a single-line type license plate, the license plate processing engine 395 can provide transformation data 380 as representation data 390 for the system 100 to perform additional operations.

[0084] Figure 4 A single-line representation 490 is shown as an example of a two-line type license plate 405.

[0085] like Figure 4 As shown, the double-line type license plate 405 represented by the transformation data 380 may include a width 410 and a height 413.

[0086] The license plate processing engine 395 can process dual-line license plates 405 by determining the upper part 450 and the lower part 460 of the dual-line license plate. The license plate processing engine 395 can also determine the upper height 415 and the lower height 425 corresponding to the upper and lower parts, respectively.

[0087] Generally, the license plate processing engine 395 can determine different regions based on the identified key point information. As previously described, the system 100 can determine a first region of the license plate (e.g., upper 450) based on a first set of key points and a second region of the license plate (e.g., lower 460) based on a second set of key points. The system can crop the image representing a double-line license plate into the first and second regions based on the overlapping key points in the first and second sets of key points.

[0088] The license plate processing engine 395 can crop the upper and lower parts from a two-line type license plate 405 and perform image processing techniques to transform the upper and lower parts. For example, as... Figure 4 As shown, the license plate processing engine 395 can transform the lower part 460 into a modified lower part 460b, and can transform the upper part 450 into a modified upper part 450b. In some embodiments, the license plate processing engine 395 can change at least one of the height or width of the upper and lower parts respectively during transformation 470 or 480.

[0089] For example, the lower part 460b has a modified height 435 that is different from the lower height 425, and a modified lower width 430 that is different from the width 410. As another example, the upper part 450b has a modified height 435 that is substantially aligned with the lower part 460b, and a modified upper width 420 that is different from the width 410.

[0090] Generally, the license plate processing engine 395 does not need to modify all the widths and heights of the upper and lower parts during the transformation, as long as the heights of the modified upper and lower parts can be substantially the same for alignment. For example, the license plate processing engine 395 can keep the lower part unchanged and only modify the height of the upper part to be the same as the height of the lower part.

[0091] The license plate processing engine 395 can connect the modified upper part and the modified lower part to generate a license plate like this. Figure 4 The single-line representation shown is 490.

[0092] Figure 5A An example license plate recognition engine 505 is shown, configured to process representation data 510 to generate output data 560 corresponding to the license plate being recognized. For example, Figure 5A The license plate recognition engine 505 can be similar to or the same as Figure 1 The license plate recognition engine is 160.

[0093] like Figure 5A As shown, the license plate recognition engine 505 can receive representation data 510 as input. Representation data 510 can be, for example, Figure 3B The data shown is 390 or Figure 1 The data shown is 150.

[0094] The second machine learning model 520 can be any suitable machine learning model based on a temporal network-based temporal classification (CTC) method (e.g., a CTC score or CTC loss function). The CTC method is an alternative to Hidden Markov Models, used to construct recurrent neural network models such as Long Short-Term Memory (LSTM) networks. The CTC score or CTC loss function is independent of the underlying neural network architecture and can be applied to different machine learning models to process text. In some implementations, the second machine learning model includes a neural network model configured to recognize license plate character sequences.

[0095] The second machine learning model 520 can be trained using a similar end-to-end training technique as described above, by processing training samples with character sequences from multiple known input images or videos. The second machine learning model 520 can update its model parameters while minimizing the CTC loss function. For example, when the second machine learning model 520 is a neural network model, the network parameters can be updated during backpropagation.

[0096] The trained second machine learning model 520 can process the input image (e.g., representing data 510) and generate a second model output 525 by encoding the characters in the input image. The second machine learning model can encode the input image representing a character sequence and generate a character sequence containing repeating characters. The second machine learning model 520 can distinguish between redundant and non-redundant characters using a space symbol, represented as an underscore “_” in this specification. For example, characters in a sequence adjacent to a space symbol on both sides are identified as non-redundant characters. As another example, two identical and consecutively positioned characters are identified as having at least one redundant character.

[0097] Optionally, the second machine learning model 520 may determine the position of the predicted "empty" character in the model output and use a space symbol "_" to represent the "empty" character. The term "empty" character in this specification refers to a situation where there are no "characters" at a specific position in the output sequence, for example, a "space" in text associated with a license plate.

[0098] Repeated characters in the decoding model output are useful because a single character can span multiple horizontal positions across the sequence represented in the input image, while some other techniques might predict multiple repetitions of that single character in the output and fail to correctly distinguish between predictions of the true character and repeated characters. For example, when the actual word displayed in the input is "too", some techniques might incorrectly identify the second "o" of "too" as redundant and predict it as the word "to".

[0099] Decoder 530 can decode the second model output 525 by identifying repeating characters and spaces in the predicted character sequence and removing repeating characters and spaces from the second model output 525 to generate the character sequence corresponding to the identified license plate, i.e., output data 560. Details of the decoding process will be discussed in conjunction with... Figure 5B Describe it.

[0100] Figure 5B It shows the use of Figure 5A The example process 599 shown illustrates how the decoder 530 generates output data 560. For convenience, the example process 599 is described as being executed by a system of one or more computers located in one or more locations. For example, when properly programmed, Figure 5A License plate recognition engine 505 Figure 1 License plate recognition engine 160 or Figure 1 System 100 can execute process 599.

[0101] like Figure 5BAs shown, the second machine learning model 520 can use CTC encoding technology to generate second model outputs 570a or 570b that include examples of character sequences. Each second model output 570a or 570b can include a predetermined length 585, for example, the number of positions of the character sequence included in the second model output 525. The length can be 5, 10, 20, and 50, depending on the specific recognition requirements. This length can be preset before or during training the second machine learning model 520. Because the second machine learning model 520 uses spaces to decode redundant characters and "empty" characters, the second model outputs 570a or 570b can include character sequences of various sizes that are recognized and are less than or equal to the predetermined length 585.

[0102] Each predicted sequence (i.e., the second model outputs 570a, 570b used to process the corresponding input) may include multiple positions 590a-z or 595a-z, each position may represent a predicted character or space associated with the license plate being identified. For example, positions 590c-k represent the sequence “AA_B_1_11”, and positions 595c-k represent the sequence “A__BB__11”. Sequences 570a and 570b may contain one or more spaces at the beginning or end, or both at the beginning and end.

[0103] Decoder 530 can identify one or more spaces and redundant characters from the second model output (e.g., sequences 570a and 570b). In some implementations, decoder 530 can determine that two or more identified characters are repeated if they are the same characters and are adjacent to each other without spaces. For example, decoder 530 can determine that one of the two "A"s in positions 590c and 590d of sequence 570a is redundant, and one of the two "1"s in positions 590j and 590k is redundant because the two "A"s and the two "1"s are adjacent to each other without one or more spaces in between. Similarly, decoder 530 can determine that one of the two "B"s in positions 595f and 595g of sequence 570b is redundant. Therefore, decoder 530 can remove spaces and redundant characters and generate output data 580a of the second model output 570a as "AB11", and output data 580b of the second model output 570b as "AB1". It should be understood that decoder 530 can be used in conjunction with... Figure 5B The sequences 570a and 570b shown are processed in different configurations to handle character sequences.

[0104] Figure 6 This is a flowchart of an example process 600 for generating output data corresponding to objects identified from the input data. For convenience, example process 600 is described as being executed by a system of one or more computers located in one or more locations. For example, when properly programmed, Figure 1 System 100 can execute process 600.

[0105] The system can receive input data 610 including one or more objects. As previously described, the system may include a first machine learning model trained to process the input data. The input data may include one or more images or videos acquired by one or more image sensors. The input data may capture one or more scenes with one or more objects, each object being associated with at least text and a specific type. In some implementations, the one or more objects may include one or more vehicle license plates.

[0106] The system can process input data to generate a first model output 620. More specifically, the first machine learning model can generate the first model output by processing an input image or video that includes one or more objects. The first model output may include data that at least identifies the presence of a specific object in the input data. In some implementations, the first model output may also include a cropped image representing the object identified from the original input image frame.

[0107] When the first model output indicates a specific object in the input image, the first machine learning model can also generate a prediction associated with the first model output, which specifies the features of the identified object. The features may include at least one of the following: a background color detected in the object, keypoint information associated with the identified object, or a confidence level associated with the prediction of the object.

[0108] The system can determine the type of a specific object based on features 630. More specifically, the system can be configured to determine the type of the object to be identified based at least on key point information. The type can include single-line type and multi-line type. The object to be identified can be a single-line type with a single line of text or a multi-line type with multiple lines of text.

[0109] The system can generate multiple pairs of keypoints and calculate the distance between each pair. The system can then compare these distances to a predetermined distance threshold. Based on the comparison, the system can determine whether the identified object is a single-row or double-row type. Assuming the calculated distance meets (e.g., is greater than or equal to) the predetermined distance threshold, the system can determine that the corresponding object is a double-row type; otherwise, the corresponding object is a single-row type.

[0110] The system can generate a single-line representation of a specific object as representation data 640 based on data for a specified specific object. The system may include an image processing unit configured to perform homography transformation on a cropped image representing the corresponding object and generate transformed data representing cropped images from different viewpoints (e.g., a front view). As previously described, the system can determine whether the corresponding object is a two-line license plate. In response to determining that the corresponding object is of a two-line type, the image processing unit can also process the transformed data to generate a single-line representation of the object and provide the single-line representation as representation data for further operations. In response to determining that the corresponding object is of a single-line type, the system can directly provide the transformed data as representation data for single-line type objects. It should be understood that generating single-line representations for corresponding multi-line type (e.g., two-line type) objects can improve accuracy when the system applies the same machine learning model to recognize characters corresponding to different types of objects.

[0111] The system can process representation data to generate a second model output 650. More specifically, the system may include a second machine learning model trained end-to-end with the first machine learning model as described above. The second machine learning model is configured to process representation data (e.g., a single-line representation of a two-line type object) and generate a second model output. The second model output may include an initial sequence of predicted characters corresponding to text associated with a specific object. Notably, the second machine learning model is based on a CTC score or a CTC loss function and may encode the input representation data to recognize spaces or redundant characters. Therefore, the second model output may have one or more spaces or redundant characters in the predicted character sequence.

[0112] The system can determine whether there are one or more spaces or one or more redundancies in the second model output. As previously mentioned, the system can determine that at least one of two consecutively placed identical characters is repeated, and that a character is non-redundant if each side of the character is adjacent to a space symbol in the sequence. The system can use one or more counters to determine the total number of spaces or redundancies. For example, in a fixed-length second model output with 20 positions, the system can identify 5 spaces and 10 redundant characters.

[0113] In response to determining the presence of one or more spaces or one or more redundancies in the second model output, the system can remove one or more spaces or one or more redundancies from the second model output to generate a predicted character sequence 670 corresponding to the text associated with a specific object. As previously stated, the generated character sequence without spaces and repetitions is the output data generated by the system.

[0114] The terms "first machine learning model" and "second machine learning model" in this specification can refer to neural network models applicable to the tasks described herein. Depending on the task, the neural network can be configured to receive any type of numerical data input through training and generate any type of score, classification, or regression output based on the input.

[0115] In some cases, a neural network is a network configured to perform image processing tasks, that is, to receive an input image and process it to generate a network output. In this specification, processing an input image refers to using a neural network to process the intensity values ​​of the image's pixels. For example, the task might be image classification, where the output generated by the neural network for a given image might be a score for each object in a set of object categories, each score representing an estimated probability that the image contains an object belonging to that category. As another example, the task could be image embedding generation, where the output generated by the neural network could be a digital embedding of the input image. As yet another example, the task could be object detection, where the output generated by the neural network could identify locations in the input image that depict a specific type of object. As yet another example, the task could be image segmentation, where the output generated by the neural network could assign each pixel of the input image to one of a set of categories.

[0116] As another example, if the input to the neural network is an internet resource (such as a webpage), a document, or a portion of a document, or features extracted from an internet resource, document, or portion of a document, the task could be to classify the resource or document. That is, the output generated by the neural network for a given internet resource, document, or portion of a document could be a score for each topic in a set of topics, with each score representing an estimated probability that the internet resource, document, or portion of a document is related to the topic.

[0117] As another example, if the input to the neural network is features of the impression context of a particular ad, the output generated by the neural network could be a score representing the estimated probability that the particular ad will be clicked.

[0118] As another example, if the input to the neural network is features of personalized recommendations for the user, such as features representing the recommendation context, for example, features representing the user's previous actions, then the output generated by the neural network can be a score for each of a set of content items, each score representing an estimated probability that the user will respond positively to the recommended content item.

[0119] As another example, if the input to a neural network is a sequence of text in one language, the output generated by the neural network may be a score for each of a set of text fragments in another language, where each score represents an estimated probability that the text fragment in the other language is correctly translated into the other language.

[0120] As another example, the task could be an audio processing task. For instance, if the input to a neural network is a sequence representing spoken expression, the output generated by the neural network could be a score for each of a set of text segments, each score representing an estimated probability that the text segment is a correct transcription of the utterance.

[0121] As another example, this task could be a keyword discovery task, where the neural network's input is a sequence representing spoken utterances, and the output it generates could be an indication of whether a specific word or phrase ("hot word") was mentioned in the utterance. As yet another example, if the neural network's input is a sequence representing spoken utterances, the output it generates could identify the natural language used in the utterance.

[0122] As another example, the task can be a natural language processing or understanding task, such as an entailment task, a paraphrasing task, a text similarity task, an emotion task, a sentence completion task, a grammar task, etc., which requires operating on a text sequence of some natural language.

[0123] As another example, the task could be a text-to-speech task, where the input is natural language text or features of natural language text, and the network output is a spectrogram or other data that defines the audio as the text read aloud in natural language.

[0124] As another example, the task could be a health prediction task, where the input is a patient’s electronic health record data and the output is a prediction related to the patient’s future health, such as a predicted treatment that should be prescribed for the patient, the likelihood of the patient experiencing an adverse health event, or a predicted diagnosis for the patient.

[0125] As another example, the task can be an agent control task, where the input is an observation characterizing the state of the environment, and the output defines the action the agent should perform in response to the observation. For example, the agent can be a real-world or simulated robot, a control system of an industrial facility, or a control system that controls different types of agents.

[0126] As another example, the task could be a genomics task, where the input is a sequence representing a fragment of DNA sequence or other molecular sequence, and the output is a fragment embedding for downstream tasks, such as using unsupervised learning techniques on a dataset of DNA sequence fragments or the output of a downstream task. Examples of downstream tasks include promoter site prediction, methylation analysis, predicting the functional effects of non-coding variants, and so on.

[0127] In some cases, a machine learning task is a combination of multiple individual machine learning tasks; that is, a neural network is configured to perform multiple different individual machine learning tasks, such as two or more of the machine learning tasks mentioned above. For example, a neural network can be configured to perform multiple individual natural language understanding tasks. Optionally, the network input may include identifiers of the individual natural language understanding tasks to be performed on the network input. As another example, a neural network can be configured to perform multiple individual image processing or computer vision tasks, that is, to generate outputs in parallel for multiple different individual image processing tasks by processing a single input image.

[0128] Embodiments of the subject matter and functional operation described in this specification can be implemented in digital electronic circuits, tangibly embodied computer software or firmware, computer hardware, including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, such as one or more computer program instruction modules encoded on a tangible, non-transitory storage medium, for performing or controlling the operation of a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or one or more combinations thereof. Alternatively or additionally, program instructions can be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by the data processing apparatus.

[0129] The term "data processing apparatus" refers to data processing hardware, including various means, devices, and machines for processing data, such as programmable processors, computers, or multiple processors or computers. The apparatus may also be or include special-purpose logic circuitry, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). In addition to hardware, the apparatus may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0130] A computer program (also referred to or described as a program, software, software application, application, module, software module, script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages. It can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but does not necessarily, correspond to a file in a file system. A program may be stored as part of a file containing other programs or data, for example, one or more scripts stored in a markup language document, a single file dedicated to a problem program or multiple coordinating files, for example, a file storing one or more modules, subroutines, or portions of code. A computer program can be deployed to execute on a single computer or on multiple computers located at one site or distributed across multiple sites and interconnected via a data communication network.

[0131] For a system consisting of one or more computers, being configured to perform a specific operation or action means that the software, firmware, hardware, or a combination thereof installed on the system causes the system to perform the operation or action during operation. For one or more computer programs to be configured to perform a specific operation or action, it means that the one or more programs include instructions that cause a data processing device to perform the operation or action when executed by the device.

[0132] As used in this specification, "engine" or "software engine" refers to a software-implemented input / output system that provides outputs distinct from its inputs. An engine can be a coded functional block, such as a library, platform, software development kit ("SDK"), or object. Each engine can be implemented on any suitable type of computing device, such as a server, mobile phone, tablet, laptop, music player, e-book reader, laptop or desktop computer, personal digital assistant (PDA), smartphone, or other fixed or portable device, including one or more processors and computer-readable media. Furthermore, two or more engines can be implemented on the same computing device or on different computing devices.

[0133] The processes and logic flows described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform functions by manipulating input data and generating output. The processing and logic flows can also be executed by dedicated logic circuitry, such as FPGAs or ASICs, or by a combination of dedicated logic circuitry and one or more programmable computers.

[0134] A computer suitable for executing computer programs can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The basic elements of a computer are the central processing unit for implementing or executing instructions and one or more storage devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated into special-purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to, receiving data from or transferring data to one or more mass storage devices (e.g., disks, magneto-optical disks, or optical disks) for storing data, or both. However, a computer does not necessarily need to have such devices. Furthermore, a computer can be embedded in other devices, such as mobile phones, personal digital assistants, mobile audio or video players, game consoles, Global Positioning System (GPS) receivers, or portable storage devices, such as Universal Serial Bus (USB) flash drives, to name a few.

[0135] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, such as semiconductor storage devices, such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0136] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer with a display device, such as a cathode ray tube (CRT) or liquid crystal display (LCD) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse, trackball, or presence-sensitive display or other surface through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's device in response to a request received from the web browser. Additionally, the computer can interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smartphone), running a messaging application, and receiving response messages from the user.

[0137] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes back-end components (e.g., as a data server), or middleware components (e.g., an application server), or front-end components (e.g., a client computer with a graphical user interface, a web browser, or an application through which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. Components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include Local Area Networks (LANs) and Wide Area Networks (WANs), such as the Internet.

[0138] A computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The client-server relationship arises from computer programs running on their respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data, such as HTML pages, to a user device, which acts as a client, for example, to display data to a user interacting with the device and to receive user input. Data generated on the user device, such as the results of user interactions, may be received at the server from the device.

[0139] In addition to the above embodiments, the following embodiments are also innovative:

[0140] Example 1 is a method comprising: receiving input data including one or more objects, wherein the input data includes one or more frames of images, wherein the one or more objects are captured in the one or more frames of images, and each object is associated with at least text and a specific type; processing the one or more frames of images using a first machine learning model to generate a first model output, the output including data identifying the presence of a specific object among the one or more objects and specifying features of the specific object, wherein the features include at least a plurality of keypoints associated with the specific object; determining the type of the specific object based on the features, the type including single-line type and multi-line type; generating a single-line representation of the specific object as representation data based on the data specifying the specific object; and processing the representation data using a second machine learning model to generate a second model output, the second model output including an initial predicted character sequence corresponding to the text associated with the specific object; wherein the first machine learning model and the second machine learning model are simultaneously trained on the same training dataset including a set of objects.

[0141] Example 2 is the method described in Example 1, wherein the characteristics of the specific object further include at least one of color or confidence level of the presence of the specific object in the input data, wherein the color includes at least one of background color or text color associated with the specific object.

[0142] Example 3 is the method described in Example 1 or Example 2, wherein a specific object corresponds to a license plate, and determining the type of the specific object includes: determining one or more pairs of key points among a plurality of key points associated with the vehicle license plate; calculating the distance between each pair of key points; determining whether at least one of the distances of each pair exceeds a distance threshold; in response to determining that at least one of the distances exceeds the distance threshold, determining that the vehicle license plate is of a multi-line type; and in response to determining that the distances do not exceed the distance threshold, determining that the vehicle license plate is of a single-line type.

[0143] Example 4 is the method described in Example 3, wherein generating a single-line representation of a specific object as representation data includes: in response to determining that the vehicle license plate is of a multi-line type, generating a single-line representation of the multi-line type vehicle license plate as representation data by modifying the data of the multi-line type license plate; and in response to determining that the vehicle license plate is of a single-line type, providing the data of the specified single-line type license plate as representation data to the second machine learning model.

[0144] Example 5 is the method of any one of Examples 3 or 4, wherein generating a single-line representation of a multi-line type license plate as representation data includes: modifying data representing a multi-line type vehicle license plate by correcting a multi-line type vehicle license plate image; identifying a first portion and a second portion of a corrected image of the multi-line type license plate, wherein the first portion is different from the second portion; adjusting the first portion to at least match the height of the second portion; and aligning the adjusted first portion and the second portion into a single line to form a single-line representation of the multi-line type license plate.

[0145] Example 6 is the method of any one of Examples 1 to 5, wherein generating the second model output includes: for each position of the initial predicted character sequence, generating a corresponding initial predicted character for the character at each position, wherein each initial predicted character includes at least one of a space, number, letter, or symbol character, wherein each initial prediction is associated with a corresponding probability of the initial predicted character at that position, and the respective corresponding probabilities form a probability vector; wherein the initial predicted character sequence has a fixed length, and the probability vector also has the same fixed length.

[0146] Example 7 is the method of any one of Examples 1 to 6, further comprising: determining whether one or more spaces or one or more redundancies exist in the initial predicted character sequence; and in response to determining that one or more spaces or one or more redundancies exist in the initial predicted character sequence, removing one or more spaces or one or more redundancies from the initial predicted character sequence to generate a predicted character sequence corresponding to text associated with a specific object.

[0147] Example 8 is a system comprising one or more computers and one or more storage devices storing instructions, which, when executed by the one or more computers, cause the one or more computers to perform a corresponding operation, including the method of any of Examples 1 to 7.

[0148] Example 9 is one or more computer-readable storage media for storing instructions that, when executed by one or more computers, cause the one or more computers to perform respective operations, the respective operations including the methods of any one of Examples 1 to 7.

[0149] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope that may be claimed, but rather as descriptions of features of particular embodiments of a particular invention. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations, or even claimed in this manner from the outset, in some cases one or more features may be removed from the claimed combination, and the claimed combination may be a sub-combination or a variation of a sub-combination.

[0150] Similarly, although operations are described in a specific order in the accompanying drawings, this should not be construed as requiring such operations to be performed in the specific order shown or sequentially, or to perform all the illustrated operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments. It should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0151] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require the specific order or sequence shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method, characterized in that, include: Receive input data including one or more objects, wherein the input data includes one or more frames of images, wherein the one or more objects are captured in the one or more frames of images and each object is associated with at least text and a specific type; The image frame or multiple frames are processed using a first machine learning model to generate a first model output, the first model output including data that identifies the presence of a specific object among the one or more objects and specifies the features of the specific object, wherein the features include at least a plurality of key points associated with the specific object; Determining the type of the specific object based on the features, wherein the type includes single-line type and multi-line type, wherein the specific object corresponds to a vehicle license plate, and determining the type of the specific object based on the features includes: Identify one or more pairs of key points from a plurality of key points associated with the vehicle license plate; Calculate the distance between each pair of one or more pairs of key points; Determine whether at least one of the respective distances exceeds a distance threshold; In response to determining that at least one of the respective distances exceeds the distance threshold, the vehicle license plate is determined to be of a multi-line type; and In response to determining that none of the respective distances exceed the distance threshold, the vehicle license plate is determined to be of the single-line type; Generate a single-line representation of the specified object as representation data based on the data of the specified object; and The representation data is processed using a second machine learning model to generate a second model output, which includes an initial predicted character sequence corresponding to text associated with the particular object; The first machine learning model and the second machine learning model are trained simultaneously on the same training dataset, which includes a set of objects.

2. The method according to claim 1, characterized in that, The characteristics of the specific object further include at least one of color or confidence level of the presence of the specific object in the input data, wherein the color includes at least one of background color or text color associated with the specific object.

3. The method according to claim 1, characterized in that, The generation of a single-line representation of the specified object based on data of the specified object as representation data includes: In response to determining that the vehicle license plate is of a multi-line type, a single-line representation of the multi-line type vehicle license plate is generated as the representation data by modifying the data specifying the multi-line type vehicle license plate; and In response to determining that the vehicle license plate is of the single-line type, the data specifying the single-line type vehicle license plate is provided as the representation data to the second machine learning model.

4. The method according to claim 3, characterized in that, The method of generating a single-line representation of the multi-line type vehicle license plate as the representation data by modifying the data specifying the multi-line type vehicle license plate includes: The data representing the multi-line vehicle license plate is modified by correcting the multi-line vehicle license plate image; Identify a first portion and a second portion of the corrected image of the multi-line type vehicle license plate, wherein the first portion is different from the second portion; Adjust the first part to at least match the height of the second part; and The adjusted first and second parts are aligned into a single line to form a single-line representation of the multi-line type vehicle license plate.

5. The method according to claim 1, characterized in that, The process of processing the representation data using a second machine learning model to generate the output of the second model includes: For each position of the initial predicted character sequence, a corresponding initial predicted character is generated for each position, wherein each initial predicted character includes at least one of a space, a number, a letter, or a symbol character, wherein each initial predicted character is associated with a corresponding probability of the initial predicted character at that position, and wherein the corresponding probability forms a probability vector; wherein the initial predicted character sequence has a fixed length, and the probability vectors have the same fixed length.

6. The method according to any one of claims 1 to 5, characterized in that, Also includes: Determine whether there are one or more spaces or one or more redundancies in the initial predicted character sequence; as well as In response to determining that one or more spaces or one or more redundancies exist in the initial predicted character sequence, the one or more spaces or one or more redundancies are removed from the initial predicted character sequence to generate a predicted character sequence corresponding to the text associated with the specific object.

7. A system, characterized in that, One or more storage devices comprising one or more computers and storage instructions, wherein when the instructions are executed by the one or more computers, the one or more computers perform corresponding operations, the operations including: Receive input data including one or more objects, wherein the input data includes one or more frames of images, wherein the one or more objects are captured in the one or more frames of images and each object is associated with at least text and a specific type; The image frame or multiple frames are processed using a first machine learning model to generate a first model output, the first model output including data that identifies the presence of a specific object among the one or more objects and specifies the features of the specific object, wherein the features include at least a plurality of key points associated with the specific object; Determining the type of the specific object based on the features, wherein the type includes single-line type and multi-line type, wherein the specific object corresponds to a vehicle license plate, and determining the type of the specific object based on the features includes: Identify one or more pairs of key points from a plurality of key points associated with the vehicle license plate; Calculate the distance between each pair of one or more pairs of key points; Determine whether at least one of the respective distances exceeds a distance threshold; In response to determining that at least one of the respective distances exceeds the distance threshold, the vehicle license plate is determined to be of a multi-line type; and In response to determining that none of the respective distances exceed the distance threshold, the vehicle license plate is determined to be of the single-line type; Generate a single-line representation of the specified object as representation data based on the data of the specified object; and The representation data is processed using a second machine learning model to generate a second model output, which includes an initial predicted character sequence corresponding to text associated with the particular object; The first machine learning model and the second machine learning model are trained simultaneously on the same training dataset, which includes a set of objects.

8. The system according to claim 7, characterized in that, The characteristics of the specific object further include at least one of color or confidence level of the presence of the specific object in the input data, wherein the color includes at least one of background color or text color associated with the specific object.

9. The system according to claim 7, characterized in that, The generation of a single-line representation of the specific object based on data specifying the specific object as the representation data includes: In response to determining that the vehicle license plate is of a multi-line type, a single-line representation of the multi-line type vehicle license plate is generated as the representation data by modifying the data specifying the multi-line type vehicle license plate; and In response to determining that the vehicle license plate is of the single-line type, the data specifying the single-line type vehicle license plate is provided as the representation data to the second machine learning model.

10. The system according to claim 9, characterized in that, The method of generating a single-line representation of the multi-line type vehicle license plate as the representation data by modifying the data specifying the multi-line type vehicle license plate includes: The data representing the multi-line vehicle license plate type is modified by correcting the multi-line vehicle license plate image; Identify a first portion and a second portion of the corrected image of the multi-line type vehicle license plate, wherein the first portion is different from the second portion; Adjust the first part to at least match the height of the second part; and The adjusted first and second parts are aligned into a single line to form a single-line representation of the multi-line type vehicle license plate.

11. The system according to claim 7, characterized in that, The process of processing the representation data using a second machine learning model to generate the output of the second model includes: For each position of the initial predicted character sequence, a corresponding initial predicted character is generated for that position, wherein each initial predicted character includes at least one of a space, a number, a letter, or a symbol character, wherein each initial predicted character is associated with a corresponding probability of the initial predicted character at that position, and wherein the corresponding probability forms a probability vector; wherein the initial predicted character sequence has a fixed length, and wherein the probability vectors have the same fixed length.

12. The system according to any one of claims 7 to 11, characterized in that, The operation also includes: Determine whether there are one or more spaces or one or more redundancies in the initial predicted character sequence; and In response to determining that one or more spaces or one or more redundancies exist in the initial predicted character sequence, the one or more spaces or one or more redundancies are removed from the initial predicted character sequence to generate a predicted character sequence corresponding to the text associated with the specific object.

13. One or more computer-readable storage media, characterized in that, Used to store instructions that, when executed by one or more computers, cause the one or more computers to perform the method described in any one of claims 1 to 6.

Images