Vehicle and license plate information integrated identification method and system

Abstract

The invention provides a vehicle and license plate information integrated identification method and system, and the method comprises the steps: carrying out the multi-scale feature extraction of an input image, and obtaining a shared feature map containing semantic information; wherein the input image comprises at least one vehicle. And performing context information aggregation processing on the shared feature map to obtain a target representation vector. Performing interactive processing on the joint query vector and the target representation vector to execute regression prediction of vehicle detection, license plate positioning and license plate character recognition in parallel; wherein the joint query vector comprises vehicle query, license plate key point query and character query, and the joint query vector is generated through a step-by-step expansion mechanism. According to the mode, integrated parallel regression of vehicle detection, license plate positioning and character recognition tasks in a single reasoning process can be realized, so that the comprehensive accuracy of vehicle and license plate information recognition and the system execution efficiency in a complex traffic scene are improved.

Description

Technical Field

[0001] This application relates to the field of vehicle and license plate recognition technology, and in particular to an integrated recognition method and system for vehicle and license plate information. Background Technology

[0002] With the rapid development of smart city traffic monitoring and autonomous driving technologies, real-time extraction of vehicle and license plate information from video streams has become a core requirement for traffic control and security systems. In application scenarios such as fast passage through parking lots, evidence collection of traffic violations on highways, and analysis of vehicle behavior trajectories, the recognition system is required not only to have extremely high accuracy but also excellent reasoning efficiency.

[0003] Currently, mainstream processing methods typically employ a serial cascaded architecture. This involves first using a vehicle detection model to locate the vehicle region, then using a license plate localization model to locate the license plate within that region, and finally using a character recognition model to identify the license plate image. This staged, multi-model approach results in a significant waste of computational resources in repetitive feature extraction, and the lack of semantic interaction between models leads to noticeable delays in the recognition process. Furthermore, the serial logic inevitably leads to error accumulation; even small deviations in the preceding vehicle localization can cause subsequent license plate recognition failures, making it difficult to handle complex scenarios such as significantly tilted or obscured license plates. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide an integrated method and system for vehicle and license plate information recognition. By utilizing the contextual aggregation of multi-scale shared feature maps and the stepwise expansion mechanism from vehicle to license plate key points and then to character queries, it is possible to achieve integrated parallel regression of vehicle detection, license plate localization and character recognition tasks in a single inference process, thereby improving the overall accuracy and system execution efficiency of vehicle and license plate information recognition in complex traffic scenarios.

[0005] In a first aspect, the present invention provides an integrated identification method for vehicle and license plate information, comprising: Multi-scale feature extraction is performed on the input image to obtain a shared feature map containing semantic information; wherein the input image contains at least one vehicle.

[0006] The shared feature map is processed by contextual information aggregation to obtain the target representation vector.

[0007] The joint query vector and the target representation vector are interactively processed to perform regression predictions for vehicle detection, license plate localization, and license plate character recognition in parallel. The joint query vector includes vehicle query, license plate key point query, and character query, and is generated through a hierarchical expansion mechanism.

[0008] During an inference process, the output image contains the vehicle bounding box coordinates, license plate corner coordinates, and license plate character sequence information for each vehicle in the input image.

[0009] In an optional implementation, the step of performing multi-scale feature extraction on the input image to obtain a shared feature map containing semantic information includes: The input image is processed by a backbone network composed of multiple residual modules to generate hierarchical feature maps with different spatial resolutions. The hierarchical feature maps contain semantic features and edge texture information of the vehicle region and license plate region in the corresponding input image.

[0010] The hierarchical feature maps are integrated through a multi-scale fusion mechanism to obtain an initial shared feature map.

[0011] A positional encoding is introduced into the initial shared feature map, and the positional encoding is fused with the initial shared feature map to output a shared feature map; wherein, the positional encoding is used to characterize the spatial positional relationship of each feature vector in the shared feature map in the input image.

[0012] In an optional implementation, the step of performing context information aggregation on the shared feature map to obtain the target representation vector includes: The shared feature map is input into the preset encoder; the preset encoder has a multi-scale deformable attention mechanism.

[0013] By using a pre-defined encoder, contextual information is aggregated from hierarchical feature maps with different spatial resolutions in the shared feature map to generate a target representation vector.

[0014] In an optional implementation, the joint query vector is generated through the following steps: Vehicle queries are obtained by filtering the target representation vector; vehicle queries include vehicle content queries and vehicle location queries.

[0015] Starting with vehicle search, initialize license plate key point search.

[0016] Based on the key points of the license plate, the coordinates of the corresponding corner points of the license plate are queried, and the center curve of the license plate is generated by fitting a Bézier curve.

[0017] Point sampling is performed on the center curve of the license plate to generate a license plate character query.

[0018] By combining vehicle queries, license plate key point queries, and license plate character queries, a joint query vector is obtained.

[0019] In an optional implementation, the step of performing point sampling on the center curve of the license plate to generate a license plate character query includes: Equidistant sampling is performed on the center curve of the license plate to obtain a preset number of sampling points, and license plate character queries are generated based on the sampling points.

[0020] In an optional implementation, the steps of performing regression prediction for vehicle detection, license plate localization, and license plate character recognition in parallel include: The system generates initial vehicle detection results and initial license plate prediction results, and obtains candidate regions with high confidence through confidence screening.

[0021] The candidate region is used as the input of the refined query vector into the preset decoder. The preset decoder is used to optimize the refined query vector layer by layer with reference target representation vector, so as to obtain the vehicle bounding box coordinates, license plate corner coordinates and license plate character sequence information through synchronous regression. The preset decoder has a multi-scale deformable attention mechanism.

[0022] In an optional implementation, after the step of outputting the vehicle bounding box coordinates, license plate corner coordinates, and license plate character sequence information for each vehicle in the input image during one inference process, the method further includes: Obtain the annotation information corresponding to the input image.

[0023] A joint loss function is constructed, and the total loss value between the recognition result and the labeled information is calculated based on the joint loss function. The recognition result is the vehicle bounding box coordinates, license plate corner coordinates, and license plate character sequence information corresponding to each vehicle.

[0024] The recognition parameters required to generate the recognition results are adjusted based on the total difference loss value; the recognition parameters include backbone network parameters, preset encoder parameters, query generation parameters, and preset decoder parameters.

[0025] In an optional implementation, the step of constructing a joint loss function and calculating the total loss value between the recognition result and the labeled information based on the joint loss function includes: The bounding box regression bias of the vehicle bounding box is calculated using absolute error loss and generalized crossover ratio loss.

[0026] The prediction bias for vehicle classification is calculated using focus loss.

[0027] The corner regression bias of the license plate corner points is calculated by combining the absolute error loss and the object key point similarity loss.

[0028] The degree of mismatch between the predicted character sequence and the labeled character sequence is measured using connectionist temporal classification loss.

[0029] The total difference loss is obtained by weighting and summing the bounding box regression bias, prediction bias, corner regression bias, and mismatch degree.

[0030] In an optional implementation, after obtaining the annotation information corresponding to the input image, the method further includes: The Hungarian matching algorithm enables a one-to-one correspondence between the output recognition results and the multiple labeled targets in the annotation information.

[0031] Secondly, the present invention provides an integrated vehicle and license plate information recognition system, comprising: The feature extraction module is used to perform multi-scale feature extraction on the input image to obtain a shared feature map containing semantic information; wherein the input image contains at least one vehicle.

[0032] The aggregation processing module is used to aggregate contextual information from the shared feature map to obtain the target representation vector.

[0033] The regression prediction module is used to interact with the joint query vector and the target representation vector to perform regression prediction of vehicle detection, license plate localization and license plate character recognition in parallel. The joint query vector includes vehicle query, license plate key point query and character query, and the joint query vector is generated through a hierarchical expansion mechanism.

[0034] The information output module is used to output the vehicle bounding box coordinates, license plate corner coordinates, and license plate character sequence information for each vehicle in the input image during a single inference process.

[0035] This application provides an integrated vehicle and license plate information recognition method and system. By utilizing feature extraction from multi-scale shared feature maps, a step-by-step expansion mechanism based on vehicle-to-license plate character queries, and parallel regression prediction logic combined with Bézier curve fitting, high-precision vehicle location, license plate corner points, and character sequence information can be output synchronously during a single inference process. This eliminates the technical bottlenecks of redundant computing resources and step-by-step error accumulation in traditional serial recognition schemes, thereby enhancing the system's robustness and processing efficiency in handling complex deformation scenarios and realizing closed-loop parameter optimization capabilities based on feedback information.

[0036] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the application. The objectives and other advantages of this application are realized and obtained through the structures particularly pointed out in the description, claims and drawings.

[0037] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0039] Figure 1 A flowchart illustrating the integrated vehicle and license plate information recognition method provided in this application embodiment; Figure 2 A flowchart of the shared feature map generation method provided in the embodiments of this application; Figure 3 A flowchart of the target representation vector generation method provided in the embodiments of this application; Figure 4 The following is a flowchart of the joint query vector generation method provided in the embodiments of this application; Figure 5 This is a schematic diagram of an integrated vehicle and license plate information recognition system provided in an embodiment of this application; Figure 6 This is a schematic diagram of another integrated vehicle and license plate information recognition system provided in an embodiment of this application.

[0040] Icons: 1-Feature extraction module; 2-Aggregation processing module; 3-Regression prediction module; 4-Information output module; 5-Optimization module. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0042] To help those skilled in the art better understand this application, a brief introduction to its application scenarios and design concepts is provided.

[0043] In scenarios such as intelligent traffic management, unmanned parking lot management, autonomous driving visual perception, and security monitoring, it is necessary to process massive amounts of surveillance images in real time and accurately extract the physical location of vehicles, the precise location of license plates, and specific license plate character information. With the increase in traffic flow and the increasing complexity of recognition scenarios, how to reduce system latency and improve the ability to handle distorted license plates while ensuring extremely high recognition accuracy is currently a focus of attention in the industry.

[0044] Traditional vehicle and license plate recognition schemes mostly rely on a serial cascaded architecture that first detects the vehicle, then locates the license plate, and finally recognizes the characters. This architecture faces several challenges in practical applications: First, each stage of the task is performed by isolated models, and small localization deviations in earlier steps are amplified at each stage of the processing chain, resulting in a severe error accumulation effect. Second, multiple models extracting features separately leads to a large amount of redundant computation, significantly increasing hardware power consumption and inference time. Finally, when dealing with license plates with complex deformations such as tilt or bending caused by the shooting angle, traditional schemes often lack guidance from global semantic information, causing sampling points to deviate from the character center and resulting in a significant drop in recognition rate.

[0045] Based on this, this application provides an integrated method for recognizing vehicle and license plate information. By constructing a shared feature map containing semantic information, it enables parallel execution of vehicle detection, license plate localization, and character recognition tasks within the same feature space. This eliminates the isolation between tasks at the source, thus avoiding error accumulation. Furthermore, by leveraging the semantic complementarity between multiple tasks, the robustness of the overall recognition is significantly improved. Simultaneously, this application provides a mechanism that progressively expands from vehicle query to license plate key point query, and then to license plate character query, simulating the human visual perception process from the whole to the part. Through the interactive processing of joint query vectors under a multi-scale deformable attention mechanism, it can accurately capture the spatial constraint relationships between vehicles and license plates, and between license plates and characters, thereby achieving rapid and accurate localization of multiple targets without cumbersome image cropping. Addressing the challenge of recognizing license plates with complex deformations, this application uses Bézier curves to fit the license plate key points, generating a license plate center curve. By performing equidistant sampling on this curve, the generated character query can automatically adapt to the geometric distortion of the license plate, ensuring that the sampling points are always aligned with the character center line, greatly enhancing the system's ability to recognize tilted and curved license plates.

[0046] To facilitate understanding of this embodiment, the embodiments of this application will be described in detail below.

[0047] This application provides a method for integrated identification of vehicle and license plate information, referring to... Figure 1 The integrated vehicle and license plate information recognition method provided in this application includes: Step S101: Perform multi-scale feature extraction on the input image to obtain a shared feature map containing semantic information; wherein the input image contains at least one vehicle.

[0048] Here, raw input images containing vehicle targets are acquired using a pre-set image acquisition device. To enable the method to adapt to vehicle recognition requirements at different distances and resolutions, a hierarchical backbone network is used to process the images.

[0049] The backbone network consists of multiple residual modules or feature extraction units with similar functions, capable of progressively extracting low-level edge and texture information as well as high-level abstract semantic information based on the depth of the image. Specifically, as the image flows through the network, it generates hierarchical feature maps with different spatial resolutions. For example, early-level feature maps have higher resolution, preserving rich details of license plate edges and characters, while later-level feature maps, although lower in resolution, contain stronger global semantics, helping to accurately pinpoint vehicle outlines.

[0050] To integrate this information at different scales, a multi-scale fusion mechanism is used to associate and merge these hierarchical feature maps to obtain an initial shared feature map. Based on this, positional encoding is introduced into the initial shared feature map. The positional encoding is generated using a pre-defined trigonometric function transformation or a learnable parameter matrix to represent the spatial coordinate correspondence of each feature vector in the feature map within the original input image. The final output shared feature map not only possesses rich visual semantics but also carries precise spatial location cues.

[0051] Step S102: Perform context information aggregation processing on the shared feature map to obtain the target representation vector.

[0052] Here, after obtaining the shared feature map, it is input into the preset encoder for refinement. The preset encoder is usually composed of multiple symmetrical processing layers stacked together, and its core component is a multi-scale deformable attention mechanism.

[0053] Multi-scale deformable attention mechanisms do not blindly calculate all pixels when processing shared feature maps. Instead, they selectively learn and choose representative sampling points. For example, when recognizing vehicles, the encoder automatically focuses on key feature areas such as headlights and grilles, while when recognizing license plates, it focuses on the edges of the characters.

[0054] Through multi-layer transformation, the encoder can capture long-range dependencies between vehicles and license plates, and between license plates and the background. After aggregation, the original shared feature maps are transformed into a set of high-dimensional target representation vectors. These vectors are concise representations of image information, with each vector representing a deep abstraction of the potential vehicle and its environment information in the image. Based on this, the ability to model non-ideal environments such as occlusion and complex lighting is enhanced.

[0055] Step S103 involves interactively processing the joint query vector and the target representation vector to perform parallel regression predictions for vehicle detection, license plate localization, and license plate character recognition; wherein the joint query vector includes vehicle query, license plate key point query, and character query, and the joint query vector is generated through a hierarchical expansion mechanism.

[0056] Here, vehicle query filtering is performed from the target representation vector to obtain vehicle content queries representing global vehicle features and vehicle location queries representing vehicle location regions.

[0057] Starting with vehicle queries, a pre-defined mapping function is used to initialize and generate license plate key point queries. This process simulates the human search logic of shifting from observing the vehicle as a whole to focusing on the license plate.

[0058] For license plate key point queries, the corner coordinates of the license plate are predicted and a center curve of the license plate is generated using a curve fitting module. In one embodiment, Bezier curve fitting technology can be used to fit a smooth center arc based on the four corner points, and equidistant or non-uniform sampling is performed on this curve to generate character queries.

[0059] After generating the joint query vector, it is input into a predefined decoder along with the target representation vector. Within the decoder, the joint query vector interacts with the target representation vector multiple times through a multi-scale deformable attention mechanism.

[0060] During the interaction, each query vector continuously extracts key information required for its task from image features and fine-tunes its own prediction values. This parallel processing mode allows vehicle bounding box regression, corner localization, and character sequence prediction to be completed synchronously in the same feature stream.

[0061] Step S104: During one inference process, output the vehicle bounding box coordinates, license plate corner coordinates, and license plate character sequence information corresponding to each vehicle in the input image.

[0062] Here, after the joint query vectors complete their interaction and iteration, each query vector is parsed using pre-defined regression and classification heads. The entire process requires only a single forward inference to complete the predictions for all tasks.

[0063] For each vehicle to be identified, the method will simultaneously output the vehicle bounding box coordinates in the image coordinate system, the precise coordinates of the four license plate corner points, and the license plate character sequence composed of multiple character probabilities.

[0064] To ensure a one-to-one correspondence between the predicted results and the actual vehicles in the image, the Hungarian matching algorithm is used as the target assignment logic. The Hungarian matching algorithm calculates the comprehensive cost of consistency between the predicted results and the actual targets in terms of category, location, and character, finding a unique match for each predicted vehicle identification information. Finally, it directly outputs structured information containing all recognition dimensions.

[0065] In one embodiment, reference is made to Figure 2 Step S102 includes the following steps S201-S203.

[0066] Step S201: The input image is processed based on a backbone network composed of multiple residual modules to generate hierarchical feature maps with different spatial resolutions; wherein, the hierarchical feature maps contain semantic features and edge texture information of the vehicle region and license plate region in the corresponding input image.

[0067] Here, the preprocessed input image is fed into the backbone network. The backbone network is preferably a ResNet50 with a deep residual structure. As the image passes through the backbone network, it sequentially goes through multiple convolutional stages composed of residual modules.

[0068] Specifically, the backbone network performs downsampling operations on the image, thereby generating hierarchical feature representations at different depths. These hierarchical feature maps are mainly extracted from the third, fourth, and fifth stages of the backbone network, and are denoted as hierarchical feature maps C3, C4, and C5, respectively. In terms of spatial resolution, hierarchical feature maps C3, C4, and C5 correspond to one-eighth, one-sixteenth, and one-thirty-second resolutions of the original input image, respectively.

[0069] These hierarchical feature maps carry visual cues from different dimensions: hierarchical feature map C3 has high spatial resolution, containing a large amount of edge texture information, and can clearly depict the outline of the license plate characters and the subtle features of the license plate corners. Although hierarchical feature map C5 has lower spatial resolution, it has deep semantic features after multiple nonlinear transformations, and can effectively identify the vehicle's category attributes and overall posture. Hierarchical feature map C4 serves as an intermediate layer, balancing semantic strength and spatial accuracy. This multi-level extraction method ensures that subsequent processing can perceive both the macroscopic information of the vehicle and capture the microscopic details of the license plate.

[0070] Step S202: The hierarchical feature maps are integrated through a multi-scale fusion mechanism to obtain an initial shared feature map.

[0071] Here, during the fusion process, a 1x1 convolutional layer is first applied to each layer's feature map to uniformly map their channel dimensions to a preset dimension, such as 256. Next, the fusion operation is performed following a top-down path. The deep layer's feature map C5 is upsampled by a factor of two using bilinear interpolation to make its spatial dimensions consistent with those of layer feature map C4, and then element-wise summed with layer feature map C4. The summation result is then upsampled again and fused with layer feature map C3.

[0072] This fusion mechanism allows shallow feature maps to receive semantic guidance from deep feature maps, while the deep feature maps supplement the shallow feature maps with precise location information. The fused result contains target information from different receptive fields, forming an initial shared feature map. This initial shared feature map integrates large-scale vehicle information with small-scale license plate information, providing a unified feature representation for parallel recognition.

[0073] Step S203: Introduce positional encoding into the initial shared feature map and fuse the positional encoding with the initial shared feature map to output a shared feature map; wherein, the positional encoding is used to characterize the spatial positional relationship of each feature vector in the shared feature map in the input image.

[0074] Here, since convolutional neural networks have translation invariance when processing images, positional encoding needs to be introduced in order for the subsequent attention mechanism to perceive the absolute and relative spatial positions of each vector in the feature map.

[0075] Specifically, a position encoding scheme based on sine and cosine functions is adopted. For any feature point in the initial shared feature map, a position vector is calculated using sine and cosine functions of different frequencies, based on the feature point's x-coordinate and y-coordinate in the feature map. The dimension of the position vector is consistent with the channel dimension of the initial shared feature map. The calculation logic of position encoding is as follows: The formula for calculating the horizontal position coordinates is: ; .

[0076] in, This represents the component values ​​of the location encoding in a specific feature dimension. This represents the spatial coordinates of the feature point in the horizontal direction of the shared feature map. This represents the total dimension of the feature vector, i.e., the channel depth of the shared feature map, which is usually preset to 256. The index number representing the feature dimension. This represents the even-numbered index position in the feature dimension, which is encoded using a sine function. This represents the odd-numbered index position in the feature dimension, which is encoded using a cosine function. `sin` represents the sine function, which imparts spatial frequency characteristics to the feature point through periodic numerical changes. `cos` represents the cosine function, which, together with the sine function, forms a complete positional representation. `10000` represents a preset constant base, used to adjust the frequency of positional information changes under different feature dimension indices, ensuring that the positional encoding is discriminative over long distances.

[0077] Similarly, the same calculation logic is used for vertical position coordinates as for horizontal position coordinates. In practice, only the horizontal spatial position coordinates in the above formula need to be included. Replace with vertical spatial coordinates This will give you the position code value in the vertical direction.

[0078] By combining the horizontal and vertical positional encoding values ​​(e.g., through vector concatenation or element-wise summation), a complete spatial location parameter is generated. Fusing this spatial location parameter into the initial shared feature map allows each feature vector to carry explicit coordinate semantics.

[0079] In one embodiment, reference is made to Figure 3 Step S103 includes the following steps S301-S302.

[0080] Step S301: Input the shared feature map into the preset encoder; the preset encoder has a multi-scale deformable attention mechanism.

[0081] Here, after feature extraction and location encoding are completed, the generated shared feature map is input into a pre-defined encoder for deep global information integration. The pre-defined encoder consists of multiple stacked encoder layers with identical structures; in practice, six encoder layers are typically used.

[0082] Each encoder layer contains a multi-scale deformable attention module and a feedback neural network based on a fully connected layer. These two core components are connected through layer normalization and supplemented by residual connection logic to ensure the stability of the deep network when processing high-dimensional features.

[0083] By utilizing a multi-scale deformable attention mechanism, non-dense feature sampling is performed on hierarchical feature maps with different spatial resolutions, which can efficiently capture feature cues of key targets in large-size images.

[0084] Step S302: Use a preset encoder to aggregate contextual information from hierarchical feature maps with different spatial resolutions in the shared feature map to generate a target representation vector.

[0085] Here, within the pre-defined encoder, context information aggregation is achieved through associative modeling of hierarchical feature maps at different scales. For each feature point to be processed in the shared feature map, the multi-scale deformable attention mechanism first determines the two-dimensional reference coordinates corresponding to that feature point.

[0086] Subsequently, the multi-scale deformable attention mechanism automatically predicts a small number of key sampling offsets at different feature levels, as well as attention weights for these offset points, based on the current semantic content of the feature points.

[0087] Specifically, the multi-scale deformable attention mechanism follows the following computational logic: The final output feature value is equal to the weighted sum of all levels, all attention heads, and all sampling points. During the calculation, each feature level is traversed, and within each level, for each attention head, the product of the sampling point's feature vector and the corresponding attention weight is calculated. The specific location of the sampling point is determined by the reference coordinates and the predicted sampling offset, and precise feature values ​​are extracted from the corresponding level's feature map using a bilinear interpolation algorithm.

[0088] For vehicle targets in an image, the pre-defined encoder can simultaneously reference global contour information in low-resolution layers and local texture details in high-resolution layers. For example, when identifying vehicle boundaries, the encoder aggregates the spatial relationships of key parts such as headlights and windshields across layers, thereby transforming the original shared feature map into a target representation vector rich in global semantic constraints.

[0089] These target representation vectors not only encompass the apparent features of the target to be identified, but also refine the topological relationships between the target and its surrounding environment, as well as between the vehicle and the license plate. After being refined layer by layer by a six-layer encoder, the final target representation vectors exhibit high discriminativeness.

[0090] Based on this, the method's recognition performance is improved when dealing with occlusion, small targets at a distance, and complex background interference.

[0091] In one embodiment, reference is made to Figure 4 The joint query vector is generated through the following steps S401-S405.

[0092] Step S401: Perform vehicle query filtering on the target representation vector to obtain vehicle queries; vehicle queries include vehicle content queries and vehicle location queries.

[0093] Here, each target representation vector is processed by a classification prediction head to obtain a confidence score, representing the probability that a vehicle exists in the image region corresponding to that vector. Vectors are sorted from highest to lowest confidence score, and a predetermined number (e.g., 100 or 300) are selected as initial candidates. Each selected candidate vector contains two core pieces of information: a vehicle content query representing the vehicle's appearance features and semantic category, and a vehicle location query representing the vehicle's approximate coordinates in the image space. This dynamic filtering of queries from global features ensures that subsequent processing focuses on the most valuable vehicle regions in the image.

[0094] Step S402: Starting with the vehicle query, initialize the license plate key point query.

[0095] Here, after determining the vehicle search, the spatial relationship between the vehicle and the license plate is used to guide the location of the license plate.

[0096] A pre-defined mapping function (such as a multilayer perceptron) is used to non-linearly transform the content and location information in the vehicle query. Since license plates are typically located in specific areas at the front or rear of a vehicle, this transformation can narrow the vehicle-level perception range to the potential license plate area. By overlaying a set of pre-defined, learnable license plate task embedding vectors onto the vehicle query features, license plate key point queries are initialized. These license plate key point queries not only carry the vehicle's contextual information but are also given specific task instructions to find the four corner points of the license plate.

[0097] Step S403: Based on the key points of the license plate, query the coordinates of the corresponding license plate corner points, and generate the license plate center curve by fitting a Bézier curve.

[0098] Here, after obtaining the initial license plate key point query, the coordinates of the four corner points of the license plate are predicted, namely the top left, top right, bottom right, and bottom left corners. In order to cope with the distortion of the front view caused by wide-angle distortion of the camera, tilt of the shooting angle, or curvature of the license plate itself in real-world scenarios, the Bezier curve fitting module is used to use the four corner points of the license plate or the midpoint of the edge determined by the corner points as control points.

[0099] Taking a cubic Bézier curve as an example, a smooth curve describing the horizontal center direction of a license plate is constructed using defined mathematical parametric equations and control points. This license plate center curve can dynamically adapt to the geometric deformation of the license plate, whether it is a rectangular license plate or an image area that is trapezoidal or arc-shaped under perspective transformation; the curve can accurately pass through the center position of the license plate characters.

[0100] Step S404: Perform point sampling on the center curve of the license plate to generate a license plate character query.

[0101] Here, after generating the license plate center curve, fine-grained sampling is performed on this curve. To cover all character positions on the license plate, equidistant sampling logic is used.

[0102] In one embodiment, step S404 includes: Equidistant sampling is performed on the center curve of the license plate to obtain a preset number of sampling points, and license plate character queries are generated based on the sampling points.

[0103] Here, the number of sampling points is preset according to the specifications of the target license plate to adapt to the license plate standards of different countries or regions. In one embodiment, for standard license plates, the preset number is set to 7. Ideally, these 7 sampling points correspond to the center positions of the 7 characters in the license plate.

[0104] In mathematical implementation, if the center curve of the license plate is derived from the cubic Bézier curve equation... It indicates that the parameters The value range of is defined between 0 and 1, so the physical coordinates are determined by selecting uniformly distributed sample values ​​within the parameter space. Specifically, let Take 0 respectively, , , , , And 1, by substituting these parameter values ​​into the Bézier curve equation, the two-dimensional coordinates of the seven sampling points in the image space coordinate system are calculated. This sampling method ensures that regardless of whether the license plate is stretched or compressed in the image, the sampling points can automatically adjust their spacing according to the curvature change of the curve, thus always remaining on the central axis of the character sequence.

[0105] Next, a license plate character query is initialized based on the coordinates of the seven acquired sampling points. A character task embedding vector of a preset dimension is assigned to each sampling point. The character task embedding vector is usually a set of learnable parameters used to carry the character's category features and recognition instructions.

[0106] To imbue character queries with explicit spatial awareness, the two-dimensional coordinates of each sampling point are converted into high-dimensional positional features using a positional encoding mapping function. Subsequently, these high-dimensional positional features are fused with the corresponding character task embedding vector, for example, through element-wise addition or vector concatenation. Through this fusion, each generated character query contains both the character position corresponding to that point and the point's specific coordinates in the image.

[0107] Step S405: Combine the vehicle query, license plate key point query, and license plate character query to obtain a joint query vector.

[0108] Here, after generating queries at each level, query vectors from different perceptual dimensions are integrated. Vehicle queries representing the overall vehicle target, license plate key point queries representing the spatial geometric features of the license plate, and character queries representing specific character information are concatenated or stacked in a preset order.

[0109] The resulting joint query vector is a multi-task instruction vector. Within the joint query vector, a strict one-to-one correspondence is established between the vehicle, license plate corner points, and character sampling points. The joint query vector ensures that while predicting the vehicle bounding box, it can utilize the license plate corner coordinates and character sequence information for collaborative verification, achieving true parallel regression prediction.

[0110] In one embodiment, after generating a joint query vector containing multi-dimensional information, the system proceeds to a parallel regression prediction stage. This stage employs a two-stage candidate generation and result refinement mechanism, using a pre-defined decoder to transform abstract query information into precise image coordinates and character sequences.

[0111] The step of performing vehicle detection, license plate localization and license plate character recognition in parallel in step S103 includes the following steps S501-S502.

[0112] Step S501: Generate initial vehicle detection results and initial license plate prediction results, and obtain candidate regions with high confidence through confidence screening.

[0113] Here, in the first stage, preliminary prediction is first performed using the target representation vector output from the preset encoder. The target representation vector is passed through a simple linear prediction branch to calculate the confidence score of each vector position containing the vehicle target, and a rough vehicle position is initially regressed.

[0114] All potential targets are sorted based on their confidence scores, and a predetermined number (e.g., the top 300) of targets with the highest scores are selected. These selected targets and their corresponding feature vectors constitute high-confidence candidate regions. This selection mechanism filters out a large amount of background and low-quality interference in the image, ensuring that computational resources are concentrated on the most likely vehicle targets. The generated initial vehicle detection results and initial license plate prediction results provide a reliable starting point for subsequent precision refinement.

[0115] In step S502, the candidate region is used as the refined query vector and input into the preset decoder. The refined query vector is optimized layer by layer using the preset decoder reference target representation vector to synchronously regress and obtain the vehicle bounding box coordinates, license plate corner coordinates and license plate character sequence information. The preset decoder has a multi-scale deformable attention mechanism.

[0116] Here, in the second stage, the selected candidate regions are transformed into refined query vectors and input into the preset decoder. The preset decoder consists of multiple stacked decoding layers, typically employing a six-layer structure in practical implementations.

[0117] Each pre-defined decoder layer contains a multi-scale deformable attention module to enable deep interaction between the refined query vector and the target representation vector. During the interaction, the refined query vector serves as the query term, and the target representation vector serves as the reference term. The multi-scale deformable attention mechanism adaptively searches for key feature points at different spatial resolution levels of the shared feature map based on the current semantic state of each query vector.

[0118] Specifically, each decoder layer predicts the offset of the coordinate information carried by the query vector. By calculating the attention weights of the sampling points on the feature maps at different levels and performing a weighted summation on the extracted features, the query vector can continuously absorb detailed information from the image. For example, a vehicle query will further pinpoint the position of the wheels or windows to correct the bounding box, a license plate keypoint query will focus on the edges of the license plate to refine the corner coordinates, and a character query will delve into the character region to extract texture features.

[0119] During each iteration, predictions are performed synchronously using a pre-defined detection head, a keypoint regression head, and a recognition head. The detection head regresses the center point offset and width / height scaling ratio of the vehicle bounding box using a three-layer perceptron. The keypoint regression head outputs the coordinate deviations of the four license plate corner points relative to the license plate center. The recognition head is responsible for mapping character queries to a pre-defined character category space.

[0120] Through six layers of decoding, the information in the joint query vector is fully optimized. Since all regression tasks are executed in parallel within the same decoding process, information complementarity and alignment are achieved between different tasks. Finally, the vehicle bounding box coordinates, license plate corner coordinates, and license plate character sequence information are output synchronously for each vehicle target.

[0121] In one embodiment, after outputting the recognition results, to ensure that the integrated recognition method can continuously improve its accuracy and possess self-evolution capabilities, feedback annotation information and a joint loss function are introduced to achieve deep optimization of the processing logic. This process can occur either during the model training phase before the recognition method is formally applied, or during the online feedback optimization phase after the method is applied.

[0122] Following step S104, the method further includes the following steps S601-S603.

[0123] Step S601: Obtain the annotation information corresponding to the input image.

[0124] The purpose of obtaining the annotation information here is to establish a realistic reference standard. The annotation information includes the coordinates of the bounding box, the coordinates of the corner points, and the sequence of characters generated by manual annotation or a pre-set high-precision algorithm. The bounding box coordinates define the actual physical extent of the vehicle target, the corner point coordinates determine the precise positions of the four vertices of the license plate, and the character sequence corresponds to the actual text information on the license plate. This information forms the benchmark for calculating the deviation, enabling the system to quantify the gap between the current inference result and the ideal result.

[0125] Step S602: Construct a joint loss function and calculate the total loss value between the recognition result and the labeled information based on the joint loss function; the recognition result is the vehicle bounding box coordinates, license plate corner coordinates, and license plate character sequence information corresponding to each vehicle.

[0126] Here, the joint loss function consists of a weighted sum of four sub-loss terms: The first part is the vehicle bounding box regression loss, used to optimize vehicle position. The vehicle bounding box regression loss consists of the absolute error loss between the predicted and labeled bounding boxes and the generalized intersection-union (CIU) loss. The absolute error loss focuses on the pixel-level proximity of coordinate points, while the CIU loss improves the overlap of bounding boxes under complex poses by calculating the degree of overlap between the two regions and the proportion of non-overlapping regions.

[0127] .in, This represents the vehicle bounding box regression loss; and These are preset weighting coefficients used to balance the ratio of the absolute error term to the generalized intersection and union term; This represents the coordinate vector of the vehicle bounding box in the prediction result; This represents the coordinate vector of the vehicle bounding box in the annotation information; This represents the absolute error distance between the predicted coordinates and the labeled coordinates; This represents the generalized intersection-union ratio (CUIR) between the predicted bounding box and the labeled bounding box.

[0128] The second part is the vehicle classification loss, which is used to improve the class determination ability. Focus loss is used as the evaluation criterion. This loss function, by introducing an adjustment factor, can effectively solve the problem of imbalance between positive and negative samples, making the system pay more attention to those complex vehicle samples that are difficult to identify.

[0129] .in, Indicates vehicle classification loss; This indicates the preset balance parameters or weighting coefficients; This represents the adjustment factor, used to adjust the weight of the contribution of easy and difficult samples to the loss; This represents the predicted probability related to the true category; This represents the predicted probability value for the vehicle category output by the model; This refers to the category label in the annotation information; when the target is a vehicle... ; Represents a logarithmic function.

[0130] The third part is the license plate keypoint regression loss, used to calibrate the license plate corner points. This loss combines the absolute error loss and the object keypoint similarity loss. The object keypoint similarity loss simulates the visual tolerance of human perception, using a non-linear measure to address subtle deviations in the corner points, ensuring that the predicted license plate corners accurately match the license plate edges.

[0131] .in, This indicates the loss due to the regression of key points related to the license plate; and This indicates the preset weighting coefficients; This represents the predicted coordinate vector of the license plate corner point; This represents the coordinate vector of the corner point of the license plate. This represents the total number of license plate corner points; i represents the index number of the license plate corner point. This represents the coordinates of the i-th predicted corner point; Indicates the coordinates of the i-th labeled corner point; This represents the target scale factor, which typically corresponds to the area feature of the license plate region. This represents the normalization constant for the i-th key point; Represents an exponential function with the natural constant as its base; This indicates an indicator function that takes the value 1 when the condition inside the parentheses is true, and 0 otherwise. This indicates the visibility flag of the i-th labeled corner point.

[0132] The fourth part is the license plate character recognition loss. A connectionist temporal classification loss is adopted, which can measure the degree of mismatch between the predicted character probability sequence and the standard character sequence. Especially when the character lengths are different or there is background interference, it can guide the system to learn the correct text features through probability alignment.

[0133] .in, This indicates the loss in license plate character recognition; Represents the natural logarithm function; This represents the sequence of license plate characters in the label information; Indicates a possible character path; This represents the set of inverse transformations that map the labeled character sequence to all possible paths; The total length or time step of the feature sequence is represented by t; t represents the position index in the feature sequence. This represents the feature vector corresponding to the t-th position; This represents the probability of outputting a specific character given a feature at position t.

[0134] By weighted summing of the above four loss values, the final total difference loss value is obtained.

[0135] .in, This represents the total loss due to the difference; This represents the vehicle bounding box regression loss; Indicates vehicle classification loss; This indicates the loss due to the regression of key points related to the license plate; Indicates text recognition loss; These are preset weighting coefficients.

[0136] Step S603: Adjust the recognition parameters required to generate the recognition result according to the total difference loss value; wherein, the recognition parameters include backbone network parameters, preset encoder parameters, query generation parameters and preset decoder parameters.

[0137] Here, after obtaining the total difference loss value, the deviation signal is transmitted to each core component of the recognition method using the backpropagation mechanism to perform parameter updates.

[0138] The adjustment of recognition parameters covers the entire processing flow. Specifically, it includes: adjusting the parameters of the backbone network to optimize its basic ability to extract vehicle and license plate features; adjusting the parameters of the preset encoder to optimize its attention weights for context aggregation at different scales; adjusting the query generation parameters to optimize its mapping accuracy from vehicle to license plate corner points and then to Bézier curve sampling points; and adjusting the parameters of the preset decoder to improve its interaction efficiency in multi-task parallel regression.

[0139] It's important to note that this adjustment of recognition parameters has multi-dimensional application scenarios. Before the method is officially deployed, these steps correspond to the training phase. By performing these steps on a large number of known labeled sample images, the recognition parameters undergo thousands of iterations, enabling the method to achieve high-precision initial recognition capabilities. After the method is deployed, these steps correspond to the online optimization or feedback learning phase. By obtaining labeled feedback from real-world scenarios, parameter fine-tuning can be performed for specific environments (such as specific parking lot lighting or special shooting angles), ensuring that the recognition parameters always adapt to the constantly changing actual operating scenarios.

[0140] This closed-loop adjustment method based on the joint loss function enables continuous iteration of the vehicle and license plate information recognition logic, allowing the entire recognition method to maintain extremely high recognition robustness and consistency when facing various complex and dynamic traffic environments.

[0141] In one embodiment, step S602 includes the following steps S701-S705.

[0142] Step S701: Calculate the bounding box regression bias of the vehicle bounding box using absolute error loss and generalized intersection-union ratio loss.

[0143] Here, when evaluating vehicle position, the primary focus is on the degree of overlap between the predicted vehicle bounding box and the labeled bounding box in coordinate space. The bounding box regression bias consists of two complementary loss terms. The first is the absolute error loss, which calculates the Manhattan distance between the predicted bounding box's center point coordinates, width, and height and the labeled data.

[0144] Secondly, there is the Generalized Cross-Union-Ratio (GUN) loss, which addresses the gradient vanishing problem of traditional GUN loss when the predicted and labeled bounding boxes do not overlap. By calculating the minimum closure region of the two bounding boxes and considering the proportion of the non-overlapping region within the closure region, GUN loss can more objectively reflect the spatial correlation between the two geometric regions. Weighting the absolute error loss with GUN loss ensures accurate vehicle target localization across various scales.

[0145] Step S702: Use focus loss to calculate the prediction bias of vehicle classification.

[0146] Here, the focus loss introduces a modulation factor on top of the traditional cross-entropy loss, which can automatically adjust the loss weights according to the difficulty of the prediction. For easily identifiable background samples, the focus loss reduces their weights, while for difficult-to-identify or blurry vehicle targets, it increases their weights. This forces the system to focus more on feature learning for difficult samples when handling parameter adjustments, thereby improving its ability to distinguish vehicle targets in complex backgrounds.

[0147] Step S703: Calculate the corner regression deviation of the license plate corner points by combining the absolute error loss and the object key point similarity loss.

[0148] Here, the corner regression bias also consists of two parts to address the deformation of the license plate caused by different shooting angles.

[0149] The first part still uses absolute error loss to measure the straight-line distance between the predicted coordinates of the four license plate corner points and the labeled corner point coordinates. The second part introduces object keypoint similarity loss, which calculates the similarity between the predicted points and the ground truth points using an exponential function and considers the normalization scale of different keypoints. Object keypoint similarity loss can tolerate small localization noise while imposing a larger penalty for severe localization deviations. By combining the two, the four corner points of the license plate can be regressed with pixel-level accuracy, ensuring that the geometry of the license plate area is accurately restored.

[0150] Step S704: Use connectionist temporal classification loss to measure the degree of mismatch between the predicted character sequence and the labeled character sequence.

[0151] Here, the loss function iterates through all possible character permutations during calculation, handling character gaps by introducing whitespace placeholders, and sums the predicted probability paths. It eliminates the need for precise pre-segmentation or alignment of each character in the image; given only the entire character sequence labeling, it can calculate the difference between the recognition output and the actual license plate number. Connectionist temporal classification loss guides the system to autonomously learn the texture features and arrangement patterns of characters without explicit character segmentation, improving robustness in recognizing deformed and blurred characters.

[0152] Step S705: The bounding box regression bias, prediction bias, corner regression bias, and mismatch degree are weighted and summed to obtain the total difference loss value.

[0153] Here, after obtaining the deviation items in the four dimensions mentioned above, they are weighted and summed using preset weight coefficients to generate a unified total difference loss value. The weight coefficients are designed based on the degree of influence of different tasks on the overall recognition effect. For example, since character recognition and license plate localization are usually more difficult than vehicle detection, the weight of license plate-related deviation items can be appropriately increased.

[0154] Furthermore, the calculation of the total difference loss value and subsequent adjustment of recognition parameters are mainly applied in the training phase before the method is put into use, and can also be used for online feedback optimization after the method is formally applied. During the training phase, by performing the above steps in a sample library containing a large amount of labeled information, the recognition parameters can be improved through thousands of iterations, thereby constructing a high-performance integrated recognition method that can cope with complex and ever-changing traffic scenarios, ensuring that the preset performance indicators can be achieved in terms of the breadth of vehicle detection, the accuracy of license plate localization, and the depth of character recognition.

[0155] In one embodiment, after step S601, the method further includes: The Hungarian matching algorithm enables a one-to-one correspondence between the output recognition results and the multiple labeled targets in the annotation information.

[0156] In practical traffic applications, the input image often contains multiple vehicles. To ensure recall, the number of output recognition results is usually preset to a large fixed value (e.g., 100 or 300), which is more than the actual number of labeled targets in the image.

[0157] In the specific implementation process, a cost matrix is ​​first constructed with the dimension being the number of recognition results multiplied by the number of labeled targets. For each element in the matrix, that is, the matching cost between the i-th recognition result and the j-th labeled target, it is obtained by calculating the comprehensive cost function of the two. This comprehensive cost function is composed of the following three weighted dimensions.

[0158] The first is the classification cost. The predicted probability of the i-th recognition result in the vehicle category is extracted, and the focus loss or probability deviation between this probability and the true category of the j-th labeled target is calculated. The smaller the classification cost, the closer the recognition result is to the real vehicle at the semantic level.

[0159] The second is the bounding box localization cost. This involves calculating the absolute error distance and generalized intersection-union ratio (GUU) between the vehicle bounding box coordinates output by the i-th recognition result and the labeled bounding box coordinates of the j-th labeled target. By measuring the overlap and geometric distance between the two bounding boxes in space, it is determined whether the recognition result matches the labeled target in space.

[0160] Third is the cost of key point regression. Combining the coordinates of the four license plate corner points predicted by the i-th recognition result, the average deviation between these points and the coordinates of the corner points marked in the j-th labeled target is calculated.

[0161] After constructing the cost matrix, the Hungarian matching algorithm is used to solve for it. The Hungarian matching algorithm finds a set of non-conflicting elements in the matrix such that the sum of the costs of all selected matching pairs is globally minimized. This globally optimized matching strategy effectively avoids erroneous assignments caused by local optima, ensuring that each labeled target is assigned to the most representative recognition result.

[0162] Based on the processing results of the Hungarian matching algorithm, a set of one-to-one matching relationships was obtained. For the successfully matched recognition results in this set of relationships, they were defined as positive samples. The vehicle bounding box coordinates, license plate corner coordinates, and license plate character sequence information output by these recognition results were then used, along with the corresponding annotation information, to perform subsequent calculation of the total difference loss value. Recognition results that did not match any annotated target were determined as background samples.

[0163] Based on the above embodiments, this application provides an integrated vehicle and license plate information recognition system, referring to... Figure 5 The integrated vehicle and license plate information recognition system provided in this application includes: Feature extraction module 1 is used to perform multi-scale feature extraction on the input image to obtain a shared feature map containing semantic information; wherein the input image contains at least one vehicle.

[0164] The aggregation processing module 2 is used to perform context information aggregation processing on the shared feature map to obtain the target representation vector.

[0165] The regression prediction module 3 is used to interact with the joint query vector and the target representation vector to perform regression prediction of vehicle detection, license plate localization and license plate character recognition in parallel; wherein, the joint query vector includes vehicle query, license plate key point query and character query, and the joint query vector is generated through a hierarchical expansion mechanism.

[0166] Information output module 4 is used to output the vehicle bounding box coordinates, license plate corner coordinates, and license plate character sequence information for each vehicle in the input image during a single inference process.

[0167] In one embodiment, the feature extraction module 1 is further configured to: The input image is processed by a backbone network composed of multiple residual modules to generate hierarchical feature maps with different spatial resolutions. The hierarchical feature maps contain semantic features and edge texture information of the vehicle region and license plate region in the corresponding input image.

[0168] The hierarchical feature maps are integrated through a multi-scale fusion mechanism to obtain an initial shared feature map.

[0169] A positional encoding is introduced into the initial shared feature map, and the positional encoding is fused with the initial shared feature map to output a shared feature map; wherein, the positional encoding is used to characterize the spatial positional relationship of each feature vector in the shared feature map in the input image.

[0170] In one embodiment, the aggregation processing module 2 is further configured to: The shared feature map is input into the preset encoder; the preset encoder has a multi-scale deformable attention mechanism.

[0171] By using a pre-defined encoder, contextual information is aggregated from hierarchical feature maps with different spatial resolutions in the shared feature map to generate a target representation vector.

[0172] In one embodiment, the regression prediction module 3 is further configured to: Vehicle queries are obtained by filtering the target representation vector; vehicle queries include vehicle content queries and vehicle location queries.

[0173] Starting with vehicle search, initialize license plate key point search.

[0174] Based on the key points of the license plate, the coordinates of the corresponding corner points of the license plate are queried, and the center curve of the license plate is generated by fitting a Bézier curve.

[0175] Point sampling is performed on the center curve of the license plate to generate a license plate character query.

[0176] By combining vehicle queries, license plate key point queries, and license plate character queries, a joint query vector is obtained.

[0177] In one embodiment, the regression prediction module 3 is further configured to: Equidistant sampling is performed on the center curve of the license plate to obtain a preset number of sampling points, and license plate character queries are generated based on the sampling points.

[0178] In one embodiment, the regression prediction module 3 is further configured to: The system generates initial vehicle detection results and initial license plate prediction results, and obtains candidate regions with high confidence through confidence screening.

[0179] The candidate region is used as the input of the refined query vector into the preset decoder. The preset decoder is used to optimize the refined query vector layer by layer with reference target representation vector, so as to obtain the vehicle bounding box coordinates, license plate corner coordinates and license plate character sequence information through synchronous regression. The preset decoder has a multi-scale deformable attention mechanism.

[0180] In one embodiment, reference is made to Figure 6 The integrated vehicle and license plate information recognition system also includes optimization module 5, used for: Obtain the annotation information corresponding to the input image.

[0181] A joint loss function is constructed, and the total loss value between the recognition result and the labeled information is calculated based on the joint loss function. The recognition result is the vehicle bounding box coordinates, license plate corner coordinates, and license plate character sequence information corresponding to each vehicle.

[0182] The recognition parameters required to generate the recognition results are adjusted based on the total difference loss value; the recognition parameters include backbone network parameters, preset encoder parameters, query generation parameters, and preset decoder parameters.

[0183] In one embodiment, the optimization module 5 is used for: The bounding box regression bias of the vehicle bounding box is calculated using absolute error loss and generalized crossover ratio loss.

[0184] The prediction bias for vehicle classification is calculated using focus loss.

[0185] The corner regression bias of the license plate corner points is calculated by combining the absolute error loss and the object key point similarity loss.

[0186] The degree of mismatch between the predicted character sequence and the labeled character sequence is measured using connectionist temporal classification loss.

[0187] The total difference loss is obtained by weighting and summing the bounding box regression bias, prediction bias, corner regression bias, and mismatch degree.

[0188] In one embodiment, the optimization module 5 is used for: The Hungarian matching algorithm enables a one-to-one correspondence between the output recognition results and the multiple labeled targets in the annotation information.

[0189] The computer program product provided in this application includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0190] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and apparatus described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0191] Furthermore, in the description of the embodiments of this application, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0192] In the description of this application, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0193] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of protection of the claims.

Claims

1. A method for integrated identification of vehicle and license plate information, characterized in that, include: Multi-scale feature extraction is performed on the input image to obtain a shared feature map containing semantic information; wherein the input image contains at least one vehicle; The shared feature map is subjected to context information aggregation processing to obtain the target representation vector; The joint query vector is interactively processed with the target representation vector to perform regression prediction of vehicle detection, license plate localization and license plate character recognition in parallel; wherein, the joint query vector includes vehicle query, license plate key point query and character query, and the joint query vector is generated through a hierarchical expansion mechanism; During one inference process, the coordinates of the vehicle bounding box, the coordinates of the license plate corner point, and the license plate character sequence information corresponding to each vehicle in the input image are output.

2. The integrated vehicle and license plate information recognition method according to claim 1, characterized in that, The step of performing multi-scale feature extraction on the input image to obtain a shared feature map containing semantic information includes: The input image is processed by a backbone network composed of multiple residual modules to generate hierarchical feature maps with different spatial resolutions; wherein, the hierarchical feature maps contain semantic features and edge texture information corresponding to the vehicle region and license plate region in the input image; The hierarchical feature maps are integrated through a multi-scale fusion mechanism to obtain an initial shared feature map; A positional encoding is introduced into the initial shared feature map, and the positional encoding is fused with the initial shared feature map to output the shared feature map; wherein, the positional encoding is used to characterize the spatial positional relationship of each feature vector in the shared feature map in the input image.

3. The integrated vehicle and license plate information recognition method according to claim 1, characterized in that, The step of performing context information aggregation processing on the shared feature map to obtain the target representation vector includes: The shared feature map is input into a preset encoder; the preset encoder has a multi-scale deformable attention mechanism. The target representation vector is generated by aggregating contextual information from hierarchical feature maps with different spatial resolutions in the shared feature map using the preset encoder.

4. The integrated vehicle and license plate information recognition method according to claim 1, characterized in that, The joint query vector is generated through the following steps: The target representation vector is filtered for vehicle queries to obtain the vehicle query; the vehicle query includes vehicle content query and vehicle location query. Starting with the vehicle query, initialize the license plate key point query; Based on the key points of the license plate, the coordinates of the corresponding corner points of the license plate are queried, and the center curve of the license plate is generated by Bézier curve fitting. Point sampling is performed on the center curve of the license plate to generate the license plate character query. The vehicle query, the license plate key point query, and the license plate character query are combined to obtain the joint query vector.

5. The integrated vehicle and license plate information recognition method according to claim 4, characterized in that, The step of performing point sampling on the center curve of the license plate to generate the license plate character query includes: Equidistant sampling is performed on the center curve of the license plate to obtain a preset number of sampling points, and the license plate character query is initialized based on the sampling points.

6. The integrated vehicle and license plate information recognition method according to claim 5, characterized in that, The steps of performing vehicle detection, license plate localization, and license plate character recognition in parallel include regression prediction, which includes: Generate initial vehicle detection results and initial license plate prediction results, and obtain candidate regions with high confidence through confidence filtering; The candidate region is used as a refined query vector and input into a preset decoder. The preset decoder is used to optimize the refined query vector layer by layer with reference to the target representation vector, so as to synchronously regress the vehicle bounding box coordinates, the license plate corner coordinates and the license plate character sequence information; wherein, the preset decoder has a multi-scale deformable attention mechanism.

7. The integrated vehicle and license plate information recognition method according to claim 1, characterized in that, After the step of outputting the vehicle bounding box coordinates, license plate corner coordinates, and license plate character sequence information for each vehicle in the input image during a single inference process, the method further includes: Obtain the annotation information corresponding to the input image; A joint loss function is constructed, and the total loss value between the recognition result and the annotation information is calculated based on the joint loss function; the recognition result is the vehicle bounding box coordinates, the license plate corner coordinates, and the license plate character sequence information corresponding to each vehicle. The recognition parameters required to generate the recognition result are adjusted based on the total difference loss value; wherein, the recognition parameters include backbone network parameters, preset encoder parameters, query generation parameters, and preset decoder parameters.

8. The integrated vehicle and license plate information recognition method according to claim 7, characterized in that, The step of constructing a joint loss function and calculating the total loss value between the recognition result and the labeled information based on the joint loss function includes: The bounding box regression bias of the vehicle bounding box is calculated using absolute error loss and generalized intersection-union ratio loss; The prediction bias for vehicle classification is calculated using focus loss. The corner regression bias of the license plate corner points is calculated by combining the absolute error loss and the object key point similarity loss. The degree of mismatch between the predicted character sequence and the labeled character sequence is measured using connectionist temporal classification loss. The total difference loss value is obtained by weighting and summing the bounding box regression bias, the prediction bias, the corner regression bias, and the degree of mismatch.

9. The integrated vehicle and license plate information recognition method according to claim 7, characterized in that, After the step of obtaining the annotation information corresponding to the input image, the method further includes: The Hungarian matching algorithm is used to make the multiple recognition results output correspond one-to-one with the multiple labeled targets in the labeling information.

10. An integrated vehicle and license plate information recognition system, characterized in that, include: The feature extraction module is used to perform multi-scale feature extraction on the input image to obtain a shared feature map containing semantic information; wherein the input image contains at least one vehicle; The aggregation processing module is used to perform context information aggregation processing on the shared feature map to obtain the target representation vector; The regression prediction module is used to interact with the joint query vector and the target representation vector to perform regression prediction of vehicle detection, license plate localization and license plate character recognition in parallel; wherein, the joint query vector includes vehicle query, license plate key point query and character query, and the joint query vector is generated through a step-by-step expansion mechanism; The information output module is used to output the vehicle bounding box coordinates, license plate corner coordinates, and license plate character sequence information for each vehicle in the input image during a single inference process.

Images