License plate recognition method and system based on image processing

Abstract

The embodiment of the present application discloses a license plate recognition method and system based on image processing, which belongs to the field of image processing technology. The embodiment of the present application significantly improves the accuracy of license plate recognition through the special definition of feature mining indicator variables under the target attention channel and a series of complex vector processing operations. Under different attention channels, feature information related to license plate recognition can be mined more accurately, effectively reducing the interference of external factors such as light, occlusion, stains, etc. on recognition. For example, in color recognition, the color of the license plate can be determined more stably, the outline of the license plate can be judged more accurately during shape recognition, and the recognition ability of blurred or damaged characters can be improved in character recognition, thereby enhancing the reliability and stability of the entire license plate recognition technology solution, and providing better quality and efficient license plate recognition technology support for traffic management, security monitoring and other fields.

Description

Technical Field

[0001] The embodiments of the present application belong to the field of image processing technology, and specifically relate to a license plate recognition method and system based on image processing. Background Art

[0002] In the field of license plate recognition, traditional technologies have many problems. Traditional methods often analyze image features in a single way. When the lighting changes, license plate color recognition that relies solely on simple color models is prone to errors, affecting the overall recognition accuracy. When the license plate shape is blocked or deformed, or the characters are blurred or partially missing, there is a lack of effective response measures. In addition, traditional technologies do not fully utilize historical license plate recognition data and lack comprehensive reference basis. Summary of the invention

[0003] The embodiments of the present application provide a license plate recognition method and system based on image processing, which can solve or partially solve the technical problems involved in the above-mentioned background technology.

[0004] The embodiment of the present application provides a license plate recognition method based on image processing, which is applied to an image processing system. The method includes: obtaining a target multimodal image description of a historical vehicle image; wherein the historical vehicle image refers to a license plate area detection image corresponding to a license plate recognition event implemented in a target scene; obtaining an attention sequence extension model; wherein the attention sequence extension model is used to identify the license plate information of the target vehicle image to be identified from multiple attention channels, one attention channel corresponds to a license plate recognition event, each attention channel is configured with a corresponding feature mining indicator variable, and the attention channel includes an RGB color channel, an HSV color channel, a morphological channel, and a sequence keyword channel; according to the feature mining indicator variable under each attention channel, the target multimodal image description is subjected to attention vector mining to obtain a channel attention description vector of the target multimodal image description under each attention channel; wherein the channel attention description vector under an attention channel includes an image description vector in the target multimodal image description associated with the license plate recognition event of the attention channel; and the channel attention description vectors under the multiple attention channels are processed using the attention sequence extension model to generate the license plate information of the target vehicle image in the target scene.

[0005] An embodiment of the present application provides an image processing system, comprising at least one processor and a memory; the memory stores computer-executable instructions; the at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the above method.

[0006] An embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the above method are implemented.

[0007] The embodiment of the present application significantly improves the accuracy of license plate recognition through the special definition of feature mining indicator variables under the target attention channel and a series of complex vector processing operations. Under different attention channels, feature information related to license plate recognition can be mined more accurately, effectively reducing the interference of external factors such as light, occlusion, stains, etc. on recognition. For example, in color recognition, the color of the license plate can be determined more stably, the outline of the license plate can be judged more accurately during shape recognition, and the recognition ability of blurred or damaged characters can be improved in character recognition, thereby enhancing the reliability and stability of the entire license plate recognition technology solution, and providing better quality and efficient license plate recognition technology support for traffic management, security monitoring and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 A flowchart of a license plate recognition method based on image processing provided in an embodiment of the present application.

[0009] Figure 2 A schematic diagram of the structure of an image processing system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0010] Figure 1 A license plate recognition method based on image processing is shown, which is applied to an image processing system. The method includes the following steps 101 to 104.

[0011] Step 101: Obtain a target multimodal image description of a historical vehicle image; wherein the historical vehicle image refers to a license plate area detection image corresponding to a license plate recognition event implemented in a target scene.

[0012] Step 102: Obtain an attention sequence expansion model; wherein the attention sequence expansion model is used to identify the license plate information of the target vehicle image to be identified from multiple attention channels, one attention channel corresponds to a license plate recognition event, each attention channel is configured with a corresponding feature mining indicator variable, and the attention channels include RGB color channel, HSV color channel, morphological channel and sequence keyword channel.

[0013] Step 103: Perform attention vector mining on the target multimodal image description based on the feature mining indicator variables under each attention channel to obtain the channel attention description vector of the target multimodal image description under each attention channel; wherein the channel attention description vector under an attention channel includes an image description vector in the target multimodal image description that is associated with the license plate recognition event of the attention channel.

[0014] Step 104: Use the attention sequence expansion model to process the channel attention description vectors under the multiple attention channels to generate the license plate information of the target vehicle image in the target scene.

[0015] This application uses an image processing system as the execution body to achieve accurate recognition of license plate information.

[0016] First, in step 101, obtaining the target multimodal image description of the historical vehicle image is the starting step of the entire application. The historical vehicle image in this application is not an arbitrary vehicle image, but specifically refers to the license plate area detection image corresponding to the license plate recognition event implemented in the target scene. For example, in a license plate recognition scene in a parking lot, the images that have successfully recognized the license plate many times in the past belong to this kind of historical vehicle image. These images contain rich information, not only the appearance of the license plate, but also the presentation state of the license plate in different environments (such as different lighting, different angles, etc.). The target multimodal image description is a comprehensive and multi-dimensional description of these images. It may cover the color features, shape features, texture features and other information of the image. For example, for color features, the performance range of the license plate color under different lighting may be recorded; for shape features, the approximate length and width ratio of the license plate and the edge shape characteristics may be included; the texture features may involve some special texture marks on the surface of the license plate or texture changes caused by wear, etc. This multimodal image description provides a comprehensive reference for subsequent recognition work.

[0017] Next, for step 102, the step of obtaining the attention sequence expansion model plays a key architectural role in the entire license plate recognition process. This model is designed to identify the license plate information of the target vehicle image to be identified from multiple attention channels. The attention channels in this application have multiple types, each of which corresponds to a license plate recognition event. Among them, the RGB color channel is the most common color representation channel, which describes the color information in the image based on the three primary colors of red, green and blue. For example, the representation of a typical license plate color in the RGB color space may be a specific range of values, such as the RGB value of a blue license plate may be in a certain specific interval, and the red license plate also has its corresponding RGB value range. The HSV color channel has unique advantages in license plate recognition. As mentioned earlier, the HSV color space can stably maintain hue and saturation, and the hue value range in OpenCV is 0 to 179, which makes it possible to more accurately identify the license plate color through the HSV color channel under different lighting conditions. For example, under strong light, the RGB color value of the license plate may change significantly, but in the HSV color space, its hue and saturation can be relatively stable, which is convenient for accurate recognition. The morphological channel focuses on the shape and structural features of the license plate. For example, the rectangular shape of the license plate and the spacing structure between characters can be mined through the morphological channel. The sequence keyword channel may involve some specific arrangement order and combination rules of license plate characters. Each attention channel is also configured with corresponding feature mining indicator variables, which are like keys to guide the mining of specific information related to license plate recognition in each channel.

[0018] Step 103 involves performing attention vector mining on the target multimodal image description according to the feature mining indicator variables under each attention channel, and obtaining the channel attention description vector of the target multimodal image description under each attention channel. Taking the RGB color channel as an example, the feature mining indicator variable may guide the system to focus on the image description vector related to the license plate color. If the target multimodal image description contains the color change information of the license plate under different environments, then through the feature mining indicator variables under the RGB color channel, those image description vectors related to the license plate color recognition can be extracted, and these vectors are combined to form the channel attention description vector under the RGB color channel. Looking at the HSV color channel again, according to its corresponding feature mining indicator variable, the system will filter out information related to the license plate color recognition under the HSV color space from the target multimodal image description, such as the hue value distribution information corresponding to the license plate color in the hue histogram, etc., thereby constructing the channel attention description vector under the HSV color channel. For the morphological channel, the feature mining indicator variable will prompt the system to find information related to the shape and structure of the license plate from the target multimodal image description, such as the length-to-width ratio range of the license plate, character spacing and other features corresponding to the image description vector, forming the channel attention description vector under the morphological channel. The construction of the channel attention description vector under the sequence keyword channel is also based on a similar principle. The feature mining indicator variable is used to mine the image description vector related to the order of license plate character arrangement, combination rules, etc. from the target multimodal image description.

[0019] Finally, step 104 is performed, in which the channel attention description vectors under multiple attention channels are processed using the attention sequence expansion model to generate the license plate information of the target vehicle image in the target scene. In the previous steps, the channel attention description vectors have been obtained from each attention channel. For example, the description vector related to the license plate color is obtained from the RGB color channel, the description vector of the license plate color adapted to different lighting conditions is obtained from the HSV color channel, the description vector of the license plate shape structure is obtained from the morphological channel, and the description vector of the license plate character arrangement rule is obtained from the sequence keyword channel. The attention sequence expansion model will integrate the information of these different channels for processing. Just like an experienced expert, it integrates clues from various aspects to draw the final conclusion. It will perform complex calculations and judgments based on the information weights and relationships of each channel. For example, if both the RGB color channel and the HSV color channel indicate that the license plate color in the target vehicle image meets the characteristics of a blue license plate, the morphological channel determines that the shape in the image meets the rectangular structure of the license plate, and the sequence keyword channel also detects that the arrangement order of the license plate characters meets the rules of normal license plates, then the attention sequence expansion model will determine that the license plate information in the target vehicle image is a blue license plate, and accurately identify the characters on the license plate and other related information.

[0020] In the actual license plate recognition process, taking the parking lot license plate recognition scenario mentioned above as an example, when a car enters the parking lot, the image processing system first obtains the target multimodal image description in the historical vehicle image according to step 101. This description is based on the license plate recognition images of a large number of vehicles entering and leaving the parking lot. Then, according to the attention sequence expansion model in step 102, each attention channel starts to work. The RGB color channel uses its feature mining indicator variable to find license plate color related information in the target multimodal image description; the HSV color channel accurately mines license plate color information under different lighting conditions (such as strong light during the day, weak light at night, etc.) due to its adaptability to lighting changes; the morphological channel focuses on whether the shape structure of the license plate meets the standard; the sequence keyword channel detects the arrangement rules of the license plate characters. In step 103, each channel constructs its own channel attention description vector, which accurately reflects the part of the target multimodal image description related to license plate recognition. Finally, in step 104, the attention sequence expansion model comprehensively processes these vectors from different channels to accurately identify the license plate information of the vehicles entering the parking lot, including important information such as license plate color and license plate number.

[0021] Let's take a license plate recognition scenario at a highway toll station as an example. The target scenario in this application is license plate recognition when a vehicle passes through a toll station at high speed. Due to the high speed of the vehicle and complex lighting conditions (such as changes in light at the entrance and exit of a tunnel, etc.), higher accuracy and speed are required for license plate recognition. In this case, the target multimodal image description of the historical vehicle image obtained in step 101 contains various feature information of the license plate under similar high-speed driving and complex lighting conditions. The various attention channels of the attention sequence expansion model in step 102 play a key role. The HSV color channel can accurately identify the color of the license plate by virtue of its stable representation of hue and saturation under conditions of large light changes such as at the entrance and exit of a tunnel; the morphological channel can quickly determine whether the shape of the license plate of a high-speed vehicle is complete, whether there is occlusion, etc.; the sequence keyword channel can adapt to image blurring that may occur at high speeds when quickly identifying the arrangement order of license plate characters. The attention description vectors of each channel constructed in step 103 accurately reflect the key information in the target multimodal image description. Finally, in step 104, the attention sequence expansion model comprehensively processes these vectors to quickly and accurately generate the license plate information of vehicles passing through the toll station, thereby achieving efficient toll management and traffic monitoring.

[0022] In this application, the entire process from obtaining the target multimodal image description of historical vehicle images, to building an attention sequence expansion model, to mining the channel attention description vector, and finally generating the license plate information is a highly integrated and intelligent license plate recognition process. This innovative technical solution can accurately and efficiently identify license plate information in various complex target scenarios, and has broad application prospects. Whether in the fields of urban traffic management, parking lot management or highway toll collection, it can make important contributions to improving management efficiency and ensuring traffic safety.

[0023] From a deeper technical perspective, the design of the attention sequence expansion model in this application reflects the comprehensive consideration of multiple license plate recognition factors. Different attention channels are not just simple functional divisions, but are carefully designed based on different key elements of license plate recognition. The RGB color channel and the HSV color channel start from the intuitive and important aspect of color, and use different color representation models to explore the color characteristics of license plates. As a basic color representation method, the RGB color channel provides a widely used color description framework, while the HSV color channel optimizes the problem of lighting changes in license plate recognition. For example, in an actual outdoor parking scene, the color value of the license plate under the RGB color channel may fluctuate greatly during the day when the sun is bright and in the evening, but under the HSV color channel, due to its special treatment of hue and saturation, it can more stably reflect the true color characteristics of the license plate.

[0024] The role of the morphological channel in license plate recognition cannot be ignored either. It focuses on the shape structure information of the license plate, which is another key factor in license plate recognition. In the actual license plate image, the shape of the license plate is a rectangle with a specific aspect ratio, and there is a certain spacing structure between the characters on the license plate. The morphological channel can accurately extract these shape structure-related information from the target multimodal image description through its corresponding feature mining indicator variables. For example, in some special cases, such as when the license plate is partially obscured or the image is deformed to a certain extent, the morphological channel can analyze the shape structure characteristics of the remaining visible part of the license plate and compare it with the target multimodal image description in the historical vehicle image to determine whether it still meets the basic shape structure requirements of the license plate.

[0025] The sequence keyword channel focuses on the order and combination rules of license plate characters. This is based on the specific encoding rules and arrangement habits of license plate characters. The arrangement of license plate characters in different regions has certain rules. For example, the Chinese characters are in a specific position on the license plate, and the combination of letters and numbers also follows certain rules. The sequence keyword channel can mine these information related to the order and combination rules of characters from the target multimodal image description. When processing some blurred or partially occluded license plate images, even if some characters cannot be clearly identified, it is still possible to accurately infer the complete information of the license plate through the sequence keyword channel's grasp of the character arrangement rules and combined with the information from other channels.

[0026] When constructing the channel attention description vector under each attention channel, the feature mining indicator variable plays a precise guiding role. These indicator variables are set according to the specific requirements of license plate recognition and the characteristics of each attention channel. Taking the HSV color channel as an example, the feature mining indicator variable may set some parameters related to the common color hue value range of license plates. For example, for blue license plates, their hue values ​​have a relatively fixed range in the HSV color space. The feature mining indicator variable will guide the system to screen out image description vectors that fall within this hue value range in the target multimodal image description, thereby constructing a channel attention description vector that accurately reflects the color characteristics of the blue license plate.

[0027] During the implementation of the entire application, data accuracy and processing efficiency are also aspects that need to be considered. When obtaining the target multimodal image description of historical vehicle images, it is necessary to ensure the quality and integrity of the image data. For example, when collecting historical vehicle images, it is necessary to ensure that the resolution and clarity of the image meet the requirements. When the target scene is parking lot license plate recognition, if the resolution of the collected image is too low, some detailed information on the license plate may be lost, such as the edges of the characters, subtle changes in color, etc., thereby affecting subsequent recognition work. Therefore, in practical applications, appropriate image acquisition equipment and technology are needed to ensure the quality of historical vehicle images.

[0028] When the attention sequence expansion model processes the channel attention description vectors under multiple attention channels, efficient algorithms and computing resources are also required to ensure the processing speed. In actual license plate recognition scenarios, license plate information needs to be generated quickly and accurately, especially in scenarios with large traffic flow, such as highway toll stations or entrances to large parking lots. If the processing speed is too slow, vehicles will wait in line for too long, affecting traffic efficiency. Therefore, the algorithm design of the model should be optimized as much as possible to reduce unnecessary calculation steps and improve calculation efficiency.

[0029] In addition, the present application also has a certain degree of adaptive ability. As time goes by and the environment changes, the appearance of the license plate may change, such as wear and aging of the license plate. In this case, the attention sequence expansion model can adapt to these changes by continuously updating the target multimodal image description of the historical vehicle images. When new license plate images are collected and successfully recognized, these new image information can be integrated into the target multimodal image description of the historical vehicle images, so that the model can better recognize license plates with different degrees of wear and aging.

[0030] It can be seen that this application comprehensively considers multiple key elements of license plate recognition. Through the collaborative work of multiple attention channels and a carefully designed attention sequence expansion model, it is able to accurately and efficiently generate license plate information in various complex target scenarios.

[0031] In actual application scenarios, there are many details that reflect the superiority of this application. For example, in the field of security monitoring, the accuracy and timeliness of license plate recognition are crucial for the entrances and exits of some important places. The technical solution of this application can accurately identify license plate information under different weather conditions, such as sunny, cloudy, rainy or snowy days. On rainy days, there may be raindrops on the surface of the license plate, which will cause certain interference to the color and shape recognition of the license plate. The RGB color channel and the HSV color channel can ignore the color interference caused by raindrops through their color feature mining capabilities and accurately identify the background color of the license plate; the morphological channel can eliminate the irregular shape interference formed by raindrops based on the basic shape structure of the license plate and identify the rectangular outline of the license plate; the sequence keyword channel is not affected by raindrops and can still be recognized according to the arrangement pattern of the license plate characters.

[0032] For example, in some special traffic management scenarios, such as the identification of specific vehicles during traffic control, these specific vehicles may have special license plate logos or colors. Through the multiple attention channels of the attention sequence expansion model, the license plate information of these special vehicles can be accurately identified. If the license plate color of a specific vehicle is a special color, the HSV color channel can quickly identify this special color license plate among many vehicles by virtue of its accurate grasp of the hue; if the special logo is reflected in the order of the license plate characters or a specific character combination, the sequence keyword channel can quickly make a judgment.

[0033] In summary, by applying this application, the target multimodal image description of historical vehicle images can be obtained, and various features in past license plate recognition data can be used to provide a comprehensive reference for current recognition. The attention sequence expansion model recognizes license plate information through multiple attention channels. The RGB color channel provides basic color information, the HSV color channel accurately recognizes the license plate color in response to lighting changes, the morphological channel determines the shape characteristics of the license plate, and the sequence keyword channel analyzes the character arrangement rules. Multiple channels mine information from different dimensions. Mining attention vectors based on feature mining indicator variables can accurately focus on relevant information. Finally, the multi-channel vectors are comprehensively processed to obtain license plate information, thereby improving accuracy and versatility.

[0034] In some other embodiments, the method further includes: using the attention sequence expansion model to obtain a vehicle image spectrum description vector of the target vehicle image. Based on this, using the attention sequence expansion model to process the channel attention description vectors under the multiple attention channels to generate the license plate information of the target vehicle image under the target scene includes: using the attention sequence expansion model to generate the license plate information of the target vehicle image under the target scene according to the vehicle image spectrum description vector and the channel attention description vectors under the multiple attention channels.

[0035] In actual application, this embodiment further optimizes the accuracy and effectiveness of license plate recognition.

[0036] On the basis of the previous technical solution, the link of obtaining the vehicle image spectrum description vector of the target vehicle image by using the attention sequence expansion model is added. The introduction of the vehicle image spectrum description vector brings a new information dimension to license plate recognition. Spectral information has a unique meaning in image analysis. For example, in some actual image data, the spectrum can reflect the distribution of the frequency components of the image. Taking the common 800×600 resolution vehicle image as an example, when performing spectrum analysis on it, it may be found that every 100 pixels in the horizontal direction of the image may correspond to a change in a certain frequency range. For the license plate area, if the license plate occupies a range of about 200-300 pixels in the horizontal direction of the image, the frequency component corresponding to this area in the spectrum will have its unique performance. Different color areas, shape edges, etc. will have different performances in the spectrum. For the license plate area, due to its relatively fixed color and specific shape structure, its characteristics in the spectrum are obviously different from other parts of the vehicle or the background. This difference may be reflected in the amplitude of the spectrum, the frequency distribution range, etc. For example, after analyzing a large number of samples, the amplitude of the blue background part of the license plate in the spectrum may be concentrated in the numerical range of 50-100 (the actual value can be determined based on the specific spectrum analysis), and the frequency distribution range may be 0.1-0.3 Hz (also an exemplary value), which is significantly different from the spectral characteristics of other color areas or backgrounds of the vehicle body.

[0037] The attention sequence expansion model has been designed to recognize the license plate information of the target vehicle image from multiple attention channels, including RGB color channel, HSV color channel, morphological channel and sequence keyword channel. Each channel is configured with a corresponding feature mining indicator variable, through which the attention vector of the target multimodal image description can be mined to obtain the channel attention description vector under each attention channel.

[0038] When the vehicle image spectrum description vector is added, the attention sequence expansion model will comprehensively consider the information of the vehicle image spectrum description vector when processing the channel attention description vectors under multiple attention channels to generate the license plate information of the target vehicle image in the target scene. This means that the model no longer relies solely on the previous information such as color, shape structure and sequence keywords, but incorporates spectrum information into the entire recognition process.

[0039] In actual image spectrum analysis, different spectral components may correspond to different image features. For example, the high-frequency part may be related to the detailed information of the image, such as the edge of the characters on the license plate or fine texture; the low-frequency part may be related to the overall color area of ​​the image or the larger shape outline. For the target vehicle image, the spectral characteristics of the license plate area are unique in the entire vehicle image spectrum. For example, the blue background color of the license plate may have a specific frequency range and amplitude distribution in the spectrum, and the white character part will also have corresponding spectral characteristics. After testing, the amplitude of the white character part in the spectrum may be between 30-60, and the frequency range may be between 0.2-0.4 Hz. These spectral characteristics are different from the spectral characteristics of the vehicle body color, background environment, etc.

[0040] During the processing, the attention sequence expansion model analyzes the relationship between the vehicle image spectrum description vector and the channel attention description vectors under multiple attention channels according to the pre-set algorithms and rules. For example, the channel attention description vector under the RGB color channel may indicate the approximate range of the license plate color in the RGB color space, such as the red component of a blue license plate may be in the range of 0-50, the green component of 50-100, and the blue component of 150-255 in the RGB value. The channel attention description vector under the HSV color channel provides more stable hue, saturation and other information of the license plate color in the HSV color space, such as the hue value of 120-140 and the saturation of 0.8-1. The channel attention description vector under the morphological channel gives information related to the shape structure of the license plate, such as the aspect ratio of the license plate is close to 3:1 and the character spacing is 10-15 pixels. The channel attention description vector under the sequence keyword channel contains information related to the arrangement rules of the license plate characters. At this time, the spectral features in the vehicle image spectrum description vector may confirm or complement some information in these channel attention description vectors.

[0041] Take license plate color recognition as an example. For example, in the RGB color channel, the channel attention description vector determines that the license plate color may be blue, but there is a certain uncertainty due to lighting or other factors. At this time, the spectrum features corresponding to blue in the vehicle image spectrum description vector can further confirm this judgment. If the frequency components and amplitudes corresponding to the blue area in the spectrum meet the predetermined spectrum feature range of the blue license plate, for example, the frequency of the blue area in the spectrum is 0.1-0.3 Hz and the amplitude is 50-100, then the judgment that the license plate color is blue is more reliable.

[0042] Similarly, in terms of morphology, the channel attention description vector under the morphological channel may detect that the license plate is a rectangular shape, but there may be partial occlusion or image deformation, resulting in inaccurate shape judgment. The overall shape information reflected by the low-frequency part of the vehicle image spectrum description vector can assist in judgment, because the low-frequency components in the spectrum are associated with larger shape contours. If the low-frequency part of the spectrum shows a main frequency component distribution similar to a rectangle, for example, the low-frequency part shows a frequency characteristic distribution related to a rectangular shape in the range of 0-0.1 Hz, this can provide additional support for the judgment that the license plate is a rectangular shape.

[0043] In terms of character recognition, the channel attention description vector under the sequence keyword channel provides information about the arrangement rules of license plate characters, but if some characters are unclear, the high-frequency part of the vehicle image spectrum description vector may be helpful. Because the high-frequency part is related to detail information, it may contain the spectrum characteristics of the character edges. For example, the high-frequency spectrum amplitude corresponding to the edge of a normal clear character is 80-120, and the frequency is 0.5-1 Hz. Even if the character is blurred, its edge may still have certain characteristics in the spectrum. By comparing it with the known license plate character spectrum characteristics, the accuracy of character recognition can be improved.

[0044] Throughout the license plate recognition process, the attention sequence expansion model continuously optimizes the judgment of the license plate information by integrating the vehicle image spectrum description vector and the channel attention description vector under multiple attention channels. This comprehensive processing method is similar to a multi-factor decision-making process. The information provided by each vector is like a decision basis, and the model weighs and judges based on these bases.

[0045] For example, in a complex scene containing multiple color and shape interference factors, the target vehicle image may contain some objects with similar colors or shapes to the license plate. Relying solely on the previous channel attention description vector may result in misjudgment. However, when the vehicle image spectrum description vector is added, the model can better distinguish the license plate from the interference object because the spectrum information can more deeply reflect the essential characteristics of the image. For example, in this scene, there is a blue object that is similar in color to the license plate and has a certain similarity in shape. The channel attention description vector under the RGB color channel and the HSV color channel may not be able to accurately distinguish. However, from the vehicle image spectrum description vector, the spectral characteristics of the license plate include not only the frequency components corresponding to the color, but also the unique spectral characteristics caused by its specific manufacturing process, character printing and other factors, while the spectral characteristics of this blue object will be different due to its different materials and uses. For example, the spectral characteristics of the license plate have a special spectral pattern in the frequency range of 0.1-0.3 Hz and the amplitude range of 50-100, while the spectral pattern of the blue interference object in this frequency and amplitude range is obviously different. By integrating this information, the attention sequence expansion model can accurately identify the real license plate.

[0046] In addition, there may be some differences in the image spectrum of different types of vehicles. For example, the overall image spectrum of a small car and a large truck will be different due to differences in body size, material and other factors. This difference can also be used in the license plate recognition process. When the target vehicle image is a small car, the overall spectrum characteristics in its vehicle image spectrum description vector match the common spectrum characteristics of small cars, which can provide some auxiliary information for license plate recognition. For example, the overall spectrum of a small car has a relatively stable amplitude range of 20-40 in the low-frequency range of 0-0.1 Hz, and the relative position of the license plate on the small car and the proportional relationship with the body may be reflected in the spectrum. The attention sequence expansion model can locate and recognize the license plate more accurately based on these spectral information related to the vehicle type.

[0047] It can be seen that by introducing the vehicle image spectrum description vector into the license plate recognition technical solution and combining it with the channel attention description vector under multiple attention channels in the attention sequence expansion model, the accuracy and reliability of license plate recognition are greatly improved. This innovative technical solution can more effectively identify the license plate information of the target vehicle image under various complex environments and conditions, providing more powerful technical support for modern traffic management, security monitoring and other fields.

[0048] In a preferred example, any one of the multiple attention channels is a target attention channel, and the feature mining indicator variable under the target attention channel belongs to a directional feature enhancement vector; the target multimodal image description is subjected to attention vector mining based on the feature mining indicator variable under each attention channel to obtain a channel attention description vector of the target multimodal image description under each attention channel, including: mining the target multimodal image description using the attention sequence extension model to obtain an original channel attention sequence vector of the target multimodal image description; splicing the vehicle image spectrum description vector and the directional feature enhancement vector under the target attention channel to obtain an image splicing vector; and the original channel attention sequence vector is subjected to attention vector mining based on the image splicing vector to generate a channel attention description vector of the target multimodal image description under the target attention channel.

[0049] Furthermore, the splicing processing of the vehicle image spectrum description vector and the directional feature enhancement vector under the target attention channel to obtain the image splicing vector includes: fusing the vehicle image spectrum description vector and the directional feature enhancement vector under the target attention channel to obtain the image splicing vector; or, feature integration of the vehicle image spectrum description vector and the directional feature enhancement vector under the target attention channel to obtain the image splicing vector; the size of the vehicle image spectrum description vector is the same as the size of the directional feature enhancement vector under the target attention channel.

[0050] Furthermore, the target multimodal image description includes multiple multimodal image descriptions of the historical vehicle image, and the original channel attention sequence vector includes a modal vector obtained after mining each multimodal image description in the multiple multimodal image descriptions; the attention vector mining of the original channel attention sequence vector according to the image splicing vector to generate a channel attention description vector of the target multimodal image description under the target attention channel includes: performing a correlation analysis on the image splicing vector and the original channel attention sequence vector to obtain a correlation coefficient between each modal vector in the original channel attention sequence vector and the image splicing vector; performing modal optimization on each modal vector according to the correlation coefficient between each modal vector and the image splicing vector to obtain multiple modal vectors after modal optimization; and performing feature integration on the multiple modal vectors after modal optimization to obtain the channel attention description vector of the target multimodal image description under the target attention channel.

[0051] In the above-mentioned corresponding technical solutions, the design of attention channels is a key element. Any of the channels can be set as the target attention channel, and the feature mining indicator variable under the target attention channel belongs to the directional feature enhancement vector. This feature is an important innovation in the entire license plate recognition process, which provides a targeted guidance mechanism for subsequent operations.

[0052] When performing attention vector mining, the target multimodal image description is operated according to the feature mining indicator variables under each attention channel. First, the target multimodal image description is mined using the attention sequence expansion model. This process involves in-depth analysis of the target multimodal image description to obtain various feature information related to license plate recognition hidden therein. For example, for an 800×600 resolution vehicle image, the target multimodal image description covers multiple aspects of the image, including multimodal information such as color, shape, and texture. During the mining process, the attention sequence expansion model processes the image according to information such as pixel distribution, color gradient, shape edge, etc., to obtain the original channel attention sequence vector of the target multimodal image description. This original channel attention sequence vector is a complex vector representation that contains information related to license plate recognition mined from different modalities of the image. For example, in terms of color modality, it may include the approximate range of values ​​of the license plate color in the RGB color space, such as a blue license plate may correspond to information such as the red component in the range of 0-50, the green component in the range of 50-100, and the blue component in the range of 150-255; in terms of shape modality, it may include the approximate aspect ratio of the license plate, such as the aspect ratio of common license plates is close to 3:1, and the spacing between license plate characters is 10-15 pixels; in terms of texture modality, it may include some characteristic values ​​corresponding to the special texture on the license plate surface, which reflect the roughness, direction and other characteristics of the texture.

[0053] The next step involves processing the vehicle image spectrum description vector and the directional feature enhancement vector under the target attention channel. This processing aims to effectively integrate the information of the two to obtain the image splicing vector. The vehicle image spectrum description vector contains information about the image spectrum in different frequency ranges and amplitudes, which reflects the characteristics of the image in the frequency domain. For example, in the low-frequency part of the spectrum, it may be related to the overall color area of ​​the image or the larger shape outline. The amplitude of the low-frequency part may be between 0-50, and the corresponding frequency range may be 0-0.1 Hz; in the high-frequency part, it may be related to the detailed information of the image, such as the edge of the characters on the license plate or the fine texture. The amplitude of the high-frequency part may be between 80-120, and the corresponding frequency range may be 0.5-1 Hz. The directional feature enhancement vector under the target attention channel contains the enhancement feature values ​​specific to the target attention channel. For example, when the target attention channel is the RGB color channel, the directional feature enhancement vector may contain numerical information related to color enhancement, such as the value of the color enhancement of the blue license plate may be in a certain range. This range is obtained based on the analysis of a large number of blue license plate samples, aiming to highlight the characteristics of the blue license plate in the RGB color space.

[0054] There are many ways to splice the vehicle image spectrum description vector and the directional feature enhancement vector under the target attention channel, including fusion processing or feature integration. In the fusion process, the features of the two may be mixed according to a certain algorithm, so that the newly generated image splicing vector contains both the spectrum features and the directional enhancement features. For example, in the fusion algorithm, the amplitude corresponding to certain frequency ranges in the spectrum description vector may be weighted, and then added or multiplied with the corresponding values ​​in the directional feature enhancement vector to obtain the image splicing vector. In the feature integration process, more emphasis is placed on combining the features of the two according to a certain logic, such as correspondingly combining the key spectrum features in the spectrum description vector with the core enhancement features in the directional feature enhancement vector to form a new image splicing vector. In addition, in this process, the size of the vehicle image spectrum description vector is the same as the size of the directional feature enhancement vector under the target attention channel. This requirement ensures that the data can be accurately fused or integrated during the splicing process to avoid information loss or incorrect fusion due to size mismatch.

[0055] Since the target multimodal image description includes multiple multimodal image descriptions of historical vehicle images, the original channel attention sequence vector also includes the modal vector obtained after mining each multimodal image description in these multiple multimodal image descriptions. After obtaining the image splicing vector, the original channel attention sequence vector is to be mined according to the image splicing vector to generate the channel attention description vector of the target multimodal image description under the target attention channel. This process first involves a correlation analysis of the image splicing vector and the original channel attention sequence vector. Correlation analysis is a method to quantify the relationship between the two. By calculation, the correlation coefficient between each modal vector in the original channel attention sequence vector and the image splicing vector can be obtained. For example, for a modal vector related to color modality, such as the RGB value variation range of the license plate color under different lighting conditions, after correlation analysis calculation, it may be obtained that its correlation coefficient with the image splicing vector is 0.6; for a modal vector related to shape modality, such as the shape variation information of the license plate under different viewing angles, it may be obtained that its correlation coefficient with the image splicing vector is 0.4. These correlation coefficients reflect the degree of association between each modality vector and the image stitching vector.

[0056] Then, according to the correlation coefficient between each modal vector and the image stitching vector, each modal vector is modally optimized. Modal optimization is the process of adjusting the modal vector according to the correlation coefficient, aiming to strengthen the features with high correlation with the image stitching vector and weaken the features with low correlation. For example, for a modal vector with a high correlation coefficient, such as a modal vector with a correlation coefficient of 0.8, some elements in the modal vector may be weighted and strengthened during the modal optimization process. If this modal vector is related to the color of the license plate, then the value representing the color feature of the blue license plate may be weighted, such as further adjusting the value range of the blue component to 180-255 to highlight the color feature of the blue license plate; for a modal vector with a low correlation coefficient, such as a modal vector with a correlation coefficient of 0.3, if this modal vector contains some features that may be greatly disturbed by the environment, such as color features that are prone to errors when the light is strong, then these feature values ​​may be weakened, such as reducing the weight of these color feature values ​​in the entire modal vector, thereby reducing their interference with license plate recognition. In this way, multiple modal vectors after modal optimization are obtained.

[0057] Finally, the modal vectors optimized by these multiple modalities are feature integrated to obtain the channel attention description vector of the target multimodal image description under the target attention channel. In the process of feature integration, the effective features in the modal vectors optimized by each modality are merged to form a vector that focuses more on the key features related to license plate recognition under the target attention channel. For example, when the target attention channel is the RGB color channel, after the previous operation, the channel attention description vector obtained by feature integration will more accurately reflect the color characteristics of the blue license plate, including the most likely RGB value range under different lighting conditions, and the relationship between these values ​​and spectral features and directional enhancement features; in the morphological channel, the obtained channel attention description vector will more accurately reflect the shape characteristics of the license plate, such as the shape outline of the license plate can still be accurately judged in the presence of partial occlusion or deformation, because the shape-related features have been enhanced and adjusted in the modal optimization process; in terms of the sequence keyword channel, the obtained channel attention description vector will more clearly reflect the arrangement rules of the license plate characters. For blurred or partially damaged license plate characters, the recognition ability of these characters can be improved because the features related to character arrangement are enhanced in the modal optimization process.

[0058] The above technical scheme has many advantages in the license plate recognition process. In terms of color recognition, by splicing the vehicle image spectrum description vector with the directional feature enhancement vector under the target attention channel, and then undergoing operations such as correlation analysis, modal optimization and feature integration, the license plate color can be more accurately identified under the RGB color channel. For example, under different lighting conditions, the traditional color recognition method may cause misjudgment of the license plate color due to changes in light intensity, while the present scheme can accurately identify the color of the blue license plate by fusing the spectral features and directional enhancement features. Even under strong light, the RGB value of the blue license plate changes, but it can still be accurately judged as a blue license plate by the spectral features and the enhanced color feature values. In terms of shape recognition, for the morphological channel, this technical scheme can more accurately determine the shape characteristics of the license plate. For example, when the license plate is partially blocked or deformed due to the shooting angle, the traditional shape recognition method may make a misjudgment, while the present scheme can accurately judge the shape contour of the license plate based on the existing shape information and the association with the spectral features and directional enhancement features through the optimization and feature integration of the shape modal vector. In terms of character recognition, under the sequence keyword channel, this solution can more clearly identify the arrangement rules of license plate characters and improve the recognition ability of blurred or partially damaged license plate characters. For example, when some characters on the license plate become blurred due to wear or stains, traditional character recognition methods may not be able to accurately recognize the character content, but this solution can improve the recognition accuracy of blurred characters by optimizing the modal vector related to the character arrangement and utilizing the relationship with the spectral features and directional enhancement features.

[0059] Therefore, the accuracy of license plate recognition is significantly improved through the special definition of feature mining indicator variables under the target attention channel and a series of complex vector processing operations. Under different attention channels, feature information related to license plate recognition can be mined more accurately, effectively reducing the interference of external factors such as lighting, occlusion, stains, etc. on recognition. For example, in color recognition, the color of the license plate can be determined more stably, the outline of the license plate can be judged more accurately in shape recognition, and the recognition ability of blurred or damaged characters can be improved in character recognition, thereby enhancing the reliability and stability of the entire license plate recognition technology solution, and providing better quality and efficient license plate recognition technology support for traffic management, security monitoring and other fields.

[0060] In an optional embodiment, the attention sequence expansion model includes multiple sequence keyword expansion branches, and a sequence keyword expansion branch is used to characterize one attention channel among the multiple attention channels; the use of the attention sequence expansion model to generate the license plate information of the target vehicle image in the target scene according to the vehicle image spectrum description vector and the channel attention description vectors under the multiple attention channels includes: determining the recognition confidence corresponding to each sequence keyword expansion branch according to the vehicle image spectrum description vector and the channel attention description vector corresponding to each sequence keyword expansion branch; the recognition confidence corresponding to any sequence keyword expansion branch refers to the confidence weight of the license plate recognition event corresponding to any sequence keyword expansion branch implemented on the target vehicle image in the target scene; and determining the license plate information of the target vehicle image in the target scene according to the recognition confidence corresponding to each sequence keyword expansion branch.

[0061] It can be understood that the attention sequence expansion model is one of the core elements of the present technical solution, which contains multiple sequence keyword expansion branches, each of which corresponds to one of the multiple attention channels. This correspondence constructs a unique structure, providing a multi-dimensional analysis approach for license plate recognition.

[0062] When using the attention sequence expansion model to generate the license plate information of the target vehicle image in the target scene based on the vehicle image spectrum description vector and the channel attention description vectors under multiple attention channels, the recognition confidence corresponding to each sequence keyword expansion branch should be determined respectively according to the vehicle image spectrum description vector and the channel attention description vector corresponding to each sequence keyword expansion branch. Recognition confidence is a key concept, which represents the confidence weight of the license plate recognition event corresponding to any sequence keyword expansion branch implemented on the target vehicle image in the target scene. For example, when processing a target vehicle image with a resolution of 800×600, for the sequence keyword expansion branch corresponding to the RGB color channel, the vehicle image spectrum description vector may contain spectral features related to color in a specific frequency band of the spectrum (such as the amplitude of 20-40 corresponding to the low frequency band 0-0.1 Hz), and the channel attention description vector corresponding to the RGB color channel may contain the value range of the license plate color in the RGB color space (such as the red component of a blue license plate is 0-50, the green component is 50-100, and the blue component is 150-255). By processing these two vectors through a specific algorithm (based on a large amount of sample data and predefined rules), the recognition confidence of the sequence keyword expansion branch corresponding to the RGB color channel can be determined. For example, after calculation, its recognition confidence is 0.6 (this is an example value, and the actual value depends on the specific algorithm and data).

[0063] Similarly, for the sequence keyword extension branch corresponding to the HSV color channel, the spectral features in the vehicle image spectral description vector (such as the amplitude and frequency range corresponding to a certain frequency band) are combined with the channel attention description vector corresponding to the HSV color channel (including the hue, saturation and other numerical ranges of the license plate color in the HSV color space, such as the hue value in the range of 120-140 and the saturation in the range of 0.8-1), and the recognition confidence is obtained after the corresponding algorithm processing, for example, 0.7. In terms of the sequence keyword extension branch corresponding to the morphological channel, the spectral features in the vehicle image spectral description vector (such as the spectral values ​​related to the shape contour of the low-frequency part) and the channel attention description vector corresponding to the morphological channel (including the shape structure information of the license plate, such as the aspect ratio close to 3:1, the character spacing in the range of 10-15 pixels, etc.) are processed by the algorithm to obtain its recognition confidence, for example, 0.5. For the sequence keyword extension branch corresponding to the sequence keyword channel, the spectral features in the vehicle image spectral description vector (such as the spectral values ​​related to the high-frequency part and the character edge) and the channel attention description vector corresponding to the sequence keyword channel (including information related to the arrangement rules of the license plate characters) are processed to obtain its recognition confidence, for example, 0.8.

[0064] Then, according to the recognition confidence corresponding to each sequence keyword expansion branch, the license plate information of the target vehicle image in the target scene is determined. This process is similar to a comprehensive decision-making process, and the recognition confidence of each sequence keyword expansion branch participates in the determination of the final license plate information as a weight. For example, in the above example, the recognition confidence corresponding to the RGB color channel is 0.6, the recognition confidence corresponding to the HSV color channel is 0.7, the recognition confidence corresponding to the morphological channel is 0.5, and the recognition confidence corresponding to the sequence keyword channel is 0.8. If the recognition confidence of the RGB color channel and the HSV color channel is relatively high when determining the license plate color information, then when considering comprehensively, the color information provided by these two channels will have a greater weight in the final determination of the license plate color. When determining the license plate shape information, although the recognition confidence of the morphological channel is 0.5, it will still provide relevant information about the license plate shape according to its weight. For the license plate character information, since the recognition confidence of the sequence keyword channel is 0.8, the information such as the character arrangement law provided by it will play an important role in the final determination of the license plate character.

[0065] Taking color recognition as an example, by combining the vehicle image spectrum description vector with the channel attention description vector corresponding to different color channels to determine the recognition confidence, the license plate color can be recognized more accurately. Traditional license plate recognition methods may only rely on a single color space or simple color feature judgment, which is easily disturbed by factors such as lighting and environmental color. In this scheme, by comprehensively considering the spectral features and channel attention description vectors, and making comprehensive judgments with recognition confidence as weight, the license plate color can be recognized more stably under different lighting conditions. For example, under strong light, the color value of the license plate in the RGB color space may change, but through the spectral features in the vehicle image spectrum description vector and the hue and saturation information in the channel attention description vector corresponding to the HSV color channel, combined with a higher recognition confidence, the license plate color can be accurately judged.

[0066] In terms of shape recognition, the recognition confidence of the sequence keyword expansion branch corresponding to the morphological channel can more accurately determine the shape information of the license plate. When the license plate is partially occluded or deformed, the traditional shape recognition method may make a misjudgment. In this scheme, the recognition confidence is determined by combining the spectral features in the vehicle image spectral description vector (such as the features related to the shape contour in the low-frequency part) with the shape structure information in the channel attention description vector corresponding to the morphological channel (such as aspect ratio, character spacing, etc.). Even if there is a certain change in the shape, the shape contour of the license plate can be reasonably judged according to its weight.

[0067] In terms of character recognition, the recognition confidence of the sequence keyword extension branch corresponding to the sequence keyword channel plays an important role. When the characters of the license plate are partially blurred or damaged, traditional character recognition methods may be difficult to accurately recognize. In this scheme, the recognition confidence is determined by combining the spectral features in the vehicle image spectrum description vector (such as the features related to the high-frequency part and the character edge) with the character arrangement regularity information in the channel attention description vector corresponding to the sequence keyword channel. This can more effectively utilize the character arrangement regularity and the character edge-related features in the spectrum, thereby improving the recognition ability of blurred or damaged characters.

[0068] This design combines the vehicle image spectrum description vector with the attention description vector of each channel to determine the recognition confidence of the sequence keyword expansion branch, and then determines the license plate information, thereby improving the accuracy of license plate recognition. In terms of color recognition, it can overcome interference such as lighting to stably recognize colors; in shape recognition, it can accurately judge shapes in response to occlusion and deformation; in character recognition, it also has good recognition capabilities for blurred or damaged characters, which overall improves the reliability and effectiveness of license plate recognition technology and provides better technical support for related fields such as traffic management.

[0069] In other examples, there are multiple target multimodal image descriptions, one target multimodal image description has a channel attention description vector under one attention channel, and any one of the multiple sequence keyword extension branches is a target sequence keyword extension branch; the recognition confidence corresponding to each sequence keyword extension branch is determined according to the vehicle image spectrum description vector and the channel attention description vector corresponding to each sequence keyword extension branch, including: acquiring scene noise detection data of the target scene, and using the attention sequence extension model to mine the scene noise detection data to obtain the scene noise modal vector of the target scene; feature integration of the vehicle image spectrum description vector, the scene noise modal vector and multiple channel attention description vectors of multiple target multimodal image descriptions under the attention channel of the target sequence keyword extension branch to obtain the input vector of the target sequence keyword extension branch; and generating the recognition confidence corresponding to the target sequence keyword extension branch according to the input vector of the target sequence keyword extension branch.

[0070] In the next step, the attention sequence expansion model includes a feature intersection branch, and the feature intersection branch is used to extract an image description vector in which the multiple sequence keyword expansion branches are crossed in the target multimodal image description, and the feature intersection branch also has a corresponding input vector; the recognition confidence corresponding to the target sequence keyword expansion branch is generated according to the input vector of the target sequence keyword expansion branch, including: using the target sequence keyword expansion branch to extract edge elements of the input vector of the target sequence keyword expansion branch to obtain a first edge description vector of the input vector of the target sequence keyword expansion branch; using the feature intersection branch to extract edge elements of the input vector of the feature intersection branch to obtain a second edge description vector of the input vector of the feature intersection branch; using the target sequence keyword expansion branch to generate the recognition confidence corresponding to the target sequence keyword expansion branch according to the first edge description vector and the second edge description vector.

[0071] In a further embodiment, the use of the target sequence keyword expansion branch to generate the recognition confidence corresponding to the target sequence keyword expansion branch according to the first edge description vector and the second edge description vector includes: using the target sequence keyword expansion branch to perform full connection processing on the first edge description vector and the second edge description vector to obtain a fully connected vector; using the target sequence keyword expansion branch to generate the recognition confidence corresponding to the target sequence keyword expansion branch according to the fully connected vector.

[0072] In some other preferred design ideas, the target sequence keyword expansion branch includes X first residual convolution modules, and a first residual convolution module is used to generate a first edge description vector; the feature intersection branch includes Y second residual convolution modules, and a second residual convolution module is used to generate a second edge description vector, and X and Y are both positive integers; the first edge description vector and the second edge description vector are fully connected by using the target sequence keyword expansion branch to obtain a fully connected vector, including: using the target sequence keyword expansion branch to generate a fully connected parameter according to the input vector of the target sequence keyword expansion branch; the fully connected parameter includes a parameter for the The fully connected coefficient of each first residual convolution module and the fully connected coefficient of each second residual convolution module; by performing modal optimization on the first edge description vector generated by each first residual convolution module for the fully connected coefficient of each first residual convolution module, X modal optimized first edge description vectors are obtained; by performing modal optimization on the second edge description vector corresponding to each second residual convolution module for the fully connected coefficient of each second residual convolution module, Y modal optimized second edge description vectors are obtained; and feature integration is performed on the X modal optimized first edge description vectors and the Y modal optimized second edge description vectors to obtain the fully connected vector.

[0073] In the above four interrelated embodiments, there are multiple target multimodal image descriptions, and one target multimodal image description has one channel attention description vector under one attention channel. This correspondence between multiple target multimodal image descriptions and channel attention description vectors lays the foundation for subsequent complex operations and analysis. At the same time, setting any one of the multiple sequence keyword expansion branches as the target sequence keyword expansion branch provides a clear object for detailed operations under a specific branch.

[0074] A series of complex operations are required to determine the recognition confidence corresponding to each sequence keyword expansion branch. First, the scene noise detection data of the target scene is obtained. This data reflects the noise factors in the target scene that may affect license plate recognition. For example, in a traffic scene, the scene noise detection data may contain information such as the uneven distribution of light (such as the light intensity in some areas is 50-100 lux, while other areas are 150-200 lux, etc.), the complexity of the background (such as there are multiple objects in the background, the complexity of their colors and shapes can be represented by certain quantitative indicators, such as the color types are between 5-10, and the shape complexity index is in the range of 0.3-0.5, etc.). The scene noise detection data is mined using the attention sequence expansion model to obtain the scene noise modal vector of the target scene. This scene noise modal vector is a feature representation obtained after an in-depth analysis of the scene noise detection data. It contains various feature information related to noise, such as the characteristics of noise in the frequency domain (such as the amplitude of the noise spectrum in certain frequency bands in the numerical range of 10-20), the distribution characteristics of noise in the spatial domain (such as the numerical range of the position distribution ratio of noise in the image), etc.

[0075] Next, the vehicle image spectrum description vector, scene noise modal vector, and multiple channel attention description vectors of multiple target multimodal image descriptions under the attention channel of the target sequence keyword expansion branch are feature integrated to obtain the input vector of the target sequence keyword expansion branch. Taking a vehicle image with a resolution of 800×600 as an example, the vehicle image spectrum description vector may contain the amplitude and frequency range information of different frequency bands of the image on the spectrum (such as the amplitude of 20-40 for the low frequency band 0-0.1 Hz, and the amplitude of 80-120 for the high frequency band 0.5-1 Hz). The multiple channel attention description vectors of the multiple target multimodal image descriptions under the attention channel of the target sequence keyword expansion branch may contain various information related to license plate recognition, such as the value range of the license plate color in the RGB color space under the RGB color channel (such as the red component of a blue license plate is 0-50, the green component is 50-100, and the blue component is 150-255, etc.), and the shape structure information of the license plate under the morphological channel (such as the aspect ratio is close to 3:1, the character spacing is in the range of 10-15 pixels, etc.). The features of these three vectors are integrated through a specific feature integration algorithm (based on a large amount of sample data and predefined rules) to obtain the input vector of the target sequence keyword expansion branch. This input vector is a vector that integrates the channel attention features related to the vehicle image spectrum, scene noise, and target multimodal image description. It contains rich information and provides a comprehensive data basis for the subsequent generation of recognition confidence.

[0076] Then, the recognition confidence corresponding to the target sequence keyword expansion branch is generated according to the input vector of the target sequence keyword expansion branch. In this process, the attention sequence expansion model includes a feature intersection branch, which is used to extract the image description vector where multiple sequence keyword expansion branches are intersected in the target multimodal image description, and the feature intersection branch also has a corresponding input vector. First, the target sequence keyword expansion branch is used to extract the edge elements of the input vector of the target sequence keyword expansion branch to obtain the first edge description vector of the input vector of the target sequence keyword expansion branch. For example, when processing the input vector of the target sequence keyword expansion branch, edge elements related to the color change of the license plate edge, the shape edge, etc. may be extracted, and these elements are quantized and feature represented to form the first edge description vector. Among them, the edge color change may be represented in the numerical form of color difference (such as the color difference in some edge areas is in the numerical range of 10-20), and the shape edge may be represented in the numerical form of edge curvature (such as the edge curvature is in the numerical range of 0.1-0.3).

[0077] At the same time, the feature intersection branch is used to extract edge elements from the input vector of the feature intersection branch to obtain the second edge description vector of the input vector of the feature intersection branch. This process is similar to the edge element extraction of the target sequence keyword expansion branch mentioned above, but it is aimed at the input vector of the feature intersection branch. For example, the input vector of the feature intersection branch may contain information related to license plate recognition at the intersection of different attention channels, such as color and shape intersection features (for example, the correlation features of color and shape in certain areas can be represented by specific numerical values, such as a correlation degree in the range of 0.2-0.4, etc.), and edge elements are extracted from this information to form a second edge description vector.

[0078] Next, the target sequence keyword extension branch is used to generate the recognition confidence corresponding to the target sequence keyword extension branch according to the first edge description vector and the second edge description vector. In this process, the target sequence keyword extension branch is first used to perform full connection processing on the first edge description vector and the second edge description vector to obtain a fully connected vector. Here, the target sequence keyword extension branch includes X first residual convolution modules, and a first residual convolution module is used to generate a first edge description vector; the feature cross branch includes Y second residual convolution modules, and a second residual convolution module is used to generate a second edge description vector, and X and Y are both positive integers. The target sequence keyword extension branch is used to generate a fully connected parameter according to the input vector of the target sequence keyword extension branch, and this fully connected parameter includes a fully connected coefficient for each first residual convolution module and a fully connected coefficient for each second residual convolution module. For example, the fully connected coefficient for a first residual convolution module may be in the range of 0.3-0.5, and the fully connected coefficient for a second residual convolution module may be in the range of 0.2-0.4.

[0079] By performing modal optimization on the first edge description vector generated by each first residual convolution module for the full connection coefficient of each first residual convolution module, X modal optimized first edge description vectors are obtained. For example, if a first edge description vector contains the color numerical information of the edge of the license plate color (such as the blue component of the edge of the blue license plate is in the numerical range of 180-220), through the modal optimization with a full connection coefficient of 0.4, this numerical range may be adjusted to a numerical range of 190-210 to highlight the association with other factors. Similarly, by performing modal optimization on the second edge description vector corresponding to each second residual convolution module for the full connection coefficient of each second residual convolution module, Y modal optimized second edge description vectors are obtained. Finally, the features of the X modal optimized first edge description vectors and the Y modal optimized second edge description vectors are integrated to obtain a fully connected vector. This fully connected vector is the result of multiple rounds of optimization and integration, and it integrates the feature information of the first edge description vector and the second edge description vector after modal optimization.

[0080] The recognition confidence corresponding to the target sequence keyword expansion branch is generated according to the fully connected vector using the target sequence keyword expansion branch. This recognition confidence is a result of combining multiple factors, which reflects the confidence level of the license plate recognition event corresponding to the target vehicle image in the target scene. For example, after the above series of operations, the recognition confidence obtained is 0.7 (this is an example value, the actual value depends on the specific calculation and data). This value indicates that the credibility of the license plate recognition event corresponding to the target sequence keyword expansion branch is 70% under the comprehensive consideration of multiple factors such as the current vehicle image, scene noise, multi-modal image description, and feature crossover.

[0081] When processing vehicle images, by considering scene noise detection data, we can better cope with various complex actual scenes. For example, in a parking lot scene with uneven lighting, traditional license plate recognition methods may cause errors in license plate color recognition or inaccurate shape judgment due to differences in lighting. In this solution, since the scene noise modal vector is incorporated into the construction of the input vector of the target sequence keyword expansion branch, the recognition strategy can be adjusted according to the specific numerical characteristics of uneven lighting (such as different numerical ranges of light intensity), thereby improving the accuracy of license plate recognition.

[0082] When processing multimodal image description and feature intersection, various feature information in the license plate image can be more fully utilized by extracting, optimizing and integrating the first edge description vector and the second edge description vector. For example, when the license plate is partially occluded or defaced, traditional methods may not be able to accurately identify it due to the loss of some features. In this solution, the feature information of the license plate can be mined from different angles through the channel attention description vectors under multiple target multimodal image descriptions and the cross-image description vectors extracted by the feature intersection branch. Even if some features are occluded or defaced, the license plate can still be accurately identified through the association and complementarity of other related features.

[0083] When using the residual convolution module for modal optimization and full connection processing, the edge description vector can be accurately adjusted according to different full connection coefficients. For example, when there is a complex association between the color and shape of the license plate, different full connection coefficients can adjust the characteristics of the color edge description vector and the shape edge description vector according to the specific value of this association (such as the numerical range of the association between color and shape), thereby more accurately determining the characteristic information of the license plate and improving the accuracy and reliability of recognition.

[0084] It can be seen that by incorporating scene noise detection data to construct the input vector, the adaptability to complex scenes is enhanced and the interference of environmental factors on license plate recognition is reduced. The use of multi-modal image description and feature cross-branching makes license plate feature mining more comprehensive and improves the recognition ability of partially occluded or damaged license plates. The modal optimization and full connection processing of the residual convolution module can accurately adjust the feature vector and accurately capture the associated features such as license plate color and shape, thereby improving the accuracy, reliability and adaptability of license plate recognition, and providing more effective technical support for fields such as traffic management.

[0085] In an optional embodiment, determining the license plate information of the target vehicle image in the target scene according to the recognition confidence corresponding to each sequence keyword expansion branch includes: obtaining the noise factor of the license plate recognition event corresponding to each sequence keyword expansion branch; performing modal optimization on the recognition confidence corresponding to each sequence keyword expansion branch according to the noise factor of the license plate recognition event corresponding to each sequence keyword expansion branch to obtain the modally optimized recognition confidence corresponding to each sequence keyword expansion branch; and performing feature integration on the modally optimized recognition confidence corresponding to each sequence keyword expansion branch to obtain the license plate information of the target vehicle image in the target scene.

[0086] In the field of license plate recognition technology, the technical solution involved in this embodiment aims to more accurately determine the license plate information of a target vehicle image in a target scene.

[0087] First, it is clear that the license plate information of the target vehicle image in the target scene should be determined according to the recognition confidence corresponding to each sequence keyword expansion branch. The first step of this process is to obtain the noise factor of the license plate recognition event corresponding to each sequence keyword expansion branch. These noise factors reflect the interference factors that may exist in the license plate recognition event. For example, in the license plate recognition event corresponding to the RGB color channel, the noise factor may involve the degree of influence of the illumination change. Taking an actual scene as an example, in a certain period of time, the fluctuation range of the illumination intensity is between 100-200 lux. This change in illumination intensity may become a noise factor that affects the accuracy of color recognition. For the license plate recognition event corresponding to the HSV color channel, the noise factor may include the color deviation of the image acquisition device, such as the color deviation in a certain value range (for example, the hue deviation is within the range of ±5, and the saturation deviation is within the range of ±0.1). In the license plate recognition event corresponding to the morphological channel, the noise factor may be related to the blurriness of the image. For example, the blurriness of the image can be measured by edge sharpness, and the edge sharpness value is between 0.1-0.3 (the lower the value, the more blurry it is). This is the noise factor that affects shape recognition. For the license plate recognition event corresponding to the sequence keyword channel, the noise factor may be due to the reduction in character clarity caused by wear or stains on some characters. For example, the character clarity is represented by a specific value, which is between 0.2-0.4 when the wear or stains are serious.

[0088] After obtaining these noise factors, the recognition confidence corresponding to each sequence keyword expansion branch is modally optimized according to the noise factor of the license plate recognition event corresponding to each sequence keyword expansion branch, and the recognition confidence after modal optimization corresponding to each sequence keyword expansion branch is obtained. For example, in the license plate recognition event corresponding to the RGB color channel, the initial recognition confidence is 0.6. If the light intensity fluctuates greatly and the noise factor of the light intensity is at a high level (such as the noise factor weight corresponding to the fluctuation range of 150-200 lux is 0.3), then when performing modal optimization, it is necessary to adjust the recognition confidence according to this noise factor. The adjustment method may be based on a pre-set rule or algorithm, and the recognition confidence after modal optimization is obtained after calculation, for example, the adjusted value is 0.4. In the license plate recognition event corresponding to the HSV color channel, if the noise factor of the color deviation is 0.2 (when the hue deviation and saturation deviation are in the numerical range of the above example), the initial recognition confidence is 0.7. After modal optimization, the recognition confidence after modal optimization may be 0.56. For the license plate recognition event corresponding to the morphological channel, the noise factor of the image blur is 0.2 (the corresponding noise factor when the edge sharpness is between 0.1-0.3). For example, the initial recognition confidence is 0.5, which may become 0.4 after modal optimization. In the license plate recognition event corresponding to the sequence keyword channel, the noise factor of the character clarity is 0.3 (the corresponding noise factor when the character clarity is between 0.2-0.4), the initial recognition confidence is 0.8, and it may become 0.56 after modal optimization.

[0089] Finally, the recognition confidence after modal optimization corresponding to each sequence keyword expansion branch is feature integrated to obtain the license plate information of the target vehicle image in the target scene. This feature integration process is a process that comprehensively considers the results of modal optimization of each sequence keyword expansion branch. For example, when determining the color information of the license plate, the recognition confidence after modal optimization of the RGB color channel and the HSV color channel (such as 0.4 for the RGB color channel and 0.56 for the HSV color channel) will participate in the integration process according to certain weights. When determining the shape information of the license plate, the recognition confidence after modal optimization of the morphological channel (such as 0.4) will play a key role. For the license plate character information, the recognition confidence after modal optimization of the sequence keyword channel (such as 0.56) will become an important basis. Through this feature integration, various factors are comprehensively considered to finally determine the license plate information of the target vehicle image in the target scene.

[0090] This technical solution has many advantages in license plate recognition. In terms of color recognition, the color of the license plate can be determined more accurately by considering the influence of noise factors such as illumination changes on the recognition confidence. For example, if the noise factor is not considered when the light intensity fluctuates greatly, the color of the license plate may be judged incorrectly. However, this solution can more accurately identify the color of the license plate even if the light intensity fluctuates between 100-200 lux by modal optimization and feature integration of the recognition confidence of the RGB color channel and the HSV color channel. In shape recognition, considering the influence of the noise factor of the image blur degree on the recognition confidence of the morphological channel helps to more accurately judge the shape of the license plate when the image is blurred. For example, when the edge sharpness is between 0.1-0.3, the traditional method may find it difficult to accurately identify the shape of the license plate, but this solution can improve the accuracy of shape recognition through modal optimization and feature integration. In character recognition, considering the influence of the noise factor of character clarity on the recognition confidence of the sequence keyword channel, the license plate characters can be more effectively recognized for license plates with some characters worn or stained. For example, when the character clarity is between 0.2 and 0.4, this solution can use the recognition confidence after modal optimization to more accurately recognize the license plate characters.

[0091] It can be seen that by considering the noise factor of the license plate recognition event, the recognition confidence is modally optimized and the features are integrated to determine the license plate information. In color recognition, it can accurately judge the license plate color in response to changes in illumination, in shape recognition, it can accurately judge the license plate shape by dealing with image blur, and in character recognition, it can effectively recognize the license plate characters by overcoming the effects of character wear or stains, thereby improving the accuracy and reliability of license plate recognition and providing better license plate recognition technology support for traffic management and other related fields.

[0092] In some other embodiments, there are Z target vehicle images, where Z is a positive integer, and the target scene has one license plate information for one target vehicle image; the method further includes: sorting the Z target vehicle images according to a descending priority rule of the license plate information of each target vehicle image of the target scene to obtain Z sorted target vehicle images; marking the first P target vehicle images among the sorted Z target vehicle images; P is a positive integer and P is less than Z.

[0093] In the license plate recognition technology system, the technical solution proposed in this embodiment further enriches and improves the operation process related to license plate recognition.

[0094] This embodiment involves multiple target vehicle images, the number of which is Z (Z is a positive integer), and each target vehicle image has a corresponding license plate information in the target scene. Here, the one-to-one correspondence between the target scene and the license plate information is emphasized, which reflects the pertinence and particularity of license plate recognition in different scenes.

[0095] The next operation is to sort the Z target vehicle images according to the priority descending rule of the license plate information of each target vehicle image in the target scene. This priority descending rule is set based on the importance of the license plate information in the target scene. For example, in some scenarios, the accuracy of the license plate information may be the most important consideration, while in other scenarios, the speed of obtaining the license plate information or certain characteristics related to the specific vehicle type may be more critical. Taking a traffic monitoring scenario as an example, if in a toll management scenario on a highway, for the license plate recognition of vehicles passing through the toll station quickly, the accuracy of the license plate information and the speed of obtaining the license plate information may be set as high priority factors. If a vehicle has extremely high accuracy of license plate information and a very fast acquisition speed, then under this priority descending rule, the target vehicle image corresponding to this vehicle will be ranked higher.

[0096] In actual operation, for example, there are 10 (i.e., Z=10) target vehicle images, each of which has its license plate information in the target scene. According to the priority descending rule in the above-mentioned highway toll management scenario, the accuracy of the license plate information is represented by a numerical range (such as an accuracy between 0.8-1 indicates high accuracy), and the acquisition speed is also represented by a numerical range (such as an acquisition speed between 0.1-0.3 seconds indicates fast acquisition). For a certain target vehicle image, its license plate information accuracy is 0.9 and the acquisition speed is 0.15 seconds. After considering these two factors (and other factors that may exist and calculated according to certain weights), a priority value is obtained, such as 0.8. The license plate information accuracy of another target vehicle image is 0.7 and the acquisition speed is 0.25 seconds. After comprehensive calculation, the priority value is 0.6. According to the priority descending rule, the target vehicle image with a high comprehensive value of accuracy and acquisition speed will be ranked first, so these 10 target vehicle images will be sorted according to their respective calculated priority values.

[0097] After the sorting is completed, Z target vehicle images are obtained, and then the first P target vehicle images among the Z target vehicle images are labeled (P is a positive integer and P is less than Z). For example, in the above example of 10 target vehicle images (Z=10), if P=3, then the target vehicle images ranked in the top 3 in descending order of priority will be labeled. The labeling process may involve adding specific identifiers to these target vehicle images or recording specific information. This labeling operation has its unique significance in the entire license plate recognition system. The labeled target vehicle images can be used as the object of further analysis, statistics or special processing. For example, in traffic flow statistics and analysis, these labeled target vehicle images may be the focus of attention because they have a higher priority in terms of the accuracy of license plate information, acquisition speed or other related factors.

[0098] In the field of traffic management, by sorting the target vehicle images according to the priority of the license plate information and marking some of the target vehicle images, vehicles can be managed more efficiently. For example, in the traffic violation monitoring scenario, if vehicles with high-priority license plate information can be quickly and accurately identified, it is of great significance for timely handling of traffic violations. For vehicles with high-accuracy license plate information and fast acquisition speed (i.e., vehicles corresponding to the target vehicle images at the front after sorting), violations can be determined and handled more quickly.

[0099] In terms of parking lot management, sorting and marking target vehicle images according to the priority of license plate information helps optimize the parking management process. For example, a parking lot has special parking requirements, such as giving priority to providing better parking services for certain member vehicles or special vehicles. When identifying the license plate information of these vehicles, if the target vehicle images with high-priority license plate information related to special parking requirements are marked according to the technical solution of this embodiment, these vehicles can be quickly guided to the appropriate parking location, thereby improving the management efficiency of the parking lot.

[0100] In the field of security monitoring, this technical solution also has important application value. In the entrance and exit monitoring of some important places, vehicles with high license plate information priority can be marked and monitored. For example, in a highly secure area, for legal vehicles that frequently enter and exit, if their license plate information can be quickly and accurately obtained (i.e., license plate information has a high priority), these vehicles can be marked with target vehicle images and given special attention in the monitoring system. Once an abnormal situation occurs (such as abnormal vehicle behavior or sudden changes in license plate information, etc.), timely response can be made.

[0101] In this way, by sorting the target vehicle images according to the descending priority rule of the license plate information, important vehicles can be quickly located according to the needs of different scenarios. Labeling the first P target vehicle images can effectively handle traffic violations in traffic management, optimize parking processes in parking lot management, and respond to abnormal situations in security monitoring in a timely manner, thereby improving the efficiency, pertinence and safety of vehicle management and providing more effective technical means for management in related fields.

[0102] In some independent embodiments, the acquisition of the attention sequence extension model includes: acquiring a multimodal image description debugging sample of a historical image debugging sample; the historical image debugging sample refers to a license plate area detection image corresponding to a license plate recognition event implemented in a sample scene; acquiring an attention sequence extension model to be debugged; the attention sequence extension model to be debugged is used to perform license plate recognition in the sample scene based on the multiple attention channels, and each of the attention channels has its own feature mining indicator variable to be debugged, and the feature mining indicator variable belongs to a model variable of the attention sequence extension model to be debugged; according to the feature mining indicator variable to be debugged under each attention channel, the multimodal image description debugging sample is subjected to attention vector mining to obtain a channel attention description vector debugging sample of the multimodal image description debugging sample under each attention channel; Utilize the attention sequence extension model to be debugged to obtain a spectrum description vector debugging sample of a vehicle image debugging sample; utilize the attention sequence extension model to be debugged to respectively determine the recognition confidence debugging sample corresponding to each attention channel according to the channel attention description vector debugging sample and the spectrum description vector debugging sample under each attention channel; the recognition confidence debugging sample corresponding to any attention channel refers to the confidence weight of the license plate recognition event of executing any attention channel on the vehicle image debugging sample under the sample scene, and the recognition confidence debugging samples corresponding to the multiple attention channels are used to indicate the license plate information of the vehicle image debugging sample in the sample scene; according to the recognition confidence debugging sample corresponding to each attention channel, improve the model variables of the attention sequence extension model to be debugged to obtain the attention sequence extension model.

[0103] Further, the attention sequence expansion model to be debugged includes multiple sequence keyword expansion branches to be debugged, one sequence keyword expansion branch to be debugged is used to characterize one of the multiple attention channels, and each sequence keyword expansion branch to be debugged corresponds to a different training annotation of the vehicle image debugging sample; the training annotation corresponding to any sequence keyword expansion branch to be debugged is used to indicate whether the license plate recognition event corresponding to any sequence keyword expansion branch to be debugged is implemented or not implemented for the vehicle image debugging sample in the sample scenario; the improving the model variables of the attention sequence expansion model to be debugged according to the recognition confidence debugging sample corresponding to each attention channel to obtain the attention sequence expansion model includes: generating the recognition loss of each sequence keyword expansion branch to be debugged for the respective recognition confidence debugging sample according to the difference between the recognition confidence debugging sample corresponding to each sequence keyword expansion branch to be debugged and the training annotation; performing feature integration on the recognition loss of each sequence keyword expansion branch to be debugged to obtain the global recognition loss of the attention sequence expansion model to be debugged; improving the model variables of the attention sequence expansion model to be debugged according to the global recognition loss to obtain the attention sequence expansion model.

[0104] Furthermore, the recognition loss of each sequence keyword expansion branch to be debugged is subjected to feature integration to obtain the global recognition loss of the attention sequence expansion model to be debugged, including: obtaining the branch influence of each sequence keyword expansion branch to be debugged; according to the branch influence of each sequence keyword expansion branch to be debugged, modal optimization is performed on the recognition loss of each sequence keyword expansion branch to be debugged to obtain the recognition loss of each sequence keyword expansion branch to be debugged after modal optimization; and feature integration is performed on the recognition loss of each sequence keyword expansion branch to be debugged after modal optimization to obtain the global recognition loss.

[0105] In the process of developing and optimizing license plate recognition technology, the technical solution described in this embodiment presents a rigorous and systematic method for model acquisition and improvement.

[0106] First, the process of acquiring the attention sequence expansion model involves multiple key links. The starting step is to obtain the multimodal image description debugging samples of historical image debugging samples. Here, the historical image debugging samples are the license plate area detection images corresponding to the license plate recognition events implemented in the sample scene. For example, in a specific sample scene such as the license plate recognition data collection at an urban traffic intersection, a large number of historical image debugging samples are accumulated. These samples contain rich information, including multimodal image description debugging samples of license plates at different angles, different lighting conditions, and different backgrounds. Multimodal image description debugging samples may cover the color information of the license plate (such as the numerical range in the RGB color space, the blue license plate may have a red component in a certain numerical range, a green component in another numerical range, and a blue component in a specific numerical range), shape information (such as the aspect ratio of the license plate, character spacing, etc., expressed in specific numerical values), texture information (expressed in some quantitative texture feature numerical values), and other aspects.

[0107] At the same time, the attention sequence expansion model to be debugged is obtained. This model is designed to perform license plate recognition in sample scenes based on multiple attention channels, and each attention channel has its own feature mining indicator variable to be debugged, which belongs to the model variable. Taking the RGB color channel as an example, its feature mining indicator variable to be debugged may be related to how to accurately mine the color features of the license plate under different lighting conditions. This variable may represent its mining ability or tendency in a certain value or range of values.

[0108] Next, according to the feature mining indicator variables to be debugged under each attention channel, the attention vector mining is performed on the multimodal image description debugging sample to obtain the channel attention description vector debugging sample of the multimodal image description debugging sample under each attention channel. In this process, for each attention channel, according to its specific feature mining indicator variables to be debugged, the feature information related to it is extracted from the multimodal image description debugging sample and converted into a vector form. For example, under the HSV color channel, through its feature mining indicator variables to be debugged, the hue, saturation and other related feature information of the license plate color in the HSV color space may be extracted from the multimodal image description debugging sample to form a channel attention description vector debugging sample under the HSV color channel. The numerical value in this vector debugging sample may reflect the hue value range (such as 120-140) and saturation value range (such as 0.8-1) of the license plate color in a specific sample scene.

[0109] Then, the spectral description vector debugging sample of the vehicle image debugging sample is obtained by using the attention sequence extension model to be debugged. The spectral description vector debugging sample of the vehicle image debugging sample contains the characteristic information of the vehicle image in the frequency domain. For example, the low-frequency part of the spectrum may be related to the overall color area or larger shape outline of the vehicle image. The amplitude of the low-frequency part may be in a certain numerical range (such as 0-50), and the corresponding frequency range may be 0-0.1 Hz; the high-frequency part may be related to the detailed information of the image, such as the edge of the characters on the license plate or the fine texture. The amplitude of the high-frequency part may be between 80-120, and the corresponding frequency range may be in the numerical range of 0.5-1 Hz.

[0110] Then, the attention sequence extension model to be debugged is used to determine the recognition confidence debugging sample corresponding to each attention channel according to the channel attention description vector debugging sample and the spectrum description vector debugging sample under each attention channel. The recognition confidence debugging sample corresponding to any attention channel represents the confidence weight of the license plate recognition event of the attention channel for the vehicle image debugging sample in the sample scene. For example, for the RGB color channel, the recognition confidence debugging sample corresponding to the RGB color channel is calculated by a specific algorithm (based on a large amount of sample data and predefined rules) by combining its channel attention description vector debugging sample (including the feature information of the license plate color in the RGB space) and the spectrum description vector debugging sample (including the information related to the image spectrum). For example, after calculation, the recognition confidence debugging sample of the RGB color channel is 0.6, which means that in the current sample scene, the confidence level of the RGB color channel in recognizing the license plate color of the vehicle image debugging sample is 60%. Similarly, for the HSV color channel, morphology channel, sequence keyword channel, etc., their respective recognition confidence debugging samples will be calculated respectively. These recognition confidence debugging samples are used together to indicate the license plate information of the sample scene for the vehicle image debugging sample.

[0111] Next, the model variables of the attention sequence expansion model to be debugged are improved according to the recognition confidence debugging samples corresponding to each attention channel to obtain the attention sequence expansion model. In this process, since the attention sequence expansion model to be debugged contains multiple sequence keyword expansion branches to be debugged, one sequence keyword expansion branch to be debugged is used to characterize one of the multiple attention channels, and each sequence keyword expansion branch to be debugged corresponds to a different training annotation of the vehicle image debugging sample. The training annotation is used to indicate whether the license plate recognition event corresponding to the sequence keyword expansion branch to be debugged is implemented or not implemented for the vehicle image debugging sample in the sample scenario. For example, for a sequence keyword expansion branch to be debugged (such as a branch related to license plate color recognition), if the license plate color recognition is successful in the vehicle image debugging sample, the training annotation may be "implemented", and if the recognition fails, it may be "not implemented".

[0112] According to the difference between the recognition confidence debugging sample and the training annotation corresponding to each sequence keyword expansion branch to be debugged, the recognition loss of each sequence keyword expansion branch to be debugged for the respective recognition confidence debugging sample is generated. For example, if the recognition confidence debugging sample corresponding to the RGB color channel is 0.6, and the training annotation shows that the license plate color recognition should be successful (ie, "implemented"), but the actual confidence is low, then the recognition loss of the RGB color channel is calculated according to the predefined rules. This recognition loss reflects the gap between the recognition result and the expected result, and may be expressed as a certain value, such as 0.3 (this value is obtained according to the specific calculation rules and is related to the recognition confidence debugging sample and the training annotation).

[0113] Then, the recognition loss of each sequence keyword expansion branch to be debugged is feature integrated to obtain the global recognition loss of the attention sequence expansion model to be debugged. This process first requires obtaining the branch influence of each sequence keyword expansion branch to be debugged. The branch influence reflects the importance of each sequence keyword expansion branch to be debugged in the entire model. This importance may be determined based on a variety of factors, such as the frequency or degree of influence of the branch on the final license plate recognition result in a large amount of sample data. For example, the branch influence corresponding to the RGB color channel is 0.3, and the branch influence corresponding to the HSV color channel is 0.25. According to the branch influence of each sequence keyword expansion branch to be debugged, the recognition loss of each sequence keyword expansion branch to be debugged is modally optimized to obtain the recognition loss of each sequence keyword expansion branch to be debugged after modal optimization. For example, for the RGB color channel, its recognition loss is 0.3, the branch influence is 0.3, and the recognition loss after modal optimization may be adjusted to 0.2 according to a specific algorithm (this value is calculated based on a predefined algorithm, combined with the recognition loss and branch influence). Finally, the recognition loss of each sequence keyword expansion branch to be debugged after modal optimization is integrated to obtain the global recognition loss. This global recognition loss is a value that integrates the information of each sequence keyword expansion branch to be debugged, which reflects the overall gap between the entire attention sequence expansion model to be debugged and the expected result in the current sample scenario.

[0114] Finally, the model variables of the attention sequence expansion model to be debugged are improved according to the global recognition loss to obtain the attention sequence expansion model. By analyzing the global recognition loss, determine which adjustments need to be made to the model variables. For example, if the global recognition loss is mainly caused by color recognition problems (such as large recognition losses in the RGB color channel and the HSV color channel), then the model variables corresponding to these two channels (such as feature mining indicator variables) may be adjusted to improve the model's performance in license plate color recognition, thereby obtaining an optimized attention sequence expansion model.

[0115] Therefore, the attention sequence expansion model to be debugged is systematically improved based on historical image debugging samples, vehicle image debugging samples, and various debugging sample vectors and recognition confidence debugging samples. It can accurately locate the problems of the model under different attention channels, such as analyzing the deficiencies in color, shape, character recognition, etc. through recognition loss. The global recognition loss is obtained by modal optimization and feature integration using branch influence, thereby comprehensively optimizing the model variables, improving the accuracy and reliability of the license plate recognition model, and adapting to the license plate recognition needs in different sample scenarios.

[0116] Further, Figure 2FIG. 2 is a schematic diagram of the structure of an image processing system 200 provided in an embodiment of the present application. Figure 2 The image processing system 200 shown includes a processor 210, which can call and run a computer program from a memory to implement the method in the embodiment of the present application. Figure 2 As shown, the image processing system 200 may further include a memory 230. The processor 210 may call and run a computer program from the memory 230 to implement the method in the embodiment of the present application. The memory 230 may be a separate device independent of the processor 210, or may be integrated in the processor 210. Optionally, as Figure 2 As shown, the image processing system 200 may also include a transceiver 220, and the processor 210 may control the transceiver 220 to interact with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. Optionally, the image processing system 200 may implement the corresponding processes corresponding to the storage engine or the components in the storage engine (such as the processing module) or the device deployed with the storage engine in each method of the embodiment of the present application. For the sake of brevity, it is not repeated here. It should be understood that the processor of the embodiment of the present application may be an integrated circuit chip with signal processing capabilities. It can be understood that the memory in the embodiment of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. It should be noted that the memory of the system and method described herein is intended to include, but is not limited to, a memory of a suitable type.

[0117] Based on the above, a readable storage medium is provided, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the above method are implemented.

[0118] The embodiments of the embodiments of the present application are described above in conjunction with the accompanying drawings, but the embodiments of the present application are not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the embodiments of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the embodiments of the present application and the scope of protection of the embodiments of the present application, all of which are within the protection of the embodiments of the present application.

Claims

1. A license plate recognition method based on image processing, characterized in that: The method is applied to an image processing system, and the method comprises: Obtaining a target multimodal image description of a historical vehicle image; wherein the historical vehicle image refers to a license plate area detection image corresponding to a license plate recognition event implemented in a target scene; Acquire an attention sequence expansion model; wherein the attention sequence expansion model is used to identify the license plate information of the target vehicle image to be identified from multiple attention channels, one attention channel corresponds to a license plate recognition event, each attention channel is configured with a corresponding feature mining indicator variable, and the attention channels include an RGB color channel, an HSV color channel, a morphological channel, and a sequence keyword channel; Attention vector mining is performed on the target multimodal image description according to the feature mining indicator variable under each attention channel to obtain a channel attention description vector of the target multimodal image description under each attention channel; wherein the channel attention description vector under an attention channel includes an image description vector associated with the license plate recognition event of the attention channel in the target multimodal image description; Using the attention sequence expansion model to process the channel attention description vectors under the multiple attention channels, generating the license plate information of the target vehicle image in the target scene; The method further comprises: obtaining a vehicle image spectrum description vector of the target vehicle image using the attention sequence expansion model; The method of using the attention sequence expansion model to process the channel attention description vectors under the multiple attention channels to generate the license plate information of the target vehicle image under the target scene includes: using the attention sequence expansion model to generate the license plate information of the target vehicle image under the target scene according to the vehicle image spectrum description vector and the channel attention description vectors under the multiple attention channels; The attention sequence expansion model includes multiple sequence keyword expansion branches, and one sequence keyword expansion branch is used to represent one of the multiple attention channels; The method of using the attention sequence expansion model to generate the license plate information of the target vehicle image in the target scene according to the vehicle image spectrum description vector and the channel attention description vectors under the multiple attention channels includes: According to the vehicle image spectrum description vector and the channel attention description vector corresponding to each sequence keyword expansion branch, the recognition confidence corresponding to each sequence keyword expansion branch is determined respectively; the recognition confidence corresponding to any sequence keyword expansion branch refers to the confidence weight of the license plate recognition event corresponding to any sequence keyword expansion branch implemented on the target vehicle image in the target scene; Determining the license plate information of the target vehicle image in the target scene according to the recognition confidence corresponding to each sequence keyword expansion branch; The step of determining the license plate information of the target vehicle image in the target scene according to the recognition confidence corresponding to each sequence keyword expansion branch includes: Obtaining the noise factor of the license plate recognition event corresponding to each sequence keyword expansion branch; According to the noise factor of the license plate recognition event corresponding to each sequence keyword expansion branch, modal optimization is performed on the recognition confidence corresponding to each sequence keyword expansion branch to obtain the modal optimized recognition confidence corresponding to each sequence keyword expansion branch; The recognition confidence after modal optimization corresponding to each sequence keyword expansion branch is feature integrated to obtain the license plate information of the target vehicle image in the target scene.

2. The method according to claim 1, characterized in that Any one of the multiple attention channels is a target attention channel, and the feature mining indicator variable under the target attention channel belongs to a directional feature enhancement vector; The method of performing attention vector mining on the target multimodal image description according to the feature mining indicator variable under each attention channel to obtain a channel attention description vector of the target multimodal image description under each attention channel includes: Mining the target multimodal image description using the attention sequence expansion model to obtain an original channel attention sequence vector of the target multimodal image description; Performing splicing processing on the vehicle image spectrum description vector and the directional feature enhancement vector under the target attention channel to obtain an image splicing vector; Attention vector mining is performed on the original channel attention sequence vector according to the image splicing vector to generate a channel attention description vector of the target multimodal image description under the target attention channel.

3. The method according to claim 2, characterized in that The target multimodal image description includes multiple multimodal image descriptions of the historical vehicle image, and the original channel attention sequence vector includes a modal vector obtained by mining each multimodal image description in the multiple multimodal image descriptions; The performing attention vector mining on the original channel attention sequence vector according to the image splicing vector to generate a channel attention description vector of the target multimodal image description under the target attention channel includes: Performing a correlation analysis on the image splicing vector and the original channel attention sequence vector to obtain a correlation coefficient between each modal vector in the original channel attention sequence vector and the image splicing vector; According to the correlation coefficient between each modal vector and the image stitching vector, modal optimization is performed on each modal vector to obtain a plurality of modal vectors after modal optimization; The modal vectors after the optimization of the multiple modalities are feature integrated to obtain a channel attention description vector of the target multimodal image description under the target attention channel.

4. The method according to claim 2, characterized in that The step of performing splicing processing on the vehicle image spectrum description vector and the directional feature enhancement vector under the target attention channel to obtain an image splicing vector includes: The vehicle image spectrum description vector and the directional feature enhancement vector under the target attention channel are fused to obtain the image stitching vector; or, the vehicle image spectrum description vector and the directional feature enhancement vector under the target attention channel are feature integrated to obtain the image stitching vector; the size of the vehicle image spectrum description vector is the same as the size of the directional feature enhancement vector under the target attention channel.

5. The method according to claim 1, characterized in that There are multiple target multimodal image descriptions, one target multimodal image description has a channel attention description vector under one attention channel, and any one of the multiple sequence keyword extension branches is a target sequence keyword extension branch; and determining the recognition confidence corresponding to each sequence keyword extension branch according to the vehicle image spectrum description vector and the channel attention description vector corresponding to each sequence keyword extension branch, respectively, includes: Acquire scene noise detection data of the target scene, and mine the scene noise detection data using the attention sequence expansion model to obtain a scene noise modal vector of the target scene; Performing feature integration on the vehicle image spectrum description vector, the scene noise modal vector, and multiple channel attention description vectors of multiple target multimodal image descriptions under the attention channel of the target sequence keyword expansion branch to obtain an input vector of the target sequence keyword expansion branch; Generating a recognition confidence corresponding to the target sequence keyword expansion branch according to the input vector of the target sequence keyword expansion branch; The attention sequence expansion model includes a feature intersection branch, and the feature intersection branch is used to extract an image description vector in which the multiple sequence keyword expansion branches are crossed in the target multimodal image description, and the feature intersection branch also has a corresponding input vector; the recognition confidence corresponding to the target sequence keyword expansion branch is generated according to the input vector of the target sequence keyword expansion branch, including: using the target sequence keyword expansion branch to extract edge elements of the input vector of the target sequence keyword expansion branch to obtain a first edge description vector of the input vector of the target sequence keyword expansion branch; using the feature intersection branch to extract edge elements of the input vector of the feature intersection branch to obtain a second edge description vector of the input vector of the feature intersection branch; using the target sequence keyword expansion branch to generate the recognition confidence corresponding to the target sequence keyword expansion branch according to the first edge description vector and the second edge description vector; The step of using the target sequence keyword expansion branch to generate the recognition confidence corresponding to the target sequence keyword expansion branch according to the first edge description vector and the second edge description vector comprises: using the target sequence keyword expansion branch to perform full connection processing on the first edge description vector and the second edge description vector to obtain a fully connected vector; using the target sequence keyword expansion branch to generate the recognition confidence corresponding to the target sequence keyword expansion branch according to the fully connected vector; The target sequence keyword expansion branch includes X first residual convolution modules, and one first residual convolution module is used to generate a first edge description vector; the feature intersection branch includes Y second residual convolution modules, and one second residual convolution module is used to generate a second edge description vector, where X and Y are both positive integers; the first edge description vector and the second edge description vector are fully connected by using the target sequence keyword expansion branch to obtain a fully connected vector, including: using the target sequence keyword expansion branch to generate a fully connected parameter according to the input vector of the target sequence keyword expansion branch; the fully connected parameter includes a first residual convolution module for each of the first residual convolution modules; a second ... The invention relates to a method for obtaining a first edge description vector generated by each of the first residual convolution modules and a fully connected coefficient of each of the second residual convolution modules; performing modal optimization on the first edge description vector generated by each of the first residual convolution modules for the fully connected coefficient of each of the first residual convolution modules, so as to obtain X modal optimized first edge description vectors; performing modal optimization on the second edge description vector corresponding to each of the second residual convolution modules for the fully connected coefficient of each of the second residual convolution modules, so as to obtain Y modal optimized second edge description vectors; and performing feature integration on the X modal optimized first edge description vectors and the Y modal optimized second edge description vectors to obtain the fully connected vector.

6. The method according to claim 1, characterized in that There are Z target vehicle images, where Z is a positive integer, and the target scene has one license plate information for one target vehicle image; the method further includes: Arrange the Z target vehicle images according to the descending priority rule of the license plate information of each target vehicle image in the target scene to obtain the arranged Z target vehicle images; The first P target vehicle images among the sorted Z target vehicle images are labeled; P is a positive integer and P is less than Z.

7. An image processing system, characterized in that: The method comprises at least one processor and a memory; the memory stores computer-executable instructions; the at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the method according to any one of claims 1 to 6.

Images