Image processing method and device, electronic equipment and storage medium

By applying license plate recognition rules and direction gradient histogram features in license plate detection, the problem of low accuracy of license plate detection in the prior art is solved, and higher license plate recognition accuracy is achieved.

CN118942081APending Publication Date: 2024-11-12TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310524614.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-10
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Under the influence of light, angle and other factors, the existing license plate detection technology has low accuracy in detection and identification, making it difficult to effectively identify multiple license plates.

Method used

By determining the area and angle rules in the license plate recognition rules, the direction gradient histogram features of the target image are obtained and inputted into the license plate classifier to perform license plate classification to improve detection accuracy.

Benefits of technology

It improves the accuracy of license plate detection, reduces the influence of environmental factors and license plate color factors, and enhances the accuracy of license plate recognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118942081A_ABST
    Figure CN118942081A_ABST
Patent Text Reader

Abstract

The invention discloses an image processing method and device, electronic equipment and a storage medium. The embodiment of the invention relates to the technical fields of machine learning of artificial intelligence, cloud technology and the like. The method comprises the steps of determining whether a target region meeting license plate recognition rules exists in a to-be-recognized image or not, wherein the license plate recognition rules comprise an area rule set based on the license plate size and an angle rule determined based on the license plate setting angle; if the target area exists in the to-be-recognized image, obtaining an image located in the target area in the to-be-recognized image as a target image; obtaining a histogram of oriented gradient feature of the target image; and inputting the histogram of oriented gradients of the target image into a license plate classifier to obtain a license plate classification result. According to the invention, the license plate recognition rule and the license plate classifier are combined for license plate detection, so that the accuracy of the obtained license plate classification result is high.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of multimedia technology, and more specifically, to an image processing method, device, electronic device and storage medium. Background Art

[0002] As a key technology in intelligent transportation systems, vehicle recognition and detection technology is particularly important in vehicle detection, especially in the detection and recognition of multiple license plates.

[0003] Usually, license plates can be detected and recognized through edge detection and other algorithms, but due to the influence of other factors, such as light and angle, the accuracy of detection and recognition is low. Summary of the invention

[0004] In view of this, embodiments of the present application provide an image processing method, device, electronic device, and storage medium.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising: determining whether there is a target area in the image to be identified that meets the license plate recognition rules, the license plate recognition rules including an area rule based on the license plate size setting and an angle rule based on the license plate setting angle; if there is a target area in the image to be identified, obtaining an image located in the target area in the image to be identified as a target image; obtaining directional gradient histogram features of the target image; inputting the directional gradient histogram features of the target image into a license plate classifier to obtain a license plate classification result, the license plate classification result being that the target image is an image including the license plate or the target image is an image not including the license plate.

[0006] In a second aspect, an embodiment of the present application provides an image processing device, which includes: a determination module for determining whether there is a target area that meets the license plate recognition rules in the image to be identified, the license plate recognition rules including an area rule based on the size of the license plate and an angle rule based on the angle of the license plate; an image acquisition module for acquiring an image located in the target area in the image to be identified as a target image if there is a target area in the image to be identified; a feature acquisition module for acquiring directional gradient histogram features of the target image; and a classification module for inputting the directional gradient histogram features of the target image into a license plate classifier to obtain a license plate classification result, the license plate classification result being that the target image is an image including the license plate or the target image is an image not including the license plate.

[0007] Optionally, the area rule includes a first area rule and a second area rule, and the angle rule includes a first angle rule and a second angle rule; the determination module is further used to perform binarization processing on the image to be identified to obtain a first binary image, and the first binary image includes at least one connected domain; determine whether there is a first region in the first binary image, and the first region refers to a connected domain in the first binary image that satisfies the first area rule and the first angle rule; if the first region exists in the first binary image, determine the second region in the image to be identified according to the circumscribed rectangle of the first region; perform binarization processing on the second region to obtain a second binary image, and the second binary image includes at least one connected domain; determine whether there is a third region in the second binary image, and the third region refers to a connected domain in the second binary image that satisfies the second area rule and the second angle rule; if the third region exists in the second binary image, obtain the region in the image to be identified that corresponds to the third region as the target region; if the third region does not exist in the second binary image, determine that the target region does not exist in the image to be identified.

[0008] Optionally, the width of the circumscribed rectangle corresponding to each connected domain in the first binary image is parallel to the target direction; the determination module is also used to construct a circumscribed rectangle for each connected domain in the first binary image; for each connected domain in the first binary image, calculate the area ratio of the connected domain and the circumscribed rectangle corresponding to the connected domain as the area ratio corresponding to the connected domain; for each connected domain in the first binary image, if the area and area ratio of each connected domain meet the first area rule, obtain the connected domain as the first connected domain; construct the minimum circumscribed rectangle of each first connected domain; determine the angle between the minimum circumscribed rectangle of each first connected domain and the target direction; if there is a first connected domain whose angle meets the first angle rule, obtain the first connected domain whose angle meets the first angle rule as the first region; if there is no first connected domain whose angle meets the first angle rule, determine that the first region does not exist in the first binary image.

[0009] Optionally, the determination module is also used to binarize the second area to obtain a third binary image, the third binary image includes at least one connected domain; determine the endpoints of the first side and the endpoints of the second side corresponding to each connected domain in the third binary image, the first side and the second side are relative; construct a target rectangular frame according to the endpoints of the first side and the endpoints of the second side; adjust the pixel values ​​of the pixel points outside the target rectangular frame in the third binary image to the target value, and obtain the second binary image.

[0010] Optionally, the width of the circumscribed rectangle corresponding to each connected domain in the second binary image is parallel to the target direction; the determination module is also used to construct a circumscribed rectangle for each connected domain in the second binary image; for each connected domain in the second binary image, calculate the area ratio of the connected domain and the circumscribed rectangle corresponding to the connected domain as the area ratio of the connected domain; for each connected domain in the second binary image, if the area and area ratio of each connected domain meet the second area rule, obtain the connected domain as the second connected domain; construct the minimum circumscribed rectangle of each second connected domain; determine the angle between the minimum circumscribed rectangle of each second connected domain and the target direction; if there is a second connected domain whose angle meets the second angle rule, obtain the second connected domain whose angle meets the second angle rule as the third area; if there is no second connected domain whose angle meets the second angle rule, determine that there is no third area in the second binary image.

[0011] Optionally, the determination module is further used to determine a circumscribed rectangle of the first area; enlarge the circumscribed rectangle of the first area to obtain an enlarged rectangle; and determine the second area in the image to be identified according to the enlarged rectangle.

[0012] Optionally, the feature acquisition module is also used to divide the target image to obtain multiple sub-images; divide each sub-image to obtain its own image block; divide each image block to obtain its own image unit; determine the gradient and gradient direction of each pixel in each image unit according to the pixel value of each pixel in each image unit; determine the directional gradient histogram descriptor of each image unit according to multiple preset intervals and the gradient and gradient direction of each pixel in each image unit; obtain and summarize the directional gradient histogram descriptors of each image unit in each image block to obtain the descriptor of each image block; obtain and summarize the descriptors of each image block in each sub-image to obtain the descriptor of each sub-image; obtain and summarize the descriptors of multiple sub-images to obtain the directional gradient histogram features of the target image.

[0013] Optionally, the device also includes a training module, which is used to obtain training samples, the training samples include sample images and sample classification results corresponding to the sample images, the sample images include positive sample images and negative sample images, the positive sample image is an image including a license plate, the negative sample image is an image not including a license plate, and the sample classification result is that the sample image does not include a license plate or the sample image includes a license plate; determine the directional gradient histogram feature of the positive sample image as the first sample feature; determine the directional gradient histogram feature of the negative sample image as the second sample feature; train the initial classifier according to the first sample feature and the second sample feature, the sample classification result of the positive sample image, and the sample classification result of the negative sample image to obtain a license plate classifier.

[0014] Optionally, the training module is also used to obtain multiple initial sample images; if the initial sample image contains an area that satisfies the license plate recognition rule, determine that the initial sample image is an intermediate sample image; determine a sample classification result of the intermediate sample image, the sample classification result of the intermediate sample image is that the intermediate sample image includes the license plate or does not include the license plate; if the sample classification result of the intermediate sample image is that the intermediate sample image includes the license plate, determine that the intermediate sample image is a positive sample image; if the sample classification result of the intermediate sample image is that the intermediate sample image does not include the license plate, determine that the intermediate sample image is a negative sample image.

[0015] Optionally, the determination module is also used to determine a target video frame from a video stream to be identified; obtain an image in a selection box from the target video frame as an image to be identified, the selection box is determined based on at least two marking points or according to a preset rule, and the at least two marking points are determined based on a marking operation on the target video frame.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory; one or more programs are stored in the memory and configured to be executed by the processor to implement the above method.

[0017] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored, wherein the above method is executed when the program code is executed by a processor.

[0018] In a fifth aspect, an embodiment of the present application provides a computer program product or a computer program, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs the above method.

[0019] An image processing method, device, electronic device and storage medium are provided in the embodiments of the present application. In the present application, a target image corresponding to a target area is first determined by a license plate recognition rule, and then the directional gradient histogram features of the target image are classified by a license plate classifier to obtain a license plate classification result. The license plate recognition rule and the license plate classifier are combined to perform license plate detection, so that the accuracy of the obtained license plate classification result is higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 A schematic diagram showing an operating environment applicable to an embodiment of the present application;

[0022] Figure 2 A flowchart of an image processing method proposed in one embodiment of the present application is shown;

[0023] Figure 3 A schematic diagram showing a marking processing result in an embodiment of the present application is shown;

[0024] Figure 4 A flowchart of an image processing method proposed in another embodiment of the present application is shown;

[0025] Figure 5 A schematic diagram showing a process of determining a first binary image in an embodiment of the present application is shown;

[0026] Figure 6 A schematic diagram of a first binary image in an embodiment of the present application is shown;

[0027] Figure 7 A schematic diagram showing a process of determining a third area in an embodiment of the present application is shown;

[0028] Figure 8 A flowchart of an image processing method proposed in another embodiment of the present application is shown;

[0029] Fig. 9 A schematic diagram of an image to be recognized in an embodiment of the present application is shown;

[0030] Fig.10 A schematic diagram showing another marking processing result of an embodiment of the present application;

[0031] Fig.11 A block diagram of an image processing device proposed in one embodiment of the present application is shown;

[0032] Fig.12 A structural block diagram of an electronic device for executing an image processing method according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. According to the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0034] In the following description, the terms "first\second" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0036] It should be noted that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0037] The present application discloses an image processing method, device, electronic device and storage medium, which involve machine learning of artificial intelligence and cloud technology, etc.

[0038] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0039] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0040] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0041] Cloud technology refers to a hosting technology that unifies hardware, software, network and other resources within a wide area network or local area network to achieve data computing, storage, processing and sharing.

[0042] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool, which can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The backend services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the rapid development and application of the Internet industry, in the future, each item may have its own identification mark, which needs to be transmitted to the backend system for logical processing. Data of different levels will be processed separately. All kinds of industry data need strong system backing support, which can only be achieved through cloud computing.

[0043] Cloud storage is a new concept that extends and develops from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of different types of storage devices (storage devices are also called storage nodes) in the network through application software or application interfaces to work together and provide external data storage and business access functions.

[0044] At present, the storage method of the storage system is: create a logical volume, and when creating a logical volume, allocate physical storage space for each logical volume. The physical storage space may be composed of disks of a storage device or several storage devices. The client stores data on a logical volume, that is, stores the data on the file system. The file system divides the data into many parts, each of which is an object. The object contains not only data but also additional information such as data identification (ID, ID entity). The file system writes each object into the physical storage space of the logical volume, and the file system records the storage location information of each object, so that when the client requests to access the data, the file system can allow the client to access the data according to the storage location information of each object.

[0045] The process of the storage system allocating physical storage space to a logical volume is as follows: based on the estimated capacity of the objects stored in the logical volume (this estimate often has a large margin relative to the actual capacity of the objects to be stored) and the grouping of independent redundant disk arrays (RAID, Redundant Array of Independent Disks), the physical storage space is pre-divided into stripes. A logical volume can be understood as a stripe, thereby allocating physical storage space to the logical volume.

[0046] With the development and improvement of smart cities, intelligent license plate detection and recognition in various traffic scenarios are gradually becoming popular. For example, the entrance and exit of the parking lot adopts license plate recognition technology, and the motor vehicles entering can automatically calculate the usage time and charge fees. At the same time, it can display the parking space occupancy in real time, which is convenient for the owner to dispatch, saving a lot of manpower and material costs, and greatly improving the utilization rate and management efficiency of the parking lot. For another example, the smart community can automatically identify the license plate number, dispatch the free parking position of internal personnel's vehicles and the parking fee of external personnel's vehicles. At the same time, license plate recognition related technologies are gradually applied to a series of occasions such as supervising road traffic violations, recovering stolen vehicles, searching for wanted vehicles, and dispatching special vehicles. The accuracy of the license plate recognition algorithm depends to a large extent on the license plate positioning algorithm. In other words, improving the license plate detection and positioning algorithm will help the accuracy of the license plate recognition algorithm.

[0047] The license plate can be located by the following license plate detection and positioning algorithm:

[0048] 1. License plate positioning method based on edge detection. The so-called "edge" refers to the set of pixels whose surrounding pixel grayscale has a step change. The two sides of the "edge" belong to two regions. The grayscale of each region is uniform, and there are certain differences in the characteristics of the grayscale of the two regions. The task of edge detection is to accurately locate the edge and suppress noise. There are many detection methods, such as Roberts edge operator, Prewitt operator, Sobel operator and Laplace edge detection. However, although the edge detection algorithm can accurately determine the edge, it is difficult to accurately determine whether the edge is the edge of the license plate, resulting in a low license plate detection accuracy.

[0049] 2. License plate location method based on color segmentation. The license plate location method based on color segmentation consists of color segmentation and target location modules. It uses a multi-layer perceptron network to segment the color image, and then uses the projection method to segment the potential license plate area. However, the color of the license plate is not single. For example, the license plate is a yellow and green license plate. As a result, the license plate location method based on color segmentation is difficult to accurately locate license plates with various colors, resulting in a low license plate detection accuracy.

[0050] 3. License plate location method based on wavelet transform. Wavelet analysis is an important analysis tool applied to image processing, with the characteristics of a "microscope". The multi-resolution characteristics of wavelet analysis make the high-frequency wavelet coefficients of wavelet decomposition coefficients in different directions have different characteristics. Therefore, the use of directional wavelets can reflect the changes of images in any direction at different resolutions. The multi-scale decomposition characteristics of wavelet analysis are more in line with human visual mechanisms. The basic idea is to decompose the original signal into a series of sub-band signals with different spatial resolutions, different frequency characteristics and directional characteristics through operations such as scaling and translation. These sub-band signals have good time-frequency characteristics. By utilizing these characteristics, local analysis of the signal in the time domain and frequency domain can be achieved. However, when the image to be identified includes a large amount of background information, it is difficult to filter out the interference information in the background information, resulting in a low accuracy rate of license plate detection.

[0051] 4. License plate positioning method based on genetic algorithm. The license plate positioning method based on genetic algorithm uses genetic algorithm to optimize the search of the image, combines the regional feature vector to construct the fitness function, and finally finds the optimal positioning parameters of the license plate area. License plate positioning is the process of finding an optimal area that meets the "license plate area characteristics". In essence, it is a problem of finding the optimal positioning parameters from the parameter space, and finding the optimal solution in the parameter space is what genetic algorithms are good at. However, genetic algorithms are only for license plates of specific colors. When the license plates have various colors, it is still difficult to accurately locate license plates of various colors, resulting in low license plate detection accuracy.

[0052] 5. License plate location method based on mathematical morphology. This method is to perform a binary operation on the original image, and then perform morphological operations such as corrosion denoising and dilation to mark the connected domain and filter out the non-license plate area, so as to locate the license plate. However, it is difficult to filter out the interference information in the background information using this method, resulting in a low license plate detection accuracy.

[0053] 6. License plate location method based on grayscale image texture feature analysis. Most traditional texture feature analysis and location algorithms are based on grayscale images for analysis. Therefore, the algorithm needs to preprocess the image, convert the color image into a grayscale image, and then perform row scanning to find the license plate line segments contained in each row of the image, and record their starting coordinates and lengths. If there are several consecutive rows with at least one license plate line segment, and the number of rows is greater than a certain threshold, it is considered that a candidate area for the license plate is found in the row direction, and the starting row and height of the candidate area are determined; perform column scanning in the area where the license plate may exist to determine the starting row and height of the candidate area for the license plate, as well as the starting column coordinates and length, thereby determining a license plate area; continue to search in other areas where the license plate may exist until all the candidate areas for the license plate are found. However, this method is sensitive to environmental factors such as lighting and weather when acquiring the image. In the case of insufficient light or bad weather, the image may not accurately represent the information of the license plate, resulting in a decrease in the accuracy of license plate detection.

[0054] In order to improve the accuracy of license plate detection, the inventor has proposed an image processing method, device, electronic device and storage medium according to the embodiment of the present application after long-term research. First, the target image corresponding to the target area is determined by the license plate recognition rule, and the background information is effectively removed, thereby avoiding the influence of interference information in the background information on the license plate detection and improving the detection accuracy. Furthermore, the directional gradient histogram features of the target image are classified by the license plate classifier to obtain the license plate classification result. The license plate recognition rule and the license plate classifier are combined to perform license plate detection, so that the accuracy of the obtained license plate classification result is higher. At the same time, when the license plate classifier classifies the directional gradient histogram features of the target image, it is not classified according to the color features of the license plate, which can effectively reduce the influence of environmental factors and license plate color factors and improve the accuracy of license plate recognition.

[0055] like Figure 1As shown, the operating environment applicable to the embodiment of the present application includes a terminal 20 and a server 10, and the terminal 20 and the server 10 are connected through a wired network or a wireless network. The terminal 20 can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart home appliance, a vehicle terminal, an aircraft, a wearable device terminal, a virtual reality device, and other terminal devices that can display pages, or run other applications that can call page display applications (such as instant messaging applications, shopping applications, search applications, game applications, forum applications, map traffic applications, etc.).

[0056] The server 10 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The server 10 may be used to provide services for applications running on the terminal 20.

[0057] Among them, the terminal 20 can send the image to be identified to the server 10, the server 10 can process the image to be identified to obtain a target image, then determine the directional gradient histogram features of the target image, and input the directional gradient histogram features of the target image into the license plate classifier to obtain a license plate classification result. After that, the server 10 feeds back the obtained license plate classification result target information to the terminal 20.

[0058] In some possible implementations, the server 10 may train an initial classifier based on positive sample images including license plates and negative sample images not including license plates to obtain a license plate classifier, and deploy the license plate classifier locally on the server 10 .

[0059] In another embodiment, the terminal 20 may process the image to be identified to obtain a target image, determine the directional gradient histogram features of the target image, and input the directional gradient histogram features of the target image into a license plate classifier to obtain a license plate classification result.

[0060] Optionally, the server 10 may store the acquired license plate classifier in a cloud storage system, and when executing the image processing method of the present application, the terminal 20 acquires the license plate classifier from the cloud storage system.

[0061] For the convenience of description, in the following embodiments, the image processing is performed by an electronic device as an example.

[0062] See also Figure 2 , Figure 2A flowchart of an image processing method proposed in one embodiment of the present application is shown. The method can be applied to an electronic device, which can be Figure 1 The terminal 20 or the server 10 in the embodiment of the present invention comprises:

[0063] S110, determining whether there is a target area that meets the license plate recognition rules in the image to be recognized, where the license plate recognition rules include an area rule set based on the size of the license plate and an angle rule determined based on the angle at which the license plate is set.

[0064] The image to be identified may refer to an image to be subjected to license plate detection, and the image to be identified may be a color image. The image to be identified may include at least one vehicle, and the vehicle in the image to be identified may be equipped with a license plate, or may not be equipped with a license plate. The image to be identified may also include other elements other than the vehicle, for example, the image to be identified includes pedestrians and road signs.

[0065] As an implementation manner, a video stream to be identified may be obtained, a video frame including a vehicle may be determined in the video stream to be identified, and then the video frame including the vehicle may be used as an image to be identified.

[0066] As another implementation, a target video frame may be determined from a video stream to be identified; and an image in a selection box may be obtained from the target video frame as the image to be identified. The selection box is determined based on at least two marking points or according to a preset rule, and the at least two marking points are determined based on a marking operation on the target video frame.

[0067] The target video frame may refer to each frame in the video stream to be identified, or may refer to a video frame including a vehicle in the video stream to be identified, or may refer to a video frame where the video to be identified stays when the video to be identified is paused during viewing.

[0068] In one embodiment, after determining the target video frame, a marking operation for the target video frame may be received, at least two marking points may be determined according to the marking operation, a rectangular frame may be determined according to the at least two marking points, and the image within the rectangular frame may be used as the image to be identified. For example, the point of action of each click operation in the target video frame may be used as a marking point, and the two marking points with the farthest distance from the at least two marking points may be used as the diagonal points of the rectangle, and a rectangle may be constructed with the diagonal points of the rectangle as a selection frame. For another example, the two marking points with the closest distance from the at least two marking points may be used as the diagonal points of the rectangle, and a rectangle may be constructed with the diagonal points of the rectangle as a selection frame.

[0069] In another embodiment, the selection box can also be determined according to preset rules. The preset rules can be determined according to the characteristics of the video in the video to be identified, and then the selection box can be determined according to the pre-set rules. For example, if the video stream to be identified is a city road video stream, the preset rule can be to select a rectangular box around the partial area where the motor vehicle road is located as the selection box. For another example, if the video stream to be identified is a country road video stream, the preset rule can be to select a rectangular box around the partial area where the entire country road is located as the selection box.

[0070] In another embodiment, the preset rule may be to select a rectangular box of a fixed area in the target video frame as the selection box. For example, the selection box may be a rectangular box of a 600×600 partial image in the center area of ​​the target video frame, or a rectangular box of a 600×600 partial image at the upper left side of the target video frame.

[0071] In this embodiment, the image to be recognized can be divided into multiple regions (which can be divided according to color continuity and fixed size, etc.), and the license plate recognition rule is used to select the region matching the license plate in the image to be recognized. The license plate rule may include an area rule and an angle rule. The image to be recognized can be divided into multiple regions (which can be divided according to color continuity and fixed size, etc.), the area rule is used to filter out the region whose area meets the area rule in the image to be recognized, and the angle rule is used to filter out the region whose angle meets the angle rule in the image to be recognized. The region that meets both the angle rule and the area rule is used as the target region.

[0072] Generally speaking, the size of the license plate installed on the vehicle is usually a fixed value, for example, the size of the license plate installed on the front of the vehicle is 440mm×140mm, and the size of the license plate installed on the rear of the vehicle is 440×220; the area rule can be determined according to the size of the license plate. At the same time, the setting angle of the license plate installed on the vehicle is also fixed. The setting angle of the license plate refers to the angle between the width of the license plate and the target direction. The angle rule can be determined according to the setting angle of the license plate. The target direction can be the horizontal direction or the vertical direction. Usually, the setting angle of the license plate is 0 degrees, that is, the width of the license plate is parallel to the target direction. For example, the angle between the width of the license plate and the horizontal plane is 0 degrees (parallel to the horizontal direction), and the angle rule can be determined according to the setting angle of the license plate being 0 degrees.

[0073] As an implementation method, the area rule includes that the area of ​​the region is within a first area interval, and the angle rule is that the angle between the width of the smallest circumscribed rectangle of the region and the target direction is less than the first angle. The image to be identified can be divided into multiple segmented regions according to the color of each region in the image to be identified and the continuity of the region, and the segmented regions that meet both the area rule and the angle rule are selected as the target region. Among them, the first area interval and the first angle can be set based on demand. For example, the first area interval can be (96400, 100000), and the first angle is 20 degrees.

[0074] As another implementation, the area rule includes that the area ratio of the connected domain to the circumscribed rectangle of the connected domain (the width of the circumscribed rectangle is parallel to the horizontal direction) is greater than the first area ratio, and the area of ​​the connected domain is within the second area interval (to filter out connected domains with too large or too small areas); the angle rule is that the angle between the width of the minimum circumscribed rectangle of the connected domain and the horizontal direction is less than the second angle. For example, the first area ratio can be 0.6, the second area interval can be (96500, 100100), and the second angle is 22 degrees.

[0075] When the image to be identified is obtained, the image to be identified can be binarized to obtain a binary image, which includes at least one connected domain, and a connected domain that satisfies both the area rule and the angle rule is screened from the binary image as a selected connected domain, and the area corresponding to the selected connected domain in the image to be identified is used as the target area. The image to be identified is the same size as the binary image obtained, and the area in the image to be identified corresponding to the selected connected domain refers to an area in which the pixel coordinates in the image to be identified are the same as the pixel coordinates of the selected connected domain (the pixel coordinates of the selected connected domain refer to the pixel coordinates of the selected connected domain in the binary image corresponding to the image to be identified).

[0076] S120: If there is a target area in the image to be identified, obtain an image located in the target area in the image to be identified as a target image; and obtain a directional gradient histogram feature of the target image.

[0077] If the target area exists, the image located in the target area in the image to be identified is used as the target image. If the target area does not exist, it is determined that the image to be identified does not include the license plate, and it is directly determined that the image to be identified does not include the license plate.

[0078] As an implementation manner, the entire image in the target area of ​​the image to be identified may be used as the target image, or a portion of the image in the target area (eg, 80% of the image in the target area) may be used as the target image.

[0079] The directional gradient histogram feature refers to the HOG feature (Histogram of Oriented Gradient). The HOG feature is a feature descriptor used for object detection in computer vision and image processing. It constructs features by calculating and counting the gradient direction histogram of the local area of ​​the image. The local feature here refers to dividing the image into multiple sub-blocks (Blocks, which are also image blocks below in this embodiment), and the features in each Block are combined to form the final feature.

[0080] As an implementation mode, a method for acquiring directional gradient histogram features of a target image may include: dividing the target image to obtain a plurality of sub-images; dividing each sub-image to obtain an image block of each sub-image; dividing each image block to obtain an image unit of each image block; determining the gradient and gradient direction of each pixel in each image unit according to the pixel value of each pixel in each image unit; determining the directional gradient histogram descriptor of each image unit according to a plurality of preset intervals and the gradient and gradient direction of each pixel in each image unit; acquiring and summarizing the directional gradient histogram descriptors of each image unit in each image block to obtain a descriptor of each image block; acquiring and summarizing the descriptors of each image block in each sub-image to obtain a descriptor of each sub-image; acquiring and summarizing the descriptors of multiple sub-images to obtain the directional gradient histogram features of the target image.

[0081] In this embodiment, a detection window of size 128*32 can be set, and the target image is divided into multiple sub-images through the detection window; for each sub-image, it is divided into blocks of size 4*4 with a step size of 2*2, and each block is an image block; for each image block, it is divided into units of size 2*2 as image units (one image block corresponds to 4 image units). In this embodiment, the size of the detection window, the size of the block, the size of the unit, and the size of the step size can be determined according to the actual scene. This application does not limit the size of the detection window, the size of the block, the size of the unit, and the size of the step size. The above size is only one example.

[0082] In this embodiment, the preset interval can be set based on demand. For example, the preset interval is 9 preset intervals obtained by dividing the gradient histogram into 9 intervals (bins). For example, the preset interval is 12 preset intervals obtained by dividing the gradient histogram into 12 intervals (bins).

[0083] For each image unit, determine the gradient and gradient direction of each pixel in the image unit, determine the preset interval to which the pixel value of the pixel belongs based on the gradient and gradient direction of each pixel, traverse each pixel in the image unit, obtain multiple pixels corresponding to each preset interval, and construct the gradient histogram of the image unit based on the number of multiple pixels corresponding to each preset interval. Then, obtain the number of pixels in each preset interval in the gradient histogram of the image unit as the directional gradient histogram descriptor of the image unit. For example, when the preset interval includes 9 preset intervals, the directional gradient histogram descriptor of the image unit includes 9 values.

[0084] The directional gradient histogram descriptors of each image unit in each image block can be summarized to obtain the directional gradient histogram descriptor of each image block. For example, the image block includes 4 image units, the directional gradient histogram descriptor of each image unit includes 9 values, and the directional gradient histogram descriptor of the image block includes 36 values.

[0085] For each sub-image, the directional gradient histogram descriptors of each image block can be summarized to obtain the directional gradient histogram descriptor of the sub-image. For example, the sub-image is divided into 63×15 image blocks according to the step size of 2*2 and the block size of 4*4. The directional gradient histogram descriptor of each image block includes 36 values, so the directional gradient histogram descriptor of the sub-image is 36×63×15 values.

[0086] The directional gradient histogram descriptors of the sub-images are obtained, and the directional gradient histogram descriptors of the sub-images are directly summarized to obtain the summarized results as the directional gradient histogram features of the target image.

[0087] S130, inputting the directional gradient histogram feature of the target image into a license plate classifier to obtain a license plate classification result, where the license plate classification result is that the target image is an image including the license plate or the target image is an image not including the license plate.

[0088] The license plate classifier can be obtained by training the initial classifier through positive sample images including the license plate and negative sample images not including the license plate, wherein the initial classifier can be SVM (support vector machines). SVM is a binary classification model, and its basic model is a linear classifier with the largest interval defined in the feature space, which is different from the perceptron; SVM also includes kernel techniques, which makes it a nonlinear classifier in essence. The learning strategy of SVM is to maximize the interval, which can be formalized as a problem of solving convex quadratic programming, which is also equivalent to the problem of minimizing the regularized hinge loss function. The learning algorithm of SVM is the optimization algorithm for solving convex quadratic programming.

[0089] As an implementation method, the directional gradient histogram features of the positive sample image and the directional gradient histogram features of the negative sample image can be determined according to the method of S120 above, and the initial classifier can be trained based on the directional gradient histogram features of the positive sample image, the directional gradient histogram features of the negative sample image, the classification results of the positive sample image (including the license plate) and the classification results of the negative sample image (excluding the license plate) to obtain a license plate classifier.

[0090] After obtaining the license plate classifier, if the directional gradient histogram of the target image is obtained, the directional gradient histogram of the target image is directly input into the license plate classifier to obtain the license plate classification result. After obtaining the license plate classification result, if the license plate classification result is that the target image includes the license plate, the image to be identified can be further marked or information extracted. If the license plate classification result is that the target image does not include the license plate, it is determined that the image to be identified does not include the license plate, and a prompt message can be output to prompt that the image to be identified does not include the license plate.

[0091] As an implementation, marking the image to be identified may include: adding a rectangular frame to select the target area in the image to be identified according to the target area where the target image is located (since the license plate classification result is that the target image includes the license plate, and the target image is an image within the target area, the rectangular frame also selects the license plate), so as to obtain a marking result. For example, Figure 3 a in FIG. 1 shows a schematic diagram of an image to be recognized. Figure 3 b shows Figure 3 The mark processing result corresponding to a in Figure 3 The b in FIG. 3 includes a rectangular frame 301 for selecting a license plate.

[0092] As an implementation method, the information extraction process may include: extracting the license plate information in the target image to obtain an information extraction result, for example, the image to be identified is as follows: Figure 3 When shown as a in , the information extraction result is: A9V6JI.

[0093] In a possible implementation, if the image to be identified is determined from the video stream to be identified, each image to be identified in the video stream to be identified can be marked and / or information extracted, and the processing results of each image to be identified can be summarized. For example, the information extraction results of each image to be identified are summarized to obtain all the license plates included in each image to be identified. For another example, the marking processing results of each image to be identified are summarized to obtain multiple images marked with rectangular frames.

[0094] In this embodiment, the target image corresponding to the target area is first determined by the license plate recognition rule, and then the directional gradient histogram features of the target image are classified by the license plate classifier to obtain the license plate classification result. The license plate recognition rule and the license plate classifier are combined to perform license plate detection, so that the accuracy of the obtained license plate classification result is higher, thereby making the accuracy of the license plate detection result of the image to be identified determined according to the license plate detection result of the target image higher.

[0095] See also Figure 4 , Figure 4 A flowchart of an image processing method proposed in another embodiment of the present application is shown. The method can be applied to an electronic device, which can be Figure 1 The terminal 20 or the server 10 in the embodiment of the present invention comprises:

[0096] S210 , binarizing the image to be recognized to obtain a first binary image, where the first binary image includes at least one connected domain.

[0097] As an implementation, a first pixel value threshold may be set, and the pixel values ​​of the pixels in the image to be identified whose pixel values ​​exceed the first pixel value threshold are adjusted to a first value (e.g., 0), and the pixel values ​​of the pixels in the image to be identified whose pixel values ​​do not exceed the first pixel value threshold are adjusted to a second value (e.g., 1), so as to obtain a first binary image. The first pixel value threshold may be a value set based on demand, and this application does not limit it. For example, the first pixel value threshold is 120.

[0098] As another possible implementation, Figure 5 As shown, the image to be identified is processed by wavelet transform to obtain a grayscale image; the grayscale image is processed by vertical edge detection to obtain a vertical edge image; the grayscale image is processed by horizontal edge detection to obtain a horizontal edge image; the vertical edge image and the horizontal edge image are superimposed to obtain a superimposed image; the superimposed image is sequentially subjected to Gaussian filtering (which may be 8-field Gaussian filtering) and image enhancement processing to obtain a first preprocessed image; the horizontal edge image is sequentially subjected to image enhancement processing and Gaussian filtering processing (which may include sequentially performing Gaussian filtering twice) to obtain a second preprocessed image; the difference between the first preprocessed image and the second preprocessed image is calculated as a target sub-image; the target sub-image is sequentially subjected to morphological dilation processing, median filtering processing (which may include sequentially performing median filtering processing twice), image enhancement processing and binarization processing to obtain a first binary image.

[0099] like Figure 6 As shown, Figure 6 a in it is the image to be recognized. Figure 6 The b in Figure 6 The first binary image corresponding to a in .

[0100] In some possible implementations, Opencv can be used according to Figure 5 The steps shown are used to determine the first binary image. Opencv is a cross-platform computer vision and machine learning software library that can run on Linux, Windows, Android and MacOS operating systems. It is lightweight and efficient - it consists of a series of C functions and a small number of C++ classes, and provides interfaces for languages ​​such as Python, Ruby, and MATLAB, and implements many common algorithms in image processing and computer vision.

[0101] S220, determining whether a first region exists in the first binary image, where the first region refers to a connected domain in the first binary image that satisfies a first area rule and a first angle rule.

[0102] In this embodiment, the area rule includes a first area rule and a first angle rule, and a connected domain satisfying the first area rule and the first angle rule can be determined in the first binary image as the first region. The first area rule is used to filter out connected domains whose areas do not match the size of the license plate, and the angle rule is used to filter out connected domains whose directions do not match the setting angle of the license plate.

[0103] As an implementation method, a circumscribed rectangle can be constructed for each connected domain in the first binary image; for each connected domain in the first binary image, the area ratio of the connected domain and the circumscribed rectangle corresponding to the connected domain is calculated as the area ratio corresponding to the connected domain; for each connected domain in the first binary image, if the area and area ratio of each connected domain satisfy the first area rule, the connected domain is obtained as the first connected domain; the minimum circumscribed rectangle of each first connected domain is constructed; the angle between the minimum circumscribed rectangle of each first connected domain and the target direction is determined; if there is a first connected domain whose angle satisfies the first angle rule, the first connected domain whose angle satisfies the first angle rule is obtained as the first region; if there is no first connected domain whose angle satisfies the first angle rule, it is determined that the first region does not exist in the first binary image.

[0104] Among them, the area and area ratio satisfying the first area rule may include: the area of ​​the connected domain is within the third area interval, and the area ratio is greater than the second area ratio, the third area interval and the second area ratio can be values ​​set based on demand, and this application does not limit them; when the setting angle of the license plate is 0 degrees with the target direction, the first angle rule can be that the angle is less than the third angle, and the third angle can be 30 degrees.

[0105] S230: If the first region exists in the first binary image, determine a second region in the image to be identified according to a circumscribed rectangle of the first region.

[0106] If the first region exists in the first binary image, the second region can be directly determined in the image to be recognized based on the circumscribed rectangle of the first region.

[0107] As an implementation method, a rectangle that matches the circumscribed rectangle of the first area can be directly determined in the image to be identified as a matching rectangle (the position of the matching rectangle in the image to be identified and the position of the circumscribed rectangle of the first area in the first binary image are the same, and the matching rectangle has the same size as the circumscribed rectangle of the first area), and the area within the matching rectangle in the image to be identified is used as the second area.

[0108] As another implementation, the bounding rectangle of the first region may be obtained, and the bounding rectangle of the first region may be enlarged to obtain an enlarged rectangle; and the second region may be determined in the image to be identified based on the enlarged rectangle. The enlarged rectangle of the first region may be enlarged with the centroid of the bounding rectangle of the first region as the center (the enlargement factor may be, for example, 2 times or 2.5 times), and then a matching rectangle matching the enlarged rectangle may be determined in the image to be identified, and the area within the matching rectangle in the image to be identified is used as the second region.

[0109] S240: Binarize the second region to obtain a second binary image, where the second binary image includes at least one connected domain.

[0110] As an implementation, a second pixel value threshold may be set, and the pixel values ​​of the pixels in the second region whose pixel values ​​exceed the first pixel value threshold are adjusted to the first value (e.g., 0), and the pixel values ​​of the pixels in the second region whose pixel values ​​do not exceed the second pixel value threshold are adjusted to the second value (e.g., 1), so as to obtain a second binary image. The second pixel value threshold may be a value set based on demand, which is not limited in this application. For example, the first pixel value threshold is 120.

[0111] As another implementation, a Sobel operation may be performed on the second region to obtain a second binary image, wherein the Sobel operation may include: sequentially performing Gaussian filtering, color image to grayscale image conversion, Sobel edge detection, binarization, and morphological closing operations.

[0112] As an implementation method, the second area is binarized to obtain a third binary image, which includes at least one connected domain; according to the endpoints of the first side and the endpoints of the second side corresponding to each connected domain in the third binary image, the first side is relative to the second side; a target rectangular frame is constructed according to the endpoints of the first side and the endpoints of the second side; the pixel values ​​of the pixel points outside the target rectangular frame in the third binary image are adjusted to the target value to obtain a second binary image.

[0113] When the angle between the setting angle of the license plate and the horizontal direction is 0 degrees, the license plate is set horizontally, the first side refers to the left side, and the second side refers to the right side. At this time, the endpoint of the first side corresponding to each connected domain in the third binary image refers to the endpoint close to the leftmost side of all connected domains in the third binary image, and the endpoint of the second side corresponding to each connected domain in the third binary image refers to the endpoint close to the rightmost side of all connected domains in the third binary image. Similarly, when the angle between the setting angle of the license plate and the vertical direction is 0 degrees, the license plate is set vertically, the first side refers to the upper side, and the second side refers to the lower side. At this time, the endpoint of the first side corresponding to each connected domain in the third binary image refers to the endpoint close to the uppermost side of all connected domains in the third binary image, and the endpoint of the second side corresponding to each connected domain in the third binary image refers to the endpoint close to the lowermost side of all connected domains in the third binary image.

[0114] The width of the target rectangle is parallel to the target direction. For example, if the target direction is horizontal, the width of the target rectangle is also horizontal. The target value may refer to the pixel value of the part that is not a connected domain in the third binary image. For example, if the value of the connected domain in the third binary image is 1 and the value of the part that is not a connected domain is 0, then the target value is 0.

[0115] S250: Determine whether there is a third region in the second binary image, where the third region refers to a connected domain in the second binary image that satisfies a second area rule and a second angle rule.

[0116] In this embodiment, the area rule further includes a second area rule, and the angle rule further includes a second angle rule. A connected domain that satisfies the second area rule and the second angle rule can be determined in the second binary image as the third region. The second area rule is used to filter out connected domains whose areas do not match the size of the license plate, and the angle rule is used to filter out connected domains whose directions do not match the setting angle of the license plate.

[0117] As an implementation method, a circumscribed rectangle can be constructed for each connected domain in the second binary image; for each connected domain in the second binary image, the area ratio of the connected domain and the circumscribed rectangle corresponding to the connected domain is calculated as the area ratio of the connected domain; for each connected domain in the second binary image, if the area and area ratio of each connected domain satisfy the second area rule, the connected domain is obtained as the second connected domain; the minimum circumscribed rectangle of each second connected domain is constructed; the angle between the minimum circumscribed rectangle of each second connected domain and the target direction is determined; if there is a second connected domain whose angle satisfies the second angle rule, the second connected domain whose angle satisfies the second angle rule is obtained as the third area; if there is no second connected domain whose angle satisfies the second angle rule, it is determined that there is no third area in the second binary image.

[0118] Among them, the area and area ratio satisfying the second area rule may include: the area of ​​the connected domain is within the fourth area interval, and the area ratio is greater than the third area ratio, the fourth area interval and the third area ratio may be values ​​set based on demand, and this application does not limit them; the second angle rule may be that the included angle is less than the fourth angle, and the fourth angle may be 30 degrees.

[0119] In this embodiment, steps S240-S250 can be simplified as follows: Figure 7 ,like Figure 7 As shown, the first area is enlarged to obtain the second area, and then the Sobel operation is performed on the second area to obtain a third binary image. The endpoints of the first side and the second side of the circumscribed rectangle are determined in the third binary image to construct a target rectangular frame, and the pixel values ​​of the pixels in the non-target rectangular frame are set to the target values ​​to obtain the second binary image, and then the third area is screened according to the second area rule and the second angle rule.

[0120] S260: If the third region exists in the second binary image, obtain a region in the image to be identified that corresponds to the third region as a target region; if the third region does not exist in the second binary image, determine that the target region does not exist in the image to be identified.

[0121] If the third region does not exist in the second binary image, it is determined that the target region does not exist in the image to be identified.

[0122] If there is a third area in the second binary image, the bounding rectangle of the third area can be obtained, and a matching rectangle that matches the bounding rectangle of the third area is determined in the image to be processed (the position of the matching rectangle in the image to be identified and the position of the bounding rectangle of the third area in the second binary image are the same, and the matching rectangle has the same size as the bounding rectangle of the third area), and the area in the image to be processed that is within the matching rectangle is taken as the target area.

[0123] If there is a third area in the second binary image, the position information of the third area (for example, pixel coordinates, the pixel coordinates of the third area refer to the pixel coordinates of the third area in the second binary image) can be obtained, and the area in the image to be processed with the same position information as the third area is determined as the target area.

[0124] S270, obtaining directional gradient histogram features of the target image; inputting the directional gradient histogram features of the target image into a license plate classifier to obtain a license plate classification result, the license plate classification result being that the target image is an image including the license plate or the target image is an image not including the license plate.

[0125] Among them, the description of S270 refers to the description of S120-S130 above, and will not be repeated here.

[0126] In this embodiment, the first area is determined by the first area rule and the first angle rule, and the second area is determined based on the first area. Then, the third area is determined based on the second area rule and the second angle rule. Multiple recognitions are achieved, making it more likely that the determined third area includes a license plate, thereby improving the accuracy of the target image determined according to the area rule and the angle rule, and further improving the accuracy of the classification result of the target image.

[0127] See also Figure 8 , Figure 8 A flowchart of an image processing method proposed in another embodiment of the present application is shown. The method can be applied to an electronic device, which can be Figure 1 The terminal 20 or the server 10 in the embodiment of the present invention comprises:

[0128] S310: Obtain training samples.

[0129] Among them, the training samples include sample images and sample classification results corresponding to the sample images, the sample images include positive sample images and negative sample images, the positive sample images are images including license plates, the negative sample images are images not including license plates, and the sample classification results are that the sample images do not include license plates or the sample images include license plates.

[0130] The initial sample image may be identified directly through manual screening, and the initial sample image including the license plate may be determined as the positive sample image, and the sample classification result of the positive sample image may be determined as including the license plate. At the same time, the initial sample image not including the license plate may be determined as the positive sample image, and the sample classification result of the positive sample image may be determined as not including the license plate.

[0131] As an implementation method, S310 may include: acquiring multiple initial sample images; if the initial sample image contains an area that satisfies the license plate recognition rule, determining that the initial sample image is an intermediate sample image; determining a sample classification result of the intermediate sample image, the sample classification result of the intermediate sample image is that the intermediate sample image includes the license plate or does not include the license plate; if the sample classification result of the intermediate sample image is that the intermediate sample image includes the license plate, determining that the intermediate sample image is a positive sample image; if the sample classification result of the intermediate sample image is that the intermediate sample image does not include the license plate, determining that the intermediate sample image is a negative sample image.

[0132] If the initial sample image includes an area that satisfies the license plate recognition rule, the initial sample image is determined to be an intermediate sample image; otherwise, the initial sample image is discarded.

[0133] The process of determining whether an initial sample image includes an area that satisfies a license plate recognition rule may include: binarizing the initial sample image to obtain a fourth binary image, the fourth binary image including at least one connected domain; determining whether a fourth area exists in the fourth binary image, the fourth area refers to a connected domain that satisfies a first area rule and a first angle rule in the fourth binary image; if the fourth area exists in the fourth binary image, determining a fifth area in the initial sample image according to a circumscribed rectangle of the fourth area; binarizing the fifth area to obtain a fifth binary image, the fifth binary image including at least one connected domain; determining whether a sixth area exists in the fifth binary image, the sixth area refers to a connected domain that satisfies a second area rule and a second angle rule in the fifth binary image; if the sixth area exists in the fifth binary image, obtaining an area in the initial sample image corresponding to the sixth area as an area in the initial sample image that satisfies the license plate recognition rule; if the sixth area does not exist in the fifth binary image, determining that the initial sample image does not include an area that satisfies the license plate recognition rule.

[0134] Among them, the width of the circumscribed rectangle corresponding to each connected domain in the fourth binary image is parallel to the target direction; determining whether there is a fourth area in the fourth binary image, including: constructing a circumscribed rectangle for each connected domain in the fourth binary image; for each connected domain in the fourth binary image, calculating the area ratio of the connected domain and the circumscribed rectangle corresponding to the connected domain as the area ratio corresponding to the connected domain; for each connected domain in the fourth binary image, if the area and area ratio of each connected domain meet the first area rule, obtaining the connected domain as the third connected domain; constructing the minimum circumscribed rectangle of each third connected domain; determining the angle between the minimum circumscribed rectangle of each third connected domain and the target direction; if there is a third connected domain whose angle meets the first angle rule, obtaining the third connected domain whose angle meets the first angle rule as the fourth area; if there is no third connected domain whose angle meets the first angle rule, determining that there is no fourth area in the fourth binary image.

[0135] Determining a fifth area in the initial sample image according to the circumscribed rectangle of the fourth area includes: determining the circumscribed rectangle of the fourth area; enlarging the circumscribed rectangle of the fourth area to obtain a new enlarged rectangle; and determining the fifth area in the initial sample image according to the new enlarged rectangle.

[0136] Binarization is performed on the fifth region to obtain a fifth binary image, including: binarization is performed on the fifth region to obtain a sixth binary image, the sixth binary image including at least one connected domain; determining the endpoints of the first side and the endpoints of the second side corresponding to each connected domain in the sixth binary image, the first side and the second side being opposite; constructing a new target rectangular frame according to the endpoints of the first side and the endpoints of the second side corresponding to each connected domain in the sixth binary image; adjusting the pixel values ​​of the pixels outside the new target rectangular frame in the sixth binary image to the target values, to obtain the fifth binary image.

[0137] The width of the circumscribed rectangle corresponding to each connected domain in the second binary image is parallel to the target direction; determining whether there is a sixth region in the fifth binary image, including: constructing a circumscribed rectangle for each connected domain in the fifth binary image; for each connected domain in the fifth binary image, calculating the area ratio of the connected domain and the circumscribed rectangle corresponding to the connected domain as the area ratio of the connected domain; for each connected domain in the fifth binary image, if the area and area ratio of each connected domain meet the second area rule, obtaining the connected domain as the fourth connected domain; constructing the minimum circumscribed rectangle of each fourth connected domain; determining the angle between the minimum circumscribed rectangle of each fourth connected domain and the target direction; if there is a fourth connected domain whose angle meets the second angle rule, obtaining the fourth connected domain whose angle meets the second angle rule as the sixth region; if there is no fourth connected domain whose angle meets the second angle rule, determining that there is no sixth region in the fifth binary image.

[0138] After the intermediate sample image is obtained, the sample classification result of the intermediate sample image can be determined by manual screening. The sample classification result of the intermediate sample image includes the license plate or does not include the license plate.

[0139] S320, determining a histogram of directional gradients feature of the positive sample image as a first sample feature; and determining a histogram of directional gradients feature of the negative sample image as a second sample feature.

[0140] Among them, the process of determining the first sample feature can be: dividing the positive sample image to obtain multiple positive sub-images; dividing each positive sub-image to obtain a positive image block of each positive sub-image; dividing each positive image block to obtain a positive image unit of each positive image block; determining the gradient and gradient direction of each pixel in each positive image unit according to the pixel value of each pixel in each positive image unit; determining the directional gradient histogram descriptor of each positive image unit according to multiple preset intervals and the gradient and gradient direction of each pixel in each positive image unit; obtaining and summarizing the directional gradient histogram descriptors of each image unit in each positive image block to obtain a descriptor of each positive image block; obtaining and summarizing the descriptors of each positive image block in each positive sub-image to obtain a descriptor of each positive sub-image; obtaining and summarizing the descriptors of multiple positive sub-images to obtain the directional gradient histogram feature of the positive sample image as the first sample feature.

[0141] Among them, the process of determining the second sample feature can be: dividing the negative sample image to obtain multiple negative sub-images; dividing each negative sub-image to obtain a negative image block of each negative sub-image; dividing each negative image block to obtain a negative image unit of each negative image block; determining the gradient and gradient direction of each pixel in each negative image unit according to the pixel value of each pixel in each negative image unit; determining the directional gradient histogram descriptor of each negative image unit according to multiple preset intervals and the gradient and gradient direction of each pixel in each negative image unit; obtaining and summarizing the directional gradient histogram descriptors of each image unit in each negative image block to obtain the descriptor of each negative image block; obtaining and summarizing the descriptors of each negative image block in each negative sub-image and the sliding step size to obtain the descriptor of each negative sub-image; obtaining and summarizing the descriptors of multiple negative sub-images to obtain the directional gradient histogram feature of the negative sample image as the second sample feature.

[0142] In this embodiment, for each positive sample image, the area in the positive sample image that meets the license plate recognition rules can be obtained as the target positive sample image, and the directional gradient histogram feature of the target positive sample image is determined according to the above-mentioned process of determining the directional gradient histogram feature of the positive sample image as the first sample feature. Similarly, for each negative sample image, the area in the negative sample image that meets the license plate recognition rules can be obtained as the target negative sample image, and the directional gradient histogram feature of the target negative sample image is determined according to the above-mentioned process of determining the directional gradient histogram feature of the negative sample image as the first sample feature.

[0143] S330. Train the initial classifier according to the first sample feature and the second sample feature, the sample classification result of the positive sample image, and the sample classification result of the negative sample image to obtain a license plate classifier.

[0144] The first sample feature can be input into the initial classifier to obtain a first predicted classification result, and the second sample feature can be input into the initial classifier to obtain a second predicted classification result. Then, based on the sample classification result of the positive sample image, the sample classification result of the negative sample image, the first predicted classification result, and the second predicted classification result, the loss value is determined, and the initial classifier is trained according to the determined loss value until the training is completed to obtain a license plate classifier. Among them, the function for determining the loss value can be a cross entropy loss function, an L1 loss function, or an L2 loss function, etc. The end of training can mean that the number of iterations reaches a set number, or the loss value of the initial classifier after iteration is less than a preset gap. The preset gap can be a range of loss values ​​set based on scenarios and requirements, which is not limited in this application.

[0145] After the license plate classifier is trained, the model parameters of the license plate classifier can be obtained, and a yml (YAML Aint Markup Language, yml is a markup language) file can be obtained according to the model parameters, so as to configure other classifiers in the initial state through the yml file to obtain the license plate classifier.

[0146] S340, determine whether there is a target area that meets the license plate recognition rules in the image to be recognized; if there is a target area in the image to be recognized, obtain the image located in the target area in the image to be recognized as the target image; obtain the directional gradient histogram features of the target image; input the directional gradient histogram features of the target image into the license plate classifier to obtain the license plate classification result.

[0147] The description of S340 refers to the description of S110 to S130 above, which will not be repeated here.

[0148] In this embodiment, a license plate classifier is obtained by training the positive sample images and the negative sample images, and the classification effect of the license plate classifier is good, thereby improving the classification result of the target image. At the same time, the positive sample images and the negative sample images are screened by the license plate recognition rules, so that the difference between the positive sample images and the negative sample images is small, so that the license plate classifier obtained by training the positive sample images and the negative sample images can effectively learn the small difference between the positive sample images and the negative sample images, thereby improving the classification performance and accuracy of the license plate classifier.

[0149] In order to explain the technical solution of the present application more clearly, the image processing method of the present application is explained below in conjunction with an exemplary scenario. In this scenario, the initial sample image includes 1000 images, the third area interval in the first area rule in the license plate recognition rule and the fourth area interval in the second area rule are both (95600, 100000), the second area ratio in the first area rule and the third area ratio in the second area rule are both 0.75, the third angle in the first angle rule and the fourth angle in the second angle rule are both 30 degrees, the target direction is horizontal, the target direction of the license plate is horizontal, and the setting angle of the license plate is 0 degrees (the width of the license plate is parallel to the horizontal direction).

[0150] 1. Training of license plate classifier

[0151] For each initial sample image, determine whether the initial sample image includes an area that meets the license plate recognition rule:

[0152] Binarize the initial sample image F1 to obtain a binary image F2, which includes at least one connected domain; construct a bounding rectangle for each connected domain in the binary image F2; and calculate the area ratio of each connected domain and the bounding rectangle corresponding to the connected domain as the area ratio corresponding to the connected domain; if the area of ​​each connected domain is (95600, 100000) and the area ratio is greater than 0.7, determine that the connected domain is a third connected domain, and traverse in this way to determine all the third connected domains of the binary image F2. Construct the minimum bounding rectangle of each third connected domain; determine the angle between the minimum bounding rectangle of each third connected domain and the horizontal direction; if there is a third connected domain with an angle less than 30 degrees, obtain the third connected domain with an angle less than 30 degrees as the fourth area F3 corresponding to the initial sample image. If there is no third connected domain with an angle that meets the first angle rule, determine that the initial sample image is discarded.

[0153] For the initial sample image F2 in which the fourth region (actually a connected domain) exists, the circumscribed rectangle of the fourth region F3 is determined; with the centroid of the fourth region F3 as the enlargement center, the circumscribed rectangle of the fourth region is enlarged by 2 times to obtain the enlarged rectangle E1; the area corresponding to the enlarged rectangle E1 in the initial sample image F2 is used as the fifth region F4.

[0154] The fifth region F4 is binarized to obtain a binary image F5, which includes at least one connected domain. A new target rectangular frame E2 is constructed based on the leftmost endpoint and the rightmost endpoint corresponding to all connected domains in the binary image F5. The leftmost endpoint and the left and right endpoints corresponding to each connected domain in the binary image F5 are used to construct the new target rectangular frame E2. The pixel values ​​of the pixel points in the binary image F5 outside the new target rectangular frame E2 are adjusted to 0 to obtain a binary image F6, which includes at least one connected domain.

[0155] Construct a circumscribed rectangle for each connected domain in the binary image F6; calculate the area ratio of the connected domain and the circumscribed rectangle corresponding to the connected domain; if the area of ​​each connected domain is (95600, 100000) and the area ratio is greater than 0.7, determine the connected domain as the fourth connected domain; construct the minimum circumscribed rectangle of each fourth connected domain; determine the angle between the minimum circumscribed rectangle of each fourth connected domain and the horizontal direction; if there is a fourth connected domain with an angle less than 30 degrees, obtain the fourth connected domain with an angle less than 30 degrees as the sixth region F7; if there is no fourth connected domain with an angle less than 30 degrees, discard the initial sample image.

[0156] According to the above steps, all 1000 initial sample images are traversed, 800 initial sample images with the sixth area are determined as 800 intermediate sample images, and the sample classification results of each of the 800 intermediate sample images are determined. It is determined that 600 positive sample images include license plates as the sample classification results, and 200 negative sample images do not include license plates as the sample classification results. According to the 2:1 ratio between the positive sample images and the negative sample images, training samples are constructed, and the training samples include 400 positive sample images, 200 negative sample images, and the sample classification results of each of the 600 sample images.

[0157] For each positive sample image, an image in the positive sample image located in the sixth area of ​​the positive sample image is determined as the target positive sample image, and for each negative sample image, an image in the negative sample image located in the sixth area of ​​the negative sample image is determined as the target negative sample image.

[0158] For the target sample image (target positive sample image or target negative sample image), the oriented gradient histogram feature is extracted according to the following process: the target sample image is divided according to the detection window of 128*32 to obtain multiple target sample sub-images; each target sample sub-image is divided into 4*4 blocks according to a step size of 2*2 to obtain a target sample image block for each target sample sub-image; each target sample image block is divided into 2*2 units to obtain a target sample image unit for each target sample image block. According to the pixel value of each pixel point in each target sample image unit, the gradient and gradient direction of each pixel point in each target sample image unit are determined; according to multiple preset intervals and the gradient and gradient direction of each pixel point in each target sample image unit, the directional gradient histogram descriptor of each target sample image unit is determined; the directional gradient histogram descriptors of the four target sample image units in each target sample image block are obtained and summarized to obtain the descriptor of each target sample image block; the descriptors and sliding steps of the (63×15) target sample image blocks in each target sample sub-image are obtained and summarized to obtain the descriptor of each target sample sub-image; the descriptors of multiple target sample sub-images are obtained and summarized to obtain the directional gradient histogram features of the target sample image.

[0159] The directional gradient histogram feature of the target positive sample image is used as the first sample feature, and the directional gradient histogram feature of the target negative sample image is used as the second sample feature. The first sample feature can be input into the initial classifier (which can be a support vector machine) to obtain a first predicted classification result, and the second sample feature can be input into the initial classifier to obtain a second predicted classification result, and then the cross entropy loss value is determined according to the sample classification result of the positive sample image, the sample classification result of the negative sample image, the first predicted classification result and the second predicted classification result, and the initial classifier is trained according to the determined cross entropy loss value until the training is completed to obtain a license plate classifier.

[0160] 2. License plate detection

[0161] Acquire and play the video stream to be identified. When the video stream to be identified is paused (for example, by pressing the space bar or clicking the play interface), determine the video frame where the video stream to be identified stays as the target video frame, monitor at least two click operations on the target video frame, take the action point of each click operation as a marking point to determine at least two marking points, determine a rectangular box as a selection box based on the two marking points with the farthest distance, and take the image in the selection box as the image to be identified.

[0162] The image to be identified is binarized to obtain a binary image F8, which includes at least one connected domain; it is determined whether there is a first region in the binary image F8, the first region refers to a connected domain in the first binary image that satisfies a first area rule and a first angle rule; if the first region exists in the binary image F8, a second region is determined in the image to be identified according to the circumscribed rectangle of the first region; the second region is binarized to obtain a binary image F9, which includes at least one connected domain; it is determined whether there is a third region in the binary image F9, the third region refers to a connected domain in the second binary image that satisfies a second area rule and a second angle rule; if the third region exists in the binary image F9, an area corresponding to the third region in the image to be identified is obtained as a target area; if the third region does not exist in the second binary image, it is determined that the target region does not exist in the image to be identified, and it is directly determined that the image to be identified does not include a license plate.

[0163] Obtain a target area in the image to be identified, obtain a target image F10 corresponding to the target area, and determine the directional gradient histogram features of the target image F10 (similar to the above-mentioned steps of obtaining the directional gradient histogram features of the target sample image, which will not be repeated); input the directional gradient histogram features of the target image into the license plate classifier to obtain a license plate classification result, and the license plate classification result is that the target image is an image including the license plate or the target image is an image not including the license plate.

[0164] If the license plate classification result includes the license plate, the license plate information in the license plate can be extracted according to the license plate detection algorithm (such as optical character recognition technology), for example, Fig. 9 The image to be identified is shown, and the license plate information extracted is: M83059.

[0165] If the license plate classification result includes the license plate, the license plate can be marked to obtain a marking result, for example, Fig.10 The image to be identified is shown in a in the figure. The labeling result is as follows Fig.10 As shown in b, Fig.10 The 1001 in b is the marked rectangular box.

[0166] In this scenario, the target image is determined based on the traditional digital image processing algorithm, and the machine learning HOG+SVM method is used as an aid to effectively improve the detection rate of license plate positioning.

[0167] When the video stream to be identified is a video obtained by a camera at an intersection or highway, scenarios such as license plate recognition, intersection vehicle counting, and traffic flow calculation can be performed based on the classification results of the target image.

[0168] See also Fig.11 , Fig.11A block diagram of an image processing device proposed in an embodiment of the present application is shown, wherein the device 1100 includes:

[0169] A determination module 1110 is used to determine whether there is a target area that meets the license plate recognition rules in the image to be recognized, and the license plate recognition rules include an area rule set based on the size of the license plate and an angle rule determined based on the angle of the license plate;

[0170] The image acquisition module 1120 is used to acquire an image located in the target area in the image to be identified as a target image if there is a target area in the image to be identified;

[0171] A feature acquisition module 1130 is used to acquire the directional gradient histogram features of the target image;

[0172] The classification module 1140 is used to input the directional gradient histogram feature of the target image into the license plate classifier to obtain a license plate classification result, where the license plate classification result is whether the target image is an image including the license plate or the target image is an image not including the license plate.

[0173] Optionally, the area rule includes a first area rule and a second area rule, and the angle rule includes a first angle rule and a second angle rule; the determination module 1110 is also used to perform binarization processing on the image to be identified to obtain a first binary image, and the first binary image includes at least one connected domain; determine whether there is a first region in the first binary image, and the first region refers to a connected domain in the first binary image that satisfies the first area rule and the first angle rule; if the first region exists in the first binary image, determine the second region in the image to be identified according to the circumscribed rectangle of the first region; perform binarization processing on the second region to obtain a second binary image, and the second binary image includes at least one connected domain; determine whether there is a third region in the second binary image, and the third region refers to a connected domain in the second binary image that satisfies the second area rule and the second angle rule; if the third region exists in the second binary image, obtain the region in the image to be identified that corresponds to the third region as the target region; if the third region does not exist in the second binary image, determine that the target region does not exist in the image to be identified.

[0174] Optionally, the width of the circumscribed rectangle corresponding to each connected domain in the first binary image is parallel to the target direction; the determination module 1110 is also used to construct a circumscribed rectangle for each connected domain in the first binary image; for each connected domain in the first binary image, calculate the area ratio of the connected domain and the circumscribed rectangle corresponding to the connected domain as the area ratio corresponding to the connected domain; for each connected domain in the first binary image, if the area and area ratio of each connected domain meet the first area rule, obtain the connected domain as the first connected domain; construct the minimum circumscribed rectangle of each first connected domain; determine the angle between the minimum circumscribed rectangle of each first connected domain and the target direction; if there is a first connected domain whose angle meets the first angle rule, obtain the first connected domain whose angle meets the first angle rule as the first region; if there is no first connected domain whose angle meets the first angle rule, determine that the first region does not exist in the first binary image.

[0175] Optionally, the determination module 1110 is also used to perform binarization processing on the second area to obtain a third binary image, where the third binary image includes at least one connected domain; determine the endpoints of the first side and the endpoints of the second side corresponding to each connected domain in the third binary image, where the first side is relative to the second side; construct a target rectangular frame based on the endpoints of the first side and the endpoints of the second side; adjust the pixel values ​​of the pixel points outside the target rectangular frame in the third binary image to the target value to obtain a second binary image.

[0176] Optionally, the width of the circumscribed rectangle corresponding to each connected domain in the second binary image is parallel to the target direction; the determination module 1110 is also used to construct a circumscribed rectangle for each connected domain in the second binary image; for each connected domain in the second binary image, calculate the area ratio of the connected domain and the circumscribed rectangle corresponding to the connected domain as the area ratio of the connected domain; for each connected domain in the second binary image, if the area and area ratio of each connected domain meet the second area rule, obtain the connected domain as the second connected domain; construct the minimum circumscribed rectangle of each second connected domain; determine the angle between the minimum circumscribed rectangle of each second connected domain and the target direction; if there is a second connected domain whose angle meets the second angle rule, obtain the second connected domain whose angle meets the second angle rule as the third area; if there is no second connected domain whose angle meets the second angle rule, determine that there is no third area in the second binary image.

[0177] Optionally, the determination module 1110 is further configured to determine a circumscribed rectangle of the first region; enlarge the circumscribed rectangle of the first region to obtain an enlarged rectangle; and determine a second region in the image to be identified according to the enlarged rectangle.

[0178] Optionally, the feature acquisition module 1130 is also used to divide the target image to obtain multiple sub-images; divide each sub-image to obtain its own image block; divide each image block to obtain its own image unit; determine the gradient and gradient direction of each pixel in each image unit according to the pixel value of each pixel in each image unit; determine the directional gradient histogram descriptor of each image unit according to multiple preset intervals and the gradient and gradient direction of each pixel in each image unit; obtain and summarize the directional gradient histogram descriptors of each image unit in each image block to obtain a descriptor of each image block; obtain and summarize the descriptors of each image block in each sub-image to obtain a descriptor of each sub-image; obtain and summarize the descriptors of multiple sub-images to obtain the directional gradient histogram features of the target image.

[0179] Optionally, the device also includes a training module, which is used to obtain training samples, the training samples include sample images and sample classification results corresponding to the sample images, the sample images include positive sample images and negative sample images, the positive sample image is an image including a license plate, the negative sample image is an image not including a license plate, and the sample classification result is that the sample image does not include a license plate or the sample image includes a license plate; determine the directional gradient histogram feature of the positive sample image as the first sample feature; determine the directional gradient histogram feature of the negative sample image as the second sample feature; train the initial classifier according to the first sample feature and the second sample feature, the sample classification result of the positive sample image, and the sample classification result of the negative sample image to obtain a license plate classifier.

[0180] Optionally, the training module is also used to obtain multiple initial sample images; if the initial sample image contains an area that satisfies the license plate recognition rule, determine that the initial sample image is an intermediate sample image; determine a sample classification result of the intermediate sample image, the sample classification result of the intermediate sample image is that the intermediate sample image includes the license plate or does not include the license plate; if the sample classification result of the intermediate sample image is that the intermediate sample image includes the license plate, determine that the intermediate sample image is a positive sample image; if the sample classification result of the intermediate sample image is that the intermediate sample image does not include the license plate, determine that the intermediate sample image is a negative sample image.

[0181] Optionally, the determination module 1110 is also used to determine a target video frame from a video stream to be identified; obtain an image in a selection box from the target video frame as an image to be identified, the selection box is determined based on at least two marking points or according to a preset rule, and the at least two marking points are determined based on a marking operation on the target video frame.

[0182] It should be noted that the device embodiment in the present application corresponds to the aforementioned method embodiment. The specific principles in the device embodiment can be found in the contents of the aforementioned method embodiment and will not be repeated here.

[0183] Fig.12 A structural block diagram of an electronic device for executing an image processing method according to an embodiment of the present application is shown. The electronic device may be Figure 1 The terminal 20 or the server 10, etc., need to be explained that, Fig.12 The computer system 1200 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0184] like Fig.12 As shown, the computer system 1200 includes a central processing unit (CPU) 1201, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1202 or the program loaded from the storage part 1208 to the random access memory (RAM) 1203, such as executing the method in the above embodiment. In RAM 1203, various programs and data required for system operation are also stored. CPU 1201, ROM 1202 and RAM 1203 are connected to each other through bus 1204. Input / output (I / O) interface 1205 is also connected to bus 1204.

[0185] The following components are connected to the I / O interface 1205: an input section 1206 including a keyboard, a mouse, etc.; an output section 1207 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the I / O interface 1205 as needed. A removable medium 1211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1210 as needed so that a computer program read therefrom is installed into the storage section 1208 as needed.

[0186] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 1209, and / or installed from a removable medium 1211. When the computer program is executed by a central processing unit (CPU) 1201, various functions defined in the system of the present application are executed.

[0187] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0188] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0189] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0190] As another aspect, the present application further provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or may exist independently without being assembled into the electronic device. The above computer-readable storage medium carries computer-readable instructions, and when the computer-readable storage instructions are executed by a processor, the method in any of the above embodiments is implemented.

[0191] According to one aspect of the embodiments of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the method in any of the above embodiments.

[0192] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0193] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation method of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable an electronic device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation method of the present application.

[0194] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary techniques in the art that are not disclosed in the present application. It should be understood that the present application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

[0195] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Determine whether there is a target area that meets the license plate recognition rule in the image to be recognized, wherein the license plate recognition rule includes an area rule set based on the size of the license plate and an angle rule determined based on the angle at which the license plate is set; If the target area exists in the image to be identified, obtaining an image located in the target area in the image to be identified as a target image; Obtaining directional gradient histogram features of the target image; The directional gradient histogram feature of the target image is input into a license plate classifier to obtain a license plate classification result, wherein the license plate classification result is that the target image is an image including a license plate or the target image is an image not including a license plate.

2. The method according to claim 1, characterized in that The area rule includes a first area rule and a second area rule, and the angle rule includes a first angle rule and a second angle rule; and determining whether there is a target area that meets the license plate recognition rule in the image to be recognized includes: Binarization is performed on the image to be identified to obtain a first binary image, wherein the first binary image includes at least one connected domain; Determine whether there is a first region in the first binary image, where the first region refers to a connected domain in the first binary image that satisfies a first area rule and a first angle rule; If the first region exists in the first binary image, determining a second region in the image to be identified according to a circumscribed rectangle of the first region; Binarizing the second region to obtain a second binary image, wherein the second binary image includes at least one connected domain; Determine whether there is a third region in the second binary image, where the third region refers to a connected domain in the second binary image that satisfies a second area rule and a second angle rule; If the third region exists in the second binary image, obtaining a region in the image to be identified that corresponds to the third region as the target region; If the third area does not exist in the second binary image, it is determined that the target area does not exist in the image to be identified.

3. The method according to claim 2, characterized in that The width of the circumscribed rectangle corresponding to each connected domain in the first binary image is parallel to the target direction; and determining whether the first region exists in the first binary image includes: constructing a bounding rectangle for each connected domain in the first binary image; For each connected domain in the first binary image, calculating an area ratio of the connected domain and a circumscribed rectangle corresponding to the connected domain as an area ratio corresponding to the connected domain; For each connected domain in the first binary image, if the area and area ratio of each connected domain satisfy the first area rule, acquiring the connected domain as a first connected domain; Constructing a minimum circumscribed rectangle of each of the first connected domains; Determine the angle between the minimum circumscribed rectangle of each of the first connected domains and the target direction; If there is a first connected domain whose angle satisfies the first angle rule, obtaining the first connected domain whose angle satisfies the first angle rule as the first region; If there is no first connected domain whose angle satisfies the first angle rule, it is determined that the first region does not exist in the first binary image.

4. The method according to claim 2, characterized in that: The binarization process is performed on the second region to obtain a second binary image, including: Binarizing the second region to obtain a third binary image, wherein the third binary image includes at least one connected domain; The endpoints of the first side and the endpoints of the second side corresponding to each connected domain in the third binary image, the first side and the second side are opposite; Constructing a target rectangular frame according to the endpoints of the first side and the endpoints of the second side; The pixel values ​​of the pixels outside the target rectangular frame in the third binary image are adjusted to target values ​​to obtain a second binary image.

5. The method according to claim 2, characterized in that: The width of the circumscribed rectangle corresponding to each connected domain in the second binary image is parallel to the target direction; and determining whether the third area exists in the second binary image includes: constructing a bounding rectangle for each connected domain in the second binary image; For each connected domain in the second binary image, calculating an area ratio of the connected domain and a circumscribed rectangle corresponding to the connected domain as the area ratio of the connected domain; For each connected domain in the second binary image, if the area and area ratio of each connected domain satisfy the second area rule, acquiring the connected domain as a second connected domain; Constructing a minimum circumscribed rectangle of each of the second connected domains; Determine the angle between the minimum circumscribed rectangle of each of the second connected domains and the target direction; If there is a second connected domain whose angle satisfies the second angle rule, obtaining the second connected domain whose angle satisfies the second angle rule as the third region; If there is no second connected domain whose angle satisfies the second angle rule, it is determined that the third region does not exist in the second binary image.

6. The method according to claim 2, characterized in that The step of determining a second area in the image to be recognized according to the circumscribed rectangle of the first area includes: Determine a bounding rectangle of the first area; Enlarging the circumscribed rectangle of the first area to obtain an enlarged rectangle; The second area is determined in the image to be recognized according to the enlarged rectangle.

7. The method according to claim 1, characterized in that Determining the directional gradient histogram feature of the target image includes: Dividing the target image to obtain a plurality of sub-images; Dividing each of the sub-images to obtain image blocks of each of the sub-images; Dividing each of the image blocks to obtain image units of each of the image blocks; Determining the gradient and gradient direction of each pixel point in each image unit according to the pixel value of each pixel point in each image unit; Determine a directional gradient histogram descriptor for each of the image units according to a plurality of preset intervals and the gradients and gradient directions of the respective pixels in each of the image units; Acquire and summarize the directional gradient histogram descriptors of each image unit in each of the image blocks to obtain the descriptors of each of the image blocks; Acquire and summarize the descriptors of the respective image blocks in each of the sub-images to obtain the descriptors of the respective sub-images; The descriptors of the respective sub-images are acquired and summarized to obtain the directional gradient histogram features of the target image.

8. The method according to claim 1, characterized in that The method for obtaining the license plate classifier comprises: Acquire training samples, the training samples including sample images and sample classification results corresponding to the sample images, the sample images including positive sample images and negative sample images, the positive sample images are images including license plates, the negative sample images are images not including license plates, and the sample classification results are that the sample images do not include license plates or the sample images include license plates; Determine a directional gradient histogram feature of the positive sample image as a first sample feature; Determine a directional gradient histogram feature of the negative sample image as a second sample feature; The initial classifier is trained according to the first sample feature and the second sample feature, the sample classification result of the positive sample image and the sample classification result of the negative sample image to obtain the license plate classifier.

9. The method according to claim 8, characterized in that The obtaining of training samples comprises: Acquire multiple initial sample images; If the initial sample image has an area that satisfies the license plate recognition rule, determining the initial sample image as an intermediate sample image; Determine a sample classification result of the intermediate sample image, the sample classification result of the intermediate sample image being whether the intermediate sample image includes a license plate or does not include a license plate; If the sample classification result of the intermediate sample image is that the intermediate sample image includes a license plate, determining that the intermediate sample image is a positive sample image; If the sample classification result of the intermediate sample image is that the intermediate sample image does not include a license plate, it is determined that the intermediate sample image is a negative sample image.

10. The method according to claim 1, characterized in that The method for acquiring the image to be identified includes: Determine a target video frame from a video stream to be identified; An image in a selection box is obtained from the target video frame as the image to be identified, wherein the selection box is determined based on at least two marking points or according to a preset rule, and the at least two marking points are determined based on a marking operation on the target video frame.

11. An image processing device, characterized in that: The device comprises: A determination module, used to determine whether there is a target area that meets the license plate recognition rules in the image to be recognized, wherein the license plate recognition rules include an area rule set based on the size of the license plate and an angle rule determined based on the angle at which the license plate is set; An image acquisition module, configured to acquire an image located in the target area in the image to be identified as a target image if the target area exists in the image to be identified; A feature acquisition module, used to acquire the directional gradient histogram features of the target image; The classification module is used to input the directional gradient histogram feature of the target image into a license plate classifier to obtain a license plate classification result, wherein the license plate classification result is that the target image is an image including a license plate or the target image is an image not including a license plate.

12. An electronic device, characterized in that: include: one or more processors; Memory; One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs being configured to execute the method according to any one of claims 1-10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 10.