Video stream-based license plate recognition method and device, computing device, and storage medium

Abstract

The application discloses a license plate recognition method based on a video stream, and comprises the following steps: obtaining a video stream to be recognized; inputting the video stream to be recognized into a first slow channel network and a first fast channel network of a trained vehicle and license plate detection model in sequence to detect the positions of vehicles and license plates, and obtaining license plate region images and time sequence features of sequence frames; inputting the license plate region images and the time sequence features of the sequence frames into a second slow channel network and a second fast channel network of a trained license plate recognition model in sequence to recognize the license plates, and obtaining license plate recognition results and time sequence features of the sequence frames; and outputting license plate recognition results of the video stream to be recognized according to the license plate recognition results and the time sequence features of the sequence frames. The scheme can improve the speed and accuracy of license plate recognition on a low-power device.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and image recognition technology, and specifically to a license plate recognition method, apparatus, computing device, and storage medium based on video streams. Background Technology

[0002] Accurate and rapid license plate recognition in various complex scenarios can effectively improve the efficiency of traffic enforcement, parking management, and road traffic flow. Currently, license plate recognition is mainly based on deep learning methods, which directly locate the license plate in the image and recognize its content.

[0003] Existing technology discloses a license plate recognition system based on deep learning. When a vehicle is detected entering a video, the recognition function is triggered. A license plate recognition localization model identifies and locates the relative position of the license plate in the image; a license plate content recognition model identifies and locates the category, size, color, and text information of the license plate. This method requires two models for detection, making the detection process cumbersome. Furthermore, treating the license plate as a small target within a large scene results in low detection efficiency and requires high computational power, making it unsuitable for low-computing-power platforms.

[0004] Therefore, there is a need for a license plate recognition method based on video streams that can improve the accuracy and efficiency of license plate recognition in complex video stream scenarios on lightweight devices, in order to solve the problems existing in the above-mentioned technologies. Summary of the Invention

[0005] In view of the above problems, in order to improve the implementation of license plate recognition in complex video streams on low computing power devices, this solution proposes a license plate recognition method, device, computing device and storage medium based on video stream, which can reduce the dependence of license plate recognition on image resolution, reduce learning difficulty and improve recognition accuracy.

[0006] According to a first aspect of the present invention, a license plate recognition method based on video stream is provided, comprising: first, acquiring a video stream to be recognized; then, sequentially inputting the video stream to be recognized into a first slow channel network and a first fast channel network of a trained vehicle and license plate detection model for vehicle and license plate position detection, thereby obtaining license plate region images and temporal features of the sequence frames; next, sequentially inputting the license plate region images and temporal features of the sequence frames into a second slow channel network and a second fast channel network of a trained license plate recognition model for license plate recognition, thereby obtaining license plate recognition results and temporal features of the sequence frames; and finally, outputting the license plate recognition result of the video stream to be recognized based on the license plate recognition results and temporal features of the sequence frames.

[0007] The license plate recognition method provided in this solution learns the spatial position information between the vehicle and the license plate, and determines the approximate area of ​​the license plate based on the relative position between the vehicle and the license plate. This reduces the dependence of the license plate on image resolution and lowers the learning difficulty. By extracting different features from the input video frames at alternating speeds, it can integrate recognition information from multiple frames while significantly reducing the computational resource requirements and improving recognition accuracy and speed.

[0008] Optionally, in the above method, the trained vehicle and license plate detection model includes a first slow channel network and a first fast channel network connected in parallel, as well as a first ConvLSTM feature fusion network. The first slow channel network includes one first slow channel module, and the first fast channel network includes five first fast channel modules connected in series.

[0009] Optionally, in the above method, the first fast channel module uses MobileNetv2 as the backbone, FPN as the neck module, and the head module of CenterNet as the prediction branch to extract the first feature of the input image at a first time step; the first slow channel module uses MobileNetv3 as the backbone, FPN as the neck module, and the head module of CenterNet as the prediction branch to extract the second feature of the input image at a second time step, where the first time step is smaller than the second time step; the first ConvLSTM feature fusion network is used to fuse the first feature obtained by the first fast channel network with the second feature obtained by the first slow channel network, so as to obtain the vehicle and license plate location and temporal features of the input image based on the fused features.

[0010] Optionally, in the above method, the trained license plate recognition model includes a parallel second slow channel network, a second fast channel network, and a second ConvLSTM feature fusion network. The second slow channel network includes one second slow channel module, and the second fast channel network includes three cascaded second fast channel modules.

[0011] Optionally, in the above method, the second fast-channel module uses MobileNetV2 as the backbone and CRNN as the recognition and prediction network to extract the third feature of the input image at a third time step. The second slow-channel module uses MobileNetV3 as the backbone and CRNN as the recognition and prediction network to extract the fourth feature of the input image at a fourth time step, where the third time step is smaller than the fourth time step. The second ConvLSTM feature fusion network is used to fuse the third feature obtained by the second fast-channel network with the fourth feature obtained by the second slow-channel network, so as to obtain the license plate recognition result and temporal features of the input image based on the fused features.

[0012] Alternatively, in the above method, the license plate area detection task can be performed through the following steps:

[0013] S1. Input the current frame image into the first slow channel network of the trained vehicle and license plate detection model for detection, obtain the vehicle and license plate positions and first temporal features of the current frame image, and determine the first license plate region image of the current frame image based on the vehicle and license plate positions of the current frame image.

[0014] S3. Input the next frame image and the first temporal features into the first fast channel network of the trained vehicle and license plate detection model for detection, and obtain the vehicle and license plate positions and the third temporal features of the next frame image. Based on the vehicle and license plate positions of the next frame image, determine the second license plate region image of the next frame.

[0015] Repeat steps S1 and S3 until vehicle and license plate location detection is completed for all sequence frames.

[0016] Alternatively, in the above method, the license plate recognition task can be performed through the following steps:

[0017] S2. Input the first license plate area image into the second slow channel network of the trained license plate recognition model for recognition, and obtain the license plate recognition result and second temporal features of the current frame image;

[0018] S4. Input the second license plate area image and the second temporal features into the second fast channel network of the trained license plate recognition model for recognition, and obtain the license plate recognition result and the fourth temporal features of the next frame image;

[0019] Repeat steps S2 and S4 until license plate recognition is completed for all sequence frames.

[0020] According to a second aspect of the present invention, a license plate recognition device based on a video stream is provided, characterized in that it comprises: an acquisition module, a license plate region detection module, a license plate recognition module, and an output module. The acquisition module is used to acquire the video stream to be recognized;

[0021] The license plate region detection module is used to sequentially input the video stream to be identified into the first slow channel network and the first fast channel network of the trained vehicle and license plate detection model to detect the vehicle and license plate positions, and obtain the license plate region image and temporal features of the sequence frames;

[0022] The license plate recognition module is used to input the license plate region image and temporal features of the sequence frame into the second slow channel network and the second fast channel network of the trained license plate recognition model for license plate recognition, so as to obtain the license plate recognition result and temporal features of the sequence frame; the output module is used to output the license plate recognition result of the video stream to be recognized based on the license plate recognition result and temporal features of the sequence frame.

[0023] According to a third aspect of the present invention, a computing device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor performs the license plate recognition method based on video stream as described in the first aspect.

[0024] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, comprising storing a computer program capable of being loaded by a processor and executing the video stream-based license plate recognition method as described in the first aspect.

[0025] According to the present invention, the license plate detection problem is transformed into learning the spatial location of the vehicle and license plate, and the approximate license plate area is determined based on the distance between the center point of the vehicle and the center point of the license plate. This reduces the dependence of license plate detection on the resolution of the input image and lowers the learning difficulty of the algorithm. Furthermore, by alternately performing feature extraction tasks using fast and slow channel feature extraction networks, a balance between computational speed and accuracy can be maintained. By integrating the license plate recognition results from multiple frames, the recognition accuracy is significantly improved. Therefore, the video stream-based license plate recognition method provided by this solution can run on low-computing-power devices and is suitable for high-precision, real-time license plate recognition of large-scale license plate video streams.

[0026] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0027] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0028] Figure 1 A schematic diagram of the structure of a computing device 100 according to an embodiment of the present invention is shown;

[0029] Figure 2 A schematic diagram of the network structure of a trained vehicle and license plate detection model according to an embodiment of the present invention is shown.

[0030] Figure 3 A network structure diagram of a trained license plate recognition model according to an embodiment of the present invention is shown;

[0031] Figure 4 A flowchart illustrating a video stream-based license plate recognition method 400 according to an embodiment of the present invention is shown.

[0032] Figure 5 A schematic diagram of an interleaved model of a license plate recognition method based on video stream according to an embodiment of the present invention is shown.

[0033] Figure 6 A schematic diagram of the structure of a video stream-based license plate recognition device 600 according to an embodiment of the present invention is shown. Detailed Implementation

[0034] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0035] License plate recognition based on video streams involves dynamically capturing vehicle information using a camera to obtain a video stream containing the vehicle, and then processing the video stream to identify the license plate information. Current technologies treat the entire image as the detection object, separating license plate image segmentation and license plate content recognition into two steps. This process is computationally intensive and easily affected by external factors. Furthermore, cumulative errors can easily occur between the two models, thus affecting the overall recognition accuracy.

[0036] Compared to image object detection, video object detection is highly redundant, containing a large amount of temporal and spatial locality. Fully utilizing temporal context relationships can solve the problem of a large amount of redundancy between consecutive frames in a video stream and improve detection speed.

[0037] Therefore, in order to improve the accuracy and speed of license plate recognition in complex scenarios and reduce the computational load of the model, this solution proposes a license plate recognition method based on video streams. This method can be applied to low-computing-power devices. By using deep learning algorithms to obtain the spatial relative position of the vehicle and license plate, the dependence of license plate recognition on the input image resolution can be reduced, thus lowering the learning difficulty of the algorithm. Furthermore, by combining the slowfast algorithm, two networks with different frame rates are used for feature extraction, which can improve recognition accuracy while reducing the computational load of the model.

[0038] Figure 1 A schematic diagram of the structure of a computing device 100 according to an embodiment of the present invention is shown. Figure 1 As shown, in the basic configuration 102, the computing device 100 typically includes a system memory 106 and one or more processors 104. A memory bus 108 can be used for communication between the processors 104 and the system memory 106.

[0039] Depending on the desired configuration, processor 104 can be any type of processor, including but not limited to: microprocessors (μP), microcontrollers (μC), digital information processors (DSPs), or any combination thereof. Processor 104 may include one or more levels of cache such as L1 cache 110 and L2 cache 112, processor core 114, and registers 116. Example processor core 114 may include an arithmetic logic unit (ALU), a floating-point unit (FPU), a digital signal processing core (DSP core), or any combination thereof. Example memory controller 118 may be used with processor 104, or in some implementations, memory controller 118 may be an internal part of processor 104.

[0040] Depending on the desired configuration, system memory 106 can be any type of memory, including but not limited to: volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.), or any combination thereof. Physical memory in a computing device typically refers to volatile RAM, and data on a disk needs to be loaded into physical memory before it can be read by processor 104. System memory 106 may include operating system 120, one or more applications 122, and program data 124. In some embodiments, application 122 may be configured to execute instructions on the operating system using program data 124 by one or more processors 104. Operating system 120 may be, for example, Linux, Windows, etc., and includes program instructions for handling basic system services and performing hardware-dependent tasks. Application 122 includes program instructions for implementing various user-desired functions, and application 122 may be, for example, a browser, instant messaging software, software development tools (such as integrated development environments IDEs, compilers, etc.), but is not limited to these. When application 122 is installed in computing device 100, driver modules may be added to operating system 120.

[0041] When computing device 100 starts up, processor 104 reads and executes program instructions from memory 106 of operating system 120. Application 122 runs on operating system 120, utilizing interfaces provided by operating system 120 and underlying hardware to implement various user-expected functions. When user starts application 122, application 122 is loaded into memory 106, and processor 104 reads and executes program instructions from memory 106 of application 122.

[0042] The computing device 100 also includes a storage device 132, which includes a removable storage device 136 and a non-removable storage device 138, both of which are connected to a storage interface bus 134.

[0043] The computing device 100 may also include an interface bus 140 that facilitates communication from various interface devices (e.g., output devices 142, peripheral interfaces 144, and communication devices 146) to the basic configuration 102 via a bus / interface controller 130. Example output devices 142 include a graphics processing unit 148 and an audio processing unit 150. They may be configured to facilitate communication with various external devices such as displays or speakers via one or more A / V ports 152. Example peripheral interfaces 144 may include a serial interface controller 154 and a parallel interface controller 156, which may be configured to facilitate communication with external devices such as input devices (e.g., keyboards, mice, pens, voice input devices, touch input devices) or other peripherals (e.g., printers, scanners, etc.) via one or more I / O ports 158. Example communication devices 146 may include a network controller 160, which may be arranged to facilitate communication with one or more other computing devices 162 via a network communication link through one or more communication ports 164.

[0044] A network communication link can be an example of a communication medium. A communication medium can typically be embodied in computer-readable instructions, data structures, program modules in a modulated data signal such as a carrier wave or other transmission mechanism, and can include any information delivery medium. A “modulated data signal” can be a signal whose data set, or whose modifications, can be encoded with information in the signal. As a non-limiting example, a communication medium can include wired media such as wired networks or leased lines, and various wireless media including sound, radio frequency (RF), microwave, infrared (IR), or other wireless media. The term computer-readable medium as used herein can include both storage media and communication media. In the computing device 100 according to the invention, application 122 includes instructions for performing the video stream-based license plate recognition method 400 of the invention.

[0045] The lightweight video stream license plate recognition method provided by this invention includes two tasks: license plate detection and license plate information recognition. The license plate detection part combines license plate detection and vehicle detection. Therefore, this solution designs a vehicle and license plate detection model to detect the approximate area of ​​the vehicle and license plate, so as to assist in license plate detection by using the relative positions between the vehicle and license plate in the video stream.

[0046] Based on the algorithms of slow and fast networks, this scheme uses two networks with different feature extraction accuracies to extract different features from video frames. The fast network uses a small time step and a high frame rate to quickly extract weak features from video frames, resulting in poor accuracy. The slow network uses a large time step and a low frame rate to extract precise features from video frames. The larger the time step, the lower the accuracy of feature extraction; the smaller the time step, the higher the accuracy. Video frames can be processed alternately using these two feature extraction networks with different accuracies. Finally, a ConvLSTM feature fusion structure is used to fuse the features of the slow and fast networks, and the vehicle and license plate locations are detected based on the fused features.

[0047] Figure 2 A schematic diagram of a trained vehicle and license plate detection model according to an embodiment of the present invention is shown. Figure 2 As shown, the trained vehicle and license plate detection model includes a first slow channel network and a first fast channel network connected in parallel, as well as a first ConvLSTM feature fusion network. The first slow channel network includes one first slow channel module, and the first fast channel network contains five first fast channel modules connected in series.

[0048] Specifically, the system's computational efficiency can be improved by using asynchronous mode and quantization models. Specifically, a fast-channel network is used to extract image features, while a slow-channel network is used to compute and update the stored features. Furthermore, the quantization model can compress the model, eliminating the need for rescaling.

[0049] According to one embodiment of the present invention, the first fast-channel module uses MobileNet v2 as the backbone, FPN as the neck module, and the CenterNet head module as the prediction branch to extract the first feature of the image at a first time step. Here, "fast-channel" refers to a faster sampling frame rate, which maintains fidelity in temporal resolution; therefore, the first feature is a weak feature with lower accuracy.

[0050] The first slow-channel module uses MobileNet v3 as the backbone, FPN as the neck module, and CenterNet's head module as the prediction branch. It primarily extracts spatial features from the input image using a second time step. The first time step is smaller than the second time step; for example, the first time step is 2, and the second time step is 16. This means the slow channel has a lower sampling frame rate, extracting one frame every 16 frames. Therefore, the second feature is a strong feature with higher accuracy.

[0051] In this network, MobileNetv2 primarily uses the bottleneck residual module for feature extraction. MobileNetv3 adds an SE structure to the bottleneck residual module, using ReLU6(x+3) / 6 to approximate the sigmoid function, and reducing the number of convolutional kernels in the first convolutional layer (from 32 to 16). MobileNetv3's model initialization and inference times are slightly slower than MobileNetv2. Therefore, this approach uses MobileNetv3 for the slow-channel network and MobileNetv2 for the fast-channel network.

[0052] Feature maps from different time dimensions and feature scales of the FPN feature pyramid network can be fused. Finally, CenterNet is used to perform global average pooling and feature vector concatenation on the output of each channel, resulting in three prediction branches: license plate location heatmap prediction branch, distance prediction branch from vehicle center point to license plate center point, and license plate size prediction branch.

[0053] When training the pre-built model, 5000 video streams of vehicles in the checkpoint scene can be collected. Each video stream has a frame rate of 25fps and a duration of 30 seconds. The collected video streams are labeled with the vehicle and license plate positions and license plate recognition results in each frame, and used as training data input to the model. The model is iteratively trained for 160 epochs based on the CenterNet loss function to obtain the trained vehicle and license plate detection model. Each time, the first slow channel module is followed by 5 times the first fast channel module.

[0054] The first ConvLSTM feature fusion network is used to fuse the first feature obtained by the first fast-channel network with the second feature obtained by the first slow-channel network. ConvLSTM transforms the 2D input in LSTM into a 3D tensor, with the last two dimensions being spatial dimensions (rows and columns). For each time step t, ConvLSTM replaces some connection operations in LSTM with convolution operations, that is, it makes predictions based on the current input and local neighboring past states. Finally, after feature fusion, it obtains the vehicle and license plate detection boxes and temporal features of the input video frame.

[0055] Figure 3 A schematic diagram of the structure of a trained license plate recognition model according to an embodiment of the present invention is shown. Figure 3 As shown, the trained license plate recognition model includes a parallel second fast channel network and a second slow channel network, as well as a second ConvLSTM feature fusion network. The second fast channel network includes three cascaded second fast channel modules, and the second slow channel network includes one second slow channel module.

[0056] According to one embodiment of the present invention, the second fast-channel module uses MobileNetV2 as the backbone and CRNN as the recognition and prediction network to extract temporal features of the input image at a third time step. The second slow-channel network uses MobileNetV3 as the backbone and CRNN as the recognition and prediction network to extract spatial features of the input image at a fourth time step, where the third time step is smaller than the fourth time step. That is, the sampling frame rate of the second fast-channel network is greater than the sampling frame rate of the second slow-channel network.

[0057] Among them, CRNN network, or Convolutional Recurrent Neural Network, is used for end-to-end recognition of text sequences of variable length, transforming text recognition into a time-dependent sequence learning problem, thereby effectively improving the accuracy of text recognition.

[0058] When training the pre-built license plate recognition model, 5000 video streams of vehicles in the checkpoint scene can be collected. Each video stream has a frame rate of 25fps and a duration of 30 seconds. The collected video streams are labeled with the vehicle and license plate positions and license plate recognition results in each frame, and used as training data input to the model. The model is iteratively trained for 200 cycles based on the CTC loss function to obtain the trained license plate recognition model. Each second slow channel module is followed by three second fast channel modules.

[0059] The second ConvLSTM feature fusion network is used to fuse the third feature obtained from the second fast channel network with the fourth feature obtained from the second slow channel network to obtain the license plate recognition result and temporal features of the input image.

[0060] Figure 4 A flowchart illustrating a video stream-based license plate recognition method 400 according to an embodiment of the present invention is shown. Figure 4 As shown, the method begins with step S410, which acquires the video stream to be identified.

[0061] The video stream to be identified can be a vehicle video stream collected by road camera monitoring equipment, a vehicle video stream collected at parking lot entrances and exits or highway toll booths. Since the video segments containing vehicles in the video stream may not be continuous, the video stream data to be identified needs to be processed.

[0062] For example, the acquired raw video stream can be sliced ​​into video segments from when the vehicle enters the video capture frame to when the vehicle leaves the video capture frame.

[0063] Then, step S420 is executed, in which the video stream to be identified is sequentially input into the first slow channel network and the first fast channel network of the trained vehicle and license plate detection model to detect the vehicle and license plate positions, thereby obtaining the license plate region image and temporal features of the sequence frames.

[0064] Specifically, this may include: S1, inputting the current frame image into the first slow-channel network of the trained vehicle and license plate detection model for detection, obtaining the vehicle and license plate positions and first temporal features of the current frame image. Then, determining the first license plate region image of the current frame image based on the vehicle and license plate positions. S3, inputting the next frame image and the first temporal features into the first fast-channel network of the trained vehicle and license plate detection model for detection, obtaining the vehicle and license plate positions and third temporal features of the next frame image, and determining the second license plate region image of the next frame based on the vehicle and license plate positions of the next frame image. Repeating steps S1 and S3 until the vehicle and license plate position detection of all sequence frames is completed.

[0065] By alternating between fast and slow feature extraction networks to extract features from different frames, computational redundancy can be reduced, achieving a balance between speed and accuracy.

[0066] Then, step S430 is executed, in which the license plate area image and temporal features of the sequence frame are sequentially input into the second slow channel network and the second fast channel network of the trained license plate recognition model for license plate recognition, so as to obtain the license plate recognition result and temporal features of the sequence frame.

[0067] Specifically, this may include: S2, inputting the first license plate region image into the second slow channel network of the trained license plate recognition model for recognition, obtaining the license plate recognition result and second temporal feature of the current frame image. S4, inputting the second license plate region image and second temporal feature into the second fast channel network of the trained license plate recognition model for recognition, obtaining the license plate recognition result and fourth temporal feature of the next frame image. Repeating steps S2 and S4 until license plate recognition of all sequence frames is completed.

[0068] By alternating between fast and slow feature extraction networks to extract features from different frames, computational redundancy can be reduced, achieving a balance between speed and accuracy.

[0069] Finally, step S440 is executed to output the license plate recognition result of the video stream to be recognized based on the license plate recognition result of the sequence frame and the temporal features.

[0070] Multiple license plate recognition results from a sequence of frames can be voted on and cross-referenced to obtain the most accurate result. For example, the frequency of each license plate character can be counted, and the character with the highest frequency can be selected as the final license plate recognition result.

[0071] Therefore, the steps for license plate recognition in video streams can be summarized as follows: Input image t into the first slow channel network to obtain the approximate location of vehicle t and the corresponding license plate area, as well as the first temporal feature. Then, extract the approximate license plate area of ​​image t and input the approximate license plate area of ​​image t into the second slow channel network to obtain the license plate recognition result of t and the second temporal feature.

[0072] Image t+1 and the first temporal feature are input into the first fast-channel network to obtain the approximate location of the vehicle at t+1, the corresponding license plate area, and the third temporal feature. The approximate license plate area of ​​image t+1 is then extracted. This approximate license plate area and the third temporal feature are input into the second fast-channel network to obtain the license plate recognition result and temporal feature at t+1. Following the alternating slow and fast rules, the above steps are repeated until the vehicle leaves the video frame, at which point the license plate recognition result is output.

[0073] Figure 5 A schematic diagram of an interleaved model of a license plate recognition method according to an embodiment of the present invention is shown. Figure 5 As shown, first, the input vehicle video stream sequence frames: ... t-3 l t-2 l t-1 l t l t+1 l t+2 ...Then, based on the principle of alternating between fast and slow channels, the current frame is first input into the first slow channel network and the second slow channel network, and then the next frame is input into the first fast channel network and the second fast channel network, and so on, until all sequence frames have completed license plate recognition, obtaining the license plate recognition results for all sequence frames: ...D t-3 D t-2 D t-1 D t D t+1 D t+2 Finally, based on temporal features, the results of multiple license plate recognitions are integrated to obtain the final license plate recognition result.

[0074] Figure 6 A schematic diagram of a video stream-based license plate recognition device 600 according to an embodiment of the present invention is shown. Figure 6 As shown, the license plate recognition device 600 based on video stream includes: an acquisition module 610, a license plate area detection module 620, a license plate recognition module 630, and an output module 640.

[0075] The acquisition module 610 acquires the video stream to be recognized. The license plate region detection module 620 sequentially inputs the video stream to be recognized into the first slow channel network and the first fast channel network of the trained vehicle and license plate detection model to detect vehicle and license plate positions, obtaining license plate region images and temporal features of the sequence frames. The license plate recognition module 630 sequentially inputs the license plate region images and temporal features of the sequence frames into the second slow channel network and the second fast channel network of the trained license plate recognition model to recognize license plates, obtaining license plate recognition results and temporal features of the sequence frames. The output module 640 outputs the license plate recognition result of the video stream to be recognized based on the license plate recognition result and temporal features of the sequence frames.

[0076] This application discloses a computer-readable storage medium, including a computer program that can be loaded by a processor and executed as described above in the video stream-based license plate recognition method 400.

[0077] The computer-readable storage medium may be any tangible medium that contains or stores a program that may be used by or in connection with an instruction execution system, apparatus, or device; the program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, or any suitable combination thereof.

[0078] This invention transforms the license plate detection problem into learning the spatial locations of vehicles and license plates. It determines the approximate license plate area based on the distance between the vehicle's center point and the license plate's center point, reducing the dependence of license plate detection on input image resolution and lowering the algorithm's learning difficulty. Furthermore, by alternately executing feature extraction tasks using fast and slow channel feature extraction networks, a balance between computational speed and accuracy is maintained. By integrating multi-frame license plate recognition results, recognition accuracy is significantly improved. Therefore, the video stream-based license plate recognition method provided in this invention can run on low-computing-power devices and is suitable for high-precision, real-time license plate recognition in large-scale video stream scenarios.

[0079] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0080] Similarly, it should be understood that, in order to streamline this disclosure and aid in understanding one or more of the various aspects of the invention, in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof. However, this method of disclosure should not be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into this detailed description, wherein each claim itself is a separate embodiment of the invention.

[0081] Those skilled in the art will understand that modules, units, or components of the devices disclosed in the examples herein can be arranged in the devices described in this embodiment, or alternatively, can be located in one or more devices different from the devices in this example. The modules in the foregoing examples can be combined into a single module or, in addition, can be divided into multiple sub-modules.

[0082] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0083] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0084] Furthermore, some of the embodiments are described herein as methods or combinations of method elements that can be implemented by a processor of a computer system or by other means of performing functions. Therefore, a processor having the necessary instructions for implementing a method or method element forms means for implementing that method or method element. Furthermore, the elements described herein in the apparatus embodiments are examples of means for implementing functions performed by elements for the purposes of carrying out the invention.

[0085] As used herein, unless otherwise specified, the use of ordinal numbers such as “first,” “second,” “third,” etc., to describe ordinary objects merely indicates different instances of similar objects and is not intended to imply that the objects being described must have a given order in time, space, ordering, or any other manner.

[0086] Although the invention has been described with respect to a limited number of embodiments, those skilled in the art will understand from the foregoing description that other embodiments are conceivable within the scope of the invention described herein. Furthermore, it should be noted that the language used in this specification has been chosen primarily for readability and edibility purposes, and not for the purpose of interpreting or limiting the subject matter of the invention. Therefore, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The disclosure of the invention is illustrative rather than restrictive, and the scope of the invention is defined by the appended claims.

Claims

1. A license plate recognition method based on video streams, suitable for execution in a computing device, characterized in that, include: Obtain the video stream to be identified; The video stream to be identified is sequentially input into the first slow channel network and the first fast channel network of the trained vehicle and license plate detection model for vehicle and license plate location detection, resulting in license plate region images and temporal features of the sequence frames, including: S1. Input the current frame image into the first slow channel network of the trained vehicle and license plate detection model for detection, obtain the vehicle and license plate positions and first temporal features of the current frame image, and determine the first license plate region image of the current frame image based on the vehicle and license plate positions of the current frame image. S3. Input the next frame image and the first temporal feature into the first fast channel network of the trained vehicle and license plate detection model for detection, and obtain the vehicle and license plate positions and the third temporal feature of the next frame image. Based on the vehicle and license plate positions of the next frame image, determine the second license plate region image of the next frame. Repeat steps S1 and S3 until vehicle and license plate location detection is completed for all sequence frames; The license plate region image and temporal features of the sequence frames are sequentially input into the second slow channel network and the second fast channel network of the trained license plate recognition model for license plate recognition, resulting in the license plate recognition results and temporal features of the sequence frames, including: S2. Input the first license plate area image into the second slow channel network of the trained license plate recognition model for recognition, and obtain the license plate recognition result and second temporal features of the current frame image; S4. Input the second license plate area image and the second temporal features into the second fast channel network of the trained license plate recognition model for recognition, and obtain the license plate recognition result and the fourth temporal features of the next frame image; Repeat steps S2 and S4 until license plate recognition for all sequence frames is completed; The license plate recognition results of the video stream to be recognized are output based on the license plate recognition results of the sequence frames and the temporal features.

2. The license plate recognition method according to claim 1, characterized in that, The trained vehicle and license plate detection model includes a first slow channel network and a first fast channel network connected in parallel, as well as a first convLSTM feature fusion network. The first slow channel network includes one first slow channel module, and the first fast channel network includes five first fast channel modules connected in series.

3. The license plate recognition method according to claim 2, characterized in that, The first fast channel module uses MobileNet v2 as the backbone, FPN as the neck module, and CenterNet's head module as the prediction branch to extract the first feature of the input image through the first time step. The first slow channel module uses MobileNet v3 as the backbone, FPN as the neck module, and CenterNet's head module as the prediction branch to extract the second feature of the input image through a second time step, where the first time step is smaller than the second time step. The first ConvLSTM feature fusion network is used to fuse the first feature obtained by the first fast channel network with the second feature obtained by the first slow channel network, so as to obtain the vehicle and license plate location and temporal features of the input image based on the fused features.

4. The license plate recognition method according to claim 1, characterized in that, The trained license plate recognition model includes a parallel second slow channel network, a second fast channel network, and a second convLSTM feature fusion network. The second slow channel network includes one second slow channel module, and the second fast channel network includes three cascaded second fast channel modules.

5. The license plate recognition method according to claim 4, characterized in that, The second fast-channel module uses MobileNetv2 as the backbone and CRNN as the recognition and prediction network to extract the third feature of the input image through the third time step. The second slow channel module uses MobileNetv3 as the backbone and CRNN as the recognition and prediction network to extract the fourth feature of the input image through a fourth time step, wherein the third time step is smaller than the fourth time step. The second ConvLSTM feature fusion network is used to fuse the third feature obtained by the second fast channel network with the fourth feature obtained by the second slow channel network, so as to obtain the license plate recognition result and temporal features of the input image based on the fused features.

6. A license plate recognition device based on a video stream, used to perform the license plate recognition method as described in claim 1, characterized in that, include: The acquisition module is used to acquire the video stream to be identified; The license plate region detection module is used to sequentially input the video stream to be identified into the first slow channel network and the first fast channel network of the trained vehicle and license plate detection model to detect the vehicle and license plate positions, and obtain the license plate region image and temporal features of the sequence frames; The license plate recognition module is used to input the license plate region image and temporal features of the sequence frame into the second slow channel network and the second fast channel network of the trained license plate recognition model for license plate recognition, so as to obtain the license plate recognition result and temporal features of the sequence frame; The output module is used to output the license plate recognition results of the video stream to be recognized based on the license plate recognition results of the sequence frames and the temporal features.

7. A computing device, characterized in that: The device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor performing a license plate recognition method based on a video stream as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, Includes a computer program that stores a license plate recognition method based on a video stream, as described in any one of claims 1-5, which can be loaded by a processor and executed.

Images