Intelligent vehicle monitoring system based on deep twinning
By introducing deep twin technology and multi-objective tracking model into the vehicle monitoring system, combining high- and low-level cameras and license plate recognition systems, the problems of inefficiency and limited application scenarios of traditional vehicle monitoring systems are solved, and efficient and accurate vehicles are achieved open monitoring.
Patent Information
- Application Number
- CN202411509629.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-28
- Publication Date
- 2025-02-14
AI Technical Summary
Traditional vehicle monitoring systems rely on manual monitoring, are inefficient and only suitable for closed management, limiting their application scenarios.
The intelligent vehicle monitoring system based on depth twins is adopted, combining high-position cameras, low-position cameras, computers, object detection models, multi-target tracking models and license plate recognition systems to realize open management and real-time monitoring of vehicles.
Real-time monitoring and identification of vehicles is realized, monitoring efficiency and accuracy are improved, and vehicles can be effectively managed in an open environment, covering a wider monitoring area and reducing blind spots.
Smart Images
Figure CN119449996A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision and artificial intelligence technology, and in particular to a vehicle intelligent monitoring system based on deep twinning. Background Art
[0002] Intelligent transportation system, also known as intelligent transport system, is to effectively and comprehensively apply advanced science and technology (information technology, computer technology, data communication technology, sensor technology, electronic control technology, automatic control theory, operations research, artificial intelligence, etc.) to transportation, service control and vehicle manufacturing, strengthen the connection between vehicles, roads and users, and thus form a comprehensive transportation system that ensures safety, improves efficiency, improves the environment and saves energy. In the current intelligent transportation system, vehicle monitoring is an important part of ensuring road safety.
[0003] Traditional vehicle monitoring systems mainly rely on manual monitoring, which is inefficient and easily affected by subjective factors. They are only suitable for closed management of vehicles. Closed management refers to the use of barriers to restrict vehicles to a closed area where traffic is prohibited during daily vehicle management, by checking license plates when entering and exiting. In simple terms, the area where vehicles are managed cannot be open to the public, which greatly limits the application of vehicle monitoring systems. Closed management has caused vehicle monitoring systems to lose many application scenarios.
[0004] To this end, we propose a vehicle intelligent monitoring system based on deep twins. Summary of the invention
[0005] The present invention mainly solves the technical problems existing in the above-mentioned prior art and provides a vehicle intelligent monitoring system based on deep twinning.
[0006] In order to achieve the above-mentioned objectives, the present invention adopts the following technical scheme: a vehicle intelligent monitoring system based on deep twins, including a high-position camera, a low-position camera, a low-position computer, a high-position computer, a unity client, a target detection model, a multi-target tracking model and a license plate recognition system. The high-position camera is arranged in the area to be monitored, such as a street lamp, a pole or a wall of a high-rise building. The low-position camera is embedded in the entrance speed bump of the coverage area of the high-position camera. The high-position camera and the low-position camera are respectively connected to the high-position computer and the low-position computer through electrical signals, and the high-position computer is connected to the unity client through electrical signals.
[0007] Preferably, when setting up the camera, the image of the high-position camera under which the low-position camera is located and the position and direction of the camera when shooting the image are recorded. According to the direction of the license plate in the field of view of the low-position camera, the field of view angle of the camera device and the position and direction recorded in the previous step, the line of sight Ray between the camera and the license plate is generated. The line of sight Ray is simplified and represented by the ray abstracted as a point cluster composed of N continuous points with intervals. The point cluster is calculated with the latest predicted frame of the trajectory in the high-position camera to determine in which frame each point in the point cluster is located. The trajectory with the most points in the frame is the vehicle trajectory corresponding to the license plate.
[0008] Preferably, the low-level computer is responsible for license plate recognition, and the high-level computer is responsible for vehicle detection and tracking. The high-level computer and the low-level computer communicate through a socket, and the low-level computer sends information including the license plate and the direction and position of the sight line Ray that generates the license plate. After being received by the high-level computer, this information participates in license plate matching and trajectory calculation to achieve the association between the license plate and the trajectory. In this way, the high-level computer includes the trajectory containing the license plate information in the monitored area, and transmits these trajectories to the unity client in real time. The unity client generates hundreds of vehicle models in advance, and the unity client moves the vehicle model corresponding to the trajectory ID by identifying the trajectory ID and the latest position of the trajectory, and determines the direction of the vehicle model's front face based on the direction of the previous position.
[0009] Preferably, the target detection model includes a backbone network, a classification task and a regression task.
[0010] Preferably, the backbone network extracts image features of the input image, the backbone network refers to various deep neural networks, such as VGG Net, GoogLeNet or ResNet, and identifies the category branch of the target of interest in the image, i.e., the classification task, which is performed based on the features output by the backbone network, and the regression branch that marks the position of the target of interest in the image with a rectangular box, i.e., the regression task, which is also performed based on the features output by the backbone network. Generally, the classification task and the regression task are collectively referred to as the detection head of the model.
[0011] Preferably, the target detection model generates anchor boxes of different sizes and proportions based on each pixel on the feature map output by the feature extraction network to cover the target on the entire original image. However, the target to be detected usually accounts for a small proportion of the entire image, and many of the generated anchor boxes do not frame the target to be detected. At this time, the IoU, that is, the degree of overlap between the generated anchor boxes and the pre-marked ground truth box, is calculated. The larger the IoU value, the higher the degree of overlap. For training, all anchor boxes are divided into two categories, one is a positive sample, whose IoU value with the real target is greater than 0.5; the other is a negative sample, which does not overlap with any real target or has an IoU value with the real target of less than 0.3. Usually, the number of negative samples will far exceed the number of positive samples. Then, a certain proportion of the positive and negative samples are sampled and sent to the classification and regression networks to train the category information and position offset respectively, and the loss function is obtained. The picture is input during reasoning to finally obtain the prediction result.
[0012] Preferably, the multi-target tracking model adopts multi-target tracking based on Tracking by detection, and uses the inference results of the upstream detection model to be sent to the downstream for similarity matching, matching the predicted frames of the same instance between frames at different times, wherein the similarity measurement is generally an intersection-over-union ratio, feature similarity or a combination of multiple ones. When using IOU to match the same instance between two frames, it is generally not calculated using the current frame and the previous frame, but using the current frame and the next frame estimated by the previous frame. Generally, a Kalman filter is used to predict the next frame. If the new object after similarity matching successfully matches the target of the previous frame, it is added to the trajectory of the instance. If it does not match, it becomes a new trajectory. If the existing trajectory does not match the new detection frame, it will be destroyed.
[0013] Preferably, the license plate recognition system includes an image preprocessing module, a license plate positioning module, a character segmentation module and a character recognition module.
[0014] Preferably, the image preprocessing module performs grayscale, denoising and binarization processing on the input vehicle image to improve the effect of subsequent processing.
[0015] Preferably, the license plate positioning module determines the specific position of the license plate in the image by using color features, shape features or texture features in the image.
[0016] Preferably, the character segmentation module segments the located license plate area into individual characters in preparation for character recognition.
[0017] Preferably, the character recognition module uses optical character recognition technology to convert the segmented character image into computer-readable text information, namely the license plate number.
[0018] The present invention provides a vehicle intelligent monitoring system based on deep twinning, which has the following beneficial effects:
[0019] 1. This vehicle intelligent monitoring system based on deep twins integrates high-position cameras, low-position cameras, low-position computers, high-position computers, unity clients, target detection models, multi-target tracking models and license plate recognition systems. It can realize open management of vehicles without using props that restrict vehicle operation. Vehicle information can be obtained only by monitoring cameras. It can be deployed on simple public roads and sections without hindering the operation of vehicles in public sections. It realizes real-time monitoring and identification of vehicles and improves monitoring efficiency and accuracy.
[0020] 2. This vehicle intelligent monitoring system based on deep twinning, by setting up high-position cameras and low-position cameras, realizes the association between license plates and vehicles through the cooperation of high-position cameras and thus obtains vehicle information and manages vehicles. It can also cover a wider monitoring area and reduce monitoring blind spots.
[0021] 3. This vehicle intelligent monitoring system based on deep twins, by setting up a license plate recognition system and a target detection model, the combination of the license plate recognition system and the target detection model enables the system to quickly and accurately identify and track specific vehicles.
[0022] 4. This vehicle intelligent monitoring system based on deep twins sets up a multi-target tracking model. The application of the multi-target tracking model enables the system to track multiple vehicles at the same time, thereby improving the practicality of the system.
[0023] 5. This vehicle intelligent monitoring system based on deep twinning adopts deep twinning technology to enhance the generalization ability of the model and improve the accuracy of recognition and tracking. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 A schematic diagram of the license plate recognition, vehicle detection and tracking algorithm framework of the present invention;
[0025] Figure 2 It is a schematic diagram of the multi-eye license plate matching algorithm flow of the present invention;
[0026] Figure 3 It is a schematic diagram of the license plate matching and vehicle tracking algorithm of the present invention. DETAILED DESCRIPTION
[0027] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings described below are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.
[0028] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.
[0029] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.
[0030] In the description of the embodiments of the present invention, it should be noted that the terms "center", "upper", "lower", "inner", "outer", "side", etc. indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the invented product is usually placed when in use, which is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.
[0031] In the description of the embodiments of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the embodiments of the present invention can be understood according to specific circumstances.
[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0033] Embodiment 1: A vehicle intelligent monitoring system based on deep twinning, such as Figure 1As shown, it includes a high-position camera, a low-position camera, a low-position computer, a high-position computer, a unity client, a target detection model, a multi-target tracking model and a license plate recognition system. The high-position camera is set in the area to be monitored, such as a street lamp, a pole or a wall of a high-rise building. The low-position camera is embedded in the entrance speed bump of the coverage area of the high-position camera. The high-position camera and the low-position camera are respectively connected to the high-position computer and the low-position computer through electrical signals. The high-position computer is connected to the unity client through electrical signals. The low-position computer is responsible for license plate recognition. The high-position computer is responsible for vehicle detection and tracking. The high-position computer and the low-position computer communicate through sockets. The low-position computer sends a video containing a license plate and a generated license plate. The direction and position of the line Ray. After being received by the high-level computer, this information is involved in license plate matching and trajectory calculation to realize the association between license plates and trajectories. In this way, the high-level computer contains the trajectory containing license plate information in the monitoring area and transmits these trajectories to the unity client in real time. The unity client generates hundreds of vehicle models in advance. The unity client moves the vehicle model corresponding to the trajectory ID by identifying the trajectory ID and the latest position of the trajectory, and determines the direction of the vehicle model based on the direction of the previous position. The target detection model includes the backbone network, classification tasks and regression tasks. The backbone network extracts the image features of the input image. The backbone network refers to various deep neural networks, such as VGG Net, GoogLeNet or ResNet, identify the category branch of the target of interest in the image, that is, the classification task, which is based on the features output by the backbone network. The regression branch that marks the position of the target of interest in the image with a rectangular box, that is, the regression task, is also based on the features output by the backbone network. Generally, the classification task and the regression task are collectively referred to as the detection head of the model. The target detection model will generate anchor boxes of different sizes and proportions based on each pixel on the feature map output by the feature extraction network to cover the target on the entire original image. However, the target to be detected usually accounts for a small proportion of the entire image, and many of the generated anchor boxes do not frame the target to be detected. At this time, the IoU, that is, the overlap between the generated anchor boxes and the pre-marked ground truth box, is calculated. The larger the IoU value, the higher the overlap. For training, all anchor boxes are divided into two categories, one is positive samples, whose IoU value with the real target is greater than 0.5; the other is negative samples, which do not overlap with any real target or have an IoU value with the real target less than 0.3. Usually, the number of negative samples will far exceed the number of positive samples. Then, a certain proportion of the positive and negative samples are sampled and sent to the classification and regression networks to train the category information and position offset respectively, and the loss function is obtained. The image is input during inference to finally obtain the prediction result. The multi-target tracking model adopts the Trackingby Multi-target tracking of detection, using the inference results of the upstream detection model to send to the downstream for similarity matching, matching the predicted frames of the same instance between frames at different times, where the similarity measurement is generally the intersection-union ratio, feature similarity or multiple combinations. When using IOU to match the same instance between two frames, it is generally not calculated on the current frame and the previous frame, but on the current frame and the next frame estimated by the previous frame. Generally, Kalman filtering is used to predict the next frame. If the new object after similarity matching successfully matches the target of the previous frame, it will be added to the trajectory of the instance. If it does not match, it will become a new trajectory. If the existing trajectory does not match the new detection frame, it will be destroyed. The license plate recognition system includes an image preprocessing module, a license plate positioning module, a character segmentation module and a character recognition module. The image preprocessing module will grayscale and denoise the input vehicle image. and binarization processing to improve the effect of subsequent processing. The license plate positioning module uses the color features, shape features or texture features in the image to determine the specific position of the license plate in the image. The character segmentation module segments the located license plate area into individual characters to prepare for character recognition. The character recognition module uses optical character recognition technology to convert the segmented character image into computer-readable text information, namely the license plate number. By integrating high-position cameras, low-position cameras, low-position computers, high-position computers, unity clients, target detection models, multi-target tracking models and license plate recognition systems, it is possible to achieve open management of vehicles without using props that restrict vehicle operation. Vehicle information can be obtained only by monitoring cameras. It can be deployed on simple public roads and sections without hindering the operation of vehicles in public sections. It realizes real-time monitoring and recognition of vehicles and improves monitoring efficiency and accuracy. .
[0034] Embodiment 2: Based on Embodiment 1, Figure 2As shown, when setting up the camera, record which high-position camera image the low-position camera is under and the position and direction of the camera shooting the image. Generate the sight line Ray between the camera and the license plate according to the direction of the license plate in the field of view of the low-position camera, the field of view angle of the camera device, and the position and direction recorded in the previous step. Simplify the sight line Ray and abstract the ray into a point cluster composed of N continuous points with intervals. Calculate the point cluster and the latest predicted frame of the trajectory in the high-position camera to determine which frame each point in the point cluster is in. The trajectory with the most points in the frame is the vehicle trajectory corresponding to the license plate. The low-position computer is responsible for license plate recognition, and the high-position computer is responsible for vehicle detection and tracking. The low-position computer sends the direction and position of the sight line Ray containing the license plate and the generated license plate. After the high-level computer receives this information, it participates in license plate matching and trajectory calculation to realize the association between license plates and trajectories. In this way, the high-level computer contains the trajectories containing license plate information in the monitoring area, and transmits these trajectories to the unity client in real time. The unity client generates hundreds of vehicle models in advance. The unity client identifies the trajectory ID and the latest position of the trajectory to move the vehicle model corresponding to the trajectory ID, and determines the direction of the vehicle model's front end based on the direction of the previous position. By setting up high-level cameras and low-level cameras, and through the cooperation of high-level cameras and low-level cameras, the license plate and vehicle are associated to obtain vehicle information and manage vehicles. It can also cover a wider monitoring area and reduce monitoring blind spots.
[0035] Embodiment 3: Based on Embodiment 1 and Embodiment 2, Figure 3As shown in the figure, the target detection model includes a backbone network, a classification task and a regression task. The backbone network extracts image features of the input image. The backbone network refers to various deep neural networks, such as VGG Net, GoogLeNet or ResNet. The category branch that identifies the target of interest in the image is the classification task. The classification task is based on the features output by the backbone network. The regression branch that marks the position of the target of interest in the image with a rectangular box is the regression task. The regression task is also based on the features output by the backbone network. Generally, the classification task and the regression task are collectively referred to as the detection head of the model. The target detection model will generate anchor boxes of different sizes and proportions based on each pixel on the feature map output by the feature extraction network to cover the targets on the entire original image. However, the targets to be detected usually do not account for a high proportion of the entire image, and many of the generated anchor boxes do not frame the targets to be detected. At this time, the IoU, that is, the degree of overlap between the generated anchor boxes and the pre-marked ground truth box, will be calculated. The larger the IoU value, the higher the degree of overlap. For training, all anchor boxes are Boxes are divided into two categories, one is positive samples, whose IoU value with the real target is greater than 0.5; the other is negative samples, which do not overlap with any real target or have an IoU value with the real target less than 0.3. Usually, the number of negative samples will far exceed the number of positive samples. Then a certain proportion of the positive and negative samples are sampled and sent to the classification and regression networks to train the category information and position offset respectively to obtain the loss function. The picture is input during inference to finally obtain the prediction result. By setting up the license plate recognition system and the target detection model, the combination of the license plate recognition system and the target detection model enables the system to quickly and accurately identify and track specific vehicles.
[0036] Embodiment 4: Based on Embodiment 1, Embodiment 2 and Embodiment 3, Figure 3 As shown in the figure, the multi-target tracking model adopts multi-target tracking based on Tracking by detection, and uses the inference results of the upstream detection model to send to the downstream for similarity matching, matching the prediction frames of the same instance between frames at different times, where the similarity measurement is generally the intersection-union ratio, feature similarity or multiple combinations. When using IOU to match the same instance between two frames, it is generally not calculated by the current frame and the previous frame, but by the current frame and the next frame estimated by the previous frame. Generally, Kalman filtering is used to predict the next frame. If the new object after similarity matching successfully matches the target of the previous frame, it will be added to the trajectory of the instance. If it does not match, it will become a new trajectory. If the existing trajectory does not match the new detection frame, it will be destroyed. By setting up a multi-target tracking model and applying the multi-target tracking model, the system can track multiple vehicles at the same time, thereby improving the practicality of the system.
[0037] Embodiment 5: Based on Embodiment 1, Embodiment 2, Embodiment 3 and Embodiment 4, Figure 3 As shown, the license plate recognition system includes an image preprocessing module, a license plate positioning module, a character segmentation module and a character recognition module. The image preprocessing module will grayscale, denoise and binarize the input vehicle image to improve the effect of subsequent processing. The license plate positioning module uses the color features, shape features or texture features in the image to determine the specific position of the license plate in the image. The character segmentation module divides the located license plate area into single characters in preparation for character recognition. The character recognition module uses optical character recognition technology to convert the segmented character image into computer-readable text information, namely the license plate number. By adopting deep twin technology, the generalization ability of the model is enhanced and the accuracy of recognition and tracking is improved.
[0038] The working principle of the present invention is as follows: the high-position camera is responsible for capturing high-definition images of vehicles and transmitting the image data to the low-position computer. The low-position computer performs preliminary processing on the images, such as compression and format conversion, and then sends the processed images to the high-position computer. The high-position computer further processes the image data and executes the algorithms of the target detection model and the multi-target tracking model to identify and track the vehicles. The license plate recognition system, under the control of the high-position computer, performs license plate positioning and character recognition on the detected vehicles and finally obtains the license plate number. This information is then transmitted to the unity client in real time. The client updates the display status of the vehicle model according to the trajectory ID and the location information of the vehicle model to ensure that the front direction of the vehicle model is consistent with the actual vehicle. In this way, the system can monitor the vehicle's running trajectory in real time and accurately identify vehicle information, providing strong technical support for intelligent traffic management.
[0039] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.
Claims
1. A vehicle intelligent monitoring system based on deep twin, characterized in that: It includes a high-position camera, a low-position camera, a low-position computer, a high-position computer, a unity client, a target detection model, a multi-target tracking model and a license plate recognition system. The high-position camera is set at the area to be monitored, such as a street lamp, a pole or a wall of a high-rise building, and the low-position camera is embedded at the entrance speed bump of the coverage area of the high-position camera.
2. The vehicle intelligent monitoring system based on deep twinning according to claim 1 is characterized in that: The high-position camera and the low-position camera are respectively connected to the high-position computer and the low-position computer via electrical signals.
3. The vehicle intelligent monitoring system based on deep twinning according to claim 1 is characterized in that: The high-level computer is connected to the unity client via electrical signals.
4. The vehicle intelligent monitoring system based on deep twinning according to claim 1 is characterized in that: The high-order computer and the low-order computer communicate via a socket.
5. The vehicle intelligent monitoring system based on deep twinning according to claim 1 is characterized in that: The target detection model includes a backbone network, a classification task and a regression task.
6. The vehicle intelligent monitoring system based on deep twinning according to claim 1 is characterized in that: The multi-target tracking model adopts multi-target tracking based on Tracking by detection.
7. The vehicle intelligent monitoring system based on deep twinning according to claim 1 is characterized in that: The license plate recognition system includes an image preprocessing module, a license plate positioning module, a character segmentation module and a character recognition module.