License plate scanning vehicle moving control method and system

Abstract

The invention provides a license plate scanning vehicle moving control method and system, and the method comprises the following steps: carrying a camera on a license plate, collecting a video of a positive region of the license plate through the camera, associating the video with vehicle information, and building a historical database; performing preprocessing and data calibration on a historical video, and establishing a sample feature set; constructing a target detection model, and inputting the sample feature set into the target detection model for iterative training to obtain a trained target detection model for outputting a mobile phone and face bounding box; extracting the attitude angle and relative distance between the mobile phone and the license plate, scanning duration, face orientation angle and center offset distance characteristics from the mobile phone and face bounding box, and calculating the confidence of the mobile phone to scan the license plate according to the extracted characteristics; the scanning action is accurately judged by comparing with a preset value; and finally, after the scanning action is confirmed, information is uploaded in time and a vehicle owner is notified to move the vehicle, so that the efficiency and accuracy of license plate scanning and vehicle moving are improved, and meanwhile, user privacy leakage is avoided.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of automobiles, in particular to a license plate scanning and car moving control method and system. BACKGROUND

[0002] With the rapid development of the automobile industry, the governments at all levels have proposed the policy of promoting the automobile industry by replacing old cars with new ones, but the phenomenon of easy car purchase and difficult parking is still becoming increasingly obvious; the phenomenon of random parking in many parking spaces is caused by the fact that the number of vehicles is more than the number of parking spaces; and the phenomenon of road and community congestion and random road occupation is also becoming increasingly serious, so a license plate scanning and car moving control method and system are needed.

[0003] A method for notifying a car owner to move a car is disclosed in CN111539274B, when a user encounters the situation that someone else's vehicle occupies a parking space or blocks a road in life, the user can realize the function of notifying the car owner to move the car through a client, the client can obtain a vehicle image of the vehicle, and trigger two operations: identifying license plate information based on the vehicle image and determining a license plate attribution based on the license plate information of the vehicle, the server can send the license plate information of the vehicle to a notification device of the license plate attribution of the vehicle, the notification device can query the corresponding car owner contact information according to the received license plate information and send a car moving notification because the notification device stores the corresponding relationship between the license plate information of the license plate attribution and the car owner contact information.

[0004] In the existing scheme, the detection object is single, only the license plate information is concerned and the recognition of the scanning mobile phone and the face is neglected, misjudgment is prone to occur, abnormal processing is not considered, the scanning exception cannot be effectively responded to, and the accuracy and reliability of the car moving notification are seriously affected. SUMMARY

[0005] Therefore, the application provides a license plate scanning and car moving control method and system, the scanning mobile phone and the face are recognized, multi-dimensional features are extracted to calculate the scanning confidence, the scanning action is accurately determined by comparing with a preset value, information is uploaded in time according to the scanning action and the car owner is notified to move the car, and the accuracy and reliability of the car moving notification are improved.

[0006] The technical scheme of the application is as follows: in the first aspect, the application provides a license plate scanning and car moving control method, including the following steps: S1, a camera is mounted on a license plate, a video of a front area of the license plate is collected by the camera, and the video is associated with vehicle information to establish a historical database; S2, historical videos of the historical database are obtained, the historical videos are preprocessed and data calibrated, and a sample feature set is established; S3, constructing a target detection model based on the network trained model, inputting the sample feature set into the target detection model for iterative training to obtain the trained target detection model for outputting the mobile phone and face bounding box; S4, extracting the posture angle and relative distance of the mobile phone and license plate, scanning time length, face orientation angle and distance from the center offset feature from the mobile phone and face bounding box, and calculating the confidence of the mobile phone scanning the license plate according to the extracted features; S5, comparing the confidence of the mobile phone scanning the license plate with the preset scanning confidence, and when the confidence of the mobile phone scanning the license plate is greater than the preset scanning confidence, determining the scanning action, uploading the confirmation information to the server and pushing the message notification to the owner user, reminding the owner to move the car in time.

[0007] On the basis of the above technical scheme, preferably, the camera is mounted on the license plate in step S1, the video of the surrounding area of the license plate is collected by the camera, and the vehicle information is associated to establish a historical database, including the following sub-steps: S11, installing a camera on the front and rear license plates of the vehicle, and collecting the video of the front area of the license plate through the camera; S12, integrating a positioning module and a communication module in the vehicle, acquiring vehicle positioning information and vehicle configuration information in real time according to the positioning module and the communication module, the positioning module and the communication module being in communication connection with the vehicle central control system, the positioning module being used for acquiring the vehicle positioning information, and the communication module being used for acquiring the vehicle configuration information, the vehicle configuration information including vehicle type, license plate number, color, owner name and phone number; S13, associating the video with the vehicle positioning information and the vehicle configuration information according to time sequence to establish a historical database.

[0008] On the basis of the above technical scheme, preferably, the historical video of the historical database is obtained in step S2, and the historical video is preprocessed and data calibrated to establish a sample feature set, including the following sub-steps: S21, obtaining historical videos covering different scenes and angles in the historical database; S22, extracting key frame images of the historical video according to fixed time intervals, and performing geometric correction, image enhancement and size standardization processing on the key frame images to obtain standard enhanced images; S23, segmenting the mobile phone and face area of the standard enhanced image by using a pre-trained segmentation model, outputting a mask, a bounding box and a confidence as calibration data, and associating the calibration data with the corresponding standard enhanced image; S24, pre-setting a confidence threshold, traversing all calibration data, screening calibration data with a confidence greater than the pre-set confidence threshold to obtain calibrated image data, and integrating and constructing a sample feature set.

[0009] Based on the above technical scheme, preferably, the step S3 comprises the following sub-steps: The target detection model is constructed based on a YOLOv5 basic network structure; the target detection model comprises an input layer, a backbone network, a bidirectional feature fusion layer and a detection head, wherein, The input layer receives the calibrated image data, performs normalization and channel adjustment; The backbone network comprises a plurality of CSP modules and SE attention modules, the first and last of the plurality of CSP modules are connected in turn, and the plurality of SE attention modules are respectively arranged between adjacent two CSP modules, for extracting local features of image regions, and the input layer is connected with the input end of the first CSP module; The bidirectional feature fusion layer comprises a top-down sampling module, a bottom-up sampling module and a plurality of CBAM modules, the top-down sampling module fuses high-level and low-level features through upsampling and concatenation, the bottom-up sampling module fuses low-level and high-level features through downsampling and concatenation, the top-down sampling module comprises a plurality of downsampling convolution layers, the bottom-up sampling module comprises a plurality of upsampling convolution layers, the first and last of the plurality of downsampling convolution layers and upsampling convolution layers are connected in turn, and the last downsampling convolution layer is connected with the first upsampling convolution layer, and the input end of each downsampling convolution layer is connected across two adjacent upsampling convolution layers for feature fusion, the output end of each SE attention module is connected across the corresponding downsampling convolution layer for feature fusion, and the output end of each upsampling convolution layer is connected with the corresponding CBAM module for focusing on the region around the license plate; The number of detection heads is multiple, the scales of the plurality of detection heads are different, the input ends of the plurality of detection heads are respectively connected with the output ends of the corresponding CBAM modules, and the mobile phone and face bounding boxes are outputted.

[0010] Based on the above technical scheme, preferably, in step S3, the sample feature set is inputted into the target detection model for iterative training to obtain a trained target detection model for outputting mobile phone and face bounding boxes, comprising the following sub-steps: S31, the sample feature set is expanded by using Mosaic data enhancement, and the model parameters are initialized; S32, the training parameters are set, the loss function is defined, and the Adam optimizer is used, the initial learning rate is 0.001, and the cosine annealing learning rate scheduling strategy is used for training; S33, the expanded sample feature set is inputted into the target detection model for repeated iterative training, the model parameters are updated according to the calculated loss value, until the model parameters reach convergence, and the trained target detection model is obtained; S34, the image data to be detected is inputted into the trained target detection model, and the mobile phone and face bounding boxes are outputted.

[0011] On the basis of the above technical solutions, preferably, in step S4, the attitude angle and relative distance of the mobile phone and the license plate, the scanning duration, the face orientation angle and the center offset distance feature are extracted from the mobile phone and the face bounding box, and the confidence of the mobile phone scanning the license plate is calculated according to the extracted features, including the following sub-steps: S41, an polar coordinate system with the center point of the license plate as the origin is established, and the radial distance and angle are obtained; S42, the center point coordinates of the mobile phone and the face are extracted from the mobile phone and the face bounding box respectively, and are converted into polar coordinates relative to the center point of the license plate, to obtain the mobile phone polar coordinates and the face polar coordinates; S43, PCA analysis is performed on the pixel region in the mobile phone bounding box, the principal component direction is found, the included angle between the principal component direction and the horizontal axis of the image is calculated, the mobile phone rotation angle is obtained, and the attitude angle of the mobile phone and the license plate is calculated according to the difference between the angle of the mobile phone polar coordinates and the mobile phone rotation angle; S44, the duration of the simultaneous existence of the mobile phone and the face bounding box in the interval continuous frames is obtained as the scanning duration, the radial distance of the mobile phone polar coordinates is obtained as the relative distance of the mobile phone, and the radial distance of the face polar coordinates is obtained as the center offset distance; S45, the image of the face bounding box is extracted, the pre-trained face key point detection model is used to identify the image of the face bounding box, the center point coordinates of the left eye and the right eye are extracted, and the center line vector is calculated by connecting the center points of the left eye and the right eye; S46, the included angle between the center line vector and the horizontal axis is calculated by using the inverse tangent function, and the face orientation angle is obtained; S47, the confidence of the mobile phone scanning the license plate is obtained by weighted calculation according to the attitude angle and relative distance of the mobile phone and the license plate, the scanning duration, the face orientation angle and the center offset distance feature.

[0012] On the basis of the above technical solutions, preferably, in step S5, the confidence of the mobile phone scanning the license plate is compared with the preset scanning confidence, and when the confidence of the mobile phone scanning the license plate is greater than the preset scanning confidence, it is determined as a scanning action, the confirmation information is uploaded to the server and is pushed to the vehicle owner user message notification, reminding the vehicle owner to move the vehicle in time, including the following sub-steps: S51, the preset scanning confidence, the attitude angle threshold range of the mobile phone and the license plate, the relative distance threshold range of the mobile phone and the license plate, the scanning duration threshold range, the face orientation angle threshold range and the center offset distance threshold range are set; The confidence of the mobile phone scanning the license plate is compared with the preset scanning confidence, and when the confidence of the mobile phone scanning the license plate is greater than the preset scanning confidence, it is determined as a scanning action; Otherwise, the attitude angle and relative distance of the mobile phone and the license plate, the scanning time length, the face orientation angle and the distance from the center offset distance are compared with the corresponding attitude angle threshold range of the mobile phone and the license plate, the relative distance threshold range of the mobile phone and the license plate, the scanning time length threshold range, the face orientation angle threshold range and the distance from the center offset distance threshold range in turn; If the attitude angle of the mobile phone and the license plate is not in the attitude angle threshold range of the mobile phone and the license plate, it is marked as a scanning posture anomaly, if the scanning time length is not in the scanning time length threshold range, it is marked as a scanning time that is too short, if the face orientation angle is not in the face orientation angle threshold range, it is marked as a face position anomaly, if the relative distance of the mobile phone and the license plate is not in the relative distance threshold range of the mobile phone and the license plate, it is marked as a distance anomaly, and if the face distance from the center offset distance is not in the face distance from the center offset distance threshold range, it is marked as a face not aligned with the license plate. S52, the features not in the corresponding threshold range are obtained, and auxiliary voice broadcast information is generated to remind pedestrians to re-scan; S53, according to the determined scanning action, confirmation information is uploaded to the server and pushed to the car owner user message notification to remind the car owner to move the car in time.

[0013] In a second aspect, the application also provides a license plate scanning and car moving control system, which is implemented by using the license plate scanning and car moving control method, and the system comprises: A database establishment module is configured to carry a camera on the license plate, collect videos of the front area of the license plate through the camera, associate the videos with vehicle information, and establish a historical database. A data processing module is configured to obtain historical videos of the historical database, pre-process and data calibrate the historical videos, and establish a sample feature set. An image detection module is configured to construct a target detection model based on a network training model, input the sample feature set into the target detection model for iterative training, obtain the trained target detection model, and output the mobile phone and face bounding boxes. A feature analysis module is configured to extract the attitude angle and relative distance of the mobile phone and the license plate, the scanning time length, the face orientation angle and the distance from the center offset distance features from the mobile phone frame and the face bounding box, and calculate the confidence of the mobile phone scanning the license plate according to the extracted features. A determination notification module is configured to compare the confidence of the mobile phone scanning the license plate with a preset scanning confidence, determine a scanning action when the confidence of the mobile phone scanning the license plate is greater than the preset scanning confidence, upload confirmation information to the server and push it to the car owner user message notification to remind the car owner to move the car in time.

[0014] In a third aspect, the present application also provides an electronic device, comprising at least one processor, at least one memory, a communication interface and a bus; wherein the processor, the memory and the communication interface complete the communication among each other through the bus; the memory stores a license plate scanning and vehicle moving control method program which can be executed by the processor, and the license plate scanning and vehicle moving control method program is configured to implement a license plate scanning and vehicle moving control method.

[0015] In a fourth aspect, the present application also provides a computer readable storage medium, wherein the storage medium stores a license plate scanning and vehicle moving control method program, and the license plate scanning and vehicle moving control method program is executed to implement a license plate scanning and vehicle moving control method.

[0016] The license plate scanning and vehicle moving control method and system of the present application have the following beneficial effects relative to the prior art: (1) The video is collected by the camera carried on the license plate and is associated with the vehicle information to construct a historical database, the historical video is preprocessed and data is calibrated, a target detection model is constructed and trained, the mobile phone and face boundary boxes are output, multi-dimensional features are extracted to calculate the scanning confidence, the scanning action is accurately determined by comparison with a preset value, and finally the information is uploaded in time after the scanning action is confirmed and the vehicle owner is notified to move the vehicle, so that the efficiency and accuracy of the license plate scanning and vehicle moving are improved, and the user privacy is avoided to be leaked; (2) The perception ability for small targets such as mobile phones and faces is improved by embedding SE attention modules and CBAM modules in the set backbone network and bidirectional feature fusion layer, the cross connection of the top-down and bottom-up sampling paths is used in the bidirectional feature fusion layer to realize the complementation of high-level and low-level semantic information, and a plurality of detection heads of different scales are used to cover the target detection requirements from close-up to long shot; (3) The plurality of features are calculated by weighted comprehensive calculation to obtain the confidence of the mobile phone scanning license plate, so that the effectiveness of the mobile phone scanning license plate can be more comprehensively and accurately determined, and the accuracy and reliability of the scanning action recognition are improved; (4) The scanning action can be more accurately determined to be effective or not by comparing the plurality of features with the preset threshold range, and when the scanning is abnormal, the passerby is reminded to rescan by voice broadcast, so that the scanning operation is reperformed to improve the success rate of scanning. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0018] Figure 1The flow chart of the license plate scanning and vehicle removal control method of the present application; Figure 2 The model structure diagram of the license plate scanning and vehicle removal control method of the present application. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0020] As Figures 1-2 shown in the first aspect, the present application provides a license plate scanning and vehicle removal control method, comprising the following steps: S1, a camera is mounted on the license plate, a video of the front area of the license plate is collected by the camera, and the video is associated with vehicle information to establish a historical database.

[0021] The step S1 comprises the following sub-steps: S11, a camera is mounted on the front and rear license plates of the vehicle, and a video of the front area of the license plate is collected by the camera; It should be noted that a wide-angle high-definition camera is embedded at the front and rear license plates of the vehicle, which supports low-illumination environment, has dustproof and waterproof functions, is suitable for rainy and snowy weather, and is connected to the vehicle power supply through the vehicle OBD interface, and has a hibernation mode.

[0022] S12, a positioning module and a communication module are integrated in the vehicle, vehicle positioning information and vehicle configuration information are obtained in real time according to the positioning module and the communication module, the positioning module and the communication module are both in communication connection with the vehicle central control system, the positioning module is used to obtain the vehicle positioning information, and the communication module is used to obtain the vehicle configuration information, the vehicle configuration information includes vehicle type, license plate number, color, vehicle owner's name and telephone number.

[0023] It should be noted that a GPS positioning module is used to obtain the vehicle positioning information in real time, and a 5G communication module and a CAN communication module are integrated, the 5G communication module and the CAN communication module are both in communication connection with the vehicle central control system, the 5G communication module realizes data transmission between the vehicle and the cloud server, and the CAN communication module is used to return the configuration information of the vehicle.

[0024] S13, the video is associated with the vehicle positioning information and the vehicle configuration information according to time sequence to establish a historical database.

[0025] It should be noted that the historical database with time sequence is constructed, which is convenient for managing the vehicle information of the user.

[0026] S2, acquire historical videos of the historical database, and pre-process and data calibrate the historical videos to establish a sample feature set.

[0027] Step S2 includes the following sub-steps: S21, acquire historical videos covering different scenes and angles in the historical database; It should be noted that the scene types are divided according to weather, light, and angle, and video clips are randomly extracted under each scene type to ensure uniform distribution of samples. Repetitive videos are removed through key frame similarity comparison, and the videos are segmented to extract key clips of people scanning license plates with mobile phones.

[0028] S22, key frame images of the historical videos are extracted at fixed time intervals, and the key frame images are geometrically corrected, image enhanced, and size standardized to obtain standard enhanced images.

[0029] It should be noted that 1 frame of key frame image of historical video is extracted per second, the license plate corner points are detected and corrected for inclined viewing angle through Harris operator to ensure that the license plate is horizontal; the CLAHE algorithm is used to enhance the dark details of the key frame image, and the noise of the image is removed through frequency domain filtering, and finally the enhanced key frame image is uniformly scaled to 512x512 pixels to maintain the aspect ratio.

[0030] S23, a pre-trained segmentation model is used to segment the standard enhanced image mobile phone and face area, and the mask, bounding box and confidence are output as calibration data, and are associated with the corresponding standard enhanced image; It should be noted that the pre-trained Mask R-CNN segmentation model is used to segment the standard enhanced image mobile phone and face area, which is fine-tuned based on the COCO dataset, adds license plate scanning scene data, mobile phone and face annotation, and associates the calibration data with the standard enhanced image in the form of key-value pairs.

[0031] S24, a confidence threshold is pre-set, all calibration data are traversed, calibration data with a confidence greater than the pre-set confidence threshold are screened, calibrated image data are obtained, and a sample feature set is integrated and constructed.

[0032] It should be noted that if the confidence of the mobile phone is >0.85 and the confidence of the face is >0.85, the calibrated image data is retained, and the retained image data is integrated into a sample feature set.

[0033] In this embodiment, through multi-dimensional scene sample coverage and high-precision data calibration, reliable data support is provided for subsequent target detection, and the detection accuracy is effectively improved.

[0034] S3, constructing a target detection model based on the network trained model, inputting the sample feature set into the target detection model for iterative training to obtain a trained target detection model for outputting a mobile phone and a face bounding box.

[0035] In this embodiment, the target detection model is constructed based on a YOLOv5 basic network structure; the target detection model comprises an input layer, a backbone network, a bidirectional feature fusion layer and a detection head, wherein, The input layer receives the calibrated image data, performs normalization and channel adjustment; The backbone network comprises a plurality of CSP modules and SE attention modules, the first and last of the plurality of CSP modules are connected in turn, and the plurality of SE attention modules are respectively inserted between adjacent two CSP modules, for extracting image region local features, and the input layer is connected with the input end of the first CSP module; The bidirectional feature fusion layer comprises a top-down sampling module, a bottom-up sampling module and a plurality of CBAM modules, the top-down sampling module fuses high-level and low-level features through upsampling and splicing, the bottom-up sampling module fuses low-level and high-level features through downsampling and splicing, the top-down sampling module comprises a plurality of down-sampling convolution layers, the bottom-up sampling module comprises a plurality of up-sampling convolution layers, the first and last of the plurality of down-sampling convolution layers and up-sampling convolution layers are connected in turn, and the last down-sampling convolution layer is connected with the first up-sampling convolution layer, and the input end of each down-sampling convolution layer is connected across two adjacent up-sampling convolution layers for feature fusion, the output end of each SE attention module is connected across the corresponding down-sampling convolution layer for feature fusion, and the output end of each up-sampling convolution layer is connected with the corresponding CBAM module for focusing on the region around the license plate. The number of detection heads is a plurality, the scales of the plurality of detection heads are different, the input ends of the plurality of detection heads are respectively connected with the output ends of the corresponding CBAM modules, and the boundary box and the target confidence are output.

[0036] It should be noted that the SE attention module and the CBAM module are embedded in the backbone network and the bidirectional feature fusion layer to improve the perception ability of small targets such as mobile phones and faces, the bidirectional feature fusion layer realizes the complementation of high-level and low-level semantic information through the cross connection of the top-down and bottom-up sampling paths, and a plurality of detection heads of different scales are used to cover the target detection requirements from close-up to long shot.

[0037] In addition, the SE attention module is inserted after each CSP module to dynamically adjust the channel weight, so that the model focuses on the key area of the mobile phone screen and the face contour, improves the small target recall rate, and the calculation expression is:

[0038] In the formula, GAP() is a global average pooling, deltais a RELU activation function, and σ is Sigmoid is an activation function, W1 and W2 are parameters of the fully connected layer, X is an input feature map, and X' is an adjusted feature map.

[0039] In addition, a CBAM is connected after each up-sampling convolution layer in the bidirectional feature fusion layer, and the surrounding area of the license plate is focused through the channel attention and spatial attention dual mechanisms, wherein, Channel attention:

[0040] In the formula, M c (F) is a channel attention weight for weighting each channel of the original feature map, F is an input feature map, AvgPool(F) is a global average pooling operation on the input feature map F, MaxPool(F) is a global maximum pooling operation on the input feature map F, MLP is a multi-layer perceptron, and σ is Sigmoid an activation function; Spatial attention:

[0041] In the formula, M s (F) is a spatial attention weight for weighting the spatial position of the original feature map, F is an input feature map, AvgPool(F) is a global average pooling, MaxPool(F) is a global maximum pooling, Conv is a convolution operation, and [AvgPool(F); MaxPool(F)] is a concatenation of the results of the global average pooling and the global maximum pooling in the channel dimension.

[0042] Step S3 includes the following sub-steps: Based on the YOLOv5 basic network structure, a target detection model is constructed; S31, Mosaic data augmentation is used to expand the sample feature set, and the model parameters are initialized; It should be noted that the Mosaic data augmentation randomly crops four different training images to form a new image, and adjusts the corresponding target bounding box coordinates, which can enrich the background and context information of the training data and increase the diversity of the data.

[0043] S32, set the training parameters, define the loss function, and use the Adam optimizer with an initial learning rate of 0.001, and use the cosine annealing learning rate scheduling strategy for training; It should be noted that the loss function includes a bounding box regression loss, a confidence loss and a classification loss, the bounding box regression loss is a CIoU loss, which comprehensively considers the overlapping area, center point distance and aspect ratio of the bounding box, and can more accurately measure the difference between the predicted bounding box and the real bounding box; the confidence loss and the classification loss are binary cross entropy losses.

[0044] S33, input the expanded sample feature set into the target detection model for repeated iteration training, update the model parameters according to the calculated loss value, until the model parameters reach convergence, and obtain the trained target detection model; S34, input the image data to be detected into the trained target detection model, and output the bounding box of the mobile phone and the face.

[0045] S4, from the mobile phone and face bounding box, extract the posture angle and relative distance of the mobile phone and the license plate, the scanning time length, the face orientation angle and the distance offset distance features, and calculate the confidence of the mobile phone scanning the license plate according to the extracted features.

[0046] Step S4 includes the following sub-steps: S41, establish a polar coordinate system with the license plate center point as the origin, and obtain the radial distance and angle; S42, extract the center point coordinates of the mobile phone and the face from the mobile phone and face bounding box respectively, and convert them into polar coordinates relative to the license plate center point to obtain the mobile phone polar coordinates and the face polar coordinates; The license plate center point coordinates are O ( x 0, y 0), the mobile phone bounding box center point coordinates are x p , y p ), and the face bounding box center point coordinates are x f , y f ), the center point coordinates of the mobile phone and the face are converted into polar coordinates relative to the license plate center point to obtain the mobile phone polar coordinates r p , theta p ) and the face polar coordinates r f , theta f ), the expression is:

[0047] In the formula, r p is the radial distance of the mobile phone relative to the license plate center point, theta pthe angle of the mobile phone relative to the center point of the license plate; r f the radial distance of the face relative to the center point of the license plate, theta f the angle of the face relative to the center point of the license plate.

[0048] The embodiment can determine the positions of the mobile phone and the face in the coordinate system with the license plate as the center by extracting the center point coordinates of the mobile phone and the face bounding box and converting them into polar coordinates, which is helpful for subsequent analysis of the relative positional relationship between the mobile phone and the face and the license plate.

[0049] S43, performing PCA analysis on the pixel region in the mobile phone bounding box to find the principal component direction, calculating the included angle between the principal component direction and the horizontal axis of the image to obtain the mobile phone rotation angle, and calculating the pose angle of the mobile phone and the license plate according to the difference between the angle of the mobile phone polar coordinates and the mobile phone rotation angle; It should be noted that the pixel matrix S in the mobile phone bounding box is centered S c =S-μ, μ is the mean vector, and the covariance matrix is calculated according to the pixel matrix after the centering processing.

[0050] Eigenvalue decomposition is performed on the covariance matrix to find the eigenvector corresponding to the maximum eigenvalue, and the direction of the vector is the principal component direction.

[0051] Let the included angle between the principal component direction vector and the horizontal axis be α, which is calculated through the vector dot product formula.

[0052] In the formula, is the principal component direction vector, is the horizontal axis direction vector.

[0053] Then the pose angle of the mobile phone and the license plate β =| theta p - a |.

[0054] S44, obtaining the duration of the simultaneous existence of the mobile phone and the face bounding box in the interval continuous frames as the scanning time length, taking the radial distance of the mobile phone polar coordinates as the mobile phone relative distance, and taking the radial distance of the face polar coordinates as the center offset distance.

[0055] It should be noted that the number of frames in which the mobile phone and the face bounding box exist simultaneously in the continuous frames is counted N， and multiplied by the frame interval time Δt, that is, t=N×Δt; the mobile phone relative distance d p =r p ; off-center offset distance d o = r f .

[0056] The scan duration in the embodiment reflects the duration of the user's scanning operation, and the phone relative distance and the off-center offset distance respectively describe the distance relationship between the phone and the face and the center of the license plate. The two dimensions of time and space can provide a basis for judging the scanning action. Longer scan duration and appropriate relative distance are conducive to confirming effective scanning action.

[0057] S45, extracts the image of the face bounding box, identifies the image of the face bounding box based on a pre-trained face key point detection model, extracts the center point coordinates of the left eye and the right eye, and connects the center points of the left eye and the right eye to calculate a connecting vector; S46, calculates the included angle between the connecting vector and the horizontal axis by using an inverse tangent function to obtain a face orientation angle; It should be noted that the center point coordinates of the left and right eyes are extracted by the pre-trained face key point detection model, and the connecting vector is calculated. The positional relationship of the left and right eyes can reflect the orientation information of the face. By calculating the connecting vector, basic data can be provided for accurately judging the orientation of the face. The face orientation angle is obtained by calculating the included angle between the connecting vector and the horizontal axis by using an inverse tangent function. The face orientation angle can directly represent the orientation of the face.

[0058] S47, according to the posture angle and the relative distance of the phone and the license plate, the scan duration, the face orientation angle, and the off-center offset distance, weighted calculation is performed to obtain the confidence of the phone scanning the license plate.

[0059] The calculation expression is: Z= w 1 f 1(β)+ w 2 f 2( d p )+ w 3 f 3(t)+ w 4 f 4(γ)+ w 5 f 5( d o ) In the formula, w 1 is the weight of the posture angle of the phone and the license plate, f 1 is the normalization function of the posture angle of the phone and the license plate, w 2 is the weight of the phone relative distance, f 2 is the normalization function of the phone relative distance,w 3 is a weight of the scanning duration, f 3 is a normalization function of the scanning duration, w 4 is a weight of the face orientation angle, f 4 is a normalization function of the face orientation angle, w 5 is a weight of the center offset distance, f 5 is a normalization function of the center offset distance.

[0060] In this embodiment, multiple features are integrated by weighted calculation to obtain the confidence of the mobile phone scanning the license plate, and the effectiveness of the mobile phone scanning the license plate can be more comprehensively and accurately judged by comprehensively considering multiple features, thereby improving the accuracy and reliability of the scanning action recognition.

[0061] S5, comparing the confidence of the mobile phone scanning the license plate with the preset scanning confidence, when the confidence of the mobile phone scanning the license plate is greater than the preset scanning confidence, determining that it is a scanning action, uploading the confirmation information to the server and pushing the message notification to the owner user, reminding the owner to move the car in time.

[0062] Step S5 includes the following sub-steps: S51, presetting the scanning confidence, the posture angle threshold range of the mobile phone and the license plate, the relative distance threshold range of the mobile phone and the license plate, the scanning duration threshold range, the face orientation angle threshold range, and the center offset distance threshold range; comparing the confidence of the mobile phone scanning the license plate with the preset scanning confidence, when the confidence of the mobile phone scanning the license plate is greater than the preset scanning confidence, determining that it is a scanning action; otherwise, comparing the posture angle and the relative distance of the mobile phone and the license plate, the scanning duration, the face orientation angle, and the center offset distance with the corresponding posture angle threshold range of the mobile phone and the license plate, the relative distance threshold range of the mobile phone and the license plate, the scanning duration threshold range, the face orientation angle threshold range, and the center offset distance threshold range in sequence; if the posture angle of the mobile phone and the license plate is not in the posture angle threshold range of the mobile phone and the license plate, it is marked as a scanning posture abnormality, if the scanning duration is not in the scanning duration threshold range, it is marked as a scanning time being too short, if the face orientation angle is not in the face orientation angle threshold range, it is marked as a face position abnormality, if the relative distance of the mobile phone and the license plate is not in the relative distance threshold range of the mobile phone and the license plate, it is marked as a distance abnormality, and if the face center offset distance is not in the face center offset distance threshold range, it is marked as a face not aligned with the license plate; S52, obtaining the features not in the corresponding threshold range, and generating auxiliary voice broadcast information to remind pedestrians to rescan; S53, according to the determined scanning action, uploading confirmation information to the server and pushing a message notification to the vehicle owner, reminding the vehicle owner to move the vehicle in time.

[0063] It should be noted that by presetting the threshold range and comparing multiple features, it can be more accurately determined whether the scanning action is effective, avoiding misjudgment. When the scanning is abnormal, the passerby is reminded to re-scan through voice broadcast, and the passerby is reminded that the scanning action is abnormal, so as to re-scan operation, improve the success rate of scanning; At the same time, the vehicle owner is notified to move the vehicle in time, which is convenient for the vehicle owner to handle in time.

[0064] In a second aspect, the present application also provides a license plate scanning and moving control system, which is realized by using the license plate scanning and moving control method. The system comprises: A database establishment module is configured to carry a camera on the license plate, collect videos of the front area of the license plate through the camera, and associate the videos with vehicle information to establish a historical database. A data processing module is configured to obtain historical videos of the historical database, pre-process and data calibrate the historical videos, and establish a sample feature set. An image detection module is configured to construct a target detection model based on a network training model, input the sample feature set into the target detection model for iterative training, obtain the trained target detection model, and output mobile phone and face bounding boxes. A feature analysis module is configured to extract the posture angle and relative distance of the mobile phone and the license plate, the scanning time, the face orientation angle, and the distance offset distance feature from the mobile phone frame and the face bounding box, and calculate the confidence of the mobile phone scanning the license plate according to the extracted features. A determination notification module is configured to compare the confidence of the mobile phone scanning the license plate with a preset scanning confidence, determine the scanning action when the confidence of the mobile phone scanning the license plate is greater than the preset scanning confidence, upload confirmation information to the server and push a message notification to the vehicle owner, and remind the vehicle owner to move the vehicle in time. It should be noted that the system corresponds to the above-mentioned license plate scanning and moving control method. All implementation manners in the above-mentioned method embodiments are applicable to the embodiments of the system, and the same technical effects can also be achieved.

[0065] Those skilled in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software mode depends on the specific application and design constraints of the technical solution. The professional technicians can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0066] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and the module described above can refer to the corresponding process in the foregoing method embodiment, and will not be repeated here.

[0067] In the embodiments provided by the present application, it should be understood that the disclosed system and method can be implemented in other ways. For example, the device embodiments described above are only schematic. The division of the units is only a logical function division. There can be another division during actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between the units can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or in other forms.

[0068] The units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, that is, can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0069] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit.

[0070] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the part of the present application that essentially contributes to the prior art or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk, and various program code storage media.

[0071] Moreover, it is pointed out that, in the system and method of the present application, obviously, the components or steps can be decomposed and / or recombined. These decompositions and / or recombination should be considered as equivalents of the present application. Also, the steps of performing the above series of processes can naturally be executed in time sequence according to the order of description, but do not necessarily have to be executed in time sequence. Some steps can be executed in parallel or independently of each other. It can be understood by those skilled in the art that all or any steps or components of the method and device of the present application can be implemented in hardware, firmware, software or a combination thereof in any computing device (including processors, storage media, etc.) or network of computing devices, using the basic programming skills of those skilled in the art upon reading the description of the present application.

[0072] Therefore, the object of the present application can also be achieved by running a program or a set of programs on any computing system. The computing system can be a commonly known general-purpose system. Therefore, the object of the present application can also be achieved only by providing a program product containing program code for implementing the method or device. That is, such a program product also constitutes the present application, and a storage medium storing such a program product also constitutes the present application. Obviously, the storage medium can be any commonly known storage medium or any storage medium developed in the future. It is also pointed out that, in the device and method of the present application, obviously, the components or steps can be decomposed and / or recombined. These decompositions and / or recombination should be considered as equivalents of the present application. Also, the steps of performing the above series of processes can naturally be executed in time sequence according to the order of description, but do not necessarily have to be executed in time sequence. Some steps can be executed in parallel or independently of each other.

[0073] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A license plate scanning boot control method, characterized by, Comprise the following steps: S1, the license plate is loaded with a camera, the video of the front area of the license plate is collected by the camera, and the vehicle information is associated to establish a historical database; S2, obtain the historical video of the historical database, and pre-process and data calibrate the historical video to establish a sample feature set; S3, a target detection model is constructed based on a network training model, the sample feature set is input into the target detection model for iterative training, and a trained target detection model is obtained, which is used for outputting mobile phone and face bounding boxes; S4, from the mobile phone and face bounding box, the posture angle and relative distance of the mobile phone and the license plate, the scanning time, the face orientation angle and the distance offset distance features are extracted, and the confidence of the mobile phone scanning the license plate is calculated according to the extracted features; S5, the confidence of the mobile phone scanning the license plate is compared with the preset scanning confidence, and when the confidence of the mobile phone scanning the license plate is greater than the preset scanning confidence, it is determined as a scanning action, the confirmation information is uploaded to the server and pushed to the vehicle owner user message notification, reminding the vehicle owner to move the vehicle in time.

2. The license plate scanning boot control method of claim 1, wherein: In step S1, the camera is mounted on the license plate, the video of the surrounding area of the license plate is collected by the camera, and the vehicle information is associated to establish a historical database, comprising the following sub-steps: S11, install a camera on the front and rear license plates of the vehicle, and collect the video of the front area of the license plate through the camera; S12, the vehicle integrates a positioning module and a communication module, and obtains vehicle positioning information and vehicle configuration information in real time according to the positioning module and the communication module; the positioning module and the communication module are both in communication connection with the vehicle central control system; the positioning module is used to obtain the vehicle positioning information, and the communication module is used to obtain the vehicle configuration information; the vehicle configuration information includes vehicle type, license plate number, color, vehicle owner's name and telephone number; S13, the video is associated with the vehicle positioning information and the vehicle configuration information according to time sequence to establish a historical database.

3. The license plate scanning boot control method of claim 2, wherein: In step S2, the historical video of the historical database is obtained, and the historical video is pre-processed and data calibrated to establish a sample feature set, comprising the following sub-steps: S21, obtain the historical video covering different scenes and angles in the historical database; S22, extract key frame images of the historical video at fixed time intervals, and perform geometric correction, image enhancement and size standardization processing on the key frame images to obtain standard enhanced images; S23, a pre-trained segmentation model is used to segment the mobile phone and face area of the standard enhanced image, and outputs a mask, a bounding box and a confidence as calibration data, and is associated with the corresponding standard enhanced image; S24, a confidence threshold is preset, all calibration data is traversed, calibration data with a confidence greater than the preset confidence threshold is screened, calibrated image data is obtained, and a sample feature set is integrated and constructed.

4. The license plate scanning boot control method of claim 3, wherein: In step S3, the target detection model is constructed based on the network training model, comprising the following sub-steps: The target detection model is constructed based on the YOLOv5 basic network structure; the target detection model comprises an input layer, a backbone network, a bidirectional feature fusion layer and a detection head, wherein, the input layer receives the calibrated image data, performs normalization and channel adjustment; The backbone network comprises a plurality of CSP modules and SE attention modules, the first and last of the plurality of CSP modules are connected in sequence, and the plurality of SE attention modules are respectively arranged between adjacent two CSP modules, for extracting local features of image regions and inputting layers connected with the input end of the first CSP module; The bidirectional feature fusion layer comprises a top-down sampling module, a bottom-up sampling module and a plurality of CBAM modules, the top-down sampling module fuses high-level and low-level features through upsampling and splicing, the bottom-up sampling module fuses low-level and high-level features through downsampling and splicing, the top-down sampling module comprises a plurality of down-sampling convolutional layers, the bottom-up sampling module comprises a plurality of up-sampling convolutional layers, the first and last of the plurality of down-sampling convolutional layers and up-sampling convolutional layers are connected in sequence, and the last down-sampling convolutional layer is connected with the first up-sampling convolutional layer, and the input end of each down-sampling convolutional layer is connected across two adjacent up-sampling convolutional layers for feature fusion, the output end of each SE attention module is connected across a corresponding down-sampling convolutional layer for feature fusion, and the output end of each up-sampling convolutional layer is connected with a corresponding CBAM module for focusing on the region around the license plate; The number of detection heads is a plurality, the scales of the plurality of detection heads are different, the input end of each detection head is connected with the output end of a corresponding CBAM module, and the mobile phone and face bounding boxes are outputted.

5. The license plate scanning boot control method of claim 4, wherein: In step S3, the sample feature set is inputted into the target detection model for iterative training to obtain the trained target detection model for outputting the mobile phone and face bounding boxes, comprising the following sub-steps: S31, the sample feature set is expanded by using Mosaic data enhancement, and the model parameters are initialized; S32, the training parameters are set, the loss function is defined, and the Adam optimizer is used, the initial learning rate is 0.001, and the cosine annealing learning rate scheduling strategy is used for training; S33, the expanded sample feature set is inputted into the target detection model for repeated iterative training, the model parameters are updated according to the calculated loss value, until the model parameters reach convergence, and the trained target detection model is obtained; S34, the image data to be detected is inputted into the trained target detection model, and the mobile phone and face bounding boxes are outputted.

6. The license plate scanning boot control method of claim 5, wherein: In step S4, the pose angle and relative distance between the mobile phone and the license plate, the scanning time length, the face orientation angle and the distance offset from the center are extracted from the mobile phone and face bounding boxes, the confidence of the mobile phone scanning the license plate is calculated according to the extracted features, comprising the following sub-steps: S41, an polar coordinate system with the center point of the license plate as the origin is established, and the radial distance and angle are obtained; S42, the center point coordinates of the mobile phone and face are extracted from the mobile phone and face bounding boxes respectively, and are converted into polar coordinates relative to the center point of the license plate to obtain the mobile phone polar coordinates and face polar coordinates; S43, PCA analysis is performed on the pixel region in the mobile phone bounding box, the principal component direction is found, the included angle between the principal component direction and the horizontal axis of the image is calculated to obtain the mobile phone rotation angle, and the pose angle between the mobile phone and the license plate is calculated according to the difference between the angle of the mobile phone polar coordinates and the mobile phone rotation angle. S44, obtain the duration of the simultaneous existence of the mobile phone and the face bounding box in the interval continuous frames as the scanning time length, take the radial distance of the mobile phone polar coordinate as the relative distance of the mobile phone, and take the radial distance of the face polar coordinate as the distance offset distance from the center; S45, extract the image of the face bounding box, identify the image of the face bounding box based on the pre-trained face key point detection model, extract the center point coordinates of the left eye and the right eye, and connect the center points of the left eye and the right eye to calculate the connecting vector; S46, calculate the angle between the connecting vector and the horizontal axis by using the inverse tangent function to obtain the face orientation angle; S47, obtain the confidence of the mobile phone scanning the license plate according to the posture angle and the relative distance of the mobile phone and the license plate, the scanning time length, the face orientation angle, and the distance offset distance feature weighted calculation.

7. The license plate scanning boot control method of claim 6, wherein: In step S5, the confidence of the mobile phone scanning the license plate is compared with the preset scanning confidence. When the confidence of the mobile phone scanning the license plate is greater than the preset scanning confidence, it is determined as a scanning action, the confirmation information is uploaded to the server and pushed to the car owner user message notification, reminding the car owner to move the car in time, including the following sub-steps: S51, preset scanning confidence, posture angle threshold range of mobile phone and license plate, relative distance threshold range of mobile phone and license plate, scanning time length threshold range, face orientation angle threshold range and distance offset distance threshold range; According to the confidence of the mobile phone scanning the license plate and the preset scanning confidence, when the confidence of the mobile phone scanning the license plate is greater than the preset scanning confidence, it is determined as a scanning action; Otherwise, the posture angle and the relative distance of the mobile phone and the license plate, the scanning time length, the face orientation angle, and the distance offset distance are compared with the corresponding posture angle threshold range of the mobile phone and the license plate, the relative distance threshold range of the mobile phone and the license plate, the scanning time length threshold range, the face orientation angle threshold range, and the distance offset distance threshold range, respectively; If the posture angle of the mobile phone and the license plate is not in the posture angle threshold range of the mobile phone and the license plate, it is marked as a scanning posture abnormality, if the scanning time length is not in the scanning time length threshold range, it is marked as a scanning time too short, if the face orientation angle is not in the face orientation angle threshold range, it is marked as a face position abnormality, if the relative distance of the mobile phone and the license plate is not in the relative distance threshold range of the mobile phone and the license plate, it is marked as a distance abnormality, and if the face distance offset distance is not in the face distance offset distance threshold range, it is marked as a face not aligned with the license plate; S52, obtain the features not in the corresponding threshold range, and generate auxiliary voice broadcast information to remind pedestrians to re-scan; S53, according to the determined scanning action, upload the confirmation information to the server and push it to the car owner user message notification to remind the car owner to move the car in time.

8. A license plate scanning boot control system implemented using the license plate scanning boot control method according to any one of claims 1 to 7, characterized by: The system comprises: A database establishment module is configured to carry a camera on a license plate, collect a video of a front area of the license plate through the camera, and associate the video with vehicle information to establish a historical database; A data processing module is configured to obtain historical videos of the historical database, pre-process and data calibrate the historical videos, and establish a sample feature set; An image detection module is configured to construct a target detection model based on a network training model, input a sample feature set into the target detection model for iterative training, and obtain a trained target detection model configured to output a mobile phone and a face bounding box; A feature analysis module is configured to extract a pose angle and a relative distance of the mobile phone and the license plate, a scanning time length, a face orientation angle, and a distance offset feature from the mobile phone frame and the face bounding box, and calculate a confidence level of the mobile phone scanning the license plate according to the extracted features. A determination notification module is configured to compare the confidence level of the mobile phone scanning the license plate with a preset scanning confidence level, determine a scanning action when the confidence level of the mobile phone scanning the license plate is greater than the preset scanning confidence level, upload confirmation information to a server and push a message notification to a vehicle owner user, and remind the vehicle owner to move the vehicle in time.

9. An electronic device, comprising: The device comprises at least one processor, at least one memory, a communication interface, and a bus; wherein the processor, the memory, and the communication interface communicate with each other through the bus; the memory stores a vehicle license plate scanning and moving control method program executable by the processor, and the vehicle license plate scanning and moving control method program is configured to implement the vehicle license plate scanning and moving control method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a vehicle license plate scanning and moving control method program, and the vehicle license plate scanning and moving control method program is implemented when executed to implement the vehicle license plate scanning and moving control method according to any one of claims 1 to 7.

Images