Illegal parking detection method, device and system and electronic equipment

Abstract

The invention provides an illegal parking detection method, device and system and electronic equipment, and belongs to the field of data processing, and the method comprises the steps: intercepting an aerial image from a patrol video stream currently pushed by an unmanned aerial vehicle; detecting the position coordinates of each to-be-detected vehicle in the aerial image; and taking the to-be-detected vehicle of which the position coordinate is matched with the preset no-parking area as a concerned vehicle, and generating illegal parking alarm information of the illegal parking vehicle under the condition that the current illegal parking duration of the concerned vehicle exceeds the illegal parking lower limit duration. In this way, the unmanned aerial vehicle is used for comprehensive inspection, the problem that a detection area is limited or a blind area exists is solved in a high-altitude high-angle shooting mode, the inspection video stream comprehensively covers the inspection area, and then tracking detection, illegal parking vehicle determination and illegal parking alarm information generation are performed on the aerial image intercepted in the real-time inspection video stream. Therefore, the illegal parking detection is more efficient and comprehensive, the detection accuracy is improved, and the inspection cost is reduced.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a method, device, system and electronic equipment for detecting illegal parking. Background Art

[0002] Vehicle parking supervision is of great significance in maintaining traffic order, ensuring safety, improving urban management efficiency and improving the quality of life of residents. In the current development of intelligent transportation systems, vehicle parking detection technology has become an important component, aiming to effectively manage urban road resources through technical means and improve road traffic efficiency and safety. However, the existing vehicle parking detection technology still faces many challenges, among which low efficiency is one of the prominent problems.

[0003] At present, illegal parking detection is usually carried out by human supervision or video surveillance. Among them, the human supervision method requires traffic management personnel to conduct regular inspections, which cannot fully cover all areas in real time, is inefficient and has high labor costs. The video surveillance method is to obtain real-time video of a specific area for remote personnel to view and check. Although this method can cover all areas, it is prone to blind spots and blind spots, and also requires manual judgment, making it difficult to achieve comprehensive detection and low accuracy.

[0004] Therefore, there is an urgent need for an efficient, highly accurate, comprehensive and low-cost illegal parking detection method. Summary of the invention

[0005] In view of this, an object of the present invention is to provide a method, device, system and electronic equipment for illegal parking detection, which greatly reduce the cost of illegal parking detection and improve the efficiency, accuracy and comprehensiveness of illegal parking detection.

[0006] In order to achieve the above purpose, the technical solution adopted by the embodiment of the present invention is as follows:

[0007] In a first aspect, the present invention provides a method for detecting illegal parking, which is applied to a cloud service device, wherein the cloud service device is communicatively connected to a drone, and the method comprises:

[0008] Intercepting an aerial image from the inspection video stream currently pushed by the drone; wherein the inspection video stream is obtained by the drone inspecting and photographing the inspection area according to a preset inspection route;

[0009] Detecting the aerial image to obtain vehicle information of the aerial image; wherein the vehicle information includes the position coordinates of each vehicle to be detected in the aerial image;

[0010] The vehicle to be tested whose position coordinates match the preset no-parking zone is regarded as a concerned vehicle, and the shooting time of the aerial image is counted as the latest illegal parking time of the concerned vehicle;

[0011] According to the latest illegal parking time, obtain the current illegal parking duration of each concerned vehicle;

[0012] The concerned vehicle whose current illegal parking duration exceeds the lower limit of illegal parking duration is regarded as an illegally parked vehicle, and illegal parking alarm information of the illegally parked vehicle is generated.

[0013] Optionally, the step of detecting the aerial image to obtain vehicle information of the aerial image includes:

[0014] Performing tracking detection on the aerial image to obtain a detection result; wherein the detection result includes a detection frame and an identification number of each vehicle to be detected;

[0015] For each of the vehicles to be tested, based on the pixel information of the detection frame, the position coordinates of the vehicle to be tested in the latitude and longitude coordinate system are converted.

[0016] Optionally, the step of performing tracking detection on the aerial image to obtain a detection result includes:

[0017] Using a preset target detector to generate a detection frame for each vehicle to be detected in the aerial image;

[0018] Retrieving the tracking frame predicted from the previous frame of the aerial image, respectively calculating the similarity between the detection frame and each of the tracking frames, and constructing a cost matrix according to the similarity;

[0019] The cost matrix is ​​optimized by using the Hungarian algorithm to obtain a matching result between the detection frame and the tracking frame; wherein the matching result includes the mismatched detection frame and the tracking frame, and the successfully matched detection frame and the tracking frame;

[0020] Using the identification number of the tracking frame that matches the detection frame as the identification number of the successfully matched detection frame, and updating the state of the successfully matched tracking frame to a confirmed state;

[0021] Deleting the mismatched tracking frame whose number of mismatches reaches the mismatch upper limit, using the mismatched detection frame as a new tracking frame, and assigning a unique identification number to the mismatched detection frame;

[0022] A preset Kalman filter is used to predict and update each of the tracking frames.

[0023] Optionally, the step of converting the position coordinates of the vehicle to be tested in a latitude and longitude coordinate system based on the pixel information of the detection frame includes:

[0024] Extracting core pixel coordinates of the vehicle to be tested from the detection frame;

[0025] The core pixel coordinates are transformed in combination with the position coordinates of the drone when acquiring the image to be tested, as well as the camera intrinsic parameters and camera extrinsic parameters of the drone, to obtain the position coordinates of the vehicle to be tested in the latitude and longitude coordinate system.

[0026] Optionally, the vehicle information also includes an identification number of each vehicle to be tested;

[0027] The step of obtaining the current illegal parking duration of each concerned vehicle according to the latest illegal parking time includes:

[0028] For each of the vehicles of interest, extracting the historical illegal parking times of the vehicle of interest based on the identification number of the vehicle of interest;

[0029] From each of the historical illegal parking moments, a time period that is continuous with the latest illegal parking moment of the concerned vehicle is extracted to obtain the current illegal parking duration of the concerned vehicle.

[0030] Optionally, the step of generating illegal parking alarm information of the illegally parked vehicle includes:

[0031] capturing a detection frame image of the illegally parked vehicle from the aerial image;

[0032] Inputting the detection frame image into a preset target detection model, and using the target detection model to intercept the license plate image of the illegally parked vehicle;

[0033] Inputting the license plate image into a preset license plate recognition network, and using the license plate recognition network to generate the license plate number of the illegally parked vehicle;

[0034] Based on the license plate number, illegal parking alarm information is generated.

[0035] Optionally, the cloud service device is also in communication connection with a terminal device, and after the step of generating the illegal parking alarm information of the illegally parked vehicle, the method further includes:

[0036] The preset database is queried to obtain the terminal device corresponding to the illegally parked vehicle, and the illegal parking alarm information is sent to the terminal device.

[0037] In a second aspect, the present invention provides a parking violation detection device, which is applied to a cloud service device, wherein the cloud service device is communicatively connected to a drone, and the parking violation detection device comprises an image acquisition module, an image detection module, a first timing module, a second timing module, and a parking violation alarm module;

[0038] The image acquisition module is used to capture the aerial image from the inspection video stream currently pushed by the drone; wherein the inspection video stream is obtained by the drone inspecting and photographing the inspection area according to a preset inspection route;

[0039] The image detection module is used to detect the aerial image to obtain vehicle information of the aerial image; wherein the vehicle information includes the position coordinates of each vehicle to be detected in the aerial image;

[0040] The first timing module is used to take the vehicle to be tested whose position coordinates match the preset no-parking area as a concerned vehicle, and count the shooting time of the aerial image as the latest illegal parking time of the concerned vehicle;

[0041] The second timing module is used to obtain the current illegal parking duration of each of the concerned vehicles according to the latest illegal parking time;

[0042] The illegal parking alarm module is used to treat the concerned vehicle whose current illegal parking duration exceeds the lower limit of illegal parking duration as an illegally parked vehicle, and generate illegal parking alarm information of the illegally parked vehicle.

[0043] In a third aspect, the present invention provides a parking violation detection system, including a drone and a cloud service device, wherein the drone is communicatively connected to the cloud service device;

[0044] The drone is used to perform inspection and filming of the inspection area according to a preset inspection route, obtain an inspection video stream, and push the inspection video stream to the cloud service device in real time;

[0045] The cloud service device is used to implement the illegal parking detection method described in the first aspect.

[0046] In a fourth aspect, the present invention provides an electronic device, comprising a processor and a memory, wherein the memory stores a computer program that can be executed by the processor, and the processor can execute the computer program to implement the illegal parking detection method as described in the first aspect.

[0047] In a fifth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the illegal parking detection method as described in the first aspect.

[0048] The illegal parking detection method, device, system and electronic device provided by the embodiments of the present invention include: intercepting an aerial image from the inspection video stream currently pushed by the drone, the inspection video stream is obtained by the drone inspecting the inspection area according to a preset inspection route; detecting the aerial image to obtain the position coordinates of each vehicle to be tested in the aerial image; taking the vehicle to be tested whose position coordinates match the preset no-parking area as a vehicle of interest, and counting the shooting time of the aerial image as the latest illegal parking time of the vehicle of interest; obtaining the current illegal parking duration of each vehicle of interest based on the latest illegal parking time; determining that the vehicle of interest whose current illegal parking duration exceeds the lower limit of the illegal parking duration is an illegally parked vehicle, and generating illegal parking alarm information for the illegally parked vehicle. In this way, drones are first used for comprehensive inspections, which solves the problem of limited detection areas or blind spots, allowing the inspection video stream to fully cover the inspection area. Then, by detecting the aerial images captured in the real-time inspection video stream, illegally parked vehicles in the image are identified and illegal parking alarm information is generated, making illegal parking detection more efficient and comprehensive, improving detection accuracy, and eliminating the need for manual inspections, greatly reducing inspection costs.

[0049] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0051] Figure 1 A schematic structural diagram of a parking violation detection system provided in an embodiment of the present invention is shown.

[0052] Figure 2 A block diagram of an electronic device provided by an embodiment of the present invention is shown.

[0053] Figure 3 One of the flow charts of the illegal parking detection method provided by an embodiment of the present invention is shown.

[0054] Figure 4 Shows Figure 3 Schematic diagram of the process of some sub-steps of step 12.

[0055] Figure 5 Shows Figure 4 A flowchart of some sub-steps of step 121 in FIG.

[0056] Figure 6Shows Figure 4 A flowchart of some sub-steps of step 122 in FIG.

[0057] Figure 7 Shows Figure 3 Schematic diagram of the process flow of some sub-steps of step 14.

[0058] Figure 8 Shows Figure 3 Schematic diagram of the process of some sub-steps of step 15.

[0059] Fig. 9 The second flowchart of the illegal parking detection method provided by the embodiment of the present invention is shown.

[0060] Fig.10 A block diagram of a parking violation detection device provided by an embodiment of the present invention is shown.

[0061] Explanation of the accompanying drawings: 10 - illegal parking detection system; 110 - drone; 120 - cloud service device; 130 - terminal device; 20 - electronic device; 210 - memory; 220 - processor; 230 - communication module; 30 - illegal parking detection device; 310 - image acquisition module; 320 - image detection module; 330 - first timing module; 340 - second timing module; 350 - illegal parking alarm module. DETAILED DESCRIPTION

[0062] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0063] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0064] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0065] The parking violation detection method provided by the embodiment of the present invention can be applied to Figure 1 In the illegal parking detection system 10 shown, the illegal parking detection system 10 includes a drone 110, a cloud service device 120 and multiple terminal devices 130. The cloud service device 120 can be communicated with the drone 110 and the multiple terminal devices 130 respectively through the network, radio, etc.

[0066] The drone 110 is used to perform inspection and shooting in the inspection area according to a preset inspection route, obtain a real-time inspection video stream, and push the inspection video stream to the cloud service device 120.

[0067] The cloud service device 120 is used to obtain illegal parking alarm information of illegally parked vehicles by using the illegal parking detection method provided by the embodiment of the present invention.

[0068] The terminal device 130 may be a management device of the traffic control department, or may be a wearable editing device such as a mobile phone, a laptop computer, a personal computer, a tablet computer, a sports watch, etc. of each driver.

[0069] The database of the cloud service device 120 can record the correspondence between each vehicle and the driver's terminal device 130, so that after the cloud service device 120 generates illegal parking alarm information for the illegally parked vehicle, the illegal parking alarm information will be sent to the management device, and at the same time, the illegal parking alarm information will be sent to the terminal device 130 of the driver corresponding to the illegally parked vehicle via text message or app information, so as to remind the traffic management department and the owner of the illegally parked vehicle of the illegal parking.

[0070] The cloud service device 120 may be an independent server or a server cluster.

[0071] Please refer to Figure 2 , is a block diagram of an electronic device 20, which may be Figure 1The cloud service device 120 in the parking violation detection system 10 shown. The electronic device 20 includes a memory 210, a processor 220 and a communication module 230. The memory 210, the processor 220 and the communication module 230 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0072] The memory 210 is used to store programs or data and can be, but not limited to, a random access memory, a read-only memory, a programmable read-only memory, an erasable read-only memory, an electrically erasable read-only memory, and the like.

[0073] The processor 220 is used to read / write data or programs stored in the memory 210 and execute corresponding functions. For example, Figure 1 The processor 220 of the cloud service device 120 in the illustrated illegal parking detection system 10 executes the computer program stored in the memory 210 to implement the illegal parking detection method.

[0074] The communication module 230 is used to establish a communication connection between the electronic device 20 and other communication terminals through the network, and to send and receive data through the network. Figure 1 In the illegal parking detection system 10 shown, the cloud service device 120 transmits and receives data with the drone 110 and the terminal device 130 respectively through the communication module 230.

[0075] It should be understood that Figure 2 The structure shown is only a schematic diagram of the structure of the electronic device 20. The electronic device 20 may also include Figure 2 More or fewer components as shown, or with Figure 2 Different configurations are shown. Figure 2 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0076] Please refer to Figure 3 The embodiment of the present invention provides a method for detecting illegal parking, including steps 11 to 15, so that Figure 1 The cloud service device 120 in the parking violation detection system 10 shown can Figure 2 The structure shown in FIG. 11 and FIG. 15 are executed to implement the illegal parking detection method.

[0077] Step 11: capture the aerial image from the inspection video stream currently pushed by the drone.

[0078] Here, the inspection video stream is obtained by the drone inspecting the inspection area according to the preset inspection route. In addition, the frame rate of the screenshot aerial image can be flexibly set, for example, it can be 15 frames per second, 20 frames per second, or 2 frames per second, and the specific value of the frame rate is not limited.

[0079] Step 12: Detect the aerial image to obtain vehicle information of the aerial image.

[0080] The vehicle information includes the position coordinates of each vehicle to be tested in the aerial image.

[0081] Step 13, the vehicle to be tested whose position coordinates match the preset no-parking area is regarded as a concerned vehicle, and the shooting time of the aerial image is counted as the latest illegal parking time of the concerned vehicle.

[0082] Step 14, according to the latest illegal parking time, obtain the current illegal parking duration of each concerned vehicle.

[0083] Step 15, the concerned vehicle whose current illegal parking duration exceeds the lower limit of illegal parking duration is regarded as an illegally parked vehicle, and illegal parking alarm information of the illegally parked vehicle is generated.

[0084] For example, in combination Figure 1 In the parking violation detection system 10 shown, for any inspection area, the inspection personnel can formulate an inspection route, and upload the inspection route and the inspection area in a corresponding relationship to the cloud service device 120, and store it in the database of the cloud service device 120. Alternatively, the cloud service device 120 can generate an inspection route for each inspection area according to a preset rule, and store the inspection video and the inspection area in a corresponding relationship in the database.

[0085] Thus, for each inspection area, according to the preset inspection cycle, the inspection route of the inspection area is retrieved from the database, an inspection instruction is generated, and the inspection instruction is sent to the drone 110. After receiving the inspection instruction, the drone 110 regularly inspects and shoots the inspection area according to the inspection cycle and inspection route, obtains the inspection video stream, and uploads the inspection video stream to the cloud service device 120 in real time.

[0086] After receiving the inspection video stream, the cloud service device 120 captures the image at a preset frame rate to obtain the current aerial image (i.e., the current frame), and then detects the aerial image to obtain the position coordinates of each vehicle to be tested in the aerial image. The cloud service device 120 can read the location file that records the geographic information of the prohibited parking area from the database, and take the vehicle to be tested whose location coordinates match the location file as the vehicle of interest, and count the shooting time corresponding to the aerial image as the latest illegal parking time of the vehicle of interest. Then, based on the latest illegal parking time, the current illegal parking duration of each vehicle of interest is obtained, and the vehicle of interest whose current illegal parking duration exceeds the lower limit of illegal parking duration is taken as an illegally parked vehicle, and illegal parking alarm information of the illegally parked vehicle is generated.

[0087] Then, after receiving a new patrol video stream, the above process (ie, step 11 to step 15) is repeated to obtain the illegal parking detection results of each patrol video stream in real time.

[0088] The current method of detecting illegal parking through video surveillance is to use a fixed camera to shoot, and extract the position (or bounding box) of the vehicle and the position (or bounding box) of the no-parking area from the image, and determine whether it is illegal parking based on the relationship between the two. In this method, the shooting height and shooting angle of the fixed camera are limited, the detection area is limited, and there are dead angles and blind spots, resulting in the image not being able to fully cover the entire area. At the same time, it is very easy for the image to be difficult to distinguish between small distances due to reasons such as the shooting angle, resulting in errors in the positions of the extracted vehicles and no-parking areas, which in turn affects the accuracy of the detection.

[0089] The above-mentioned illegal parking detection method first uses a drone for comprehensive inspection through steps 11 to 15, and solves the problem of limited detection area or blind spots by taking aerial photos from high altitude, so that the inspection video stream fully covers the inspection area, and then detects the aerial images captured in the real-time inspection video stream, determines the illegally parked vehicles in the image, and generates illegal parking alarm information, thereby making illegal parking detection more efficient and comprehensive, improving the detection accuracy, and eliminating the need for manual inspections, greatly reducing the inspection costs. At the same time, only the position coordinates of the vehicle need to be obtained from the image, reducing the factors affecting the detection accuracy, thereby further improving the detection accuracy.

[0090] Among them, the method of detecting the aerial image in step 12 can be flexibly set. For example, a preset neural network model can be used to generate vehicle information of the aerial image, or model detection + coordinate system conversion can be used to obtain vehicle information of the aerial image. The above two methods are only examples, and their specific implementation methods are not limited.

[0091] In order to obtain the position coordinates of the vehicle to be tested in the latitude and longitude coordinate system, and simultaneously locate the trajectory of the same vehicle in different aerial images, so as to improve the accuracy of the current illegal parking time of each vehicle to be tested, the concept of tracking detection and coordinate system conversion positioning is adopted in step 12. Figure 4 , step 12 may include steps 121 to 122.

[0092] Step 121, performing tracking detection on the aerial image to obtain a detection result.

[0093] Here, the detection result includes the detection frame and identification number of each vehicle to be tested.

[0094] Step 122 : for each vehicle to be tested, based on the pixel information of the detection frame, convert the position coordinates of the vehicle to be tested in the longitude and latitude coordinate system.

[0095] In step 121, in order to make the same vehicle under test have the same identity in the detection of different aerial images to avoid misjudgment of the vehicle under test, the concept of using deepsort algorithm for target detection and multi-target tracking to detect the aerial images is adopted in step 121. Figure 5 , step 121 detects the aerial image through steps 1211 to 1216.

[0096] Step 1211, using a preset target detector to generate a detection frame for each vehicle to be detected in the aerial image.

[0097] The target detector can be obtained by training a YOLOv5s model, a YOLOv5x model, a YOLOv5l model, a YOLOv5m model, a YOLOv5n model, or any other model with a target detection function using a sample image containing a vehicle. Thus, by inputting the aerial image into the target detector, the detection frame of each vehicle to be detected generated by the target detector can be obtained.

[0098] Step 1212, retrieve the tracking frame predicted from the previous frame of the aerial image, calculate the similarity between the detection frame and each tracking frame, and construct a cost matrix based on the similarity.

[0099] It should be noted that there may be multiple tracking frames predicted from the previous frame of aerial image, and the tracking frame may be a detection frame obtained by using a Kalman filter to perform motion estimation on the motion position of each detection frame in the previous frame of aerial image in the next frame of aerial image.

[0100] For each detection frame, the IOU (i.e., intersection over union) between the detection frame and each tracking frame can be calculated respectively, and the intersection over union is used as the similarity between the detection frame and the tracking frame. Feature matching can also be used to calculate the similarity between the detection frame and the tracking frame. The Jaccard index of the detection frame and the tracking frame can also be used as the similarity. The above method of calculating similarity is only an example, and its specific implementation method is not limited.

[0101] In addition, the cost matrix can be constructed by referring to the cost matrix construction method of the Hungarian algorithm, or any other cost matrix construction method can be used, which will not be described in detail here.

[0102] Step 1213, using the Hungarian algorithm to optimize the cost matrix, to obtain the matching result between the detection box and the tracking box.

[0103] The matching results include mismatched detection frames and tracking frames, as well as successfully matched detection frames and tracking frames.

[0104] The cost matrix is ​​optimized by the Hungarian algorithm to match the detection frame in the current aerial image with the tracking frame of the previous aerial image, and obtain the tracking frame with the minimum matching cost for each detection frame. Therefore, for the detection frame and tracking frame with the minimum matching cost, if the matching cost of the two is less than the preset matching cost upper limit, the match is successful, otherwise the detection frame and the tracking frame are mismatched.

[0105] Step 1214: The identification number of the tracking frame that matches the detection frame is used as the identification number of the successfully matched detection frame, and the state of the successfully matched tracking frame is updated to a confirmed state.

[0106] For the successfully matched detection frame and tracking frame, it indicates that the two are the boundary frames of the same vehicle to be tested in different frame images, so the detection frame inherits the identification number of the detection frame and is updated to the confirmation state through the status of the tracking frame.

[0107] Step 1215 , delete the mismatched tracking frame whose mismatch times reach the mismatch upper limit, use the mismatched detection frame as a new tracking frame, and assign a unique identification number to the mismatched detection frame.

[0108] Here, the specific value of the mismatch upper limit can be set flexibly, for example, it can be 30 times or 40 times, and its specific data can be adjusted according to the detection requirements. For the mismatched detection frame, it indicates that it is a newly appeared vehicle and a new identification number is assigned. For the mismatched tracking frame whose mismatch times reach the mismatch upper limit, it means that the vehicle to be tested corresponding to the tracking frame has driven out of the drone's aerial field of view, and the tracking frame can be deleted at this time.

[0109] Through the above steps 1214 and 1215 , the tracking frame is updated.

[0110] Step 1216, using a preset Kalman filter, perform prediction updates on each tracking frame.

[0111] In step 1216, a Kalman filter is used to predict the state of each tracking frame in the next frame of aerial image, so as to be used in tracking detection in the next frame of aerial image.

[0112] Thus, through the above steps 1211 to 1216, through tracking and detection, the vehicle to be tested can maintain the same identity (same identification number) in different frames of aerial images, avoiding vehicle identity confusion, and at the same time facilitating more accurate subsequent acquisition of the current illegal parking duration of the vehicle of interest, thereby further improving the accuracy of illegal parking detection.

[0113] After obtaining the detection frame and identification number of each vehicle to be detected in the aerial image of the current frame through the above steps 1211 to 1216, refer to Figure 6 The process of obtaining the position coordinates in step 122 can be implemented as steps 1221 to 1222.

[0114] Step 1221, extracting the core pixel coordinates of the vehicle to be tested from the detection frame.

[0115] The core pixel coordinates may be the pixel coordinates of any one or several pixels in the detection frame, for example, the pixel coordinates of the central pixel of the detection frame, or the pixel coordinates of the outermost pixel of the detection frame, and the specific selection is not limited.

[0116] Step 1222, combining the position coordinates of the drone when acquiring the image to be tested, as well as the camera intrinsic parameters and camera extrinsic parameters of the drone, transform the core pixel coordinates to obtain the position coordinates of the vehicle to be tested in the latitude and longitude coordinate system.

[0117] Here, in order to obtain the position coordinates of the vehicle to be tested in the longitude and latitude coordinate system, when the core pixel coordinates are converted, the rotation matrix between the world coordinate system and the camera coordinate system, and the translation vector between the world coordinate system and the camera coordinate system can be introduced into the camera extrinsic parameters, where the world coordinate system is a coordinate system composed of longitude, latitude and altitude, that is, the x-axis and y-axis represent longitude and latitude, respectively.

[0118] Therefore, the conversion formula between position coordinates and core pixel coordinates can be expressed as:

[0119]

[0120] in, represents the position coordinates, u and v represent the longitude and latitude respectively, (u0, v0) represents the core pixel coordinates, f x and f yRespectively represent the focal length components in the x-axis direction and the y-axis direction, R represents the rotation matrix between the world coordinate system and the camera coordinate system, T represents the translation vector between the world coordinate system and the camera coordinate system, (X w , Y w , Z w ) represents the coordinates of the drone camera in the world coordinate system when acquiring aerial images.

[0121] Through the above steps 1221 to 1222, the position coordinates of the vehicle to be tested in the latitude and longitude coordinate system are quickly and accurately obtained.

[0122] After obtaining the position coordinates of the vehicle to be tested, in step 13, if the position coordinates of the vehicle to be tested are within the prohibited parking area, the vehicle to be tested is regarded as a focused vehicle, and the shooting time of the aerial image of the current frame is counted as the latest illegal parking time of the focused vehicle.

[0123] Therefore, in step 14, for each concerned vehicle, the earliest illegal parking time that is temporally continuous with the latest illegal parking time can be extracted from the detection results of the historical aerial images in combination with the identification number to obtain the current illegal parking time of the concerned vehicle. Figure 7 , step 14 may include steps 141 to 142.

[0124] Step 141, for each vehicle of interest, based on the identification number of the vehicle of interest, extract the historical illegal parking time of the vehicle of interest.

[0125] Step 142, extracting the time period that is continuous with the latest illegal parking time of the concerned vehicle from each historical illegal parking time, so as to obtain the current illegal parking time of the concerned vehicle.

[0126] It should be noted that time continuity means that the adjacent time intervals are equal to the capture period of the aerial image. For example, in the case of 2 frames per minute, the adjacent time interval is equal to 30 seconds to be considered time continuous. If it is greater than 30 seconds, it is not considered time continuous.

[0127] For example, if the frame rate of the aerial photography time is 2 frames per minute, and the latest illegal parking time of vehicle A (A is the identification number) is 11 minutes and 10 seconds, and the historical illegal parking times are 01 minutes and 00 seconds, 9 minutes and 10 seconds, 9 minutes and 40 seconds, 10 minutes and 10 seconds, and 10 minutes and 40 seconds, then the consecutive illegal parking times are 9 minutes and 10 seconds, 9 minutes and 40 seconds, 10 minutes and 10 seconds, 10 minutes and 10 seconds, 10 minutes and 40 seconds, and 11 minutes and 10 seconds, and the current illegal parking time is 2 minutes.

[0128] Through the above steps 141 to 142, the current illegal parking duration of each concerned vehicle is quickly and accurately obtained. Furthermore, in step 15, the illegally parked vehicle can be determined according to whether the current illegal parking duration exceeds the lower limit of illegal parking duration.

[0129] In addition, in step 15, the method of generating illegal parking alarm information can be flexibly set, for example, a prompt SMS or prompt message including the location coordinates of the illegally parked vehicle, the latest illegal parking time and the current illegal parking duration can be generated, or a prompt SMS or prompt message including the location coordinates of the illegally parked vehicle can be generated. The specific implementation method is not limited.

[0130] In order to enable the traffic management department to obtain the information of illegally parked vehicles, so as to quickly intervene in management, or to quickly notify the owner to move the vehicle, etc., the idea of ​​identifying and obtaining the identity of the vehicle is introduced in the process of generating illegal parking alarm information in step 15. Figure 8 The process of generating illegal parking alarm information in step 15 may include steps 151 to 154.

[0131] Step 151, capturing a detection frame image of the illegally parked vehicle from the aerial image.

[0132] Step 152, input the detection frame image into a preset target detection model, and use the target detection model to capture the license plate image of the illegally parked vehicle.

[0133] Step 153, input the license plate image into a preset license plate recognition network, and use the license plate recognition network to generate the license plate number of the illegally parked vehicle.

[0134] Step 154, generating illegal parking alarm information based on the license plate number.

[0135] Among them, the target detection model and the license plate recognition network are both pre-trained models. For the target detection model, multiple vehicle images with license plates can be collected, and the vehicle images can be annotated according to the bounding box of the license plate to obtain training samples, and the training samples are used to train the YOLOv5s model, YOLOv5x model, YOLOv5l model, YOLOv5m model, YOLOv5n model or any other model with target detection function to obtain the target detection model. Similarly, multiple license plate images are collected, and the license plate number is used as the label of the license plate image to obtain training samples, and the training samples are used to train the LPRNet recognition network or any other model with image text extraction and recognition function to obtain the license plate recognition network.

[0136] Furthermore, the target detection model and the license plate recognition network are deployed on the cloud service device and can be called when executing the above steps 151 to 154.

[0137] In addition, in order to accurately obtain the license plate number, all aerial images with illegally parked vehicles can be identified in step 151 to obtain a detection frame image for each, and then steps 152 to 153 are executed on each detection frame image to obtain multiple license plate numbers. Finally, the license plate number with the largest vote rate or proportion is selected as the license plate number of the illegally parked vehicle.

[0138] After the illegal parking alarm information is generated through the above steps 151 to 155, in order to facilitate the notification of traffic control personnel to intervene in management, or to notify the owner to move the car, refer to Fig. 9 The illegal parking detection method provided in the embodiment of the present invention also includes step 16, which is executed after step 15 to issue an illegal parking notification.

[0139] Step 16, query the preset database, obtain the terminal device corresponding to the illegally parked vehicle, and send the illegal parking alarm information to the terminal device.

[0140] Here, the license plate number, illegal parking period, illegal parking duration, and time of sending illegal parking alarm information of the illegally parked vehicle can be recorded in the illegal parking table, which can be stored in the database. Therefore, after obtaining the license plate number of the illegally parked vehicle, the illegal parking table can be queried to check whether the illegally parked vehicle has sent illegal parking alarm information, and whether the interval between the time of sending illegal parking alarm information last time and the current time is within the preset single alarm effective time. If not, the illegal parking alarm information is sent to the terminal device 130. If not, no alarm is issued, so as to avoid repeated alarms for the same illegal parking of the illegally parked vehicle.

[0141] The database of the cloud service device 120 can record the correspondence between each vehicle and the driver's terminal device 130. Therefore, after the cloud service device 120 generates illegal parking alarm information for illegally parked vehicles, it sends the illegal parking alarm information to the management device, and at the same time sends the illegal parking alarm information to the terminal device 130 of the driver corresponding to the illegally parked vehicle in the form of text messages or app messages, so as to remind the traffic management department to intervene in the management and notify the owner of the illegally parked vehicle to move the vehicle quickly.

[0142] Based on the same concept as the above-mentioned illegal parking detection method, refer to Fig.10 The embodiment of the present invention also provides a parking violation detection device 30, which can be applied to Figure 1 The cloud service device 120 in the illegal parking detection system 10 shown includes an image acquisition module 310 , an image detection module 320 , a first timing module 330 , a second timing module 340 and an illegal parking alarm module 350 .

[0143] The image acquisition module 310 is used to capture aerial images from the inspection video stream currently pushed by the drone.

[0144] Among them, the inspection video stream is obtained by the drone inspecting the inspection area according to the preset inspection route.

[0145] The image detection module 320 is used to detect the aerial image and obtain the vehicle information of the aerial image.

[0146] The vehicle information includes the position coordinates of each vehicle to be tested in the aerial image.

[0147] The first timing module 330 is used to take the vehicle to be tested whose position coordinates match the preset no-parking area as a concerned vehicle, and count the shooting time of the aerial image as the latest illegal parking time of the concerned vehicle.

[0148] The second timing module 340 is used to obtain the current illegal parking duration of each concerned vehicle according to the latest illegal parking time.

[0149] The illegal parking alarm module 350 is used to treat the concerned vehicle whose current illegal parking duration exceeds the lower limit of illegal parking duration as an illegally parked vehicle, and generate illegal parking alarm information of the illegally parked vehicle.

[0150] The above-mentioned illegal parking detection device 30, through the coordinated action of the image acquisition module 310, the image detection module 320, the first timing module 330, the second timing module 340 and the illegal parking alarm module 350, first uses a drone for comprehensive inspection, and solves the problem of limited detection area or blind spots by taking aerial photos from high altitude, so that the inspection video stream fully covers the inspection area, and then detects the aerial images intercepted from the real-time inspection video stream, determines the illegally parked vehicles in the image, and generates illegal parking alarm information, so that the illegal parking detection is more efficient and comprehensive, and the detection accuracy is improved. At the same time, no manual inspection is required, which greatly reduces the inspection cost. At the same time, only the position coordinates of the vehicle need to be obtained from the image, which reduces the factors affecting the detection accuracy, thereby further improving the accuracy.

[0151] For the specific implementation and effect of the illegal parking detection device 30, please refer to the description of the implementation of the illegal parking detection method above, such as, for the specific implementation and effect of the image acquisition module 310, please refer to the description of the relevant content of step 11 above, for the specific implementation and effect of the image detection module 320, please refer to the description of the relevant content of step 12 above, for the specific implementation and effect of the first timing module 330, please refer to the description of the relevant content of step 13 above, for the specific implementation and effect of the second timing module 340, please refer to the description of the relevant content of step 14 above, for the specific implementation and effect of the illegal parking alarm module 350, please refer to the description of the relevant content of step 15 above, and will not be repeated here.

[0152] In addition, each module in the above-mentioned illegal parking detection device 30 can be implemented in whole or in part by software, hardware, or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor 220 in the electronic device 20 in the form of hardware, or can be stored in the memory 210 of the electronic device 20 in the form of software, so that the processor 220 can call and execute the operations corresponding to the above-mentioned modules to implement the illegal parking detection method provided above.

[0153] Based on the same inventive concept as the above-mentioned illegal parking detection method, an embodiment of the present invention also provides an electronic device 20, including a processor 220 and a memory 210, the memory 210 stores a computer program that can be executed by the processor 220, and the processor 220 can execute the computer program to implement the illegal parking detection method proposed in the embodiment of the present invention.

[0154] The embodiment of the present invention further provides a storage medium on which a computer program is stored. When the computer program is executed by the processor 220, the illegal parking detection method proposed in the embodiment of the present invention is implemented.

[0155] Based on the same concept as the above-mentioned illegal parking detection method, the embodiment of the present invention also provides an illegal parking detection system 10, referring to Figure 1 , including a drone 110, a cloud service device 120 and multiple terminal devices 130. The cloud service device 120 can be communicated with the drone 110 and the multiple terminal devices 130 respectively through the network, radio, etc.

[0156] The drone 110 is used to perform inspection and shooting in the inspection area according to a preset inspection route, obtain a real-time inspection video stream, and push the inspection video stream to the cloud service device 120.

[0157] The cloud service device 120 is used to implement the illegal parking detection method provided above.

[0158] In summary, the illegal parking detection method, device, system and electronic device provided by the embodiments of the present invention have at least the following beneficial effects:

[0159] (1) More comprehensive inspection of the inspection area: Using drones for comprehensive inspections, by taking aerial photos from above, the problem of limited inspection areas or blind spots is solved, so that the inspection video stream fully covers the inspection area;

[0160] (ii) Making illegal parking detection more efficient and comprehensive: detecting the aerial images captured from the inspection video stream shot by the drone, identifying the illegally parked vehicles in the images, and generating illegal parking alarm information, thereby making illegal parking detection more efficient and comprehensive and improving the detection accuracy;

[0161] (3) No manual inspection is required, which greatly reduces the inspection cost.

[0162] In several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0163] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0164] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk.

[0165] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for detecting illegal parking, characterized in that: Applied to a cloud service device, the cloud service device is communicatively connected with a drone, and the method comprises: Intercepting an aerial image from the inspection video stream currently pushed by the drone; wherein the inspection video stream is obtained by the drone inspecting and photographing the inspection area according to a preset inspection route; Detecting the aerial image to obtain vehicle information of the aerial image; wherein the vehicle information includes the position coordinates of each vehicle to be detected in the aerial image; The vehicle to be tested whose position coordinates match the preset no-parking zone is regarded as a concerned vehicle, and the shooting time of the aerial image is counted as the latest illegal parking time of the concerned vehicle; According to the latest illegal parking time, obtain the current illegal parking duration of each concerned vehicle; The concerned vehicle whose current illegal parking duration exceeds the lower limit of illegal parking duration is regarded as an illegally parked vehicle, and illegal parking alarm information of the illegally parked vehicle is generated.

2. The method for detecting illegal parking according to claim 1, characterized in that: The step of detecting the aerial image to obtain vehicle information of the aerial image includes: Performing tracking detection on the aerial image to obtain a detection result; wherein the detection result includes a detection frame and an identification number of each vehicle to be detected; For each of the vehicles to be tested, based on the pixel information of the detection frame, the position coordinates of the vehicle to be tested in the latitude and longitude coordinate system are converted.

3. The method for detecting illegal parking according to claim 2, characterized in that: The step of tracking and detecting the aerial image to obtain a detection result includes: Using a preset target detector to generate a detection frame for each vehicle to be detected in the aerial image; Retrieving the tracking frame predicted from the previous frame of the aerial image, respectively calculating the similarity between the detection frame and each of the tracking frames, and constructing a cost matrix according to the similarity; The cost matrix is ​​optimized by using the Hungarian algorithm to obtain a matching result between the detection frame and the tracking frame; wherein the matching result includes the mismatched detection frame and the tracking frame, and the successfully matched detection frame and the tracking frame; Using the identification number of the tracking frame that matches the detection frame as the identification number of the successfully matched detection frame, and updating the state of the successfully matched tracking frame to a confirmed state; Deleting the mismatched tracking frame whose number of mismatches reaches the mismatch upper limit, using the mismatched detection frame as a new tracking frame, and assigning a unique identification number to the mismatched detection frame; A preset Kalman filter is used to predict and update each of the tracking frames.

4. The method for detecting illegal parking according to claim 2, characterized in that: The step of converting the position coordinates of the vehicle to be tested in the latitude and longitude coordinate system based on the pixel information of the detection frame comprises: Extracting core pixel coordinates of the vehicle to be tested from the detection frame; The core pixel coordinates are transformed in combination with the position coordinates of the drone when acquiring the image to be tested, as well as the camera intrinsic parameters and camera extrinsic parameters of the drone, to obtain the position coordinates of the vehicle to be tested in the latitude and longitude coordinate system.

5. The method for detecting illegal parking according to any one of claims 1 to 4, characterized in that: The vehicle information also includes the identification number of each vehicle to be tested; The step of obtaining the current illegal parking duration of each concerned vehicle according to the latest illegal parking time includes: For each of the vehicles of interest, extracting the historical illegal parking times of the vehicle of interest based on the identification number of the vehicle of interest; From each of the historical illegal parking moments, a time period that is continuous with the latest illegal parking moment of the concerned vehicle is extracted to obtain the current illegal parking duration of the concerned vehicle.

6. The method for detecting illegal parking according to any one of claims 1 to 4, characterized in that: The step of generating the illegal parking alarm information of the illegally parked vehicle comprises: capturing a detection frame image of the illegally parked vehicle from the aerial image; Inputting the detection frame image into a preset target detection model, and using the target detection model to intercept the license plate image of the illegally parked vehicle; Inputting the license plate image into a preset license plate recognition network, and using the license plate recognition network to generate the license plate number of the illegally parked vehicle; Based on the license plate number, illegal parking alarm information is generated.

7. The method for detecting illegal parking according to claim 6, characterized in that: The cloud service device is also in communication connection with the terminal device. After the step of generating the illegal parking alarm information of the illegally parked vehicle, the method further includes: The preset database is queried to obtain the terminal device corresponding to the illegally parked vehicle, and the illegal parking alarm information is sent to the terminal device.

8. A parking violation detection device, characterized in that: Applied to a cloud service device, the cloud service device is connected to the drone for communication, and the illegal parking detection device includes an image acquisition module, an image detection module, a first timing module, a second timing module and an illegal parking alarm module; The image acquisition module is used to capture the aerial image from the inspection video stream currently pushed by the drone; wherein the inspection video stream is obtained by the drone inspecting and photographing the inspection area according to a preset inspection route; The image detection module is used to detect the aerial image to obtain vehicle information of the aerial image; wherein the vehicle information includes the position coordinates of each vehicle to be detected in the aerial image; The first timing module is used to take the vehicle to be tested whose position coordinates match the preset no-parking area as a concerned vehicle, and count the shooting time of the aerial image as the latest illegal parking time of the concerned vehicle; The second timing module is used to obtain the current illegal parking duration of each of the concerned vehicles according to the latest illegal parking time; The illegal parking alarm module is used to treat the concerned vehicle whose current illegal parking duration exceeds the lower limit of illegal parking duration as an illegally parked vehicle, and generate illegal parking alarm information of the illegally parked vehicle.

9. A parking violation detection system, characterized in that: It includes a drone and a cloud service device, wherein the drone is communicatively connected with the cloud service device; The drone is used to perform inspection and filming of the inspection area according to a preset inspection route, obtain an inspection video stream, and push the inspection video stream to the cloud service device in real time; The cloud service device is used to implement the illegal parking detection method as described in any one of claims 1 to 7.

10. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a computer program that can be executed by the processor, and the processor can execute the computer program to implement the illegal parking detection method as described in any one of claims 1 to 7.

Images