Transport vehicle sealing status compliance inspection method, device and readable storage medium

Abstract

An embodiment of the present invention provides a method, device and readable storage medium for checking the compliance of the sealing status of a transport vehicle, including: obtaining a monitoring video, and performing the following steps on each frame in sequence: identifying the current frame, and determining the transport vehicle image area and the truck bed image area that appear; for any truck bed image area, determining the corresponding sealing status; for any first truck bed image area whose sealing status is a violation status, determining the first transport vehicle image area corresponding to the first truck bed image area from the transport vehicle image area; matching the first transport vehicle image area with the transport vehicle image area that appeared in the previous frame, and determining the first transport vehicle corresponding to the first transport vehicle image area according to the matching result; for any first transport vehicle, determining the violation status score of the first transport vehicle according to the violation status; and determining the violation transport vehicle according to the violation status score of each inspected transport vehicle.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device and readable storage medium for checking the compliance of the sealing status of a transport vehicle. Background Art

[0002] In recent years, various regions have been implementing comprehensive urban renewal projects to accelerate urbanization. During this urban renewal process, construction transport vehicles have shouldered the heavy responsibility of transporting construction materials. However, fine-grained construction materials such as slag and gravel are prone to spillage and dust generation during lax transportation, impacting urban air quality and road traffic conditions.

[0003] At present, relevant departments mainly rely on manual and regular review of road surveillance videos to search for illegal transport vehicles, which has the characteristics of low search efficiency, easy omissions, and untimely punishment. Summary of the Invention

[0004] The embodiments of the present invention provide a method, a device and a readable storage medium for checking the compliance of the sealing status of a transport vehicle, so as to provide an automated solution for checking the compliance of the sealing status of a transport vehicle.

[0005] An embodiment of the present invention provides a method for checking the compliance of the sealing status of a transport vehicle, comprising:

[0006] Obtain surveillance video and perform the following steps on each frame in the order of the frames in the surveillance video:

[0007] Identify the current image frame and determine the transport vehicle image area and the truck bed image area appearing in the current image frame;

[0008] For any of the truck bed image areas, determining a sealing state corresponding to the truck bed image area;

[0009] For any first bucket image area, determining a first transport vehicle image area corresponding to the first bucket image area from the transport vehicle image area; wherein the first bucket image area is a bucket image area whose sealing state is a violation state;

[0010] For any of the first transport vehicle image regions, matching the first transport vehicle image region with a transport vehicle image region that appeared in a previous image frame, and determining the first transport vehicle corresponding to the first transport vehicle image region based on the matching result;

[0011] For any of the first transport vehicles, determining a violation status score of the first transport vehicle according to the violation status;

[0012] Identify illegal transport vehicles based on the violation status scores of each inspected transport vehicle;

[0013] The inspected transport vehicle includes the first transport vehicle.

[0014] Optionally, for any of the first transport vehicles, determining a violation status score of the first transport vehicle according to the violation status includes:

[0015] For any of the first transport vehicles, determining a violation status score of the first transport vehicle according to the relative driving direction corresponding to the first transport vehicle and the violation status;

[0016] The relative driving direction corresponding to the first transport vehicle is determined according to a first transport vehicle image area corresponding to the first transport vehicle.

[0017] Optionally, the relative driving direction includes approaching toward each other and moving away from each other;

[0018] Determining a violation status score of the first transport vehicle according to the relative driving direction corresponding to the first transport vehicle and the violation status includes:

[0019] Determining whether the confidence level of the violation status is greater than or equal to a confidence level threshold corresponding to the relative driving direction;

[0020] If the confidence level is greater than or equal to the confidence level threshold corresponding to the relative driving direction, increase the violation status score of the first transport vehicle;

[0021] The confidence threshold corresponding to the approaching direction is greater than the confidence threshold corresponding to the moving away direction.

[0022] Optionally, identifying the current image frame and determining the transport vehicle image area and the truck bed image area appearing in the current image frame includes:

[0023] Using a target detection model built based on a deep learning model to identify the current image frame, determine the transport vehicle image area appearing in the current image frame and the relative travel direction corresponding to the transport vehicle image area, and determine the truck bed image area appearing in the current image frame;

[0024] Among them, the target detection model is trained by using images that are labeled with the type of vehicle, the relative driving direction of the vehicle and the shooting direction of the surveillance video, and if the type of the vehicle is a transport vehicle, the image area of ​​the truck bed is additionally labeled as training samples.

[0025] Optionally, identifying the current image frame and determining the transport vehicle image area and the truck bed image area appearing in the current image frame includes:

[0026] Identifying the current image frame to determine a truck bed image region appearing in the current image frame and a transport vehicle image region within a preset image range;

[0027] And / or, for any first bucket image area, determining the first transport vehicle image area corresponding to the first bucket image area from the transport vehicle image area, comprising:

[0028] For any first bucket image area, determining a first transport vehicle image area corresponding to the first bucket image area and located within a preset image range from the transport vehicle image area;

[0029] The preset image range is a partial image area in the current image frame.

[0030] Optionally, matching the first transport vehicle image region with a transport vehicle image region that appeared in a previous image frame, and determining the first transport vehicle corresponding to the first transport vehicle image region according to the matching result, includes:

[0031] If the current image frame is not the first image frame in the surveillance video, matching the first transport vehicle image region with each second transport vehicle image region; the second transport vehicle image region is the first transport vehicle image region in the previous image frame;

[0032] If the first transport vehicle image region of the current image frame successfully matches one of the second transport vehicle image regions, determining that the transport vehicle corresponding to the first transport vehicle image region of the current image frame is the transport vehicle corresponding to the second transport vehicle image region;

[0033] If the current image frame is the first frame image of the surveillance video, or if the first transport vehicle image area of ​​the current image frame fails to match each of the second transport vehicle image areas, the first transport vehicle corresponding to the first transport vehicle image area is determined to be an unrecorded transport vehicle and is added to the record.

[0034] Optionally, determining the sealing state corresponding to the bucket image area includes:

[0035] Enlarging the bucket image area by a preset ratio to obtain a sealing state detection area;

[0036] The sealing state corresponding to the bucket image area is determined according to the sealing state detection area.

[0037] Optionally, determining the sealing state corresponding to the bucket image area includes:

[0038] Performing Gaussian weighted binarization on the truck bed image region to obtain a sealing state detection region;

[0039] The sealing state corresponding to the bucket image area is determined according to the sealing state detection area.

[0040] Based on the same inventive concept, an embodiment of the present invention further provides an electronic device, including:

[0041] Video acquisition module, used to acquire surveillance video;

[0042] The inspection module is used to perform the following steps on each image frame in the order of the image frames in the surveillance video:

[0043] Identify the current image frame and determine the transport vehicle image area and the truck bed image area appearing in the current image frame;

[0044] For any of the truck bed image areas, determining a sealing state corresponding to the truck bed image area;

[0045] For any first bucket image area, determining a first transport vehicle image area corresponding to the first bucket image area from the transport vehicle image area; wherein the first bucket image area is a bucket image area whose sealing state is a violation state;

[0046] For any of the first transport vehicle image regions, matching the first transport vehicle image region with a transport vehicle image region that appeared in a previous image frame, and determining the first transport vehicle corresponding to the first transport vehicle image region based on the matching result;

[0047] For any of the first transport vehicles, determining a violation status score of the first transport vehicle according to the violation status;

[0048] Identify illegal transport vehicles based on the violation status scores of each inspected transport vehicle;

[0049] The inspected transport vehicle includes the first transport vehicle.

[0050] Based on the same inventive concept, an embodiment of the present invention further provides an electronic device, comprising: a processor and a memory for storing instructions executable by the processor;

[0051] Wherein, the processor is configured to execute the instructions to implement the method for checking the compliance of the sealing status of a transport vehicle.

[0052] Based on the same inventive concept, an embodiment of the present invention further provides a readable storage medium, wherein the readable storage medium stores a computer program, and the computer program is used to implement the method for checking the compliance of the sealing status of a transport vehicle.

[0053] The beneficial effects of the present invention are as follows:

[0054] The method, device and readable storage medium for checking the compliance of the sealing status of transport vehicles provided by the embodiments of the present invention can realize automated compliance inspection of the sealing status of transport vehicles by checking the sealing status of the transport vehicle's bucket, giving a violation status score to the transport vehicle with a violation of the sealing status, and determining the violation transport vehicle based on the violation status score, which is beneficial to the law enforcement work of the relevant departments. If this solution is applied to the monitoring video collected in real time, the compliance inspection of the transport vehicle passing through the monitoring equipment can be carried out in real time, which is convenient for the relevant departments to enforce the law in a timely manner. In addition, the embodiment of the present invention first judges the compliance of the sealing status of the bucket image area, and then identifies the transport vehicle corresponding to the bucket image area with a violation of the sealing status and determines whether the vehicle is in violation, thereby effectively reducing the number of transport vehicles that are judged to be in violation, thereby effectively reducing the amount of data that needs to be processed in the entire process and speeding up the processing speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 One of the flow charts of the method for checking the compliance of the sealing status of a transport vehicle provided in an embodiment of the present invention;

[0056] Figure 2 This is one of the image frame diagrams of the surveillance video;

[0057] Figure 3 For Figure 2 Schematic diagram of image frames determining vehicle image areas and truck bed image areas;

[0058] Figure 4 One of the partial flow charts of the method for checking the compliance of the sealing status of a transport vehicle provided in an embodiment of the present invention;

[0059] Figure 5 Partial flow chart 2 of the method for checking the compliance of the sealing status of a transport vehicle provided in an embodiment of the present invention;

[0060] Figure 6 This is the second schematic diagram of the image frame of the surveillance video;

[0061] Figure 7 Partial flow chart 3 of the method for checking the compliance of the sealing status of a transport vehicle provided in an embodiment of the present invention;

[0062] Figure 8 This is a second flow chart of a method for checking the compliance of a transport vehicle's sealing status provided by an embodiment of the present invention;

[0063] Figure 9 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention;

[0064] Figure 10This is a second structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention will be further described below with reference to the accompanying drawings and examples. However, the example embodiments can be implemented in various forms and should not be understood as being limited to the embodiments described herein; on the contrary, these embodiments are provided to make the present invention more comprehensive and complete, and to fully convey the concepts of the example embodiments to those skilled in the art. The same figure marks in the figures represent the same or similar structures, and their repeated descriptions will be omitted. The words expressing position and direction described in the present invention are all explained with reference to the accompanying drawings as examples, but changes can be made as needed, and the changes made are all included in the scope of protection of the present invention. The drawings of the present invention are only used to illustrate the relative position relationship and do not represent the true proportion.

[0066] It should be noted that specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in a variety of ways different from those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The subsequent description of the specification is a preferred embodiment of the present application, but the description is for the purpose of illustrating the general principles of the present application and is not intended to limit the scope of the present application. The scope of protection of the present application shall be determined as defined by the appended claims.

[0067] The following describes in detail the method, device, and readable storage medium for checking the compliance of the sealing status of a transport vehicle provided by the embodiments of the present invention in conjunction with the accompanying drawings.

[0068] The embodiment of the present invention provides a method for checking the compliance of the sealing status of a transport vehicle, such as Figure 1 Shown, including:

[0069] S100: Obtain surveillance video.

[0070] During the specific implementation process, the monitoring video can be a video captured in real time by the monitoring device, or it can be a non-real-time video saved after being captured by the monitoring device.

[0071] S200: Select an image frame in sequence according to the order of image frames in the surveillance video.

[0072] If one image frame is successfully selected through step S200 , step S300 is executed; if it is determined through step S200 that all image frames are selected, the entire process ends.

[0073] S300: Identify the current image frame and determine the transport vehicle image area and the truck bed image area appearing in the current image frame.

[0074] In the specific implementation process, a machine learning algorithm (such as a target detection model built based on a deep learning algorithm) can be used to identify the transport vehicle image area and the truck bed image area in the image frame. The deep learning algorithm can specifically include the R-CNN (Region Convolutional Neural Networks, R-CNN) algorithm, the You Only Look Once (YOLO) algorithm, the Single Shot MultiBox Detector (SSD) algorithm, etc. It is preferred to use the YOLO algorithm to identify the image frame, especially the YOLO V5 algorithm based on DenseNet-121 as the backbone to identify the image frame. The model of the YOLO V5 algorithm is smaller and has a stronger generalization ability. Figure 2-Figure 3 As shown in the figure, after the algorithm recognizes the current image frame, it outputs the recognized vehicle image region and the truck bed image region and annotates them in the form of image frames. The vehicle image region identified by the YOLO algorithm may include image regions of other types of vehicles (such as passenger cars, buses, and construction vehicles) in addition to transport vehicles. To reduce the amount of data required for subsequent processing, these other types of vehicle image regions can be removed.

[0075] S400: For any of the truck bed image areas, determine a sealing state corresponding to the truck bed image area.

[0076] During implementation, a machine learning algorithm can be used to classify the truck bed image area and determine the corresponding sealing state. Specifically, algorithms such as Residual Network (ResNet) and GoogLeNet can be used. For example, a multi-granularity classification network algorithm based on Resnet50 can be used to classify the truck bed image area and determine the corresponding sealing state. Residual networks are characterized by ease of optimization and the ability to improve accuracy by increasing depth. The residual blocks within them use skip connections, which alleviate the vanishing gradient problem associated with increasing the depth of deep neural networks.

[0077] S500: For any first bucket image region, determine a first transport vehicle image region corresponding to the first bucket image region from the transport vehicle image regions, wherein the first bucket image region is a bucket image region whose sealing state is a violation state.

[0078] During implementation, the violation status can be set based on actual needs. For example, for heavy-duty dump trucks transporting slag, gravel, coal, and other cargo, the sealing status of the heavy-duty dump truck can be divided into four states: empty unsealed, fully loaded unsealed, fully loaded sealed, and other states. The fully loaded unsealed state can be set as the violation status, and the other three states can be set as compliance states.

[0079] Alternatively, as Figure 4 As shown, the step S500, for any first bucket image area, determines the first transport vehicle image area corresponding to the first bucket image area from the transport vehicle image area, specifically in the following manner:

[0080] S510: For any first truck bucket image area, determine the center coordinates of the first truck bucket image area.

[0081] S520: For any transport vehicle image region, determine the center coordinates of the transport vehicle image region.

[0082] S530: For any first bucket image region, determine a transport vehicle image region that satisfies the following conditions as the first transport vehicle image region:

[0083] ① The center coordinates of the transport vehicle image area are closest to the center coordinates of the first truck bed image;

[0084] ② The center coordinates of the first truck bed image area are located within the transport vehicle image area.

[0085] Furthermore, to avoid misjudgment in subsequent steps due to an incomplete image area of ​​the transport vehicle in the current frame image due to the transport vehicle just entering or leaving the current frame image, the first truck bed image area can be further set to a truck bed image area that meets the judgment conditions and has a sealing state that is a violation. The judgment conditions can be specifically set as follows: the center coordinates of the truck bed image area are within a preset area (for example, the preset area is set to be an area located at the center of the image frame, with a shape similar to the image frame, and an area of ​​80% of the image frame area), or the area of ​​the truck bed image area is greater than a preset area threshold.

[0086] S600: For any first transport vehicle image region, match the first transport vehicle image region with a transport vehicle image region that appeared in a previous image frame, and determine the first transport vehicle corresponding to the first transport vehicle image region based on the matching result.

[0087] S700: For any of the first transport vehicles, determine a violation status score of the first transport vehicle according to the violation status.

[0088] In a specific implementation, the initial violation status score of each first transport vehicle can be set to 0. For any first transport vehicle, each time an image frame is analyzed and determined to meet the violation status, the score is increased by a fixed value. (For example, if the fixed value is 1, the violation status score can represent the number of frames in which the first transport vehicle is determined to be in a violation status.)

[0089] S800: Identify illegal transport vehicles based on the violation status scores of the inspected transport vehicles.

[0090] During the specific implementation process, a violation status score threshold can be set to determine the inspected transport vehicles with a violation status score greater than the violation status score threshold as illegal transport vehicles, so as to avoid the possibility of inaccurate conclusions drawn based on a single image frame.

[0091] Afterwards, the confirmed illegal transport vehicles can be displayed to the user through a human-computer interaction interface or other means.

[0092] After completing step S800, return to step S200.

[0093] In the specific implementation process, the transport vehicles to be inspected may include the first transport vehicle (i.e., the transport vehicle that appears in the current image frame and whose sealing state is a violation state); it may also further include the second transport vehicle that appeared in the previous image frame and whose sealing state was determined to be a violation state in the previous image frame (i.e., the first transport vehicle corresponding to the previous image frame), thereby displaying all identified violation-prone transport vehicles to the user. However, if both the first transport vehicle and the second transport vehicle are used as inspected transport vehicles, the number of inspected transport vehicles will increase over time. Then, the transport vehicles in the second transport vehicle that have not appeared in a preset number of image frames as of the current image frame and have never been determined to be violation-prone transport vehicles can be removed from the set of inspected transport vehicles to reduce the amount of calculation. For example, transport vehicles are classified according to their appearance status in the current image frame: a first transport vehicle that has not been previously recorded and appears in the current image frame is marked as "Created" and given a new number for recording; a first transport vehicle that has been previously recorded in a previous image frame and also appears in the current image frame is marked as "Updated"; a second transport vehicle that has been previously recorded in a previous image frame but does not appear in the current image frame is marked as "Lost"; a second transport vehicle that has been marked as "Lost" for 12 consecutive frames and has never been determined to be an illegal transport vehicle is modified to be marked as "Deleted". When executing step S800, only transport vehicles with the "Created", "Updated", or "Lost" status are judged as illegal transport vehicles and the results are output.

[0094] In this way, the embodiment of the present invention can realize automated compliance inspection of the sealing status of transport vehicles by checking the sealing status of the bucket of the transport vehicle, giving a violation status score to the transport vehicle with a violation of the sealing status, and determining the illegal transport vehicle based on the violation status score, which is beneficial to the law enforcement work of the relevant departments. If this solution is applied to the monitoring video collected in real time, the compliance inspection of the transport vehicle passing through the monitoring equipment can be carried out in real time, which is convenient for the relevant departments to enforce the law in a timely manner. In addition, the embodiment of the present invention can effectively reduce the number of transport vehicles that are judged to be in violation by first judging the compliance of the sealing status of the bucket image area, and then identifying the corresponding transport vehicle for the bucket image area with a violation of the sealing status and judging whether the vehicle is in violation, thereby effectively reducing the amount of data that needs to be processed in the entire process and speeding up the processing speed.

[0095] When determining the truck bed image area appearing in the current image frame in step S300, the determined truck bed image area may be incomplete (for example, the determined truck bed image area does not include a covering above the truck bed opening). To ensure the accuracy of the sealing state corresponding to the truck bed image area determined in step S400, the following options are optionally implemented: the truck bed image area is expanded by a preset ratio to obtain a sealing state detection area; and the sealing state corresponding to the truck bed image area is determined based on the sealing state detection area.

[0096] During specific implementation, the area of ​​the truck bed image region may be expanded by 20% to obtain a sealing state detection region.

[0097] Furthermore, in the step S400, the sealing state corresponding to the truck bed image area is determined, and the truck bed image area can be Gaussian-weighted binarized to obtain a sealing state detection area; and the sealing state corresponding to the truck bed image area is determined based on the sealing state detection area.

[0098] In this way, binarization of the truck bed image region reduces the number of channels in the sealed state image region to one-third of the original truck bed image region, effectively reducing the computational complexity. Furthermore, Gaussian-weighted binarization of the truck bed image region can highlight cargo features within the image region, facilitating seal state identification. Furthermore, Gaussian-weighted binarization of the truck bed image region can highlight the granular and pile-like features of cargo, such as slag and gravel, in localized areas, facilitating seal state determination.

[0099] Furthermore, for the locations where certain monitoring equipment is located, the monitoring video it captures may capture not only moving transport vehicles but also transport vehicles that are not in a moving state (for example, transport vehicles are parked in roadside parking spaces, or transport vehicles are loading and unloading at work locations, etc.). Then the bucket image area of ​​the transport vehicles that are not in a moving state will be judged as a violation state by the above scheme, but in fact, the transport vehicles that are not in a moving state do not violate relevant laws and regulations, so these situations need to be screened out from the illegal transport vehicles to avoid misjudgment. Then, screening can be performed by adopting at least one of the following methods:

[0100] (1) Step S300, identifying the current image frame and determining the transport vehicle image area and the truck bed image area appearing in the current image frame, specifically includes:

[0101] The current image frame is identified to determine the bucket image area appearing in the current image frame and the transport vehicle image area within a preset image range.

[0102] The preset image range is a partial image area in the current image frame.

[0103] During the specific implementation process, after identifying the transport vehicle image area and the truck bed image area in the complete image frame, the transport vehicle image area outside the preset image range can be deleted; or the truck bed image area in the complete image frame can be identified, and the current image frame can be cropped to the preset image range before identifying the transport vehicle image area.

[0104] (2) Step S500, for any first bucket image area, determining the first transport vehicle image area corresponding to the first bucket image area from the transport vehicle image area, specifically includes:

[0105] For any first bucket image area, a first transport vehicle image area corresponding to the first bucket image area and located within a preset image range is determined from the transport vehicle image areas.

[0106] The preset image range is a partial image area in the current image frame.

[0107] In a specific implementation, the image areas of the transport vehicle's operating position, roadside parking spaces, etc. in the current image frame can be removed and used as the preset image range. In this way, when the first truck bed image area cannot be matched during step S500, the image area of ​​the transport vehicle outside the preset image range can be used as the corresponding first transport vehicle image area, resulting in a matching failure. In this way, these transport vehicles that are not sealed but do not violate relevant laws and regulations are filtered out in the subsequent judgment process. In addition, if the preset image range is set to be slightly smaller than the partial image area of ​​the image frame, the results of transport vehicles that have just entered or are about to leave the current frame image can also be filtered out, avoiding the incomplete image area of ​​the transport vehicle in the current frame image, which may lead to misjudgment in subsequent steps.

[0108] Alternatively, as Figure 5 As shown, the step S600, for any first transport vehicle image region, matching the first transport vehicle image region with a transport vehicle image region that appeared in a previous image frame, and determining the transport vehicle corresponding to the first transport vehicle image region based on the matching result, includes:

[0109] S610: Determine whether the current image frame is the first image frame of the surveillance video.

[0110] S620: Select a first transport vehicle image region of a current image frame without repetition.

[0111] If the result of step S610 is no and step S620 is successfully selected, execute step S630; if the result of step S610 is yes and step S620 is successfully selected, execute step S650; if all first transport vehicle image areas of the current image frame have been selected, execute step S700.

[0112] S630: Match the first transport vehicle image region with each second transport vehicle image region, wherein the second transport vehicle image region is the first transport vehicle image region in the previous image frame.

[0113] If the first transport vehicle image area of ​​the current image frame is successfully matched with a second transport vehicle image area through step S630, step S640 is executed; if the first transport vehicle image area of ​​the current image frame fails to be matched with each of the second transport vehicle image areas through step S630, step S650 is executed.

[0114] S640: Determine whether the first transport vehicle corresponding to the first transport vehicle image region of the current image frame is the transport vehicle corresponding to the second transport vehicle image region. Return to step S620.

[0115] S650: Determine whether the first transport vehicle corresponding to the first transport vehicle image area of ​​the current image frame is a transport vehicle that has not been recorded and add the vehicle to the record. Return to step S620.

[0116] In practice, the matching result can be determined by calculating the intersection over union (IOU) of the first and second transport vehicle image regions. IOU is a concept used in object detection and represents the overlap ratio between the generated candidate bounding box (Candidate Bound) and the original ground truth bounding box (Ground Truth Bound), that is, the ratio of their intersection to their union. Ideally, complete overlap is achieved, with a ratio of 1.

[0117] Specifically, IOU can be determined by the following formula:

[0118]

[0119] Wherein, A represents the area of ​​the first transport vehicle image region; B represents the area of ​​the second transport vehicle image region.

[0120] Then, if the IOU of the first transport vehicle image area and the second transport vehicle image area of ​​the current image frame is greater than or equal to the overlap threshold, the first transport vehicle corresponding to the first transport vehicle image area is determined to be the transport vehicle corresponding to the second transport vehicle image area; if the IOU of the first transport vehicle image area and each second transport vehicle image area of ​​the current image frame is less than the overlap threshold, the first transport vehicle corresponding to the first transport vehicle image area is determined to be an unrecorded transport vehicle and needs to be added to the record. In particular, if the IOU of the first transport vehicle image area of ​​the current image frame and multiple second transport vehicle image areas is greater than or equal to the overlap threshold, the transport vehicle corresponding to the second transport vehicle image area with the largest IOU is selected as the first transport vehicle corresponding to the first transport vehicle image area.

[0121] For example, the overlap threshold is 0.9; the intersection-and-union ratios of the first transport vehicle image area 1 and the second transport vehicle image area 1, the first transport vehicle image area 2 and the second transport vehicle image area 1, the first transport vehicle image area 2 and the second transport vehicle image area 1, and the first transport vehicle image area 2 and the second transport vehicle image area 2 are 0.88, 0.93, 0.8, and 0.85, respectively. Then, it is determined that the first transport vehicle corresponding to the first transport vehicle image area 1 is the transport vehicle corresponding to the second transport vehicle image area 2, and the transport vehicle corresponding to the first transport vehicle image area 2 is a first transport vehicle that has not been recorded and is added to the record.

[0122] Since vehicles in surveillance videos appear in consecutive image frames, the corresponding first transport vehicle can be determined by continuously matching the first transport vehicle image area in the current image frame with the transport vehicle image area in the previous frame, thereby effectively reducing the number of matches and thus reducing the amount of computation.

[0123] When a machine learning algorithm is used to implement step S400 to determine the sealing status of the bucket image area, the sealing status classification result given by the machine learning algorithm is significantly affected by the shooting direction of the bucket of the transport vehicle. In order to ensure the accuracy of the subsequent final determination of whether the transport vehicle is in violation, step S700 further determines the violation status score of the first transport vehicle based on the violation status for each of the first transport vehicles, including:

[0124] For any of the first transport vehicles, a violation status score of the first transport vehicle is determined according to the relative driving direction corresponding to the first transport vehicle and the violation status.

[0125] The relative driving direction corresponding to the first transport vehicle is determined according to a first transport vehicle image area corresponding to the first transport vehicle.

[0126] For example, Figure 6 As shown, in the surveillance video captured by monitoring equipment installed at a traffic intersection, the distance between the first transport vehicle traveling transversely and the monitoring equipment is greater. As a result, the corresponding truck bed image area of ​​the first transport vehicle traveling transversely relative to the monitoring equipment is smaller and less clear than that of the first transport vehicle traveling longitudinally relative to the monitoring equipment. When using a machine learning algorithm to determine the sealing status of the first truck bed image area, the lack of clarity in the first truck bed image area can easily lead to an erroneous conclusion. Therefore, when determining the violation status score of the first transport vehicle, the violation status score assigned to the longitudinally traveling first transport vehicle can be higher than that assigned to the transversely traveling first transport vehicle based on the violation status, thereby reducing the possibility of misjudging the transversely traveling first transport vehicle.

[0127] For another example, for the first transport vehicle traveling longitudinally, it can be further classified into two relative directions: approaching and moving away. Generally, the first transport vehicle traveling longitudinally will have a smaller image area of ​​the bucket due to the occlusion of the vehicle head, so the accuracy of the classification result of the sealing state given by the machine learning algorithm is relatively low. In order to reduce the misjudgment of the first transport vehicle, specifically, Figure 7 As shown, determining the violation status score of the first transport vehicle according to the relative driving direction corresponding to the first transport vehicle and the violation status includes:

[0128] S711. Determine whether the confidence level of the violation status is greater than or equal to a confidence level threshold corresponding to the relative driving direction.

[0129] In a specific implementation process, when determining the sealing state corresponding to the truck bed image area in step S400, the algorithm may simultaneously provide a confidence level for the result of determining the violation state.

[0130] S712: If the confidence level is greater than or equal to the confidence level threshold corresponding to the relative driving direction, increase the violation status score of the first transport vehicle.

[0131] The confidence threshold corresponding to the approaching direction is greater than the confidence threshold corresponding to the moving away direction.

[0132] In a specific implementation, the relative travel direction of the first transport vehicle can be determined by the deep learning algorithm-based object detection model implementing step S300, which identifies the image region of the first transport vehicle. Accordingly, during the object detection model training process, images labeled with the vehicle image region and the corresponding vehicle type, the relative travel direction of the vehicle relative to the direction captured by the surveillance video, and, if the vehicle type is a transport vehicle, the additionally labeled truck bed image region are used as training samples to train the object detection model. Specifically, these images can be obtained by extracting video frames from video sequences of various urban traffic scenes.

[0133] A specific example is given below to illustrate the above solution.

[0134] like Figure 8 As shown, the specific steps include:

[0135] S1001. Obtain surveillance video.

[0136] S1002: Select one image frame in sequence according to the order of image frames in the surveillance video.

[0137] If one image frame is successfully selected through step S1002, step S1003 is executed; if it is determined through step S1002 that all image frames are selected, the entire process ends.

[0138] S1003: Identify the current image frame, determine the transport vehicle image area appearing in the current image frame and the relative driving direction corresponding to the transport vehicle image area, and determine the truck bed image area appearing in the current image frame.

[0139] S1004: For any transport vehicle image region, determine the center coordinates of the transport vehicle image region.

[0140] S1005: Select a truck bed image area without duplication.

[0141] If the selection is successful in step S1005, step S1006 is executed; if all the bucket image areas are selected, step S1016 is executed.

[0142] S1006: Enlarge the area of ​​the truck bed image region by 20%, and perform Gaussian weighted binarization to obtain a sealing state detection region.

[0143] S1007. Use a multi-granularity classification network algorithm based on Resnet50 as the backbone to classify the sealing status detection area, determine the sealing status of the truck bed image area corresponding to the sealing status detection area, and give a confidence level to the sealing status of the violation state.

[0144] S1008: Determine the center coordinates of the first truck bucket image area, wherein the first truck bucket image area is the truck bucket image area whose sealing state is a violation state.

[0145] S1009: Determine whether the center coordinates of the first truck bucket image area are within a preset regular area, wherein the preset regular area is an area located at the center of the image frame, having a shape similar to the image frame, and an area of ​​80% of the image frame area.

[0146] If the result of step S1009 is yes, execute step S1010; if the result of step S1009 is no, return to step S1005.

[0147] S1010: Determine a transport vehicle image region that satisfies the following conditions simultaneously as the first transport vehicle image region:

[0148] ① The center coordinates of the transport vehicle image area are closest to the center coordinates of the first truck bed image;

[0149] ② The center coordinates of the first truck bed image area are located within the transport vehicle image area.

[0150] S1011: Determine whether the confidence level of the violation status corresponding to the first transport vehicle image region is greater than or equal to a confidence level threshold corresponding to the relative travel direction corresponding to the first transport vehicle image region, wherein the confidence level threshold corresponding to the approaching direction is greater than the confidence level threshold corresponding to the moving direction.

[0151] If the result of step S1011 is yes, execute step S1012; if the result of step S1011 is no, return to step S1005.

[0152] S1012: Match the first transport vehicle image region with each second transport vehicle image region, wherein the second transport vehicle image region is the first transport vehicle image region in the previous image frame.

[0153] If a second transport vehicle image region is successfully matched in step S1012, step S1013 is executed. If a second transport vehicle image region is not matched in step S1012, step S1014 is executed. If the current image frame is the first frame of the surveillance video, the matching result in step S1012 is a failure.

[0154] S1013: Determine that the first transport vehicle corresponding to the first transport vehicle image region of the current image frame is the transport vehicle corresponding to the second transport vehicle image region. Execute step S1015.

[0155] S1014: Determine that the first transport vehicle corresponding to the first transport vehicle image area of ​​the current image frame is an unrecorded transport vehicle, add the first transport vehicle to the record, and set the alarm flag of the first transport vehicle to 0. Execute step S1015.

[0156] S1015: Add 2 to the violation status score of the first transport vehicle. Return to step S1005.

[0157] S1016: For any transport vehicle inspected in the record, the violation status score of the inspected transport vehicle is reduced by 1. Then, the inspected transport vehicle with a violation status score of 0 in the record is deleted.

[0158] S1017: Determine a transport vehicle whose violation status score of the inspected transport vehicle in the record is greater than a preset violation status score threshold as an illegal transport vehicle, and set the corresponding alarm flag to 1.

[0159] S1018: Send an alarm message to the user for the illegal transport vehicle with the alarm flag set to 1. Return to step S1002.

[0160] Based on the same inventive concept, an embodiment of the present invention further provides an electronic device, such as Figure 9 As shown, including:

[0161] Video acquisition module M1, used to acquire surveillance video;

[0162] The inspection module M2 is used to perform the following steps on each image frame in the order of the image frames in the surveillance video:

[0163] Identify the current image frame and determine the transport vehicle image area and the truck bed image area appearing in the current image frame;

[0164] For any of the truck bed image areas, determining a sealing state corresponding to the truck bed image area;

[0165] For any first bucket image area, determining a first transport vehicle image area corresponding to the first bucket image area from the transport vehicle image area; wherein the first bucket image area is a bucket image area whose sealing state is a violation state;

[0166] For any of the first transport vehicle image regions, matching the first transport vehicle image region with a transport vehicle image region that appeared in a previous image frame, and determining the first transport vehicle corresponding to the first transport vehicle image region based on the matching result;

[0167] For any of the first transport vehicles, determining a violation status score of the first transport vehicle according to the violation status;

[0168] Identify illegal transport vehicles based on the violation status scores of each inspected transport vehicle;

[0169] The inspected transport vehicle includes the first transport vehicle.

[0170] Optionally, for any of the first transport vehicles, determining a violation status score of the first transport vehicle according to the violation status includes:

[0171] For any of the first transport vehicles, determining a violation status score of the first transport vehicle according to the relative driving direction corresponding to the first transport vehicle and the violation status;

[0172] The relative driving direction corresponding to the first transport vehicle is determined according to a first transport vehicle image area corresponding to the first transport vehicle.

[0173] Optionally, the relative driving direction includes approaching toward each other and moving away from each other;

[0174] Determining a violation status score of the first transport vehicle according to the relative driving direction corresponding to the first transport vehicle and the violation status includes:

[0175] Determining whether the confidence level of the violation status is greater than or equal to a confidence level threshold corresponding to the relative driving direction;

[0176] If the confidence level is greater than or equal to the confidence level threshold corresponding to the relative driving direction, increase the violation status score of the first transport vehicle;

[0177] The confidence threshold corresponding to the approaching direction is greater than the confidence threshold corresponding to the moving away direction.

[0178] Optionally, identifying the current image frame and determining the transport vehicle image area and the truck bed image area appearing in the current image frame includes:

[0179] Using a target detection model built based on a deep learning model to identify the current image frame, determine the transport vehicle image area appearing in the current image frame and the relative travel direction corresponding to the transport vehicle image area, and determine the truck bed image area appearing in the current image frame;

[0180] Among them, the target detection model is trained by using images that are labeled with the type of vehicle, the relative driving direction of the vehicle and the shooting direction of the surveillance video, and if the type of the vehicle is a transport vehicle, the image area of ​​the truck bed is additionally labeled as training samples.

[0181] Optionally, identifying the current image frame and determining the transport vehicle image area and the truck bed image area appearing in the current image frame includes:

[0182] Identifying the current image frame to determine a truck bed image region appearing in the current image frame and a transport vehicle image region within a preset image range;

[0183] And / or, for any first bucket image area, determining the first transport vehicle image area corresponding to the first bucket image area from the transport vehicle image area, comprising:

[0184] For any first bucket image area, determining a first transport vehicle image area corresponding to the first bucket image area and located within a preset image range from the transport vehicle image area;

[0185] The preset image range is a partial image area in the current image frame.

[0186] Optionally, matching the first transport vehicle image region with a transport vehicle image region that appeared in a previous image frame, and determining the first transport vehicle corresponding to the first transport vehicle image region according to the matching result, includes:

[0187] If the current image frame is not the first image frame in the surveillance video, matching the first transport vehicle image region with each second transport vehicle image region; the second transport vehicle image region is the first transport vehicle image region in the previous image frame;

[0188] If the first transport vehicle image region of the current image frame successfully matches one of the second transport vehicle image regions, determining that the transport vehicle corresponding to the first transport vehicle image region of the current image frame is the transport vehicle corresponding to the second transport vehicle image region;

[0189] If the current image frame is the first frame image of the surveillance video, or if the first transport vehicle image area of ​​the current image frame fails to match each of the second transport vehicle image areas, the first transport vehicle corresponding to the first transport vehicle image area is determined to be an unrecorded transport vehicle and is added to the record.

[0190] Optionally, determining the sealing state corresponding to the bucket image area includes:

[0191] Enlarging the bucket image area by a preset ratio to obtain a sealing state detection area;

[0192] The sealing state corresponding to the bucket image area is determined according to the sealing state detection area.

[0193] Optionally, determining the sealing state corresponding to the bucket image area includes:

[0194] Performing Gaussian weighted binarization on the truck bed image region to obtain a sealing state detection region;

[0195] The sealing state corresponding to the bucket image area is determined according to the sealing state detection area.

[0196] In the several embodiments provided herein, it should be understood that the device embodiments described above are merely illustrative. For example, the module division is merely a logical functional division, and actual implementations may employ alternative divisions, such as combining or integrating multiple modules or components into another system, or omitting or disabling certain features. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device, or module, and may be electrical, mechanical, or other means.

[0197] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected to achieve the purpose of the present embodiment according to actual needs.

[0198] In addition, the functional modules in the various embodiments of the present application may be integrated into a single processing module, or each module may exist physically separately, or two or more modules may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may be stored in a removable storage medium.

[0199] Since the specific manner in which each module of the electronic device performs operations has been described in detail in the embodiment of the method for checking the compliance of the sealing status of a transport vehicle, it will not be repeated here.

[0200] Based on the same inventive concept, an embodiment of the present invention further provides an electronic device, such as Figure 10 As shown, it includes: a processor 110 and a memory 120 for storing executable instructions of the processor 110;

[0201] The processor 110 is configured to execute the instructions to implement the method for checking the compliance of the sealing status of a transport vehicle.

[0202] In a specific implementation, the device may have relatively large differences due to different configurations or performances, and may include one or more processors 110, a memory 120, and a computer-readable storage medium 130. The memory 120 and / or the computer-readable storage medium 130 may include one or more applications 131 or data 132. The memory 120 and / or the computer-readable storage medium 130 may also include one or more operating systems 133, such as Windows, Mac OS, Linux, IOS, Android, Unix, FreeBSD, etc. The memory 120 and the computer-readable storage medium 130 may be temporary storage or persistent storage. The application 131 may include one or more modules ( Figure 10 (not shown), each module may include a series of instruction operations. Furthermore, the processor 110 may be configured to communicate with the computer-readable storage medium 130 and execute a series of instruction operations in the computer-readable storage medium 130 on the device. The device may also include one or more power supplies ( Figure 10 one or more network interfaces 140, the network interface 140 including a wired network interface 141 and / or a wireless network interface 142; one or more input / output / interfaces 143.

[0203] Based on the same inventive concept, an embodiment of the present invention further provides a readable storage medium, wherein the readable storage medium stores a computer program, and the computer program is used to implement the method for checking the compliance of the sealing status of a transport vehicle.

[0204] The method, device and readable storage medium for checking the compliance of the sealing status of transport vehicles provided by the embodiments of the present invention can realize automated compliance inspection of the sealing status of transport vehicles by checking the sealing status of the transport vehicle's bucket, giving a violation status score to the transport vehicle with a violation of the sealing status, and determining the violation transport vehicle based on the violation status score, which is beneficial to the law enforcement work of the relevant departments. If this solution is applied to the monitoring video collected in real time, the compliance inspection of the transport vehicle passing through the monitoring equipment can be carried out in real time, which is convenient for the relevant departments to enforce the law in a timely manner. In addition, the embodiment of the present invention first judges the compliance of the sealing status of the bucket image area, and then identifies the transport vehicle corresponding to the bucket image area with a violation of the sealing status and determines whether the vehicle is in violation, thereby effectively reducing the number of transport vehicles that are judged to be in violation, thereby effectively reducing the amount of data that needs to be processed in the entire process and speeding up the processing speed.

[0205] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0206] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0207] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0208] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0209] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A method for checking the compliance of the sealing status of a transport vehicle, characterized in that: include: Obtain surveillance video and perform the following steps on each frame in the order of the frames in the surveillance video: Identify the current image frame and determine the transport vehicle image area and the truck bed image area appearing in the current image frame; For any of the truck bed image areas, determining a sealing state corresponding to the truck bed image area; For any first bucket image area, determining a first transport vehicle image area corresponding to the first bucket image area from the transport vehicle image area; wherein the first bucket image area is a bucket image area whose sealing state is a violation state; For any of the first transport vehicle image regions, matching the first transport vehicle image region with a transport vehicle image region that appeared in a previous image frame, and determining the first transport vehicle corresponding to the first transport vehicle image region based on the matching result; For any of the first transport vehicles, determining a violation status score of the first transport vehicle according to the violation status; Identify illegal transport vehicles based on the violation status scores of each inspected transport vehicle; The inspected transport vehicle includes the first transport vehicle.

2. The method according to claim 1, wherein For any of the first transport vehicles, determining a violation status score of the first transport vehicle according to the violation status includes: For any of the first transport vehicles, determining a violation status score of the first transport vehicle according to the relative driving direction corresponding to the first transport vehicle and the violation status; The relative driving direction corresponding to the first transport vehicle is determined according to a first transport vehicle image area corresponding to the first transport vehicle.

3. The method according to claim 2, wherein The relative driving direction includes approaching toward each other and moving away from each other; Determining a violation status score of the first transport vehicle according to the relative driving direction corresponding to the first transport vehicle and the violation status includes: Determining whether the confidence level of the violation status is greater than or equal to a confidence level threshold corresponding to the relative driving direction; If the confidence level is greater than or equal to the confidence level threshold corresponding to the relative driving direction, increase the violation status score of the first transport vehicle; The confidence threshold corresponding to the approaching direction is greater than the confidence threshold corresponding to the moving away direction.

4. The method according to claim 1, wherein Identifying the current image frame and determining the transport vehicle image area and the truck bed image area appearing in the current image frame include: Identifying the current image frame to determine a truck bed image region appearing in the current image frame and a transport vehicle image region within a preset image range; And / or, for any first bucket image area, determining the first transport vehicle image area corresponding to the first bucket image area from the transport vehicle image area, comprising: For any first bucket image area, determining a first transport vehicle image area corresponding to the first bucket image area and located within a preset image range from the transport vehicle image area; The preset image range is a partial image area in the current image frame.

5. The method according to claim 1, wherein Matching the first transport vehicle image region with a transport vehicle image region that appeared in a previous image frame, and determining a first transport vehicle corresponding to the first transport vehicle image region according to the matching result, including: If the current image frame is not the first image frame in the surveillance video, matching the first transport vehicle image region with each second transport vehicle image region; the second transport vehicle image region is the first transport vehicle image region in the previous image frame; If the first transport vehicle image region of the current image frame successfully matches one of the second transport vehicle image regions, determining that the transport vehicle corresponding to the first transport vehicle image region of the current image frame is the transport vehicle corresponding to the second transport vehicle image region; If the current image frame is the first frame image of the surveillance video, or if the first transport vehicle image area of ​​the current image frame fails to match each of the second transport vehicle image areas, the first transport vehicle corresponding to the first transport vehicle image area is determined to be an unrecorded transport vehicle and is added to the record.

6. The method according to claim 1, wherein Determining the sealing state corresponding to the bucket image area includes: Enlarging the bucket image area by a preset ratio to obtain a sealing state detection area; The sealing state corresponding to the bucket image area is determined according to the sealing state detection area.

7. The method according to claim 1, wherein Determining the sealing state corresponding to the bucket image area includes: Performing Gaussian weighted binarization on the truck bed image region to obtain a sealing state detection region; The sealing state corresponding to the bucket image area is determined according to the sealing state detection area.

8. An electronic device, characterized in that: include: Video acquisition module, used to acquire surveillance video; The inspection module is used to perform the following steps on each image frame in the order of the image frames in the surveillance video: Identify the current image frame and determine the transport vehicle image area and the truck bed image area appearing in the current image frame; For any of the truck bed image areas, determining a sealing state corresponding to the truck bed image area; For any first bucket image area, determining a first transport vehicle image area corresponding to the first bucket image area from the transport vehicle image area; wherein the first bucket image area is a bucket image area whose sealing state is a violation state; For any of the first transport vehicle image regions, matching the first transport vehicle image region with a transport vehicle image region that appeared in a previous image frame, and determining the first transport vehicle corresponding to the first transport vehicle image region based on the matching result; For any of the first transport vehicles, determining a violation status score of the first transport vehicle according to the violation status; Identify illegal transport vehicles based on the violation status scores of each inspected transport vehicle; The inspected transport vehicle includes the first transport vehicle.

9. An electronic device, characterized in that: include: a processor and a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method for checking the compliance of the sealing status of a transport vehicle as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and the computer program is used to implement the method for checking the compliance of the sealing status of a transport vehicle according to any one of claims 1 to 7.

Images