Engineering transport vehicle behavior process detection method, system, medium and equipment

Abstract

The invention provides an engineering transport vehicle behavior process detection method, system, medium and equipment, and belongs to the technical field of vehicle behavior recognizing.The method comprises the steps that a streaming media codec is connected with a camera, and picture frames are obtained by decoding videos; carrying out engineering transportation vehicle identification on the picture frame by using the target detection model, and continuously tracking the identified engineering transportation vehicle; and performing target behavior detection on the tracked engineering transport vehicle, and generating corresponding alarm data according to a detection result. According to the method, the streaming media codec is connected with the camera, so that the video stream can be continuously acquired and decoded into the picture frame, and basic data is provided for subsequent detection; engineering transport vehicle identification and continuous tracking are performed on the picture frames by using the target detection model, so that accurate positioning and whole-process monitoring of the engineering transport vehicle are ensured, and no key behavior is missed; target behavior detection is carried out on the tracked engineering transport vehicle, and illegal behaviors of the engineering transport vehicle can be found in time.

Description

Technical Field

[0001] This invention belongs to the field of vehicle behavior recognition technology, and in particular relates to a method, system, medium and equipment for the process detection of the behavior of engineering transport vehicles. Background Technology

[0002] In today's society, with the acceleration of urbanization, large-scale projects such as construction sites, road reconstruction, and mining have generated a large demand for waste soil transportation. Waste soil transport vehicles typically travel on urban roads, and their numbers are vast and widely distributed. Therefore, effective management, monitoring, and tracking of these vehicles have become important issues in modern urban intelligent transportation and environmental protection management.

[0003] With the continuous development of Internet of Things (IoT) and computer vision technologies, smart cameras can collect real-time data from engineering transport vehicles. Edge gateways can then be used for deep learning research and model calculations to analyze the behavior of these vehicles, enabling camera control and identification. This allows for intelligent identification, analysis, real-time tracking, and complete recording of vehicle behavior. Edge computing, as a distributed computing model, moves data processing and logic control from the cloud closer to the data source, reducing transmission latency and improving the real-time performance and accuracy of data processing.

[0004] In existing technologies, the real-time requirements for engineering transport vehicle recognition scenarios are high, necessitating analysis and response to video streams within a short timeframe. Due to the performance limitations of ARM-based edge gateways, existing technologies face significant challenges in real-time performance. For example, during target detection, the model's inference speed may not keep pace with the input video stream, resulting in delayed detection results and hindering timely detection and handling of violations. Similarly, during PTZ tracking, computational latency may prevent the PTZ from adjusting its angle in a timely manner to accurately track the movement of engineering transport vehicles, thus affecting tracking accuracy and stability.

[0005] In practical applications, edge gateways need to communicate with the cloud or other devices to obtain model updates, upload recognition results, or receive control commands. Existing technologies have the following problems in communication and data transmission: First, communication bandwidth is limited, especially when processing large amounts of video data, which may lead to data transmission delays or losses, affecting the real-time performance and reliability of the entire system; second, the security and stability of data transmission are not high enough, and it may be affected by network interference, leading to data transmission interruptions or errors, thus affecting the normal operation of services. Summary of the Invention

[0006] In view of the shortcomings of the existing technology, the present invention provides a method, system, medium and equipment for the process-oriented detection of the behavior of engineering transport vehicles.

[0007] Firstly, the method includes, Connect the streaming media codec to the camera and obtain image frames by decoding the video; The object detection model is used to identify engineering transport vehicles in image frames and the identified engineering transport vehicles are continuously tracked. The system detects the target behavior of the tracked engineering transport vehicles and generates corresponding alarm data based on the detection results.

[0008] Furthermore, the step of connecting the streaming media codec to the camera and obtaining image frames by decoding the video specifically includes connecting the streaming media codec to the camera via a real-time streaming protocol and using a hardware codec provided by the edge gateway system to decode the video stream and obtain image frames.

[0009] Furthermore, the step of using a target detection model to identify engineering transport vehicles in image frames and continuously tracking the identified engineering transport vehicles specifically includes using a lightweight YOLOv5s target detection model to identify engineering transport vehicles in image frames and using a Re-ID+IOU hybrid tracking scheme to track the engineering transport vehicles.

[0010] Furthermore, the target behavior detection of the tracked engineering transport vehicles specifically includes detecting whether the engineering transport vehicles are tilted, detecting whether the roofs of the engineering transport vehicles are covered, recognizing the license plates of the engineering transport vehicles, and detecting whether the engineering transport vehicles are leaving the site.

[0011] Furthermore, the detection of tipping over of engineering transport vehicles specifically includes: detecting tipping over of engineering transport vehicles while the target is stationary; detecting the dynamic proportion of the outward expansion of the engineering transport vehicle frame; calculating the pixel changes in the outward expansion area over multiple consecutive frames, and if the changes exceed a threshold, detecting a pile of soil; calculating the changes in the angle between the hinge point of the carriage and the front of the vehicle, the midpoint of the rear of the carriage, the top of the hydraulic rod, and the horizontal plane over multiple consecutive frames; setting thresholds for changes in the outward expansion area, changes in the carriage angle, and a confidence threshold for detecting the pile of soil, and making a comprehensive judgment; wherein, when the calculation results for a specified number of consecutive frames all exceed the design threshold, it is determined that tipping has occurred; when tipping is detected, alarm data is generated and sent. The detection of the roof cover status of the engineering transport vehicle specifically includes: using a sliding window voting method to detect the roof cover status of the engineering transport vehicle based on the statistical results of a specific number of frames, and obtaining the detection results; after obtaining the detection results, re-performing the detection after a certain period of time; if the detection results do not change, no alarm is triggered; otherwise, if the detection results change, alarm data is generated and sent. The license plate recognition for engineering transport vehicles specifically includes using the Tesseract-OCR model to recognize Chinese, English, and numbers in the detected vehicle body area, and removing redundant language packets; when at least 3 frames of recognition results are consistent across multiple consecutive frames, the recognition result is output.

[0012] Secondly, the system includes a streaming media encoding / decoding module, a vehicle tracking module, and a vehicle behavior recognition module; The streaming media codec module is used to connect the streaming media codec to the camera and obtain image frames by decoding the video; The vehicle tracking module is used to identify engineering transport vehicles in image frames using a target detection model and to continuously track the identified engineering transport vehicles. The vehicle behavior recognition module is used to detect the target behavior of the tracked engineering transport vehicles and generate corresponding alarm data based on the detection results.

[0013] Furthermore, the streaming media codec module is specifically used to connect the streaming media codec to the camera via a real-time streaming protocol and use the hardware codec provided by the edge gateway system to decode the video stream and obtain image frames.

[0014] Furthermore, the vehicle tracking module is specifically used to identify engineering transport vehicles in image frames using a lightweight YOLOv5s target detection model, and to track engineering transport vehicles using a Re-ID+IOU hybrid tracking scheme.

[0015] Furthermore, the vehicle behavior recognition module is specifically used to detect the tilting of engineering transport vehicles, detect the roof covering status of engineering transport vehicles, perform license plate recognition on engineering transport vehicles, and detect the departure of engineering transport vehicles.

[0016] Furthermore, the vehicle behavior recognition module is specifically used to detect the tipping of the engineering transport vehicle when the target is stationary; detect the dynamic proportion of the outward expansion of the engineering transport vehicle frame; calculate the pixel changes of the outward expansion area over multiple consecutive frames, and if the changes exceed a threshold, detect the soil pile target; calculate the changes of the hinge point between the carriage and the front of the vehicle, the midpoint of the rear of the carriage, the angle between the top of the hydraulic rod and the horizontal plane over multiple consecutive frames; set thresholds for the outward expansion area changes, thresholds for the carriage angle changes, and confidence thresholds for soil pile detection, and make a comprehensive judgment; wherein, when the calculation results for a specified number of consecutive frames all exceed the design threshold, it is determined that tipping has occurred; when tipping is detected, alarm data is generated and sent. The vehicle behavior recognition module is specifically used to detect the canopy covering status of engineering transport vehicles by using a sliding window voting method based on the statistical results of a specific number of frames, and obtain the detection results; after obtaining the detection results, the detection is re-executed after a certain period of time; if the detection results do not change, no alarm is triggered; otherwise, if the detection results change, alarm data is generated and sent. The vehicle behavior recognition module is specifically used to perform Chinese, English, and number recognition on the detected vehicle body area using the Tesseract-OCR model, and remove redundant language packets; when at least 3 frames of recognition results are consistent across multiple consecutive frames, the recognition result is output. Thirdly, the computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the engineering transport vehicle behavior process-oriented detection method according to any one of claims 1-5.

[0017] Fourthly, an electronic device, characterized in that it includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a program stored in memory, it implements the steps of the engineering transport vehicle behavior process detection method according to any one of claims 1-5.

[0018] Compared with the prior art, the present invention has the following advantages: 1. This invention connects a streaming media codec to a camera, enabling continuous acquisition of video streams and decoding into image frames, providing basic data for subsequent detection; it uses a target detection model to identify and continuously track engineering transport vehicles in the image frames, ensuring accurate positioning and full-process monitoring of engineering transport vehicles without missing any key behaviors; and it performs target behavior detection on the tracked engineering transport vehicles, enabling timely detection of violations by engineering transport vehicles.

[0019] 2. By comprehensively assessing pixel changes in the extended area across multiple frames and variations in the vehicle's angle, the system determines whether engineering transport vehicles are illegally dumping waste, effectively monitoring the compliance of dumping operations and preventing indiscriminate dumping of construction waste. A sliding window voting method is employed to detect canopy occupancy status, with detection based on a specific number of frames and re-detection at reasonable time intervals to avoid frequent alarms and promptly detect changes in occupancy status, ensuring safety and environmental compliance during transportation. The Tesseract-OCR model is used for accurate license plate recognition on the vehicle body, removing redundant language packets to improve recognition efficiency and accuracy, facilitating vehicle information management and tracking. Simultaneously, departure status is detected, further improving the supervision of the entire engineering transport vehicle operation process.

[0020] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A schematic diagram of the process flow for a standardized detection method for the behavior of engineering transport vehicles according to the present invention is shown.

[0023] Figure 2 A schematic diagram of a process-oriented detection system module for engineering transport vehicle behavior according to the present invention is shown.

[0024] Figure 3 A schematic diagram of the application architecture involved in this invention is shown.

[0025] Figure 4 A schematic flowchart of the method for tracking dump trucks according to the present invention is shown. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] This invention proposes a method, system, medium, and equipment for the process-oriented detection of the behavior of engineering transport vehicles, and its application architecture is as follows: Figure 3 As shown, the main focus is on the business scenario of engineering transport vehicle behavior recognition. Combining camera gimbal target tracking, it identifies various behavioral attributes of engineering transport vehicles, such as whether the top is covered, whether it is tilted, and the vehicle body and license plate. The process records key node steps and sends the recognition data to the business platform.

[0028] In another embodiment of the invention, such as Figure 1As shown, a method for process-oriented detection of the behavior of engineering transport vehicles includes the following steps: S1. Connect the streaming media codec to the camera and obtain image frames by decoding the video.

[0029] In another embodiment of the present invention, the streaming media codec connects to the camera via the Real Time Streaming Protocol (RTSP) to pull video streams, and uses the hardware codec provided by the edge gateway system to decode the video streams and obtain image frames to store in a memory queue.

[0030] S2. Use the target detection model to identify engineering transport vehicles in the image frames and continuously track the identified engineering transport vehicles.

[0031] In another embodiment of the present invention, a lightweight YOLOv5s object detection model is used to identify engineering transport vehicles in image frames, and a Re-ID + IOU hybrid tracking scheme is used to track the engineering transport vehicles. The specific tracking method is as follows: Figure 4 As shown, including, a. The object detection model is started. The SDK is used to rotate the camera to a specified preset point and keep the image fixed to acquire image frames.

[0032] b. Perform engineering transport vehicle target detection at 4 frames per second, and use a Re-ID + IOU hybrid scheme to track moving targets.

[0033] c. Select the engineering transport vehicle that moves continuously for 2 seconds and has the largest detection area as the target for subsequent continuous tracking.

[0034] d. When the pixel position of the tracked object is close to the edge of the screen, the camera pan-tilt unit needs to be moved again to center the target. By calculating the pixel distance between the center of the tracked object and the center of the screen, and combining the built-in parameters of different manufacturers' cameras, the parameters required for PTZ control are calculated and called.

[0035] e. The target detection model continuously queries the camera pan-tilt status and waits for the camera pan-tilt control to end, repeating the above steps ad tracking process until the tracked object is lost or the camera pan-tilt reaches the rotation limit.

[0036] f. Send a command to return the camera pan-tilt unit to its initial preset position, and wait for the execution of a new process check. Optionally, since the camera pan-tilt-zoom (PTZ) rotation takes time, the target detection model continuously requests the camera status and cancels the tracking of the previous static image, waiting for the PTN rotation to stop before continuing further logical judgment and data update, thus avoiding erroneous data and control caused by network latency and PTN operation latency; after the camera PTN operation stops, target tracking is restarted, and the mobile engineering transport vehicle target closest to the center of the screen is selected as the latest tracking object.

[0037] In another embodiment of the present invention, if the tracked object occupies a small portion of the camera image, it is also necessary to calculate the required magnification factor in conjunction with the manufacturer's SDK parameters, and send PTZ control commands to the gimbal camera for magnification, while simultaneously centering the target in the center of the screen. This is mainly to ensure that the width of the target object occupies 1 / 4 to 1 / 3 of the entire screen width, so as to avoid failing to recognize the vehicle body and license plate number.

[0038] S3. Detect the target behavior of the tracked engineering transport vehicles and generate corresponding alarm data based on the detection results.

[0039] Optionally, target behavior detection can be performed on the tracked engineering transport vehicles, specifically including detecting whether the engineering transport vehicles are tipped over, detecting whether the roofs of the engineering transport vehicles are covered, recognizing the license plates of the engineering transport vehicles, and detecting whether the engineering transport vehicles are leaving the site.

[0040] In another embodiment of the present invention, target detection is performed on the behavior of engineering transport vehicles, which specifically includes: The process of monitoring the canopy covering status of engineering transport vehicles begins with the tracking process. If the status changes, it is reported to the business platform.

[0041] To detect whether engineering transport vehicles have tipped over, first report alarm data indicating that the vehicle has not tipped over; if it has tipped over, send alarm images and videos again.

[0042] The system detects the vehicle body and license plate of engineering transport vehicles and continuously updates the application with the highest threshold license plate number.

[0043] The system detects whether engineering transport vehicles have left the site and sends an alarm image, video, and the license plate number with the highest threshold after departure.

[0044] In another embodiment of the present invention, taking an engineering transport vehicle for transporting construction waste (hereinafter referred to as a construction waste truck) as an example, the method detects whether the construction waste truck is dumping by detecting the angle of the truck bed (key point detection) and combining the motion blur and texture changes of the ROI region with the judgment of the soil pile target. Specifically, this includes... Subsequent logical judgments are made when the dump truck target is stationary; the standard for the target being stationary is that the center coordinate of the target moves less than a set threshold within 2 seconds.

[0045] Based on the outward expansion dynamic ratio of the muck truck detection frame; where the outward expansion dynamic ratio is width + 50% and height + 50%.

[0046] The pixel changes in the outer region (such as the raising of the carriage or the change of the soil pile) are calculated over multiple consecutive frames. If the changes exceed the threshold, soil pile target detection is performed to determine whether a collapse may have occurred.

[0047] The changes in the angle between the hinge point between the carriage and the front of the train, the midpoint of the rear of the carriage, the top of the hydraulic rod, and the horizontal plane are calculated over multiple consecutive frames.

[0048] The threshold values ​​for changes in the outer expansion area, the car body angle, and the soil pile detection confidence threshold were set to 0.4 and 0.3, respectively, and a comprehensive judgment was made. When the scores of a specified number of consecutive frames all exceeded the design threshold (obtained from the experiment), it was determined that a collapse had occurred.

[0049] When tipping is detected, alarm data is generated and sent.

[0050] In another embodiment of the present invention, the method for detecting the roof covering status of a dump truck includes... The top 30% area of ​​the rectangular frame surrounding the dump truck is designated as the ROI area of ​​the roof.

[0051] The sliding window voting method is used to detect whether the roof of the dump truck is covered based on the statistical results of more than 70% of the frames over 5 consecutive seconds; the confidence level of each frame is required to be higher than the internally set threshold.

[0052] Once the detection results are obtained, the detection is paused to conserve resources. The detection is then resumed after a certain period of time (e.g., 1 minute). If the detection results remain unchanged, no alarm is triggered. Conversely, if the detection results change, alarm data is generated and sent.

[0053] In another embodiment of the present invention, a method for license plate recognition of dump trucks includes, OCR (Optical Character Recognition) is performed on the detected vehicle body area.

[0054] The Tesseract-OCR model is used to recognize Chinese, English, and numbers, and redundant language packets are removed.

[0055] The recognition result is output when at least 3 out of 5 consecutive recognition results are consistent.

[0056] Before the vehicle disappears, the license plate recognition results are continuously updated in memory, and the captured images are also updated in memory. After the vehicle leaves, the result with the highest threshold is output and reported.

[0057] In another embodiment of the present invention, before performing target behavior detection on the tracked engineering transport vehicles, a corresponding training labeled dataset should be selected to determine the threshold set in the above-mentioned engineering transport vehicle behavior detection process. Taking engineering transport vehicles transporting construction waste (hereinafter referred to as construction waste trucks) as an example, the training labeled dataset selection strategy includes: Rent dump trucks to record NVR (Network Video Recorder) footage from multiple angles and at different zoom levels (appropriate zoom levels required for business purposes) within the same focal length. The NVR recording requirements include, but are not limited to: dump trucks of different sizes, models, and colors; vehicle roofs half-open or fully open in multiple camera angles; dumping actions of dump trucks at different angles in different camera angles; and examples of dumped yellow soil piles, black soil piles, or garbage piles. Based on NVR recordings, a 3D model was created using the 3D modeling tool Blende. Then, using the AI ​​plugin AI Rende, material textures were generated from text (such as "dirty dump truck"), and the 3D model was animated. Animation effects were added to the roof opening and closing and tilting actions from different angles. Key data is labeled on the 3D model after the animation is rendered, such as the detection frames for the body, roof, and soil pile of the dump truck, the hinge point of the truck body (used for calculating the tipping angle), the roof status, and the soil pile color, so as to determine the threshold set in the subsequent detection of the behavior of engineering transport vehicles.

[0058] Optionally, for multi-angle NVR recordings of dump truck movement, roof opening and closing (fully open / half open / closed), and dumping actions, zoom can be manually controlled; when annotating key data, nighttime dumping data can also be added to address nighttime black-and-white mode scenarios. For example... Figure 2 As shown, an engineering transport vehicle behavior process detection system includes a streaming media encoding / decoding module, a vehicle tracking module, and a vehicle behavior recognition module.

[0059] 1. The streaming media codec module is used to connect the streaming media codec to the camera and obtain image frames by decoding the video.

[0060] In another embodiment of the present invention, the streaming media codec connects to the camera via the Real Time Streaming Protocol (RTSP) to pull video streams, and uses the hardware codec provided by the edge gateway system to decode the video stream and obtain image frames into the memory queue.

[0061] 2. The vehicle tracking module is used to identify engineering transport vehicles in image frames using a target detection model and to continuously track the identified engineering transport vehicles.

[0062] In another embodiment of the present invention, the vehicle tracking module uses a lightweight YOLOv5s object detection model to identify engineering transport vehicles in image frames, and employs a Re-ID + IOU hybrid tracking scheme to track the engineering transport vehicles. The specific tracking method is as follows: Figure 4 As shown, including, a. The object detection model is started. The SDK is used to rotate the camera to a specified preset point and keep the image fixed to acquire image frames.

[0063] b. Perform engineering transport vehicle target detection at 4 frames per second, and use a Re-ID + IOU hybrid scheme to track moving targets.

[0064] c. Select the engineering transport vehicle that moves continuously for 2 seconds and has the largest detection area as the target for subsequent continuous tracking.

[0065] d. When the pixel position of the tracked object is close to the edge of the screen, the camera pan-tilt unit needs to be moved again to center the target. By calculating the pixel distance between the center of the tracked object and the center of the screen, and combining the built-in parameters of different manufacturers' cameras, the parameters required for PTZ control are calculated and called.

[0066] e. The target detection model continuously queries the camera pan-tilt status and waits for the camera pan-tilt control to end, repeating the above steps ad tracking process until the tracked object is lost or the camera pan-tilt reaches the rotation limit.

[0067] f. Send a command to return the camera pan-tilt unit to its initial preset position, and wait for the execution of a new process check. Optionally, since the camera pan-tilt-zoom (PTZ) rotation takes time, the target detection model continuously requests the camera status and cancels the tracking of the previous static image, waiting for the PTN rotation to stop before continuing further logical judgment and data update, thus avoiding erroneous data and control caused by network latency and PTN operation latency; after the camera PTN operation stops, target tracking is restarted, and the mobile engineering transport vehicle target closest to the center of the screen is selected as the latest tracking object.

[0068] In another embodiment of the present invention, if the tracked object occupies a small portion of the camera image, it is also necessary to calculate the required magnification factor in conjunction with the manufacturer's SDK parameters, and send PTZ control commands to the gimbal camera for magnification, while simultaneously centering the target in the center of the screen. This is mainly to ensure that the width of the target object occupies 1 / 4 to 1 / 3 of the entire screen width, so as to avoid failing to recognize the vehicle body and license plate number.

[0069] 3. The vehicle behavior recognition module is used to detect the target behavior of the tracked engineering transport vehicles and generate corresponding alarm data based on the detection results.

[0070] Optionally, target behavior detection can be performed on the tracked engineering transport vehicles, specifically including detecting whether the engineering transport vehicles are tipped over, detecting whether the roofs of the engineering transport vehicles are covered, recognizing the license plates of the engineering transport vehicles, and detecting whether the engineering transport vehicles are leaving the site.

[0071] In another embodiment of the present invention, the vehicle behavior recognition module is specifically used for, The process of detecting the roof covering status of engineering transport vehicles begins with the tracking process. If the status changes, it is reported to the business platform. To detect whether engineering transport vehicles have tipped over, first report alarm data indicating that the vehicle has not tipped over; if it has tipped over, send alarm images and videos again. It detects the vehicle body and license plate of engineering transport vehicles and continuously updates the license plate number with the highest threshold within the application; The system detects whether engineering transport vehicles have left the site and sends an alarm image, video, and the license plate number with the highest threshold after departure.

[0072] In another embodiment of the present invention, taking an engineering transport vehicle for transporting construction waste (hereinafter referred to as a construction waste truck) as an example, the method detects whether the construction waste truck is dumping by detecting the angle of the truck bed (key point detection) and combining the motion blur and texture changes of the ROI region with the judgment of the soil pile target. Specifically, this includes... The subsequent logical judgment is performed when the dump truck target is stationary; the standard for the target being stationary is that the center coordinate of the target moves less than a set threshold within 2 seconds. Based on the outward expansion dynamic ratio of the dump truck detection frame; where the outward expansion dynamic ratio is width + 50% and height + 50%; Calculate pixel changes in the outer region over multiple consecutive frames (such as carriage elevation or soil pile changes). If the threshold is exceeded, perform soil pile target detection to determine whether a collapse may have occurred. The changes in the angle between the hinge point between the carriage and the front of the train, the midpoint of the rear of the carriage, the top of the hydraulic rod, and the horizontal plane are calculated in multiple consecutive frames. Thresholds are set; among them, the threshold for changes in the outer expansion area is set to 0.4, the threshold for changes in the carriage angle is set to 0.3, and the confidence threshold for soil pile detection is set to 0.3, and a comprehensive judgment is made; when the scores of a specified number of consecutive frames all exceed the design threshold (obtained from experiments), it is judged that a collapse has occurred; When tipping is detected, alarm data is generated and sent.

[0073] In another embodiment of the present invention, the method for detecting the roof covering status of a dump truck includes... The top 30% area of ​​the rectangular frame surrounding the dump truck is designated as the roof ROI area. The sliding window voting method is used to detect whether the roof of the dump truck is covered based on the statistical results of more than 70% of the frames over 5 consecutive seconds; the confidence level of each frame is required to be higher than the internally set threshold. Once the detection results are obtained, the detection is paused to conserve resources. The detection is then resumed after a certain period of time (e.g., 1 minute). If the detection results remain unchanged, no alarm is triggered. Conversely, if the detection results change, alarm data is generated and sent.

[0074] In another embodiment of the present invention, a method for license plate recognition of dump trucks includes, The OCR (Optical Character Recognition) algorithm is run on the detected vehicle body area; The Tesseract-OCR model is used to recognize Chinese, English, and numbers, and redundant language packets are removed. The recognition result is output when at least 3 out of 5 consecutive recognition results are consistent. Before the vehicle disappears, the license plate recognition results are continuously updated in memory, and the captured images are also updated in memory. After the vehicle leaves, the recognition result with the highest output threshold is reported.

[0075] In another embodiment of the present invention, the vehicle behavior recognition module is further configured to select a corresponding training labeled dataset before performing target behavior detection on the tracked engineering transport vehicle, thereby determining the threshold set in the above-mentioned process of detecting the behavior of the engineering transport vehicle. Taking an engineering transport vehicle transporting construction waste (hereinafter referred to as a construction waste truck) as an example, the strategy for selecting its training labeled dataset includes: Rent dump trucks to record NVR (Network Video Recorder) footage from multiple angles and at different zoom levels (appropriate zoom levels required for business purposes) within the same focal length. The NVR recording requirements include, but are not limited to: dump trucks of different sizes, models, and colors; vehicle roofs half-open or fully open in multiple camera angles; dumping actions of dump trucks at different angles in different camera angles; and examples of dumped yellow soil piles, black soil piles, or garbage piles. Based on NVR recordings, a 3D model was created using the 3D modeling tool Blende. Then, using the AI ​​plugin AI Rende, material textures were generated from text (such as "dirty dump truck"), and the 3D model was animated. Animation effects were added to the roof opening and closing and tilting actions from different angles. Key data is labeled on the 3D model after the animation is rendered, such as the detection frames for the body, roof, and soil pile of the dump truck, the hinge point of the truck body (used for calculating the tipping angle), the roof status, and the soil pile color, so as to determine the threshold set in the subsequent detection of the behavior of engineering transport vehicles.

[0076] Optionally, for multi-angle NVR recordings of dump truck movement, roof opening and closing (fully open / half open / closed), and dumping actions, zoom can be manually controlled; when annotating key data, nighttime dumping data can also be added to cope with nighttime black and white mode scenes.

[0077] Based on the above disclosure, the present invention also provides an electronic device. The electronic device of this embodiment includes at least one processor and at least one storage medium electrically connected to the processor. The storage medium is electrically connected to the processor, wherein the storage medium stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described above.

[0078] Based on the same inventive concept, the present invention also provides a storage medium storing instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described above.

[0079] The foregoing description and accompanying drawings fully illustrate embodiments of the invention to enable those skilled in the art to practice them. Other embodiments may include structural and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Some portions and features of some embodiments may be included or substituted for portions and features of other embodiments. Embodiments of the invention are not limited to the structures described above and shown in the accompanying drawings, and various modifications and changes may be made without departing from their scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for systematically detecting the behavior of engineering transport vehicles, characterized in that, The method includes, Connect the streaming media codec to the camera and obtain image frames by decoding the video; The object detection model is used to identify engineering transport vehicles in image frames and the identified engineering transport vehicles are continuously tracked. The system detects the target behavior of the tracked engineering transport vehicles and generates corresponding alarm data based on the detection results.

2. The method for process-oriented detection of engineering transport vehicle behavior according to claim 1, characterized in that, The step of connecting the streaming media codec to the camera and obtaining image frames by decoding the video specifically includes connecting the streaming media codec to the camera via a real-time streaming protocol and using a hardware codec provided by the edge gateway system to decode the video stream and obtain image frames.

3. The method for process-oriented detection of engineering transport vehicle behavior according to claim 1, characterized in that, The process of using a target detection model to identify engineering transport vehicles in image frames and continuously tracking the identified engineering transport vehicles specifically includes using a lightweight YOLOv5s target detection model to identify engineering transport vehicles in image frames and using a Re-ID+IOU hybrid tracking scheme to track the engineering transport vehicles.

4. The method for process-oriented detection of engineering transport vehicle behavior according to claim 1, characterized in that, The target behavior detection of the tracked engineering transport vehicles specifically includes detecting whether the engineering transport vehicles are tilted, detecting whether the roof of the engineering transport vehicles is covered, recognizing the license plates of the engineering transport vehicles, and detecting whether the engineering transport vehicles are leaving the site.

5. The method for process-oriented detection of engineering transport vehicle behavior according to claim 4, characterized in that, The detection of the tilting of engineering transport vehicles specifically includes: detecting the tilting of engineering transport vehicles while the target engineering transport vehicle is stationary; and detecting the dynamic proportion of the outward expansion of the frame of the engineering transport vehicle. Calculate the pixel changes in the extended region over multiple consecutive frames; if the changes exceed a threshold, perform soil pile target detection. The system calculates the changes in the angle between the car body and the front hinge point, the midpoint of the rear of the car body, the top of the hydraulic rod, and the horizontal plane over multiple consecutive frames; it sets thresholds for changes in the outward expansion area, car body angle changes, and soil pile detection confidence, and makes a comprehensive judgment; when the calculation results for a specified number of consecutive frames all exceed the design threshold, it is determined that a tipping has occurred; when a tipping is detected, alarm data is generated and sent. The detection of the roof cover status of the engineering transport vehicle specifically includes using a sliding window voting method to detect the roof cover status of the engineering transport vehicle based on the statistical results of a specific number of frames, and obtaining the detection results. After obtaining the detection results, the detection is re-executed after a certain period of time; if the detection results do not change, no alarm is triggered; otherwise, if the detection results change, alarm data is generated and sent. The license plate recognition for engineering transport vehicles specifically includes using the Tesseract-OCR model to recognize Chinese, English, and numbers in the detected vehicle body area, and removing redundant language packets; when at least 3 frames of recognition results are consistent across multiple consecutive frames, the recognition result is output.

6. A system for detecting the behavior of engineering transport vehicles, characterized in that, The system includes a streaming media encoding / decoding module, a vehicle tracking module, and a vehicle behavior recognition module; The streaming media codec module is used to connect the streaming media codec to the camera and obtain image frames by decoding the video; The vehicle tracking module is used to identify engineering transport vehicles in image frames using a target detection model and to continuously track the identified engineering transport vehicles. The vehicle behavior recognition module is used to detect the target behavior of the tracked engineering transport vehicles and generate corresponding alarm data based on the detection results.

7. The engineering transport vehicle behavior process detection system according to claim 6, characterized in that, The streaming media codec module is specifically used to connect the streaming media codec to the camera via a real-time streaming protocol and use the hardware codec provided by the edge gateway system to decode the video stream and obtain image frames.

8. The engineering transport vehicle behavior process-oriented detection system according to claim 6, characterized in that, The vehicle tracking module is specifically used to identify engineering transport vehicles in image frames using a lightweight YOLOv5s target detection model, and to track engineering transport vehicles using a Re-ID+IOU hybrid tracking scheme.

9. The engineering transport vehicle behavior process detection system according to claim 6, characterized in that, The vehicle behavior recognition module is specifically used to detect the tilting of engineering transport vehicles, the covering status of the roof of engineering transport vehicles, to recognize the license plates of engineering transport vehicles, and to detect the departure of engineering transport vehicles.

10. The engineering transport vehicle behavior process detection system according to claim 9, characterized in that, The vehicle behavior recognition module is specifically used to detect the tipping of the engineering transport vehicle when the target vehicle is stationary; and to detect the dynamic expansion ratio of the engineering transport vehicle frame. Calculate the pixel changes in the extended region over multiple consecutive frames; if the changes exceed a threshold, perform soil pile target detection. The system calculates the changes in the angle between the car body and the front hinge point, the midpoint of the rear of the car body, the top of the hydraulic rod, and the horizontal plane over multiple consecutive frames; it sets thresholds for changes in the outward expansion area, car body angle changes, and soil pile detection confidence, and makes a comprehensive judgment; when the calculation results for a specified number of consecutive frames all exceed the design threshold, it is determined that a tipping has occurred; when a tipping is detected, alarm data is generated and sent. The vehicle behavior recognition module is specifically used to detect the canopy covering status of engineering transport vehicles by using a sliding window voting method based on the statistical results of a specific number of frames, and obtain the detection results; after obtaining the detection results, the detection is re-executed after a certain period of time; if the detection results do not change, no alarm is triggered; otherwise, if the detection results change, alarm data is generated and sent. The vehicle behavior recognition module is specifically used to perform Chinese, English and number recognition on the detected vehicle body area using the Tesseract-OCR model, and remove redundant language packets; when at least 3 frames of recognition results in a series of consecutive frames are consistent, the recognition result is output.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the engineering transport vehicle behavior process-oriented detection method according to any one of claims 1-5.

12. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a program stored in memory, it implements the steps of the engineering transport vehicle behavior process detection method according to any one of claims 1-5.

Images