Physical simulation muck truck dumping behavior identification method, equipment and medium

Abstract

The invention discloses a physical simulation muck truck dumping behavior identification method and device and a medium, and relates to the technical field of engineering vehicle behavior analysis. The method comprises the following steps: carrying out slag car target detection on each frame of image in a monitoring area; recognizing coordinates of a plurality of key points of the target vehicle, and starting a shielding reasoning mode when the target vehicle is shielded; inputting the coordinates of the detected key points of the target vehicle into a pre-trained vehicle hopper attitude completion model; reasoning to obtain a predicted value and a confidence coefficient of a shielded key point of the vehicle through a vehicle hopper attitude completion model; based on the detected key point and a predicted value with the confidence greater than a preset threshold value, forming a hopper attitude of the target vehicle; and when it is judged that the car hopper posture is in the dumping state and the confidence coefficient of the dumping state exceeds a threshold value, early warning is triggered. According to the method, reliable reasoning based on local information from dependence on complete visual information is realized, the continuity of an identification process is ensured, and report omission is greatly reduced.

Description

Technical Field

[0001] This application relates to the field of engineering vehicle behavior analysis technology, and in particular to a physical simulation method, equipment and medium for recognizing dump truck dumping behavior. Background Technology

[0002] In scenarios such as smart construction sites and urban management enforcement, the automatic identification and monitoring of dump truck dumping behavior using fixed surveillance cameras has become an important technological means. Existing identification schemes are mostly based on the detection of key points on the vehicle, judging whether dumping has occurred by analyzing changes in geometric features such as the lifting angle of the truck bed. However, these schemes are usually based on an idealized premise that the target vehicle is always clearly and completely visible in the video sequence.

[0003] In real-world, complex operating environments, such as construction site exits and waste disposal sites, vehicles are densely packed and frequently obstruct traffic. Existing technologies face a critical flaw: when waste trucks are partially obstructed by other vehicles, buildings, or temporarily stockpiled materials, key point detection suffers significant loss or inaccuracy, causing judgment logic based on incomplete information to fail. This not only results in numerous missed detections but may also lead to false alarms due to incorrect inferences about the truck bed's posture. Therefore, in real-world industrial scenarios with strong obstruction and numerous interferences, the accuracy and reliability of existing technologies significantly decrease, making it difficult to meet actual regulatory requirements.

[0004] Based on the above analysis, the problems and shortcomings of the existing technology are as follows: Existing technologies for identifying dump truck dumping behavior cannot achieve continuous, stable, and reliable automatic identification of dump truck dumping behavior in complex monitoring scenarios with high frequency and multiple types of obstructions. Summary of the Invention

[0005] This application provides a physical simulation method, device, and medium for recognizing dump truck dumping behavior, which can solve the problem that existing dump truck dumping behavior recognition technologies cannot achieve continuous, stable, and reliable automatic recognition of dump truck dumping behavior in complex monitoring scenarios with high frequency and multiple types of obstructions.

[0006] In a first aspect, embodiments of this application provide a physical simulation method for recognizing dump truck dumping behavior. The method includes: within a monitoring area, performing dump truck target detection on each frame of image and assigning an identifier to the detected target vehicle; identifying the coordinates of multiple key points of the target vehicle, and activating an occlusion inference mode when the target vehicle is occluded; in the occlusion inference mode, inputting the coordinates of the key points detected by the target vehicle into a pre-trained truck bed posture completion model; inferring the predicted values ​​and confidence levels of the occluded key points of the vehicle through the truck bed posture completion model; constructing the truck bed posture of the target vehicle based on the detected key points and the predicted values ​​with confidence levels greater than a preset threshold; and triggering an early warning when it is determined that the truck bed posture is in a dumping state and the confidence level of the dumping state exceeds a threshold.

[0007] In one implementation of this application, the predicted values ​​and confidence levels of the occluded key points of the vehicle are inferred through a truck bed attitude completion model. Specifically, this includes: preprocessing the coordinates of the detected key points to normalize them to a relative coordinate system with the center of the vehicle detection box as the origin; the input layer of the truck bed attitude completion model is a fixed-length vector, which includes the normalized coordinates and visibility state of the key points, and the coordinates of the occluded key points are filled with zeros; the output layer of the truck bed attitude completion model is the predicted value of the normalized coordinates of the key points, and the loss function includes the localization loss of the detected key points and the prediction loss of the occluded key points.

[0008] In one implementation of this application, the method further includes: using an instance segmentation model to identify and track the outline of an occluder; using a Kalman filter to predict the position of the occluder within a preset number of frames based on the occluder's motion speed and direction; estimating the range of frames in which the target vehicle will fully appear during the occlusion based on the position within the preset number of frames; calculating the average Euclidean distance between the predicted value of the preset number of frames during the occlusion and the actual detected coordinates when visibility is restored; and determining that the inference is reliable during the occlusion if the average Euclidean distance is less than a preset pixel fault tolerance threshold.

[0009] In one implementation of this application, the method further includes: calculating the distance between the center points of the vehicle detection frames of any two vehicles in a continuous frame; if the distance between the center points is less than a preset multiple of the sum of the widths of the two vehicles, determining that the two vehicles are spatially adjacent and establishing a spatial adjacency edge; recording the timestamps for the establishment and disappearance of the spatial adjacency edge, and checking whether there are three or more spatial adjacency edges overlapping in time within the same local area; if so, and more than half of the vehicles connected by the spatial adjacency edge are determined to be in a tilted state during the overlapping time period, triggering a high-level alarm.

[0010] In one implementation of this application, the key points include wheel key points, bottom key points of the truck bed, and top key points of the truck bed; the wheel key points include the center of the left front wheel, the center of the right front wheel, the center of the left rear axle, and the center of the right rear axle; the bottom key points of the truck bed include the front corner of the bottom of the left truck bed, the rear corner of the bottom of the left truck bed, the front corner of the bottom of the right truck bed, and the rear corner of the bottom of the right truck bed; the top key points of the truck bed include the highest point of the top of the left truck bed and the highest point of the top of the right truck bed.

[0011] In one implementation of this application, the method further includes: fitting a line connecting key points of the wheels to obtain a chassis horizontal line; fitting a line connecting key points at the bottom of the truck bed to obtain a truck bed line; calculating the angle between the chassis horizontal line and the truck bed line; taking two corner points at the top of one side of the truck bed tail and calculating the length of the connecting line to obtain a first distance; taking two corner points at the top and bottom of one side of the truck bed tail and calculating the length of the connecting line to obtain a second distance.

[0012] In one implementation of this application, before inferring the predicted values ​​and confidence levels of key points of the occluded part of the vehicle through the truck bed attitude completion model, the method further includes: simulating lateral occlusion, rear occlusion, and mixed occlusion when constructing training samples, and defining the mask of discarded key points; determining the occlusion type based on the distribution pattern of visible key points, including lateral vehicle occlusion, rear vehicle occlusion, and static object occlusion; and selecting the corresponding truck bed attitude completion model according to the occlusion type.

[0013] In one implementation of this application, after triggering an early warning when the truck bed is determined to be in a tilted state and the confidence level of the tilted state exceeds a threshold, the method further includes: in the truck bed posture completion fine-tuning stage, freezing the backbone feature extraction network of the truck bed posture completion model, retraining the final fully connected regression layer; selecting the samples with the largest deviation between the truck bed posture completion predicted coordinates and the manually corrected coordinates to form a difficult sample set; and using the difficult sample set to perform a limited number of iterative training iterations on the truck bed posture completion model.

[0014] Secondly, embodiments of this application also provide a physical simulation dump truck dumping behavior recognition device, the device including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to: perform any one of the steps of a physical simulation dump truck dumping behavior recognition.

[0015] Thirdly, this application also provides a non-volatile computer storage medium for physically simulating dump truck dumping behavior recognition, which stores computer-executable instructions, configured to execute any one of the steps of physically simulating dump truck dumping behavior recognition.

[0016] This application provides a physical simulation-based method, device, and medium for recognizing dump truck dumping behavior. By introducing an occlusion inference mode and a truck bed posture completion model, it can intelligently predict the structure of the occluded part based on visible information when key points of the vehicle are partially occluded. This achieves a shift from relying on complete visual information to reliable inference based on local information, ensuring the continuity of the recognition process and greatly reducing false alarms. By designing a multi-feature fusion confidence evaluation mechanism and performing post-event verification of the inference results during occlusion, it provides a reliable quantitative indicator for the system output, effectively filtering false alarms caused by high inference uncertainty, making alarm information more accurate and reliable. Through multi-vehicle association analysis, it can identify collaborative operation modes such as clustered illegal dumping, and selects dedicated models for different occlusion types, making the system more robust when facing dynamic and complex scenarios. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A flowchart of a physical simulation method for recognizing dump truck dumping behavior is provided in this application embodiment; Figure 2 This is an internal schematic diagram of a physical simulation device for recognizing dump truck dumping behavior, provided as an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] This application provides a physical simulation method, device, and medium for recognizing dump truck dumping behavior, which solves the problem that existing dump truck dumping behavior recognition technologies cannot achieve continuous, stable, and reliable automatic recognition of dump truck dumping behavior in complex monitoring scenarios with high frequency and multiple types of obstructions.

[0020] The technical solutions proposed in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0021] Figure 1 This document presents a flowchart illustrating a physical simulation method for recognizing the dumping behavior of construction waste trucks, as provided in an embodiment of this application. Figure 1As shown in the figure, the physical simulation method for recognizing dump truck dumping behavior provided in this application includes the following steps: Step 10: Within the monitored area, perform dump truck target detection on each frame of the image and assign an identifier to the detected target vehicle.

[0022] In this step, real-time video streams are captured using fixed surveillance cameras deployed in key areas such as construction site exits and disposal sites. Each frame is preprocessed, and the model employs a YOLOv5-based architecture. It is trained and optimized using a large number of construction site scene images containing dump trucks, enabling it to accurately distinguish dump trucks from other vehicles, construction machinery, and the background environment. The model outputs bounding boxes for all dump truck targets in the image. For each detected dump truck target in each frame, a unique identifier is assigned. To achieve continuous tracking across frames, the DeepSORT multi-target tracking algorithm is integrated. By combining the target's appearance features and motion information, and using Kalman filter prediction, detection boxes belonging to the same vehicle in consecutive frames are associated, thus maintaining the same ID for the same vehicle throughout the entire video sequence. This ID serves as the vehicle's unique identification credential within the entire system.

[0023] Step 20: Identify the coordinates of multiple key points of the target vehicle, and activate the occlusion inference mode when the target vehicle is occluded.

[0024] In this step, based on the successful detection and tracking of the dump truck, for each tracked vehicle target, a deep convolutional neural network is trained to regress the two-dimensional pixel coordinates of multiple predefined key points in the image. These key points are designed according to the physical structure of the dump truck and include at least the wheel key points: center of the left front wheel, center of the right front wheel, center of the left rear axle, and center of the right rear axle; the bottom key points of the truck bed: front corner of the left bottom, rear corner of the left bottom, front corner of the right bottom, and rear corner of the right bottom; and the top feature points of the truck bed: the highest point of the left top and the highest point of the right top. The key point recognition module outputs a confidence score along with the coordinates of each key point, reflecting the model's confidence in predicting the location of that point.

[0025] Furthermore, a keypoint visibility status list is maintained for each vehicle ID. If the confidence level of a keypoint is lower than a preset visibility threshold for N consecutive frames, the keypoint is determined to be in a continuously lost state. When a vehicle has more than a certain number or specific locations of keypoints determined to be continuously lost, the vehicle is identified as being in an occluded state, and an occlusion inference mode is immediately initiated for it.

[0026] Step 30: In occlusion inference mode, input the coordinates of the key points detected by the target vehicle into the pre-trained truck bed posture completion model.

[0027] In this step, once a vehicle ID enters the occlusion inference mode, the analysis will no longer be interrupted due to missing key points, but will instead rely on the truck bed attitude completion model for continuous inference.

[0028] Step 40: Using the truck bed attitude completion model, infer the predicted values ​​and confidence levels of the key points of the vehicle that are occluded; As an optional embodiment, the predicted values ​​and confidence levels of the occluded key points of the vehicle are obtained through the truck bed attitude completion model. Specifically, this may include: Step 401: Preprocessing the coordinates of the detected key points to normalize them to a relative coordinate system with the center of the vehicle detection box as the origin; Step 402: The input layer of the truck bed attitude completion model is a fixed-length vector, which includes the normalized coordinates and visibility state of the key points. For occluded key points, the coordinates are filled with zeros; Step 403: The output layer of the truck bed attitude completion model is the predicted value of the normalized coordinates of the key points. The loss function includes the localization loss of the detected key points and the prediction loss of the occluded key points.

[0029] In this step, for key points that are determined to be invisible, the corresponding coordinate positions are filled with the zero value (0,0). The latter half of the vector is a binary mask vector of the same length, used to represent the visibility state: for visible points, the state bit is 1; for invisible points, the state bit is 0.

[0030] Furthermore, a multi-layer fully connected neural network is used. The number of neurons in the output layer corresponds to the coordinate dimension in the input vector. It directly outputs the predicted coordinates of all predefined keypoints, including both visible and invisible points, in a normalized relative coordinate system. The localization loss for detected keypoints is calculated by comparing the mean squared error between the model's predicted coordinates and the true coordinates for all keypoints marked as visible in the input. The prediction loss for occluded keypoints is calculated by comparing the mean squared error between the model's predicted coordinates and the true coordinates for all keypoints marked as invisible in the input.

[0031] Step 50: Based on the detected key points and the predicted values ​​with confidence levels greater than a preset threshold, the truck bed posture of the target vehicle is formed.

[0032] In this step, for occluded key points, the confidence level of the predicted values ​​output by the truck bed attitude completion model in step 40 depends on the confidence level of the predicted values. Only when the prediction confidence level of an occluded key point is higher than this threshold will its predicted normalized coordinates be inversely transformed back to the image pixel coordinate system and adopted as the final coordinates of the key point. If its prediction confidence level is lower than or equal to the threshold, it indicates that the model's inference uncertainty for this point is too high, and the point is still marked as invalid in this frame and does not participate in the geometric feature calculation of this frame.

[0033] Step 60: When the truck bed is determined to be tilted and the confidence level of the tilted state exceeds the threshold, an early warning is triggered.

[0034] As an optional embodiment, the method may further include: using an instance segmentation model to identify and track the outline of an occluder; using a Kalman filter to predict the position of the occluder within a preset number of frames based on the occluder's motion speed and direction; estimating the range of frames in which the target vehicle will fully appear during the occlusion based on the position within the preset number of frames; calculating the average Euclidean distance between the predicted values ​​of the preset number of frames during the occlusion and the actual detected coordinates when visibility is restored; and determining that the inference is reliable during the occlusion if the average Euclidean distance is less than a preset pixel tolerance threshold.

[0035] In this step, the reliability of the occlusion inference results is verified. That is, it is not limited to analyzing the target vehicle itself, but also actively identifies and tracks the objects that cause occlusion. The instance segmentation model is used to process the monitoring screen, which can generate pixel-level accurate segmentation masks for each independent object in the image. By analyzing the spatial positional relationship between these masks and the target dump truck detection box, the object causing the current occlusion can be accurately identified and its contour information can be extracted. In subsequent frames, the contour of the occluding object is continuously tracked and its motion trajectory is recorded.

[0036] Furthermore, based on the historical motion trajectory of the occluder, a Kalman filter is used to predict the future motion state. The Kalman filter predicts the position coordinates within a preset number of frames in the future based on the position and velocity of the occluder in the current frame and several previous frames. Based on these predicted positions and the relative spatial relationship between the target vehicle and the occluder, the approximate number of frames from which the target vehicle can be completely reproduced from the occlusion can be dynamically estimated.

[0037] Furthermore, once the target vehicle is confirmed to have fully recovered its visibility within the predicted time window, a verification process is immediately initiated. The coordinates of the occluded key points predicted by the truck bed attitude completion model in the last M frames during the occlusion period are extracted. Simultaneously, the actual detected coordinates of these key points are obtained in the first frame after the vehicle is fully reproduced. For each previously occluded key point, the average Euclidean distance between the predicted coordinates within the M frames and the actual coordinates after reproduction is calculated. A preset pixel tolerance threshold is used. If the average Euclidean distance of all key points is less than this threshold, or if the mean of the average Euclidean distances of all key points is less than this threshold, the inference of the truck bed attitude completion model during the just-ended occlusion period is deemed reliable. This verification result is recorded and linked to the inference data for that time period. If the distance exceeds the threshold, it indicates poor inference quality during the occlusion period, and the priority of related alarms can be reduced or the alarm can be marked as pending verification.

[0038] As an optional embodiment, the method may further include: calculating the center point distance between the centers of the vehicle detection frames of any two vehicles in consecutive frames; if the center point distance is less than a preset multiple of the sum of the widths of the two vehicles, determining that the two vehicles are adjacent in space and establishing a spatial adjacency edge; recording the timestamps of the establishment and disappearance of the spatial adjacency edge, and checking whether there are three or more spatial adjacency edges overlapping in time within the same local area; if so, and more than half of the vehicles connected by the spatial adjacency edges are determined to be in a tipping state during the overlapping time period, triggering a high-level alarm.

[0039] In this step, a dynamic, undirected spatial relationship graph G=(V,E) is constructed for all tracked construction waste transport vehicles in the monitoring area. The vertex set V of the graph consists of all active vehicle identifiers, and the edge set E of the graph is dynamically generated by the spatial adjacency relationships between vehicles.

[0040] During the processing of each frame, calculate the center point distance Dij between any two vertices Vi and Vj in the graph. This distance is the Euclidean distance between the center points of the detection frames of the two vehicles in the image pixel coordinate system. At the same time, obtain the detection frame widths Wi and Wj of the two vehicles.

[0041] Furthermore, a spatial adjacency multiple factor k can be set. If the center point distance between the two vehicles satisfies Dij < k*(Wi + Wj), it is determined that vehicles Vi and Vj are adjacent in space in the current frame. Then, check whether there is a spatial adjacency edge Eij between them in the spatial relationship graph; if not, create this edge and record the establishment timestamp of this edge; if it already exists, update the last active time of this edge. When the distance between the two vehicles exceeds the above condition, record the disappearance timestamp of this edge, but retain the historical record of this edge for a period of time for subsequent analysis.

[0042] Still further, periodically scan the historical edge records of the spatial relationship graph to find potential cluster behaviors, and check whether there is a local area where at least three active spatial adjacency edges exist simultaneously within the same time period, and these edges connect three or more vehicles. For each identified spatio-temporal overlapping cluster, perform behavior consistency analysis. It retrieves the number of vehicles independently determined to be in a tipping state among all vehicles connected by these adjacent edges during this time overlap segment. If it exceeds half of the total number of vehicles in the cluster, it is determined that a cluster violation tipping has occurred. At this time, trigger a high-level alarm. This alarm is different from the single-vehicle alarm. It will clearly mark the alarm type as cluster violation and record the list of all involved vehicle IDs, the core time period of the cluster activity, and the geographical area where the cluster is located. This alarm will provide supervisors with higher-dimensional decision-making information reflecting collaborative operation violations, greatly improving the efficiency of cracking down on large-scale illegal tipping behaviors.

[0043] As an optional embodiment, the key points include wheel key points, bottom key points of the truck bed, and top key points of the truck bed; the wheel key points include the center of the left front wheel, the center of the right front wheel, the center of the left rear axle, and the center of the right rear axle; the bottom key points of the truck bed include the front corner of the bottom of the left truck bed, the rear corner of the bottom of the left truck bed, the front corner of the bottom of the right truck bed, and the rear corner of the bottom of the right truck bed; the top key points of the truck bed include the highest point of the top of the left truck bed and the highest point of the top of the right truck bed.

[0044] As an optional embodiment, the method may further include: fitting a line connecting key points of the wheels to obtain a chassis horizontal line; fitting a line connecting key points at the bottom of the truck bed to obtain a truck bed line; calculating the angle between the chassis horizontal line and the truck bed line; taking two corner points at the top of one side of the truck bed tail and calculating the length of the connecting line to obtain a first distance; taking two corner points at the top and bottom of one side of the truck bed tail and calculating the length of the connecting line to obtain a second distance.

[0045] In this step, when the included angle exceeds the preset included angle threshold and continues to increase in consecutive frames, it is determined to be a tilted state from the side view; when the first distance continues to increase in consecutive frames and the second distance continues to decrease in consecutive frames, it is determined to be a tilted state from the rear view.

[0046] As an optional embodiment, before inferring the predicted values ​​and confidence levels of key points of the occluded part of the vehicle through the truck bed attitude completion model, the method may further include: Step 01: When constructing training samples, simulate side occlusion, rear occlusion, and mixed occlusion, and define the mask of discarded key points; Step 02: Determine the occlusion type based on the distribution pattern of visible key points, including side vehicle occlusion, rear vehicle occlusion, and static object occlusion; Step 03: Select the corresponding truck bed attitude completion model according to the occlusion type.

[0047] In this step, the accuracy of pose completion is improved by subdividing the occlusion scene and matching it with a dedicated model. During the model training phase, three typical occlusions are simulated and corresponding keypoint discarding rules are defined: side occlusion: discard all keypoints on one side of the vehicle; rear occlusion: discard all corner points at the rear of the truck bed; mixed occlusion: randomly discard keypoints from different parts. Based on these rules, the complete dataset is processed to generate training samples with type labels.

[0048] When occlusion inference is initiated, the type is first determined based on the distribution of visible keypoints. If the missing keypoints are concentrated on one side, it is classified as lateral occlusion; if a large number of keypoints are missing only at the rear corner of the truck bed, it is classified as rear occlusion; if the missing points are scattered and irregular, it is classified as mixed occlusion. Pre-set models specifically trained using the above three types of data are used. Based on the identified real-time occlusion type, the corresponding dedicated model is automatically scheduled and invoked for keypoint prediction, thereby improving the accuracy of completion under complex occlusion conditions.

[0049] As an optional embodiment, after triggering an early warning when the truck bed is determined to be in a tilted state and the confidence level of the tilted state exceeds a threshold, the method may further include: Step 701: During the truck bed posture completion fine-tuning stage, freeze the backbone feature extraction network of the truck bed posture completion model and retrain the final fully connected regression layer; Step 702: Select the samples with the largest deviation between the truck bed posture completion predicted coordinates and the manually corrected coordinates to form a difficult sample set; Step 703: Use the difficult sample set to perform a limited number of iterations to train the truck bed posture completion model.

[0050] In this step, the model is periodically optimized. During this phase, the backbone feature extraction network of the truck bed attitude completion model is frozen, and only the fully connected regression layers at the ends are retrained. This effectively preserves the general features learned by the model while adapting to new data, preventing catastrophic forgetting. A hard sample set is constructed by selecting samples from historical data with the largest deviations between predicted and manually corrected coordinates. This ensures that optimization resources are concentrated on the cases that the model currently struggles to accurately handle. Using the hard sample set, the model is trained iteratively a limited number of times with a small learning rate. This training method allows the model to quickly focus on improving its weak points, significantly enhancing its robustness in complex scenarios while avoiding overfitting.

[0051] The above are embodiments of the method proposed in this application. Based on the same inventive concept, embodiments of this application also provide a physical simulation-based device for recognizing the dumping behavior of construction waste trucks, which, as shown... Figure 2 As shown.

[0052] Figure 2 This is a schematic diagram of the internal structure of a physical simulation device for recognizing the dumping behavior of construction waste trucks, provided as an embodiment of this application. Figure 2 As shown, the device includes: At least one processor 201; And a memory 202 that is communicatively connected to at least one processor; The memory 202 stores instructions that can be executed by at least one processor, which are executed by at least one processor 201 to enable at least one processor 201 to: perform any one of the steps of a physical simulation for recognizing dump truck dumping behavior.

[0053] Some embodiments of this application provide corresponding to Figure 1 A non-volatile computer storage medium for physically simulating dump truck dumping behavior recognition stores computer-executable instructions, which are configured to execute any one of the steps of physically simulating dump truck dumping behavior recognition.

[0054] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for IoT devices and media are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0055] The systems, media, and methods provided in this application are one-to-one correspondences. Therefore, the systems and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the systems and media will not be repeated here.

[0056] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0057] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0058] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0059] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0060] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0061] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0062] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0063] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0064] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for recognizing a dumping behavior of a sludge truck in a physical simulation, characterized by, The method comprises: In a monitoring area, performing target detection on each frame of image, and assigning an identifier to a detected target vehicle; Identifying coordinates of a plurality of key points of the target vehicle, and starting an occlusion inference mode when the target vehicle is occluded; In the occlusion inference mode, inputting the coordinates of the detected key points of the target vehicle into a pre-trained vehicle hopper posture completion model; Through the vehicle hopper posture completion model, inferring a predicted value and a confidence of the occluded key points of the vehicle; Based on the detected key points and the predicted value with a confidence greater than a preset threshold, composing a vehicle hopper posture of the target vehicle; When determining that the vehicle hopper posture is in a dumping state and the confidence of the dumping state exceeds a threshold, triggering a warning.

2. The method of claim 1, wherein, The inferring of the predicted value and the confidence of the occluded key points of the vehicle through the vehicle hopper posture completion model specifically comprises: Preprocessing the coordinates of the detected key points to normalize to a relative coordinate system with the center of the vehicle detection frame as the origin; The input layer of the vehicle hopper posture completion model is a fixed-length vector, the vector includes the normalized coordinates and visibility state of the key points, and for the occluded key points, the coordinates are padded with zeros; The output layer of the vehicle hopper posture completion model is the normalized coordinate prediction value of the key points, and the loss function includes the positioning loss of the detected key points and the prediction loss of the occluded key points.

3. The method of claim 1, wherein the method further comprises: The method further comprises: Using an instance segmentation model to identify and track the outline of the occlusion; Based on the motion speed and direction of the occlusion, using a Kalman filter to predict the position of the occlusion within a preset number of frames; According to the positions within the preset number of frames, estimate the frame number range in which the target vehicle completely appears in the occlusion; Calculate the average Euclidean distance between the predicted value of a preset number of frames during the occlusion and the actual detection coordinates when the visibility is restored; If the average Euclidean distance is less than a preset pixel tolerance threshold, it is determined that the inference during the occlusion is reliable.

4. The method of claim 1, wherein the method further comprises: The method further comprises: Calculating the center point distance between the centers of the vehicle detection frames in any two vehicles in consecutive frames; If the center point distance is less than a preset multiple of the sum of the widths of the two vehicles, it is determined that the two vehicles are spatially adjacent, and a spatial adjacency edge is established; Record the timestamps of the establishment and disappearance of the spatial adjacency edge, and check whether there are three or more spatial adjacency edges overlapping in time in the same local area; If there are, and more than half of the vehicles connected by the spatial adjacency edges are determined to be in a dumping state within the overlapping time period, a high-level alarm is triggered.

5. The method of claim 4, wherein the method further comprises: The key points include vehicle wheel key points, vehicle hopper bottom key points, and vehicle hopper top feature points; The vehicle wheel key points include the center of the left front wheel, the center of the right front wheel, the center of the left rear wheel rear axle, and the center of the right rear wheel rear axle; The vehicle hopper bottom key points include the left hopper bottom front corner, the left hopper bottom rear corner, the right hopper bottom front corner, and the right hopper bottom rear corner; The vehicle hopper top feature points include the left hopper top highest point and the right hopper top highest point.

6. The method of claim 5, wherein the method further comprises: The method further comprises: Fitting a line to the wheel key points to obtain a chassis horizontal line, and fitting a line to the bottom of the hopper key points to obtain a hopper line; Calculating the included angle between the chassis horizontal line and the hopper line; Taking two corner points on one side of the top of the hopper tail, calculating the length of the connecting line to obtain a first distance, and taking two corner points on one side of the top of the hopper tail, calculating the length of the connecting line to obtain a second distance.

7. The method of claim 1, wherein the method further comprises: Before inferring the predicted value and confidence of the key points of the occluded part of the vehicle through the hopper posture completion model, the method further comprises: When constructing the training sample, simulating side occlusion, rear occlusion and mixed occlusion, and defining the discarded key point mask; According to the distribution mode of the visible key points, determining the occlusion type, the occlusion type including side vehicle occlusion, rear vehicle occlusion and static object occlusion; According to the occlusion type, selecting the corresponding hopper posture completion model.

8. The method of claim 1, wherein the method further comprises: After triggering the early warning when determining that the hopper posture is in the dumping state and the confidence of the dumping state exceeds the threshold, the method further comprises: In the hopper posture completion fine-tuning stage, freezing the backbone feature extraction network of the hopper posture completion model, and retraining the last fully connected regression layer; Screening the sample with the largest deviation between the hopper posture completion prediction coordinates and the manually corrected coordinates to form a difficult sample set; Using the difficult sample set to perform a limited number of iterative training on the hopper posture completion model.

9. A physical simulation of a slurry car dump behavior recognition device, characterized by, The device comprises: At least one processor; And a memory connected in communication with the at least one processor; Wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to: Perform the steps of the physical simulation of the muck car dumping behavior recognition according to any one of claims 1-8. 10.A non-transitory computer storage medium storing computer-executable instructions for physical simulation of a dump behavior of a dump truck, the computer-executable instructions comprising: The computer executable instructions are configured to: Perform the steps of the physical simulation of the muck car dumping behavior recognition according to any one of claims 1-8.

Images