A method for detecting a haul-off tipping area

Abstract

The application discloses a kind of carrying dump area detection method, it is related to computer vision technical field.The method obtains bearing state information, determines bearing reference surface according to bearing state information, and relative to bearing reference surface sets multiple candidate bearing layers that update with carrying bearing mechanism state, determines candidate space position according to the image position of cargo feature and each candidate bearing layer, and filters out candidate space position that does not meet actual bearing and release condition;According to the retained candidate space position and bearing state information, determine candidate material dropping position, associate the candidate material dropping position corresponding to the same cargo feature as candidate material dropping group, to limit the first material dropping occupancy contribution of each candidate material dropping group respectively, generate the predicted material dropping envelope of cargo;According to the positional relationship between predicted material dropping envelope and dump area, output result;So as to be able to reduce the dump area judgment error caused by fixed ground height projection and isogenic candidate position repeated statistics.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method for detecting areas that are being transported and tipped over. Background Technology

[0002] In the handling and dumping of ores, scrap steel, sand and gravel, construction waste, grain, powder, or packaged goods, it is usually necessary to determine whether the goods can be dumped into pre-designated pits, silos, stockpiles, unloading chute areas, or other permitted dumping areas. Existing solutions can acquire images of the handling operation scene, identify the goods area in the images, and then map the goods image position to a ground coordinate system or a bird's-eye view coordinate system to determine the positional relationship between the goods and the permitted dumping area.

[0003] Cargo has a significant spatial height. The top, sides, and edges of cargo captured by a camera do not necessarily correspond to their ground positions. If projection is performed solely based on a ground reference plane, the position of the same cargo feature in the bird's-eye view coordinate system will be offset by its actual height. To reduce this offset, multiple height planes parallel to the ground can be pre-defined, and projection relationships from the image plane to the bird's-eye view coordinate system can be established for each. This allows the same cargo feature to generate multiple candidate positions under different height assumptions.

[0004] However, in actual handling and tipping processes, goods do not always maintain a fixed posture relative to the ground. For example, forklift forks can rise, fall, and tilt; dump truck hoppers can tilt around their pivots; grippers can cause goods to swing; and pallets may change angle during unloading. When the handling and load-bearing mechanism changes posture, the spatial layer where the goods are primarily distributed changes synchronously with the mechanism. Continuing to use a fixed height plane parallel to the ground will generate a large number of candidate positions that do not conform to the actual geometry of the load-bearing mechanism.

[0005] Furthermore, the fact that the cargo is currently above the permitted dumping area does not necessarily mean that it will fall into that area after release. The cargo must first cross physical release boundaries such as the hopper front lip, fork front edge, pallet unloading end, or unloading gate edge, and then move to the ground or receiving surface under the translational, tilting, and gravity forces of the handling and carrying mechanism. Judging only the relationship between the cargo's current bird's-eye view position and the dumping area may mistakenly assume that cargo that has not yet reached the release boundary has entered the dumping area, or may overlook the actual boundary violation caused by horizontal displacement after release.

[0006] When the height of a cargo feature cannot be uniquely determined, the same cargo feature may generate multiple candidate drop locations. If these candidate drop locations are accumulated as complete and independent cargo occupancy, the same physical cargo feature may simultaneously make duplicate contributions both within and outside the dumping area, resulting in an expanded expected drop range, false cross-boundary reports, or conflicting results.

[0007] Therefore, there is a need for a detection method that can utilize the load-bearing structure, real-time attitude, release boundary, and tilting motion of the handling and tipping equipment to physically constrain the spatial candidate positions of goods and further determine the expected drop range of the goods. Summary of the Invention

[0008] The technical problem to be solved by this application is: when the height of the goods cannot be uniquely determined and the handling and carrying mechanism is raised, lowered, or tilted, how to eliminate spatial candidate positions that do not conform to the actual carrying and release process, avoid multiple drop candidates with the same goods characteristics being repeatedly counted in different areas, and determine the relationship between the expected drop range after the goods are released and the dumping area.

[0009] To address the aforementioned technical problems, this application acquires images of the handling and tipping scene, tipping area information, camera calibration information, and the load-bearing status information of the handling and carrying mechanism. The load-bearing status information at least characterizes the spatial position and orientation of the handling and carrying mechanism.

[0010] Based on the load-bearing status information, the load-bearing reference surface used to carry the goods, the release boundary where the goods detach from the handling and carrying mechanism, and the tilting direction from the load-bearing side to the release side are determined. The load-bearing reference surface is not a fixed ground height plane, but changes synchronously with the real-time position and attitude of the forks, hopper bottom plate, pallet, or gripper.

[0011] Multiple cargo features are identified from the cargo region of the scene image, and multiple candidate carrier layers are set relative to the carrier reference plane. Each candidate carrier layer represents a different possible normal distance of a cargo feature relative to the current carrier reference plane. The image position of each cargo feature is intersected or projected onto each candidate carrier layer to obtain multiple candidate spatial positions.

[0012] Based on the relationship between candidate spatial locations and the carrying space, release boundary, and tilting direction of the handling and tilting mechanism, candidate spatial locations located inside the carrying mechanism, outside the carrying space and deviating from the release channel, or whose movement direction is inconsistent with the current tilting direction are eliminated. This step ensures that the candidate locations processed subsequently conform to the actual physical structure of the handling and tilting equipment.

[0013] Based on the retained candidate spatial locations and carrying status information, the candidate release locations when the corresponding cargo features arrive at or cross the release boundary are determined. Then, based on the ground clearance of the release location and the translational and tilting movements of the handling and carrying mechanism, the candidate drop locations after the cargo features are released are determined.

[0014] Candidate drop locations corresponding to the same cargo feature are associated as candidate drop groups. When generating the expected drop envelope, the candidate drop group is used as the smallest drop occupancy unit, so that multiple candidate drop locations for the same cargo feature jointly limit the drop occupancy contribution once, without increasing the total contribution of the cargo feature due to the increase in the number of candidate locations.

[0015] The expected material drop envelope is generated based on each candidate material drop group, and it is determined whether the expected material drop envelope is located within the inward safety boundary of the dumping area, completely outside the dumping area, simultaneously covering both inside and outside the dumping area, or in an uncertain positional relationship.

[0016] Beneficial effects First, the candidate load-bearing layer is set relative to the load-bearing reference surface of the handling load-bearing mechanism and changes synchronously with the posture of the load-bearing mechanism, which can reduce the drift of the candidate position caused by the fixed horizontal height plane when the hopper tilts, the forks pitch, or the pallet rotates.

[0017] Second, by jointly screening candidate spatial locations based on the bearing space, release boundary, and tilting direction, candidate locations that are valid in image projection but impossible to bear or release in the physical structure of the device can be excluded.

[0018] Third, by calculating candidate release positions and candidate drop positions, the detection target is changed from the current projected position of the goods to the expected drop range after the goods are released, which is more in line with the actual safety judgment target of handling and dumping operations.

[0019] Fourth, by maintaining the homogeneity of candidate material drop locations and limiting the material drop occupancy contribution to a single drop by candidate material drop groups, the number of complete occupancy contributions formed repeatedly inside and outside the dumping area due to multiple height assumptions for the same cargo feature can be reduced.

[0020] Fifth, by comparing the expected material drop envelope with the inward safety boundary of the dumping area, anticipated cross-boundary risks can be identified before the material drop occurs, providing a basis for suspending dumping, adjusting vehicles, or supplementing image acquisition. Attached Figure Description

[0021] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an undue limitation of this application.

[0022] Figure 1 This is a schematic diagram of the architecture of the handling and tipping area detection system provided in the embodiments of this application; Figure 2 This is a schematic diagram of a handling and dumping operation scenario provided in an embodiment of this application; Figure 3 This is a flowchart of the handling and tipping area detection method provided in the embodiments of this application; Figure 4 This is a schematic diagram of the load-bearing reference surface and the follow-up candidate load-bearing layer provided in the embodiments of this application; Figure 5 A schematic diagram illustrating the screening of carrying space and release boundaries provided in an embodiment of this application; Figure 6 A schematic diagram illustrating the determination of candidate release positions and candidate material dropping positions provided in an embodiment of this application; Figure 7 A schematic diagram of the single-occupancy contribution of the candidate material feeding group provided in the embodiments of this application; Figure 8 A schematic diagram showing the expected material drop envelope located within the dumping area, provided for an embodiment of this application; Figure 9 A schematic diagram illustrating the expected material drop envelope crossing the boundary of the dumping area, provided in an embodiment of this application; Figure 10 This is a schematic diagram illustrating the space correction for the undetermined state of the material discharge boundary provided in an embodiment of this application. Figure 11 This is a schematic diagram illustrating the state maintenance of continuous image frames provided in an embodiment of this application.

[0023] Explanation of reference numerals in the attached figures 100. Image acquisition device; 200. Data processing device; 300. Handling equipment controller; 400. Tilting area; 410. Tilting area boundary; 420. Inward safety boundary; 500. Handling and bearing mechanism; 510. Bearing reference surface; 520. Release boundary; 530. Bearing space; 540. Tilting direction. Detailed Implementation

[0024] The present application will be described below with reference to the accompanying drawings and specific embodiments. The equipment type, coordinate values, distance, time, angle, scale, and judgment conditions in the embodiments are used to illustrate data processing relationships and are not intended to limit the scope of protection. Those skilled in the art can make adjustments based on the dimensions of the handling equipment, the camera installation location, the type of goods, the size of the tipping area, and operational safety requirements.

[0025] The scene coordinate system in this application is a unified spatial coordinate system used to represent the camera, the handling and supporting mechanism, the goods, the dumping area, and the receiving surface. The equipment coordinate system is fixed to the handling equipment or the handling and supporting mechanism, and the camera coordinate system is fixed to the image acquisition equipment. The data processing equipment uses camera calibration information and the pose information of the handling and supporting mechanism to transform between the above coordinate systems, so that the positions and orientations obtained from different sources can be compared in the same coordinate system.

[0026] In this application, the load-bearing reference surface is a spatial reference surface used in the handling and bearing mechanism to describe the relative load-bearing position of the goods. The load-bearing reference surface may coincide with the bottom surface of the hopper, the upper surface of the forks, the upper surface of the pallet, or the gripper support surface, or it may be obtained by fitting multiple structural key points on the handling and bearing mechanism. The candidate load-bearing layer is a spatial layer set relative to the current load-bearing reference surface, used to represent different possible normal distances of the goods features from the load-bearing reference surface.

[0027] In this application, the release boundary is the equipment structure boundary that goods must cross when changing from a load-bearing state to a release state. The release boundary divides its adjacent space into the load-bearing side, the boundary adjacent range, and the release side. The dumping direction points from the load-bearing side to the release side, defining the main departure direction of the goods under the current unloading action.

[0028] In this application, each cargo feature is assigned a feature identifier to maintain a common origin relationship. A cargo feature can form multiple candidate spatial locations on multiple candidate load-bearing layers, and further form multiple candidate dropping locations, but these candidate locations all correspond to the same local area of ​​the physical cargo. Candidate dropping locations corresponding to the same feature identifier form a candidate dropping group.

[0029] In this application, the occupancy contribution of a single material drop is the total occupancy amount generated by a local area of ​​the physical cargo represented by a cargo feature on the expected material drop range, which can also be called the source occupancy contribution. Regardless of whether a cargo feature retains one or more candidate material drop positions, the total occupancy contribution of its candidate material drop group remains unchanged and does not increase with the increase of the number of candidate load-bearing layers.

[0030] The degree of high correlation is used to represent the relative matching degree between the same cargo feature and different candidate carrier layers. The degree of high correlation can be determined based on at least one of the following: whether the candidate spatial location is located within the carrier space, whether it is consistent with the device occlusion relationship, whether it conforms to the motion direction of consecutive frames, whether it is continuous with the height relationship of adjacent cargo features, and the output result of the height recognition module. Candidate locations with hard physical contradictions are directly deleted and no longer participate in the allocation of high correlation degree.

[0031] The expected material drop envelope is generated jointly by multiple candidate material drop groups to cover the ground or bearing surface range that the cargo may reach under the current load-bearing state and uncertain cargo height. The expected material drop envelope can be formed either by the geometric union of the extended regions within the group, or by extracting continuous regions after accumulating the occupancy contributions of each candidate material drop group by occupying a grid. Regardless of the method used, the same candidate material drop group participates in the generation as a single source cell.

[0032] like Figure 1As shown, the detection system may include an image acquisition device 100, a data processing device 200, a handling equipment controller 300, a handling and carrying mechanism 500, and a detection result output terminal. The image acquisition device 100 can be fixedly installed around the material yard, pit, silo, or unloading area, or it can be installed near the handling equipment cab, chassis, robotic arm, forklift, or hopper.

[0033] The data processing device 200 receives images of the handling and tipping scene, tipping area information, camera calibration information, and load-bearing status information, and performs tasks such as cargo area determination, cargo feature extraction, construction of follow-up candidate load-bearing layers, candidate spatial location filtering, candidate release location determination, candidate material dropping location determination, candidate material dropping group establishment, expected material dropping envelope generation, and positional relationship judgment. The data processing device 200 can be an edge computing device, an industrial computer, a server, or a processing module integrated into the handling equipment controller 300.

[0034] Images of the handling and tipping scene can be single images or consecutive image frames acquired in chronological order. When using consecutive image frames, each image frame is assigned an acquisition time marker. The load-bearing status information is also assigned an acquisition time marker. The data processing device converts the load-bearing status to the image acquisition time based on these two time markers, preventing positional deviations of the load-bearing reference surface due to asynchrony between the image and device status.

[0035] Camera calibration information includes at least the camera's internal parameters and its external parameters relative to the scene coordinate system or device coordinate system. Fixed cameras can be calibrated after equipment installation using calibration plates, ground control points, or structures of known dimensions. Cameras moving with transport equipment can have their fixed installation relationship relative to the device coordinate system determined first, and then their position and orientation in the scene coordinate system updated based on the current position and attitude of the equipment.

[0036] The dumping area 400 can be defined using ground markings, pit boundaries, silo openings, work maps, geofencing, or manually configured data. The dumping area can be represented by a closed polygon, area mask, or bird's-eye view grid. The inward safety boundary 420 can be formed by shrinking the dumping area boundary 410 inward by a safe distance, determined based on the allowable scattering range, camera calibration error, equipment positioning error, and work management requirements.

[0037] The carrying status information can be obtained directly from the material handling equipment controller 300. The dump truck controller can provide the hopper shaft position, hopper tilt angle, tilt change speed, and vehicle movement speed; the forklift controller can provide the fork lifting height, mast tilt angle, fork extension status, and vehicle speed; the robotic arm controller can provide the gripper posture, gripper opening, and end effector speed.

[0038] When controller data cannot be directly read, tilt sensors, position encoders, inertial measurement units, positioning tags, or stroke sensors can be installed on the handling and carrying mechanism. It is also possible to identify hopper shafts, hopper front lips, fork roots, fork leading edges, pallet corners, or gripper key points in images, and estimate the position and orientation of the handling and carrying mechanism by combining this with a pre-established equipment size model.

[0039] The equipment dimensional model should at least record the positional relationships between the load-bearing surface, side walls, bottom plate, leading edge, pivot, unloading opening, and solid structure of the handling and bearing mechanism. For the same model of equipment, measurements can be taken once and saved during the installation and commissioning phase; for replaceable forks, pallets, or grippers, corresponding dimensional models can be saved separately for different load-bearing attachments and retrieved according to the current attachment identifier.

[0040] The cargo region can be determined using any of the following methods: semantic segmentation, instance segmentation, color or texture segmentation, or depth threshold segmentation. When the cargo region overlaps with the handling and carrying mechanism region, the image region of the handling and carrying mechanism can be determined first using the equipment size model and equipment key points, and then pixels that clearly belong to the equipment entity can be excluded from the cargo segmentation results.

[0041] Cargo features can be extracted from cargo outlines, cargo area interiors, or local cargo textures. When using outline sampling points, the image distance between adjacent sampling points is maintained within a preset range; when using grid points, points are taken inside the cargo mask according to a fixed or adaptive grid spacing; when using local feature centers, feature markers are set for continuously trackable corner points, texture block centers, or centers of locally connected regions.

[0042] To ensure that the contribution of a single material loading operation is not affected by the density of cargo feature extraction, each cargo feature can represent a defined local area of ​​the cargo. When using mask grid points, a feature represents the cargo pixel area covered by its grid cell; when using contour sampling points, a feature represents the local contour band corresponding to the sampling point and its neighboring sampling points; when using sparse local features, pixels in the cargo mask can be assigned according to the nearest feature, so that each feature represents the assigned pixel region.

[0043] The data processing equipment performs a validity check on the input data. If camera calibration information is missing, the load condition exceeds the allowable duration, the number of cargo features is insufficient, or the dumping area boundary is unavailable, the detection will not output a result allowing continued dumping, but will instead output a result indicating pending positional relationship and request supplementary data.

[0044] like Figure 3As shown, the method for detecting the handling and tipping area can be executed according to steps S1 to S7. Each step can be performed on a single image or repeatedly on consecutive image frames, while maintaining the cargo feature identifier and candidate material drop group identifier between consecutive frames.

[0045] S1: Acquire images of the handling and tipping scene, tipping area information, camera calibration information, and the load-bearing status information of the handling and carrying mechanism. The data processing equipment converts the load-bearing status to the image acquisition time and represents the tipping area, handling and carrying mechanism, and camera in a unified scene coordinate system.

[0046] S2: Determine the load reference surface, load space, release boundary, and tilting direction based on the load status information and equipment size model. The load reference surface and release boundary are updated synchronously with the lifting, translation, pitching, tilting, or opening actions of the handling load mechanism.

[0047] S3: Determine multiple cargo features with image positions from the cargo area of ​​the handling and tipping scene image, and set feature identifiers and single-drop occupancy contributions for each cargo feature; set multiple candidate load-bearing layers relative to the current load-bearing reference surface, so that each candidate load-bearing layer represents a different possible cargo height.

[0048] S4: A spatial ray is formed from the image position of each cargo feature through the optical center of the camera. The intersection point or corresponding projection position of the spatial ray with each candidate load layer is determined to obtain multiple candidate spatial positions. Then, effective candidate spatial positions are selected based on the load space, equipment entity structure, release boundary, release channel, tilting direction, occlusion relationship and continuous frame motion relationship.

[0049] S5: For valid candidate spatial locations located on the load-bearing side, predict the candidate release time and candidate release location when they reach or cross the release boundary; for valid candidate spatial locations that are already located on the release side and meet the release channel conditions, the current location or short-term extrapolated location can be determined as the candidate release location.

[0050] S5 also includes determining the candidate drop position based on the height of the candidate release position, the horizontal motion state during release, the vertical motion state, and the position of the receiving surface. When the type of goods or the motion mechanism is not suitable for using a simplified physical motion method, the drop displacement mapping relationship established by repeated tipping tests can be used to determine the candidate drop position.

[0051] S6: Group candidate drop locations with the same cargo feature into candidate drop groups, so that the candidate drop locations in the group jointly occupy the drop occupancy contribution of the cargo feature; form the drop range in the group based on the candidate drop location, degree of correlation, drop occupancy contribution, and cargo expansion range, and then merge the candidate drop groups to form the expected drop envelope.

[0052] S7: Based on the positional relationship between each candidate material drop group and the expected material drop envelope, the dumping area, and the inward safety boundary, output the result of whether dumping is allowed to continue, the dumping position is incorrect, the expected cross-boundary, or the positional relationship is pending. It can also provide the handling equipment controller with control information such as continuing dumping, pausing dumping, slowing down the tilt, adjusting the position, or supplementing the collected data.

[0053] like Figure 4 As shown, taking a tipping bucket or hopper as an example, the bearing reference surface 510 can be determined using at least three non-collinear key points on the hopper bottom plate. If the size and position of the hopper bottom plate in the equipment coordinate system are known, the bottom plate surface in the equipment coordinate system is transformed to the scene coordinate system according to the current position of the hopper shaft and the tilt angle, thus obtaining the bearing reference surface at the moment of image acquisition.

[0054] When the hopper bottom plate is not completely flat, a region of the bottom plate that can represent the main load-bearing direction can be selected for planar fitting. During fitting, abnormal critical points located on the side walls, reinforcing ribs, and attachments can be excluded. If the number of valid critical points is less than the number required to determine the load-bearing surface, the most recent reliable load-bearing reference surface is retained, but the positional relationship is left undetermined after the retention time exceeds the preset duration.

[0055] Taking the fork as an example, the upper surfaces of the two fork teeth can be fitted together as a load-bearing reference surface. When the fork is raised, the load-bearing reference surface moves upward with the lifting height; when the mast tilts forward or backward, the load-bearing reference surface rotates with the mast posture; when the fork moves laterally, the load-bearing reference surface moves synchronously with the lateral position of the fork.

[0056] Taking pallets or grippers as examples, the load-bearing reference surface can be determined using at least three corner points on the upper surface of the pallet or key structural points of the gripper's support portion. For grippers with curved support surfaces, a local tangent plane can be determined first as the load-bearing reference surface, or a curved candidate load-bearing layer can be set at a fixed normal distance from the curved support surface.

[0057] Candidate load-bearing layers are set along the normal direction of the load-bearing reference surface. The minimum relative distance between candidate load-bearing layers can start from near the load-bearing reference surface, and the maximum relative distance is determined based on the maximum height that the cargo may reach, the height of the sidewall of the load-bearing mechanism, or the visible range of the cargo area in the image.

[0058] Candidate load-bearing layers can be set at equal or non-equal intervals. More dense candidate load-bearing layers can be set for common cargo height ranges, while sparser candidate load-bearing layers can be set for rare height ranges. When using a height recognition module to obtain a rough height range, candidate load-bearing layers can be set only within this rough height range and its error margin.

[0059] Candidate carrier layers can also be generated in two stages. The first stage generates coarse candidate carrier layers with a larger spacing and performs carrier space filtering. If the same cargo feature still retains two or more adjacent coarse candidate layers, fine candidate carrier layers are added between these coarse candidate layers, thereby improving the height resolution of cargo features near the boundary without significantly increasing the overall computational load.

[0060] The candidate load-bearing layer does not need to extend indefinitely. The data processing equipment uses the load-bearing space 530 to trim the candidate load-bearing layer, retaining only the portion located within the hopper cavity, the allowable area above the forks, the vertical projection area of ​​the pallet, or the support area of ​​the gripper. The portion of the candidate load-bearing layer that falls into the equipment's physical structure does not participate in the generation of the candidate spatial position.

[0061] For a given cargo feature, the data processing device first determines whether the camera ray can intersect with the candidate carrier layer. If the ray is approximately parallel to the candidate carrier layer, the intersection point is behind the camera, the intersection point exceeds the effective distance of the scene, or the intersection point is outside the cropped range of the candidate carrier layer, no corresponding candidate spatial location is generated.

[0062] When the image acquisition time is not synchronized with the load-bearing state acquisition time, the data processing device extrapolates or pushes back the load-bearing reference plane to the image acquisition time based on the displacement, angle changes, and time intervals of the load-bearing mechanism at adjacent state sampling times. If the extrapolation time exceeds the preset upper limit or if the device state undergoes discontinuous changes, the extrapolation result is not used and the tilting continues.

[0063] In consecutive image frames, the carrier reference plane and candidate carrier layer can be updated frame by frame. To avoid the candidate carrier layer jumping back and forth due to key point positioning jitter, the position and orientation of the carrier reference plane in consecutive frames can be smoothed, provided that the maximum actual movement speed of the device is not exceeded. Alternatively, the orientation output by the device controller can be used as the main value and the image key points as correction values.

[0064] like Figure 5 As shown, the release boundary 520 is the structural boundary that goods need to cross when they detach from the handling and carrying mechanism. For a tipper or hopper, the release boundary can be a spatial segment of the hopper's front lip or a boundary band formed by the front lip along the width direction; for a fork, the release boundary can be the boundary formed by the connection of the front ends of two forks; for a pallet, the release boundary can be the pallet edge facing the unloading side; for a container with an unloading door, the release boundary can be the opening edge formed after the unloading door is opened.

[0065] The release boundary can be determined jointly by the device size model and the current load state, or it can be corrected in real time by the structural key points in the image. When the difference between the theoretical boundary determined by the device size model and the image recognition boundary is less than the allowable error, the average position of the two or the image recognition boundary can be used to correct the theoretical boundary; when the difference exceeds the allowable error, the release boundary state is marked as unreliable.

[0066] The data processing equipment classifies candidate spatial positions into the bearing-holding side, the boundary proximity range, or the release side based on their directional positional relationship relative to the release boundary. Specifically, the position of the candidate spatial position relative to the release boundary can be observed along the tilting direction: the position behind the release boundary is the bearing-holding side, the position beyond the release boundary and towards the unloading area is the release side, and the position less than the width of the proximity range from the release boundary is the boundary proximity range.

[0067] The tilting direction 540 can be determined based on the current equipment operation. When the hopper tilts forward, the slope direction from the rear to the front lip on the hopper bottom plate can be used; when the forks push the goods forward, the longitudinal forward direction of the forks can be used; when the unloading gate opens, the direction from the bearing space through the unloading opening to the outside can be used; when the gripper opens, it can be determined by the gripper opening direction and the projection direction of gravity on the gripper support surface.

[0068] For bulk or blocky cargo, the actual direction of movement may form a certain angular range around the main tipping direction. The data processing equipment can determine the permissible direction range based on the directional distribution of cargo when it crosses the release boundary in historical tipping tests. The tipping direction condition is met when the outward movement direction of the candidate spatial position falls within the permissible direction range; it is not met when it continuously points towards the side wall, the rear of the vehicle, or other non-unloading directions.

[0069] The release channel is formed by extending outward from the release boundary along the pouring direction, and its lateral range can be determined based on the width of the release boundary, the maximum lateral dimension of the cargo, and the allowable scattering range. For the hopper front lip, the release channel can form a strip-shaped space corresponding to the width of the front lip; for the discharge gate, the release channel can form a columnar or conical space corresponding to the discharge opening.

[0070] The vicinity of the release boundary is used to accommodate positioning errors at key equipment points, camera calibration errors, equipment structural deformation, and state synchronization errors. To determine this vicinity, the release boundary position can be repeatedly measured without unloading the equipment. The deviation between the theoretical and measured positions can be statistically analyzed, and the distance that covers the vast majority of normal deviations can be selected as the vicinity.

[0071] When a candidate spatial location enters the vicinity of the boundary, it is not immediately determined that it has been released. The data processing device combines the movement direction, speed, hopper angle change, and whether the cargo feature continues to enter the release side in consecutive frames to make a judgment. Only when the candidate location continuously moves towards the release side or the device has completed the release action is it updated to the release side status.

[0072] For handling and carrying mechanisms where the release boundary itself changes with the opening of the unloading gate or the opening of the gripper, the data processing equipment redetermines the release boundary based on the gate position or gripper opening at each moment. Even if the candidate spatial position is located on the theoretical release side, if the opening is insufficient to allow the goods to pass through, it can remain in a state of not being released.

[0073] For each cargo feature, the data processing device uses the camera's internal parameters to convert its image position into a spatial ray emanating from the camera's optical center. The spatial ray represents all possible positions of the image location in three-dimensional space. The data processing device then determines the intersection points of these spatial rays with each candidate carrier layer, using these intersection points as candidate spatial positions under the corresponding candidate carrier layer.

[0074] When using a pre-calibrated projection lookup table, the candidate spatial position can be directly queried according to the image position and the candidate load-bearing layer number. The lookup table should store the relative equipment coordinates of the candidate load-bearing layer as the transport and support mechanism changes, and then convert them to scene coordinates according to the current equipment pose during detection, avoiding lookup tables that are only applicable to a fixed ground plane.

[0075] After candidate spatial locations are generated, a geometric validity check is performed first. Candidate spatial locations are deleted if the intersection point is located behind the camera, exceeds the camera's effective ranging range, exceeds the boundary of the work scene, is outside the candidate load layer clipping area, or forms a numerically unstable intersection with the camera ray.

[0076] The data processing equipment further determines whether the candidate spatial location is located within the carrying space 530. Taking a hopper as an example, the inner cavity of the hopper can be represented as a three-dimensional polyhedron defined by the bottom plate and the side plate; taking a fork as an example, the space above the two forks, below the allowable height of the goods, and within the length of the forks can be represented as the carrying space; taking a pallet as an example, the allowable cargo space extending upward from the edge of the pallet can be represented as the carrying space.

[0077] Candidate spatial locations located inside the equipment base plate, side plates, fork entities, masts, pallet entities, or gripper entities are considered equipment entity structure conflict candidates and should be deleted. The equipment entity structure can be represented by the equipment dimension model, or by a combination of the equipment image mask and depth relationships.

[0078] Candidate spatial locations located on the load-bearing side of the release boundary are retained only if they are within the load space or within the transition range that allows movement towards the release boundary. Candidate spatial locations located outside the load space and unrelated to the release channel are considered incorrect height assumptions or image projection errors.

[0079] Candidate spatial locations on the release side of the release boundary are further assessed to determine whether they are within the release channel and whether they meet the tilting direction conditions. Candidate locations located within the release channel and moving towards the unloading area can be retained; candidate locations deviating from the width of the release channel, passing through the equipment sidewall, or moving towards a direction other than unloading are deleted.

[0080] The data processing equipment can also filter candidate spatial locations using occlusion relationships. If a candidate spatial location is located behind the hopper backplate or below the forks, and should be completely occluded by the equipment entity according to the current camera view, but the corresponding cargo features are clearly visible in the image, then the candidate spatial location is inconsistent with the occlusion relationship and should be deleted or have its height correlation reduced.

[0081] For multiple adjacent cargo features on the cargo surface, the spatial continuity of their candidate spatial locations can be compared. When adjacent features within the same cargo mask connectivity region form continuous surfaces on the same or adjacent candidate support layers, the high correlation between the corresponding candidate locations increases; if a candidate location causes a sudden change in the local cargo surface that exceeds the possible structure of the cargo, then its high correlation decreases.

[0082] In consecutive image frames, the data processing device associates the same local area of ​​the cargo based on feature identifiers. If the hopper is tilting forward, and the estimated motion of a candidate spatial position in consecutive frames continuously points towards the load-bearing side, or its positional change relative to the hopper exceeds the possible sliding speed of the cargo, then the candidate height assumption is inconsistent with the actual tilting action and can be deleted.

[0083] The degree of correlation can be determined by first hard screening and then soft comparison. First, candidate locations that clearly contradict the equipment entity, carrying space, release channel, or obstruction relationship are deleted; then, the remaining candidate locations are evaluated for spatial continuity, consistency of movement direction, consistency with the rough height identification result, and consistency with the historical cargo height range.

[0084] When performing soft comparisons, the same or different levels can be assigned to each consistency item. Items that fully conform receive a higher level, those close to the allowable error boundary receive a middle level, and those with slight deviations that cannot be eliminated receive a lower level. The levels are combined to obtain the initial high correlation degree of the candidate load-bearing layers, and then the sum of the high correlation degrees of all valid candidate load-bearing layers with the same cargo feature is kept to a complete correlation value.

[0085] If a cargo feature retains only one valid candidate spatial location, then that candidate location receives the full degree of high correlation for that feature. If multiple candidate locations receive the same evaluation result, the degree of high correlation can be evenly distributed. If all candidate locations are in an unreliable state, no forced distribution is enforced; instead, the cargo feature is marked as undeterminable, and supplementary data collection is triggered.

[0086] After the candidate spatial locations are selected, the data processing device saves the feature identifier, candidate carrier layer identifier, current spatial location, degree of high correlation, side relative to the release boundary, whether it is located in the release channel, and continuous frame motion state for subsequent prediction for each valid candidate spatial location.

[0087] When a candidate spatial location is situated on the load-bearing side of the release boundary, the data processing device determines its remaining distance to the release boundary along the tilting direction. This distance is not any shortest distance from the candidate location to the release boundary, but rather the distance the candidate location must travel to reach the release boundary along the currently permissible tilting direction.

[0088] When cargo features can be stably tracked across consecutive image frames, the average moving speed can be determined based on the displacement and time interval of the feature along the tipping direction in several adjacent frames. The time to reach the release boundary can then be estimated using the relationship between the remaining distance and the average moving speed. To reduce the impact of momentary jitter, the intermediate speed of the most recent frames or the average speed after removing obvious abnormal speeds can be used.

[0089] When the average moving speed is less than the minimum effective speed, the movement direction is unstable, or the feature is temporarily occluded, the candidate release time is not predicted immediately. Instead, the candidate position is kept in an unreleased state. The prediction is recalculated after the motion stabilizes in subsequent frames to avoid incorrectly extrapolating stationary goods outside the release boundary.

[0090] For goods that primarily move rigidly with the hopper, the hopper attitude when a candidate position reaches the vicinity of the front lip can be predicted based on the hopper's pivot position, current tilt angle, and tilt rate. The data processing device then converts the relative position of the candidate position in the device coordinate system to the candidate release moment as the hopper rotates, and finally to the scene coordinate system to obtain the candidate release position.

[0091] For bulk or blocky materials that slide along the bearing reference surface, a sliding displacement along the slope direction of the bearing reference surface can be added to the rigid rotational displacement of the hopper. The sliding speed can be obtained from consecutive image frames or selected from historical test data based on the type of cargo and the hopper tilt angle.

[0092] When a candidate spatial location is already located on the release side of the release boundary, and is within the release channel with the direction of movement towards the unloading area, the candidate spatial location can be determined as the candidate release location; if the equipment and goods still have significant movement, extrapolation can be performed over a short period of time not exceeding a few image frames, and the extrapolated spatial location can be used as the candidate release location.

[0093] When determining the candidate release location, physical motion can be used. The data processing equipment first determines the vertical distance between the candidate release location and the ground or receiving surface, and then determines the vertical direction of the cargo's movement during release. If the cargo does not have a significant initial vertical velocity, the descent time can be determined based on the time required for it to fall freely from that height to the receiving surface.

[0094] When a downward vertical velocity exists during release, the time it takes for the cargo to reach the receiving surface is shorter than when there is no initial vertical velocity. When an upward vertical velocity exists, first calculate the time required for the cargo to decelerate upward to the highest point and the increased height, then calculate the time required for it to fall from the highest point to the receiving surface. Add the two times together to obtain the descent time. In actual implementation, the vertical velocity given by the device controller can be used, or it can be estimated based on the vertical changes of candidate positions in consecutive image frames.

[0095] The horizontal motion state during release can consist of at least one of the following: vehicle translation speed, tangential velocity of the front lip caused by hopper rotation, sliding velocity of the cargo along the bearing reference surface, and wind field correction. After converting these velocities to the scene coordinate system, the data processing equipment takes the component parallel to the bearing surface as the horizontal motion velocity.

[0096] The data processing equipment multiplies the descent time by the horizontal speed at release to obtain the horizontal distance the goods traveled during the descent, and then superimposes this horizontal distance onto the candidate release position along the corresponding direction. The position where the superimposed movement path intersects with the ground or the receiving surface is the candidate drop position.

[0097] To reflect speed estimation errors and cargo scattering, a drop extension range can be set around the candidate drop location. The extension range can differ along the dumping direction and laterally. The longitudinal range is determined based on speed error, drop time error, and cargo flowability, while the lateral range is determined based on the release boundary width, local cargo dimensions, and historical lateral dispersion.

[0098] When goods collide significantly with the hopper lip, pit sidewall, or other structures during the unloading process, simplified physical motion methods are no longer directly applicable. In this case, the initial collision location can be used as an intermediate position, and one or more candidate unloading locations can be formed based on the post-collision displacement range obtained from historical tests, while ensuring that these locations belong to the same candidate unloading group with the original goods characteristics.

[0099] Candidate drop positions can also be determined using a calibration mapping method. When establishing the mapping relationship, multiple tipping tests are performed under representative handling height, tilt angle, tilt change rate, vehicle speed, and cargo type, and the candidate release state and actual drop position are recorded for each test.

[0100] For each set of test data, the horizontal displacement of the actual material drop position relative to the candidate release position can be used as the calibration result, and the longitudinal and lateral dispersion ranges of the test set should be recorded simultaneously. Isolated abnormal material drop positions appearing in repeated tests should be marked separately and should not be directly used to determine the normal mapping relationship.

[0101] During testing, the data processing equipment selects several sets of data from historical test data that are closest to the current load height, tilt state, vehicle speed, and cargo type. When the current state falls between these test states, interpolation can be performed according to the degree of similarity between the states; when the current state exceeds the entire test range, excessive extrapolation is not performed, but the candidate material drop range is expanded or the positional relationship is output as pending.

[0102] For goods with significantly different flow characteristics, separate material drop displacement mappings can be established for sand and gravel, scrap steel, construction waste, bagged goods, and grain or powder. When the type of goods is determined by the task information, the corresponding mapping can be directly called; when it is determined by the image recognition results, if the recognition confidence is insufficient, mappings for two or more possible types of goods can be called simultaneously, and their outputs can be kept in the same candidate drop group.

[0103] If the ground is not level, the bottom of the pit, the receiving surface of the hopper, the surface of the stockpile, or the receiving surface in the 3D scene map can be used to represent the ground. The data processing equipment gradually determines the first intersection position between the movement path and the receiving surface along the predicted movement direction, which serves as the candidate material drop position. When the receiving surface changes during continuous operation, it can be updated using depth cameras, LiDAR, or historical material drop results.

[0104] A valid candidate spatial location can result in multiple candidate dropping locations due to uncertainties in cargo type, speed range, or collision path. These candidate dropping locations all inherit the original cargo feature identifier and the original candidate load-bearing layer identifier, and are treated as different dropping probabilities within the same candidate dropping group in subsequent processing.

[0105] like Figure 7 As shown, the data processing equipment assigns a unique feature identifier to each cargo feature. All candidate drop locations generated by the same feature identifier form a candidate drop group. The candidate drop group is the smallest source unit for subsequent drop occupancy statistics and area status judgment.

[0106] The contribution of a single material drop can be determined based on the local area of ​​the cargo represented by the cargo features. When using a uniform area mask mesh, each complete mesh cell can be assigned the same contribution, while edge mesh cells have reduced contributions according to the proportion of pixels belonging to the cargo area within them. When using meshes of different areas, the contribution is determined based on the cargo pixel area represented by each mesh.

[0107] When using contour sampling points, the area of ​​the local contour band represented by the sampling point can be determined based on the contour length from the sampling point to the adjacent sampling points and the local width extending into the cargo area, and a material loading occupancy contribution can be set accordingly. When using sparse local features, the pixels of the cargo area can be assigned to the feature that is closest to it and located in the same connected region, and the contribution is determined by the number of pixels assigned.

[0108] The contribution of a single material dumping operation can be adjusted by combining the confidence level of the cargo segmentation. When the confidence level of the segmentation of the cargo area edge or dust-covered area is low, the corresponding contribution can be reduced; when the segmentation confidence level is lower than the minimum valid condition, the corresponding feature does not participate in the allowable dumping judgment, but can be retained as a pending range.

[0109] To facilitate comparisons between different image frames and different cargo regions, the single-fall occupancy contribution of all valid cargo features in an image can be adjusted to a fixed total amount, or the corresponding actual pixel area or estimated physical area can be retained. Regardless of the representation used, the total contribution of the candidate fall group for the same cargo feature must not exceed the single fall occupancy contribution of that feature before the candidate position was generated.

[0110] When a cargo feature corresponds to only one valid candidate load-bearing layer and one candidate dropping position, the candidate dropping position receives the entire dropping occupancy contribution of the cargo feature. When a cargo feature corresponds to multiple candidate dropping positions, the contribution is allocated according to the degree of correlation or dropping correlation between the candidate positions.

[0111] Before allocating the degree of correlation, the data processing equipment first converts the correlation degree of all valid candidate positions for the same cargo feature into comparable non-negative values, and then allocates them according to the proportion of each correlation degree in the total correlation degree of the feature. After allocation, the sum of the contributions obtained by each candidate position is checked. If there is a difference due to rounding, the difference is added to the candidate position with the highest correlation degree, so that the total contribution within the group is consistent with the original contribution of a single material drop.

[0112] When a candidate support layer generates multiple candidate dropping positions, the contribution allocated to that candidate support layer can be redistributed according to the relative probability of different dropping paths; if the relative probability cannot be determined, it can be distributed evenly. Regardless of whether the distribution is performed once or multiple times, the sum of the contributions of all candidate dropping positions in the candidate dropping group remains the same as the contribution of the original single dropping.

[0113] If numerical contribution allocation is not used, a discrete source identification method can be adopted. The data processing equipment first forms an intra-group envelope for the expanded regions of all candidate material placement positions within the same candidate material placement group, and then uses this intra-group envelope as a source unit to participate in the overall envelope merging. In this way, even if the group includes multiple candidate material placement positions, each position will not be repeatedly expanded as a complete and independent cargo feature.

[0114] Each candidate material drop location has an expansion range set according to the local dimensions represented by the cargo characteristics and the cargo type. Mask grid points can be expanded according to the physical width and length calculated from their grid cells; contour sampling points can be expanded along the tangent and normal directions of the cargo contour; sand and powder can be set with elliptical expansion areas based on historical scattering ranges; bagged goods can be set with rectangular or rounded rectangular expansion areas based on packaging dimensions.

[0115] When generating the expected material drop envelope using an occupancy grid, the data processing equipment establishes a bird's-eye view grid in and around the dumping area. The extended area of ​​each candidate material drop location covers several grid cells, and the contribution allocated to that candidate drop location is distributed across these grid cells. The distribution can be uniform, or grid cells closer to the candidate drop location can receive a higher contribution.

[0116] When multiple extended areas of the same candidate material drop group overlap, they should first be merged within the group before being accumulated to the overall grid. The contribution of the same grid cell within the group should not exceed the total contribution that the candidate material drop group can provide due to the overlap of multiple candidate positions, thereby avoiding repeated accumulation of candidate positions from the same source in locally overlapping areas.

[0117] After completing the intra-group processing, the data processing equipment adds up the contributions of different candidate material dropping groups to each grid cell to obtain the overall material dropping occupancy distribution. Grid cells whose occupancy contributions meet the envelope extraction criteria and are spatially continuous with the main material dropping area are merged, and their outer boundaries are extracted to obtain the expected material dropping envelope.

[0118] Envelope extraction conditions can be determined using historical validation data. Contribution conditions that cover the vast majority of actual material drop locations without excessively including isolated erroneous candidate locations are selected as envelope extraction conditions. For operations with high safety requirements, envelope extraction conditions can be reduced to expand the expected material drop envelope; for operations with high false alarm costs, continuity requirements can be increased, and the area near the boundary can be left in a pending state.

[0119] When generating the expected material drop envelope using a geometric method, the data processing equipment first generates the intra-group envelope of each candidate material drop group, then deletes isolated candidate material drop groups that are too far away from other intra-group envelopes and do not meet the minimum contribution condition, and finally takes the outer envelope, spatial union, or minimum enclosing band along the pouring direction for the remaining intra-group envelopes.

[0120] The determination of isolated candidate drop groups considers spatial distance, contribution of a single drop, and persistence across consecutive frames. Candidate drop groups that are only far away but have a high contribution or exist for multiple consecutive frames should not be directly deleted, but should be retained as out-of-area risks or pending location relationships. Candidate drop groups that are far away, have a low contribution, and only appear for a short time can be excluded as identification errors.

[0121] The data processing equipment can record the correlation chain of each candidate material drop group from cargo characteristics, candidate bearing layer, candidate spatial location, candidate release location to candidate material drop location. When the detection results show expected cross-boundary or pending location relationship, the source cargo characteristics and physical screening basis that caused the risk can be traced back to determine, which facilitates manual review.

[0122] To verify the conservation of common-source contribution, the single-stage material occupancy contribution of all cargo features and the total intra-group contribution of all candidate material occupancy groups can be calculated before and after each generation of the expected material occupancy envelope. If the difference between the two exceeds the allowable numerical error, the output of the allowable dumping result is stopped and the allocation is re-executed to prevent abnormal amplification of occupancy due to program errors or changes in the number of candidates.

[0123] like Figure 8 and Figure 9 As shown, the dumping area 400 is provided with a dumping area boundary 410 and an inward safety boundary 420. The inward safety boundary is used to reserve a safety margin for cargo spillage, equipment positioning errors, camera calibration errors, and control response distance.

[0124] The safe distance can be determined through historical material dropping tests. The data processing equipment statistically analyzes the maximum normal deviation of the actual material dropping range from the expected material dropping envelope, the response displacement of the handling equipment when it stops or pauses its tilting, and the deviation of the area boundary calibration. The larger value or combination of the above deviations is taken as the safe distance.

[0125] When all candidate dropping locations and their extended areas in the same candidate dropping group are within the inward safety boundary, the candidate dropping group is determined to be a dropping state within the area; when they are all outside the dumping area, it is determined to be a dropping state outside the area; when they are located outside or outside the dumping area, enter the boundary adjacent range, or the span within the group exceeds the allowable range, it is determined to be a dropping boundary pending state.

[0126] A candidate material drop group enters only one state set. When it is determined to be in a pending state of material drop boundary, the material drop occupation contribution of this group is not fully included in the in-region and out-of-region calculations. When numerical statistics are required, its contribution can be accumulated separately into the pending contribution, or the possible contribution within the region and the possible contribution outside the region can be formed according to the correlation between the candidate positions within the group, but the sum of the two shall not exceed the material drop occupation contribution of this group.

[0127] If the entire expected material drop envelope is within the inward safety boundary, and the contributions of the material drop groups outside the region and the undetermined material drop boundary groups do not meet the risk conditions, then the output allows continued dumping. Here, "entirely within" can be determined by all boundary points of the expected material drop envelope satisfying the region inclusion condition, or by all valid grid cells corresponding to the envelope falling into the inward safety region.

[0128] If the expected material drop envelope and the dumping area do not intersect, or if the vast majority of valid candidate material drop groups remain outside the dumping area, an error in dumping position will be output. This result indicates that, based on the current handling position and dumping action, the goods are not expected to enter the permitted dumping area.

[0129] When the expected material drop envelope covers both inside and outside the dumping area, the expected boundary crossing is not only output based on a single candidate point outside the area. The data processing equipment separately calculates the contributions of the material drop group within the area, the material drop group outside the area, and the material drop boundary undetermined group, and checks whether the material drop group outside the area forms a continuous spatial range outside the dumping area boundary.

[0130] The contribution conditions for material dumping on the inner and outer sides can be determined through validation data. For three types of samples—permitted dumping, actual boundary crossing, and uncertain boundary—the contribution ratio within the area, the contribution ratio outside the area, the undetermined contribution ratio, and the length of the continuous range outside the area are statistically analyzed. Conditions that can distinguish between actual boundary crossing and isolated error points are selected. After the conditions are determined, they are saved separately for camera viewpoint, cargo type, and dumping area size.

[0131] When the contribution from outside the region meets the boundary crossing condition, the candidate material drop group outside the region forms a continuous spatial range, and the above conditions continue to meet the continuous frame condition, the expected boundary crossing result is output. When the contribution from outside the region is insufficient, there are only isolated low contribution groups, or the relationship between inside and outside the region changes frequently with adjacent frames, the positional relationship remains undetermined.

[0132] The allowable range of position span within a group is determined based on the local dimensions represented by the cargo feature, the spacing between candidate load-bearing layers, and the error of the drop model. If the span of candidate drop positions for the same cargo feature is significantly greater than the possible drop point differences that may occur in a local area of ​​the cargo at different reasonable heights, it indicates that the reliability of the candidate positions is insufficient, and the group is designated as pending and supplementary data collection is triggered.

[0133] The pending positional relationship results can trigger the image acquisition device to switch to a side-view camera, increase the image acquisition frequency, briefly reduce the tilting speed, or maintain the posture of the transport and support mechanism. After obtaining supplementary data, the data processing device re-determines the candidate support layer and candidate material drop group, and does not use candidate positions that have been judged as unreliable.

[0134] When the detection results are sent to the handling equipment controller, control priorities can be set. Anticipated boundary crossings and tilting position errors have a higher priority than allowing continued tilting; when positional relationships are pending, the controller can pause increasing the tilt angle, but allows the equipment to adjust its lateral or longitudinal position at a safe speed.

[0135] like Figure 10 As shown, when a candidate drop group for a certain cargo feature is determined to be in an undetermined drop boundary state, a reference cargo feature can be searched within the same cargo mask connectivity region. The reference cargo feature should have a high degree of certainty, and its candidate drop group should only form states within a defined region or states outside a defined region.

[0136] Both the image distance and spatial distance between the reference cargo features and the target cargo features should be within the neighborhood. Features that are adjacent in the image but have a large difference in candidate spatial height should not be used as references to avoid incorrectly mixing the top of the cargo with the bottom near the bearing reference surface.

[0137] When at least two reference cargo features surrounding the target cargo feature are both in a dropping state within the region, and when the reference cargo features surround or clamp the target cargo feature from different directions, the high correlation between candidate positions within the target cargo feature region can be increased. When all reference cargo features are in an out-of-region state, the high correlation between candidate positions outside the region can be increased accordingly.

[0138] When reference cargo features belong to different states, are located only on one side of the target feature, or do not belong to the same cargo connectivity domain as the target feature, spatial consistency is not used to forcibly modify the target state, and the material drop boundary remains undetermined. Spatial consistency correction only changes the relative correlation of candidate positions and does not increase the material drop occupancy contribution of the target cargo feature.

[0139] like Figure 11 As shown, local areas of the same cargo are associated through feature identifiers in consecutive image frames. When a feature briefly switches from a defined state to a pending state, and the number of consecutive pending frames does not reach the state maintenance condition, the previous defined state can be maintained, while the current frame is marked as affected by occlusion or measurement fluctuations.

[0140] The status holding conditions are determined based on the image frame rate, the typical duration of the cargo crossing the release boundary or dumping area boundary, and the maximum movement speed of the equipment. The status holding duration must not exceed the minimum safe time required for the cargo to move from the inward safety boundary to the dumping area boundary, to avoid excessive holding that could mask the actual boundary crossing.

[0141] When the number of consecutive pending frames reaches the state holding condition, the spatial span of the candidate material dropping group continues to increase, the degree of correlation outside the region continues to rise, or the equipment posture changes significantly, the previous defined state is stopped, and the current effective state is updated to pending or expected cross-boundary.

[0142] When the same cargo feature reappears after disappearing in adjacent frames, it can be re-associated based on image location, motion direction, appearance description, and corresponding candidate spatial location. If reliable association is not possible, a new feature identifier is set, and the occupancy contribution of the new feature and the old feature is not retained simultaneously for more than the allowed number of transition frames.

[0143] The predicted drop envelopes of consecutive frames can be smoothed over time, but this smoothing should be performed at the candidate drop group level. The data processing equipment smooths the candidate positions and correlations of the same candidate drop group separately, and then regenerates the overall envelope. It does not directly superimpose the overall envelopes of different frames without source distinction, thus avoiding long-term persistence of old positions.

[0144] In one optional implementation, a dump truck is parked beside a rectangular hopper, and a fixed camera captures images from above the side of the vehicle. The hopper floor is 1.3 meters off the ground in a horizontal position, and the hopper is currently tilted forward 35 degrees around its rear pivot axis. The data processing equipment determines the current load-bearing reference surface based on the hopper pivot axis and key points of the front lip, and sets three candidate load-bearing layers with relative distances of 0.25 meters, 0.55 meters, and 0.85 meters along the normal direction of this load-bearing reference surface.

[0145] The target cargo features in the image form three candidate spatial locations with the three candidate load-bearing layers. The first candidate spatial location falls below the hopper bottom plate and is located within the hopper steel plate solid; the third candidate spatial location is located outside the hopper side plate and away from the front lip release channel; the second candidate spatial location is located inside the hopper cavity and close to the front lip. Therefore, the first and third candidate spatial locations are deleted, and only the second candidate spatial location is retained.

[0146] If a fixed-height plane parallel to the ground is still used, all three intersection points may be in front of the camera ray and thus retained, with one potentially projected outside the material pit. By combining the dynamic candidate load-bearing layer with the hopper entity, load-bearing space, and release channel screening, the aforementioned physically invalid locations can be eliminated before area determination.

[0147] In one optional implementation, a valid candidate spatial location is located 0.18 meters behind the front lip of the hopper. The displacement of the cargo feature along the slope direction of the hopper bottom plate is obtained by tracking three consecutive frames, corresponding to the time interval between adjacent frames. After removing one obvious jitter, the average moving speed is approximately 0.30 meters per second.

[0148] The data processing equipment divides the remaining distance by the average moving speed to determine that the cargo feature will reach the front lip in approximately 0.6 seconds. Based on the current tilting speed of the hopper, the hopper's posture 0.6 seconds later is predicted, and the position of the cargo feature reaching the front lip after rotating with the hopper and sliding along the bottom plate is determined as the candidate release position.

[0149] If subsequent image frames show that the cargo's movement speed drops to near zero, or that the cargo is blocked by other cargo, the data processing device cancels the original candidate release time and keeps the cargo's characteristics as not yet released until evidence of continuous movement toward the release side is obtained again.

[0150] In one alternative implementation, a candidate release position is 2.0 meters from the receiving surface of the hopper, resulting in a relatively low vertical velocity during release. The data processing device determines, based on this height, that the cargo's descent time to the receiving surface is approximately 0.65 seconds. The vehicle's translation and the hopper's rotation together give the cargo a horizontal velocity toward the hopper, approximately 0.8 meters per second.

[0151] The data processing equipment multiplies 0.65 seconds by 0.8 meters per second to obtain a horizontal movement distance of approximately 0.52 meters. This 0.52 meters is then superimposed onto the candidate release position along the direction of the horizontal velocity to obtain the candidate material drop position. Subsequently, based on the historical distribution range of this type of blocky cargo, an elliptical expansion area with a longitudinal dimension greater than the transverse dimension is set around the candidate material drop position.

[0152] If the hopper's front lip has a significant downward velocity, the data processing equipment shortens the descent time; if the front lip has an upward velocity, the time and height at which the goods continue to move upward are considered first, and then the total descent time is determined. This calculation sequence can be achieved using the equipment controller speed, continuous image frame rate, or calibration test data.

[0153] In one optional implementation, a cargo feature retains two candidate load-bearing layers, corresponding to two candidate material drop locations, one located within the dumping area and the other outside the dumping area. This cargo feature is generated by masked mesh cells, and the local area of ​​the cargo it represents, after conversion, yields a material drop occupancy contribution of 0.9.

[0154] The first candidate spatial location is located within the carrying space, is continuous with the adjacent cargo surface, and conforms to the continuous frame motion direction. The second candidate spatial location also meets the carrying space conditions, but deviates significantly from the coarse height identification result. Based on this, the data processing device determines the initial correlation degree of the two candidate carrying layers to be 0.55 and 0.35, respectively.

[0155] The data processing device allocates a source occupancy contribution of 0.9 according to the relative proportion of the two initial correlation degrees. Since the sum of 0.55 and 0.35 is exactly 0.9, the two candidate material dropping positions are assigned 0.55 and 0.35 respectively. If the sum of the initial correlation degrees is not 0.9, it is first converted according to the relative proportion so that the sum of the allocation results is still 0.9.

[0156] Since the two candidate locations of the same candidate material drop group are located inside and outside the dumping area respectively, the cargo characteristic is determined to be in a state of pending material drop boundary, and the 0.9 is not fully counted in the area and outside the area respectively. If the candidate location independent statistical method is adopted, the area and outside the area may each obtain 0.9, and the total contribution error increases to 1.8. This embodiment can avoid this double accumulation.

[0157] In one alternative implementation, the cargo mask is divided into twenty grid cells at an approximate bird's-eye view scale, with sixteen grid cells entirely within the cargo area and four edge grid cells each having half their area within the cargo area. The data processing device sets the complete grid cell as a base contribution unit and the half-coverage grid cell as half a base contribution unit.

[0158] The contributions of all grid cells are summed to obtain eighteen basic contribution units. To facilitate subsequent proportional condition judgments, the data processing device converts these eighteen basic contribution units into a contribution representation of a total of one. Each complete grid cell receives one-eighteenth, and each half-coverage grid cell receives one-thirty-sixth.

[0159] When any grid feature generates multiple candidate drop locations, its contribution is redistributed only within that grid feature, with one-eighteenth or one-thirty-sixth of the contribution being taken from other grids. Therefore, adding candidate load-bearing layers only changes the spatial distribution of the contribution from the same grid, without altering the total occupancy of the entire shipment.

[0160] In one optional implementation, the dumping area of ​​the crater is 5 meters wide and 7 meters long, with the inward safety boundary contracting inward by 0.3 meters relative to the boundary of the dumping area. The expected dumping envelope formed by all candidate dumping groups is 2.6 meters wide and 3.8 meters long, with a minimum distance of 0.45 meters from the edge of the envelope to the inward safety boundary.

[0161] Each candidate material dropping group is in a dropping state within the designated area, with zero contribution from outside the area, and the undetermined contribution is below the allowable threshold. The data processing equipment outputs results allowing continued dumping. Even if some goods are currently positioned above the pit boundary in the image, dumping can continue as long as the expected dropping position after release is within the safe zone.

[0162] In one alternative implementation, during a detection, the majority of the expected material drop envelope is located within the dumping area, but there exists a candidate material drop group outside the area. This outside-area group is significantly distanced from other candidate material drop groups, and its contribution to the total contribution for a single drop is only two percent, and it fails to establish a stable correlation over two consecutive frames.

[0163] The data processing equipment identifies this group as an isolated, low-contribution group and does not output a predicted cross-boundary. In subsequent frames, this candidate dumping group disappears, and the overall result remains that dumping can continue.

[0164] In another scenario, five adjacent candidate material drop groups exist outside the designated area. Their cumulative contribution reaches the criteria for determining whether the material is outside the designated area, forming a continuous strip along the boundary of the dumping area, existing for five consecutive frames. Simultaneously, the material drop group inside the designated area also maintains a stable contribution. The data processing device outputs the expected cross-boundary result and sends information to the handling equipment controller to pause the tilt and adjust the vehicle position.

[0165] In one alternative implementation, the forklift uses tilting forks to push the package into the unloading compartment. The data processing device determines the upper surfaces of the two forks as the load-bearing reference surface, the boundary formed by the connection of the front ends of the two forks as the release boundary, and the longitudinal front of the forks as the tilting direction.

[0166] When the forks tilt forward, the candidate load-bearing layers rotate synchronously with the forks. A certain candidate spatial position is located within the unloading compartment according to the fixed ground height projection, but is located below the forks relative to the current fork load-bearing reference plane. Therefore, it conflicts with the physical structure of the forks and is deleted.

[0167] After the reserved candidate positions extend beyond the leading edge of the forks, the data processing equipment determines the candidate drop position based on the forklift's forward speed, fork tilting speed, and the drop height of the package. The extended range of the package is set according to its length and width, and the expected drop range is compared with the inward safety boundary of the unloading compartment before the result is output.

[0168] In one optional implementation, when calibrating granular sand and gravel, the hopper height, hopper tilt angle, tilt change rate, vehicle speed, sand and gravel moisture content, and actual material drop range are recorded, and a large longitudinal dispersion range is saved. When calibrating scrap steel blocks, in addition to the above conditions, the block size grade is recorded, and multiple material drop displacement ranges formed by collision and tumbling are saved.

[0169] When calibrating bagged goods, the rectangular expansion range is saved based on the packaging size and release posture. During inspection, the data processing equipment selects the corresponding material drop displacement mapping according to the task information.

[0170] If the cargo identification result indicates that the cargo may be bagged cargo or block cargo, the data processing device calls the two mapping relationships respectively. The resulting candidate dropping positions all inherit the same feature identifier and are in the same candidate dropping group, so that two complete occupancy contributions are not formed due to the calling of two models.

[0171] In one optional implementation, three types of verification samples are collected during the equipment commissioning phase: permissible dumping, actual cross-boundary, and uncertain boundary. For each sample, the actual material dropping area, the expected grid occupied, the contribution within the area, the contribution outside the area, the undetermined contribution, the continuous range outside the area, and the number of consecutive frames are saved.

[0172] The data processing equipment starts with lower envelope extraction conditions and gradually increases them, comparing the coverage of the predicted material drop envelope over the actual material drop area and the number of isolated error areas under each condition. The condition that covers the vast majority of the actual material drop area while keeping isolated error areas within acceptable limits is selected.

[0173] Cross-boundary judgment criteria are determined in the same way. If most samples with low contributions from outside the region and discontinuous ranges outside the region are false alarms, the spatial continuity requirement is increased; if true cross-boundary samples only last for a few frames during rapid equipment tilting, the continuous frame requirement is reduced, but the contribution requirement outside the region is increased. The final criteria are saved according to equipment model, camera location, and cargo type.

[0174] In one optional implementation, the number and spacing of candidate carrier layers are determined based on the cargo height range, camera viewpoint, and computing resources. The number of candidate carrier layers can be gradually increased in the verification image, stopping when the recall rate of effective layers corresponding to real cargo features no longer significantly improves, or when newly added candidate layers mainly form physically invalid locations.

[0175] The load-bearing space boundary can be established based on the equipment design dimensions, or it can be corrected using no-load images and key points of the equipment. When the equipment structure deforms, the amount of material attached to the hopper increases significantly, or the load-bearing attachments are replaced, the load-bearing space should be redefined, and the original size model output should no longer be used to allow tipping.

[0176] The vicinity of the release boundary is determined by multiple key point positioning deviations, camera projection deviations, equipment vibration deviations, and time synchronization deviations. After camera replacement, installation location changes, or equipment maintenance, deviation data should be re-collected and the vicinity range updated.

[0177] The minimum effective velocity is determined based on image position measurement noise and image frame interval. When the apparent velocity of stationary goods in adjacent frames is mostly below a certain velocity, a margin can be added to that velocity as the minimum effective velocity, and candidate positions below this velocity will not be subject to immediate release prediction.

[0178] The minimum drop height is determined based on the measurement error of the receiving surface and the local dimensions of the goods. When the distance between the candidate release position and the receiving surface is less than this height, its planar position and local expansion range can be directly used as the candidate drop range to avoid unstable extrapolation of the extremely short drop process.

[0179] If the status data of the handling and bearing mechanism is missing, the most recent reliable status can be used temporarily, and the uncertainty range of the bearing reference surface and release boundary can be expanded according to the maximum movement speed of the equipment. After the maximum permissible state holding time is exceeded, the output position relationship is pending, and the expired attitude is not used to generate a deterministic dropping position.

[0180] If the camera is obstructed by dust, rain, or equipment structure, resulting in the number of cargo features falling below the minimum valid quantity, the dumping result will not be allowed to continue based on the reduced number of features. The data processing equipment may trigger lens cleaning, switch to another camera, increase lighting, or briefly maintain equipment operation.

[0181] When the position of a key point of the transport mechanism changes abruptly in adjacent image frames, and the abrupt change exceeds the maximum possible range of motion of the device between adjacent frames, the current key point identification result is considered abnormal, and the transport reference plane and release boundary are not updated. When multiple consecutive frames exhibit abnormalities, the output positional relationship is undetermined.

[0182] When the cargo segmentation area suddenly decreases or increases, the data processing device compares changes in device occlusion, changes in image exposure, and the probability of actual cargo movement. Abrupt changes that cannot be explained by device movement are not directly used to reallocate the material dropping contribution; instead, the feature identifier from the previous frame is retained and confirmation is awaited in subsequent frames.

[0183] When a long, thin protruding region is expected to appear in the material drop envelope opposite to the main material drop direction, the data processing device backtracks to generate candidate material drop groups for that region. If all relevant candidate groups come from the same error height layer, have low contribution, and do not meet the continuous frame condition, they can be deleted; if they have high contribution or exist continuously, they remain as pending risks.

[0184] The data processing equipment can save the camera calibration version, equipment size model version, cargo type mapping version, and judgment condition version used for each inspection. In case of anomalies or verification, the candidate position selection and material drop envelope generation process can be reproduced based on the version information.

[0185] In one optional implementation, the candidate load-bearing layer can be a plane parallel to the load-bearing reference surface, or a curved layer designed according to the hopper's curved bottom plate, the gripper's curved surface, or the cargo stacking shape. When using a curved layer, the data processing equipment searches for the nearest valid intersection point with the curved layer along the camera ray and trims the curved layer according to the equipment entity and the load-bearing space.

[0186] The expected material drop envelope can be represented as a two-dimensional bird's-eye view area or a three-dimensional material drop volume. When using a three-dimensional representation, a three-dimensional grid is established at the hopper opening and inside, and the candidate material drop positions are extended to adjacent height layers along the direction in which the goods may continue to move. The relationship between the three-dimensional volume and the hopper opening, inner wall, and bearing surface is then determined.

[0187] The dumping area can be a fixed area or can be dynamically switched according to the work plan. When multiple pits are set up in the same material yard, the handling equipment controller selects the dumping area according to the current cargo type, work batch or scheduling instruction, and the data processing equipment synchronously loads the corresponding dumping area boundary, inward safety boundary and judgment conditions.

[0188] The state of the handling and carrying mechanism and the image features of the cargo can be jointly input into the learning model to output candidate drop locations. Training samples must include at least the location of cargo features in the image, the pose of the handling and carrying mechanism, the location of the release boundary, the cargo type, and the measured drop location. The model output should retain the correspondence between each input cargo feature and one or more candidate drop locations.

[0189] When using the learning model, the data processing device still limits the contribution of each material drop according to the candidate material drop group. When the model outputs multiple material drop positions and relative probabilities for the same cargo feature, these relative probabilities are converted into intra-group allocation ratios; when the model does not output reliable probabilities, uniform allocation or discrete intra-group envelope methods are used, and multiple outputs are not interpreted as multiple independent cargo features.

[0190] The training data for the learning model should cover the equipment's permissible tilt angle, load height, vehicle speed, and main cargo types. If the current input state exceeds the coverage of the training samples or the model output dispersion exceeds the allowable range, the range of candidate material drop groups should be expanded, and the positional relationship should be output as pending, instead of directly outputting that continued dumping is permitted.

[0191] The test results can be used for display and recording, or directly involved in the control of the handling equipment. Control actions may include pausing tilting, reducing the tilting speed, stopping the vehicle's movement, adjusting its position towards the center of the tilting area, switching image acquisition equipment, or requesting manual confirmation.

[0192] Those skilled in the art can combine, substitute, or adjust the above embodiments without departing from the technical concept of this application. As long as the candidate positions of the cargo space are constrained by the bearing reference surface or candidate bearing layer that changes with the state of the transport bearing mechanism, the candidate dropping positions after the cargo is released are determined by the release boundary, and the expected dropping envelope is generated based on the same candidate dropping group, the processing method described in this application can be adopted.

Claims

1. A method for detecting the tipping / discarding area, characterized in that, include: Acquire images of the handling and tipping scene, tipping area information, camera calibration information, and load-bearing status information of the handling and tipping mechanism; The load-bearing reference surface, release boundary, and tilting direction from the load-bearing side to the release side are determined based on the load-bearing state information. Multiple cargo features with image positions are determined from the cargo area of ​​the transport and tipping scene image, and multiple candidate bearing layers are set relative to the bearing reference surface; for each cargo feature, multiple candidate spatial positions are determined according to the image position of the cargo feature, the camera calibration information and each candidate bearing layer, and valid candidate spatial positions are screened according to the relationship between each candidate spatial position and the bearing space of the transport bearing mechanism, the release boundary and the tipping direction; Based on the effective candidate spatial location and the carrying status information, determine the candidate dropping position of each cargo feature after it crosses the release boundary, and associate the candidate dropping positions corresponding to the same cargo feature as a candidate dropping group. Each candidate material dropping group is limited to one material dropping contribution, generating the expected material dropping envelope of the goods, and the detection result of the handling and dumping area is determined and output based on the positional relationship between the expected material dropping envelope and the dumping area.

2. The method for detecting the tipping / discarding area as described in claim 1, characterized in that, The transport and carrying mechanism includes forks, hoppers, tippers, pallets, grippers, or carrying containers with unloading openings; the carrying status information includes at least two of the following: the position of the carrying mechanism reference point, pitch attitude, and tilting action status, and is obtained from the transport equipment controller, position encoder, tilt sensor, or determined based on the key points of the carrying mechanism in the scene image and a preset equipment model.

3. The method for detecting the tipping / discarding area as described in claim 1, characterized in that, The bottom plate surface, upper surface of the forks, upper surface of the pallet, or gripper support surface in the handling and bearing mechanism that comes into contact with the goods or forms the bearing space are defined as the bearing reference surface; multiple relative distances are set along the normal direction of the bearing reference surface, and the space layers that pass through the corresponding relative distances and maintain a relative posture with the bearing reference surface are defined as candidate bearing layers, and each candidate bearing layer is updated synchronously with the state of the handling and bearing mechanism.

4. The method for detecting the handling and tipping area as described in claim 1, characterized in that, The release boundary is defined as the hopper front lip, fork front edge, pallet unloading end, unloading gate edge, or gripper release plane that the goods need to cross when moving outward from the bearing holding side; the tilting direction is determined according to at least one of the following: the slope direction of the bearing reference surface, the tilting direction of the transport bearing mechanism, the opening direction of the unloading gate, and the projection direction of the gravity direction on the bearing reference surface.

5. The method for detecting the tipping / discarding area as described in claim 1, characterized in that, Determine whether each candidate spatial location is located within the bearing space or within the release channel extending from the release boundary along the tilting direction, and determine whether it is on the bearing holding side, the boundary adjacent range, or the release side relative to the release boundary; retain candidate spatial locations located within the bearing space or meeting the release direction conditions, and eliminate candidate spatial locations located within the bearing mechanism's physical structure, deviating from the release channel, or opposite to the tilting direction.

6. The method for detecting the tipping / discarding area as described in claim 1, characterized in that, Based on the distance along the tilting direction from the effective candidate spatial location to the release boundary, the current tilting angle and tilting rate of the transport carrier, determine the candidate release time when the corresponding cargo feature arrives at or crosses the release boundary; based on the position of the transport carrier corresponding to the candidate release time, the release boundary position, and the position of the effective candidate spatial location relative to the transport carrier, determine the candidate release position.

7. The method for detecting the tipping / discarding area as described in claim 6, characterized in that, The descent time is determined based on the height of the release position above the ground and the vertical motion state at the time of release. The horizontal displacement is determined based on the descent time and the horizontal motion state at the time of release. The horizontal displacement is then superimposed on the release position to obtain the candidate drop position. Alternatively, the candidate drop position can be determined by calling the drop displacement mapping relationship established by the load height, tilt state, conveying speed and measured drop position.

8. The method for detecting the tipping / discarding area as described in claim 1, characterized in that, Configure feature identifiers for cargo features so that candidate drop locations for the same cargo feature are associated with the same identifier; based on the degree of correlation of the candidate load layer, allocate the single drop occupancy contribution of the same candidate drop group to the candidate drop locations within the group, and keep the sum of the allocation results within the group equal to the single drop occupancy contribution; generate the expected drop envelope based on the candidate drop location, drop occupancy contribution, and cargo expansion range.

9. The method for detecting the tipping / discarding area as described in claim 8, characterized in that, When all candidate dropping positions in the same candidate dropping group are within the inner safety boundary of the dumping area, it is determined to be a dropping state within the area; when they are all outside the dumping area, it is determined to be a dropping state outside the area; when they are located outside or inside the dumping area, near the boundary, or within the group, the span of the position exceeds the preset range, it is determined to be a dropping boundary pending state, so that the same cargo characteristics are not counted separately within and outside the area.

10. The method for detecting the handling and tipping area as described in any one of claims 1 to 9, characterized in that, When the expected material drop envelope is within the inward safety boundary, the output will be a permissible dumping result; when it is outside the dumping area, the output will be a position error result; when it crosses the dumping area boundary and the material drop contribution from the inner and outer sides meets the condition, the output will be a predicted cross-boundary result; otherwise, the output will be a positional relationship pending result. It also provides control information to the handling equipment controller, such as continuing to tilt, pausing tilting, adjusting position, or collecting additional data.

Images