A truck warehouse loading management method based on image recognition

Abstract

The application discloses a truck warehouse loading management method based on image recognition and belongs to the technical field of logistics management. The method comprises the following steps: when a truck enters a loading area, a bin instant image sequence is synchronously collected by using binocular cameras arranged on both sides; bin edge feature extraction is performed on each frame of image, a loading boundary line is determined, and a dynamic displacement trajectory of the bin is established according to the position change of the loading boundary line in continuous multiple frames of image; based on the dynamic displacement trajectory and a preset safety distance threshold, an effective loading space is determined, and multiple candidate loading partitions are divided; the shape size of a to-be-warehoused cargo is acquired, and a successive loading sequence containing a target loading partition and an expected placement coordinate is generated according to the space size of each candidate loading partition and the interval distance between adjacent partitions. The method realizes dynamic adaptation to a truck parking posture and automatic planning of a cargo loading sequence.

Description

Technical Field

[0001] This invention relates to the field of logistics management technology, specifically to a truck loading management method based on image recognition. Background Technology

[0002] Truck loading is a crucial link between transportation and storage in warehousing and logistics. In existing workflows, the identification of loading space and cargo placement planning after a truck enters the loading area largely relies on manual visual inspection or static schemes based on fixed parameters. A common approach is to use fixed-installation LiDAR or depth sensors to scan the empty cargo box, obtain its internal dimensions, and then use a pre-set loading template to generate a fixed cargo stacking scheme. This method assumes that the truck can accurately park in the ideal position and that the cargo box position remains constant during loading.

[0003] The aforementioned existing technical solutions have significant drawbacks. In actual operation, it is difficult for truck drivers to park the vehicle in the perfectly pre-set calibrated position in one go. After the vehicle is parked, there will always be a non-negligible lateral, longitudinal, and deflection angle between the final posture of the cargo box and the theoretical position. At the same time, during the process of loading cargo piece by piece, the vehicle suspension will experience slight settlement or tilting due to changes in force, and the original boundaries of the cargo box will also slowly change accordingly. Static scanning cannot capture these dynamic changes, resulting in a deviation between the spatial data on which subsequent loading planning is based and the actual available space from the beginning. This may lead to situations where the pre-planned loading position is executed but cargo exceeds the boundary, scrapes against the box wall, or leaves excessive gaps that waste space. In addition, existing solutions usually only arrange the cargo placement sequence based on the size and weight of the cargo, without considering the real-time accessibility of each zone within the loading area and the spatial constraints between adjacent zones to arrange the loading sequence. This can easily lead to congestion at the intersection of loading paths or a passive situation where it is difficult to load cargo later.

[0004] To address the aforementioned issues, the key challenge lies in effectively capturing the real-time positional changes of trucks as they enter the loading area and transforming them into dynamic loading space boundaries, thereby eliminating reliance on ideal parking positions. Simultaneously, it is necessary to establish a dynamic matching mechanism between the cargo loading sequence, the partition space dimensions, and the interval distance between partitions and loading entrances, based on the acquired dynamic loading space. This ensures that the generated loading sequence maintains efficient and reasonable execution in response to actual vehicle posture and spatial conditions. Summary of the Invention

[0005] The purpose of this invention is to provide an image recognition-based truck loading management method to solve the problems in existing technologies, such as loading space identification errors caused by uncertain truck parking posture and cargo box position drift during loading, as well as the inability of cargo loading sequences to dynamically adapt to real-time spatial accessibility. By synchronously acquiring image sequences of the entire cargo box loading process and extracting dynamic displacement trajectories, combined with safety distance constraints, effective loading space is dynamically determined and candidate loading zones are divided. Simultaneously, a successive loading sequence is generated based on the zone size and the interval distance between zones, achieving real-time updating of loading space boundaries and dynamic optimization of cargo loading order.

[0006] To achieve the above objectives, the present invention provides the following technical solution: The present invention provides a truck loading management method based on image recognition, comprising the following steps: During the process of a truck entering the loading area, multiple binocular cameras deployed on both sides of the loading area are used to simultaneously acquire real-time image sequences of the truck's cargo box; the edge features of the cargo box are extracted for each frame of the real-time image sequence to determine the loading boundary line of the cargo box, and a dynamic displacement trajectory of the cargo box within the loading area is established based on the positional changes of the loading boundary line in multiple consecutive frames of images; based on the dynamic displacement trajectory and a preset loading area safety distance threshold, an effective loading space for placing goods in the loading area is determined, and multiple candidate loading partitions are divided in the effective loading space; the external dimensions of the goods to be loaded are acquired, and a sequential loading sequence of goods is generated based on the spatial dimensions of each candidate loading partition and the interval distance between adjacent candidate loading partitions, wherein the sequential loading sequence includes the target loading partition of each goods and the desired placement coordinates within the target loading partition. This method uses binocular vision to perceive the dynamic position of the cargo box of a road truck in real time, accurately define the loading safety boundary, and automatically plan the loading zones and sequential loading order, thereby effectively improving the utilization rate of loading space and the efficiency of loading operations without human intervention.

[0007] As a preferred technical solution of the present invention, the loading boundary line of the cargo box is determined by extracting the edge features of each frame in the real-time image sequence. Specifically, this involves: performing grayscale processing and filtering on each frame to obtain a preprocessed image; performing edge point detection based on local gradient magnitude on the preprocessed image to obtain an initial edge point set; extracting line segments based on Hough transform from the initial edge point set to obtain multiple candidate edge segments; calculating the structural similarity between each candidate edge segment and a preset cargo box reference template edge, and selecting candidate edge segments with a structural similarity greater than a preset threshold as cargo box edge segments; and performing endpoint connection and closure processing on all selected cargo box edge segments to construct the loading boundary line of the cargo box. This method can accurately extract the real cargo box boundary from complex backgrounds and improve robustness to occlusion and lighting changes.

[0008] Furthermore, based on the dynamic displacement trajectory and the preset loading area safety distance threshold, the effective loading space for placing goods in the loading area is determined. Specifically, according to the dynamic displacement trajectory, the final stopping position of the truck when it stops in the loading area is determined; the loading boundary line of the cargo box at the final stopping position is obtained, and using this loading boundary line as a reference, it is offset inwards by a loading area safety distance threshold to obtain the inner loading boundary line; the fixed ground boundary of the loading area and the side wall boundary of the cargo box are used together as the outer loading boundary line; the three-dimensional space enclosed by the inner loading boundary line and the outer loading boundary line is determined as the effective loading space. In this way, by reserving a safety distance inside the cargo box, the determined effective loading space can prevent the goods from scratching or being squeezed by the inner wall of the cargo box, ensuring the safety of the goods and the vehicle, while providing accurate and feasible space for loading planning.

[0009] As a preferred embodiment of the present invention, multiple candidate loading zones are divided within the effective loading space. Specifically, this involves: obtaining the total quantity of goods to be loaded and the external dimensions of each goods; calculating the average occupied space volume of all goods based on the external dimensions of each goods; calculating the initial number of zones based on the average occupied space volume and the total volume of the effective loading space; dividing the bottom surface of the effective loading space into a grid according to the initial number of zones, with each grid serving as an initial loading zone; and fine-tuning the positions of the dividing lines between adjacent initial loading zones based on the maximum length and maximum width in the external dimensions of each goods, ensuring that the length and width of each candidate loading zone are not less than the external dimensions of the goods it will accommodate. This division method generates initial zones based on the statistical characteristics of the goods and makes adaptive adjustments according to the specific dimensions of the goods, enabling the division results to match the actual shape of the goods to be loaded and avoiding invalid zones with mismatched dimensions.

[0010] Furthermore, based on the spatial dimensions of each candidate loading partition and the distance between adjacent candidate loading partitions, a sequential loading sequence for goods is generated. Specifically, this involves: sorting all candidate loading partitions from low to high according to the height data in the spatial dimensions of each candidate loading partition, forming a partition height sequence; calculating the passage difficulty coefficient between each candidate loading partition and the loading area entrance based on the distance between adjacent candidate loading partitions; establishing a priority queue based on the partition height sequence and the passage difficulty coefficient, where higher-priority candidate loading partitions have lower height values ​​and lower passage difficulty coefficients; traversing the priority queue, allocating a piece of goods matching its dimensions to the current highest-priority candidate loading partition, and recording the identification of the goods and the corresponding expected placement coordinates in the sequential loading sequence; removing the candidate loading partition with the allocated goods from the priority queue, and repeating the above allocation process until all goods have been allocated. By comprehensively considering partition height and passage difficulty to optimize the loading sequence, it is possible to reduce the conflict of handling paths and the lifting action amplitude during the loading process, thereby improving loading smoothness and operational rhythm.

[0011] In a preferred embodiment of the present invention, the method further includes: after each cargo is placed into its corresponding target loading partition according to the sequential loading sequence, acquiring a real-time image of the cargo container's interior again; identifying the actual position and orientation of the placed cargo by comparing the real-time image with the desired loading image; and updating the available dimensions of the remaining candidate loading partitions in the effective loading space. This closed-loop feedback mechanism enables loading planning to dynamically respond to deviations between actual and expected loading results, improving adaptability to cargo placement errors and orientation shifts, and ensuring the accuracy and space utilization of subsequent loading sequences.

[0012] More preferably, the actual position and orientation of the placed goods are identified by comparing the differences between the real-time image and the desired loading image. Specifically, this involves: calculating a pixel-level difference image between the real-time image and the desired loading image; binarizing the difference image to obtain a change region mask; extracting the contours of all connected regions in the change region mask to obtain multiple difference regions; extracting the minimum bounding rectangle for each difference region and calculating the coordinates of the center point and the direction of the principal axis of the rectangle; and using the center point coordinates and the direction of the principal axis as the actual position and actual orientation angle of the placed goods, respectively. Through pose recognition based on image difference and geometric feature extraction, the actual occupancy status of the goods can be obtained quickly and accurately without the need for additional sensors.

[0013] The available dimensions of the remaining candidate loading partitions in the effective loading space are updated as follows: The actual position and attitude angle of the placed goods are obtained, and based on the goods' external dimensions, the actual cuboid area occupied within the effective loading space is calculated. This actual cuboid area is subtracted from the effective loading space to obtain the updated effective loading space. The bottom surface of the updated effective loading space is re-meshed to generate a new set of candidate loading partitions. The clearance distance in the height direction for each new candidate loading partition is calculated, and this clearance distance is used as the available height dimension of the new candidate loading partition. This achieves accurate reconstruction of the remaining available space after each goods placement, providing accurate real-time spatial constraints for subsequent goods allocation and minimizing space waste.

[0014] As a preferred working method, multiple binocular cameras deployed on both sides of the loading area synchronously acquire real-time image sequences of the truck's cargo box. Specifically, multiple binocular cameras are deployed at equal intervals on supports on both sides of the loading area, with the optical axis of each camera pointing towards the central axis of the loading area. A synchronous acquisition command is set for all binocular cameras, causing all cameras to capture one frame of image at the same time point, forming the raw data for one frame of panoramic stitching. The synchronous acquisition command is repeatedly executed to continuously acquire the raw data of multiple frames of panoramic stitching at fixed time intervals, and the raw data of the multiple frames of panoramic stitching are arranged in chronological order to obtain a real-time image sequence. This multi-camera synchronous acquisition method can completely cover the side contour of the cargo box, forming a seamless visual perception area, providing continuous and comprehensive image data for subsequent boundary extraction and 3D reconstruction.

[0015] Furthermore, based on the positional changes of the loading boundary line in consecutive frames of images, a dynamic displacement trajectory of the cargo box within the loading area is established. Specifically, this involves: extracting the depth value change boundary of the corresponding cargo box area in the global depth map of each frame as the cargo box outline for that frame; obtaining the loading boundary line corresponding to the global depth map of that frame, and calculating the world coordinate system coordinates of multiple key feature points of the loading boundary line; performing point set registration between the coordinates of multiple key feature points in the current frame and the coordinates of multiple key feature points in the previous frame, and obtaining the displacement vector and rotation angle of the cargo box from the previous frame to the current frame by calculating the rigid body transformation matrix; and serializing and storing the displacement vectors and rotation angles corresponding to all frames in chronological order to form the dynamic displacement trajectory of the cargo box. This dynamic displacement trajectory construction method based on point set registration can track the minute pose changes of the cargo box during the loading process with high spatiotemporal resolution, providing a reliable basis for safety boundary determination and docking position identification, and maintaining the continuity and accuracy of the trajectory even at low speeds or temporary stops.

[0016] The technical effects and advantages provided by the present invention in the above technical solution are as follows:

[0017] By simultaneously acquiring real-time image sequences of the truck's entry and parking process using binocular cameras deployed on both sides of the loading area, and extracting cargo box edge features and determining the loading boundary line for each frame, a dynamic displacement trajectory of the cargo box is constructed based on the positional changes of the loading boundary line in multiple consecutive frames. Based on this, and using the dynamic displacement trajectory and a safe distance threshold for the loading area, the inner loading boundary line is obtained by offsetting a safe distance inward from the loading boundary line corresponding to the final parking position of the cargo box. The outer loading boundary line is formed by using the fixed ground boundary of the loading area and the side wall boundary of the cargo box. These two boundary lines enclose the effective loading space. This dynamic boundary determination method directly correlates the division of the effective loading space with the truck's actual parking posture and the actual displacement during the entry process, rather than relying on an idealized fixed calibration frame. Therefore, the effective loading space can accurately adjust to the actual positional changes of the cargo box, eliminating spatial recognition misalignment caused by parking deviations and suspension deformation, and reducing interference between cargo and the box walls or excessive waste of corner space during cargo placement.

[0018] After determining the effective loading space and dividing it into multiple candidate loading zones, all zones are sorted from low to high based on the height data in the spatial dimensions of each candidate loading zone, forming a zone height sequence. Simultaneously, the access difficulty coefficient from each zone to the loading area entrance is calculated based on the distance between adjacent candidate loading zones. A priority queue is established based on the zone height sequence and access difficulty coefficient, giving higher priority to zones with lower heights and easier access. The priority queue is then traversed to match goods with suitable dimensions to the current highest priority zone, and the goods identifier and corresponding expected placement coordinates are entered into the successive loading sequence. This loading sequence generation mechanism directly links the allocation of goods to the height accessibility and entrance accessibility of zones, enabling the entire loading process to prioritize zones that are close to the loading entrance, have suitable heights, and are free from surrounding congestion. This ensures that subsequent goods are not prevented from entering deeper zones due to obstruction by earlier loaded goods, improving the smoothness of the entire loading process and the space filling rate. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0020] Figure 1 This is a flowchart of a truck loading management method based on image recognition;

[0021] Figure 2 This is a flowchart for extracting the loading boundary line of the cargo container;

[0022] Figure 3 This is a distribution map of the structural similarity of candidate edge segments;

[0023] Figure 4 This is a graph showing the relationship between candidate loading zone height and passage difficulty;

[0024] Figure 5 This is a diagram showing the grayscale difference between the real-time image and the desired loaded image, and the threshold determination using the Otsu method.

[0025] Figure 6 It is the displacement and rotation curve of the dynamic displacement trajectory of the cargo box. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] See Figure 1 This invention provides a truck loading management method based on image recognition, comprising: during the process of a truck entering a loading area, simultaneously acquiring a sequence of real-time images of the truck's cargo box using multiple binocular cameras deployed on both sides of the loading area; extracting cargo box edge features from each frame of the real-time image sequence to determine the loading boundary line of the cargo box, and establishing a dynamic displacement trajectory of the cargo box within the loading area based on the positional changes of the loading boundary line in multiple consecutive frames of images; determining the effective loading space for placing goods in the loading area based on the dynamic displacement trajectory and a preset safe distance threshold for the loading area, and dividing the effective loading space into multiple candidate loading zones; acquiring the external dimensions of the goods to be loaded, and generating a sequential loading sequence of goods based on the spatial dimensions of each candidate loading zone and the interval distance between adjacent candidate loading zones, wherein the sequential loading sequence includes the target loading zone for each goods and the desired placement coordinates within the target loading zone.

[0028] Example 1:

[0029] In specific implementation, please refer to Figure 2After acquiring each frame of real-time image, grayscale processing and filtering are performed on each frame to obtain a preprocessed image. Grayscale processing involves weighted summation of the red, green, and blue channel values ​​of each pixel in the color image, converting the color image into a single-channel grayscale image. The weighting coefficients are 0.299 for the red channel value, 0.587 for the green channel value, and 0.114 for the blue channel value. These weighting coefficients for the red, green, and blue channels are set based on the differences in human eye sensitivity to different colors. The filtering operation uses bilateral filtering. Bilateral filtering smooths image noise while preserving gradient abrupt changes at image edges through a pixel value domain kernel function. The spatial domain standard deviation of bilateral filtering is 5 pixels, and the value domain standard deviation is 50 gray levels.

[0030] Edge detection based on local gradient magnitude is performed on the preprocessed image to obtain an initial set of edge points. Edge detection first uses the Sobel operator to calculate the gradient components of the preprocessed image in the horizontal and vertical directions. The Sobel horizontal convolution kernel is... The vertical convolution kernel is The gray-level matrix of the neighborhood of each pixel in the preprocessed image is convolved with the two convolution kernels mentioned above to obtain the horizontal gradient component. and vertical gradient components Then calculate the local gradient magnitude for each pixel. The calculation formula is as follows:

[0031]

[0032] in, This represents the pixel's horizontal gradient component value calculated using the Sobel horizontal convolution kernel. This represents the pixel's vertical gradient component value calculated using the Sobel vertical convolution kernel. This represents the local gradient magnitude of a pixel, obtained by taking the square root of the sum of the squares of the horizontal and vertical gradient components. After calculating the local gradient magnitude, non-maximum suppression is applied to the local gradient magnitude map. Specifically, for each pixel, along the local gradient direction, its local gradient magnitude is compared with the local gradient magnitudes of its two neighboring pixels. If the local gradient magnitude of the current pixel is not a local maximum, then the local gradient magnitude of the current pixel is set to 0. Then, an adaptive threshold is used to binarize the non-maximum suppressed local gradient magnitude. The adaptive threshold is set to 1.2 times the average local gradient magnitude of the entire image. Pixels with local gradient magnitudes greater than the adaptive threshold are marked as initial edge points, and all initial edge points constitute the initial edge point set.

[0033] Line segments are extracted from the initial set of edge points using Hough transform, resulting in multiple candidate edge segments. During the Hough transform process, the coordinates of each initial edge point in the initial edge point set are mapped to the image coordinate system. Mapping to polar coordinate parameter space, the mapping relationship is as follows: Establish a two-dimensional accumulator array in the parameter space. The directional discrete resolution is 1 pixel. The directional discrete resolution is 1 degree. The range of angle values ​​is For each initial edge point, an accumulator vote is performed on all discrete grids traversed by the corresponding curve in the parameter space. After voting, parameter pairs with accumulator values ​​greater than a preset voting threshold (set to 90) are extracted. Each extracted parameter pair... This is the representation of a straight line in polar coordinate parameter space. For each pair of parameters, all initial edge points whose distance to the corresponding straight line is less than the distance tolerance (2 pixels) are retrieved from the initial edge point set. These initial edge points are then sorted according to their projection positions along the straight line direction, and points whose adjacent projection positions are less than the gap tolerance are merged into the same candidate edge segment (10 pixels). This results in multiple candidate edge segments.

[0034] The structural similarity between each candidate edge segment and the preset cargo box reference template edge is calculated, and candidate edge segments with a structural similarity greater than a preset threshold are selected as cargo box edge segments. The preset cargo box reference template edge is a set of two-dimensional projection line segments pre-generated based on the standard truck cargo box dimensions and camera imaging geometry, including reference line segments representing the top edge, bottom edge, side pillars, and bottom edge of the cargo box. The structural similarity is calculated as follows: First, a normalized direction vector is extracted for each candidate edge segment, normalizing the vector from the start point to the end point of the candidate edge segment into a unit vector as the direction feature. Simultaneously, the same normalized direction vector extraction is performed on each reference line segment in the cargo box reference template edge. Then, the cosine similarity between the direction vector of the candidate edge segment and the direction vector of each reference line segment is calculated, with the cosine similarity being the absolute value of the inner product of the two unit vectors. Simultaneously, the proportional difference between the length of the candidate edge segment and the length of the corresponding reference segment of the cargo box reference template is calculated. The proportional difference is calculated as follows: first, the ratio of the candidate edge segment length to the reference segment length is calculated; if the ratio is greater than 1, the reciprocal is taken; then, 1 is subtracted from this value to obtain the length proportional difference value. The structural similarity is obtained by weighted summation of the cosine similarity and the length proportional difference value, with a weight of 0.7 for cosine similarity and a weight of 0.3 for the length proportional difference value. A preset threshold of 0.75 is set, and candidate edge segments with a structural similarity greater than 0.75 are identified as cargo box edge segments.

[0035] All selected cargo box edge segments are connected and closed at their endpoints to construct the loading boundary line of the cargo box. During endpoint connection, a connection tolerance of 8 pixels is set. All cargo box edge segments are traversed. For any two cargo box edge segments, if the Euclidean distance between their endpoints is less than the connection tolerance, the two endpoints are merged into a single connection point, and the two cargo box edge segments are connected into a continuous polyline segment. After endpoint connection, if the cargo box edge segments fail to form a closed polygon, the tangent directions of the two unclosed cargo box edge segments at their open endpoints are calculated. The two cargo box edge segments are extended along the tangent directions until they intersect. The intersection point is added as a supplementary vertex, so that all cargo box edge segments form a closed polygon connected end-to-end. This closed polygon constitutes the loading boundary line of the cargo box in one frame of the image.

[0036] In some embodiments, for images acquired from different angles, when the length ratio of the cargo box reference template edge to the candidate edge line segment differs significantly due to perspective distortion, the weight of the length ratio difference value in the structural similarity can be reduced to 0.2 to improve adaptability to angle changes. In a specific implementation, the above extraction process is repeated on each frame of the real-time image sequence to obtain the cargo box loading boundary line corresponding to each frame, which is used to subsequently establish a dynamic displacement trajectory based on the positional changes of the loading boundary line in multiple consecutive frames of images.

[0037] See Figure 3 In the figure, the horizontal axis represents the structural similarity between candidate edge segments and the edge of the cargo box reference template, with a value ranging from 0 to 1. The vertical axis represents the distribution of the number of candidate edge segments within the corresponding structural similarity interval. The blue bar chart reflects the frequency of candidate edge segments in different numerical ranges of structural similarity, showing an overall distribution trend with peaks concentrated in the range of 0.2 to 0.4, indicating that the structural similarity of the vast majority of candidate edge segments is at a low level and their numbers are significantly large. The vertical line marked with red dashed lines in the figure corresponds to the preset structural similarity threshold of 0.75, which is the key judgment parameter used to filter cargo box edge segments in this embodiment. As can be seen from the figure, the number of candidate edge segments that meet the structural similarity greater than 0.75 is extremely small, significantly lower than the number of candidate segments to the left of the threshold line. This indicates that this threshold can effectively filter out most low-matching candidate segments, retaining only segments that are highly consistent with the shape features of the cargo box reference template edge as cargo box edge segments.

[0038] Example 2:

[0039] In practical implementation, the process of determining the effective loading space for placing goods in the loading area based on the dynamic displacement trajectory and the preset safe distance threshold of the loading area includes the following implementation methods: Based on the dynamic displacement trajectory, determine the final stopping position of the truck when it stops in the loading area. The dynamic displacement trajectory consists of a series of displacement vectors and rotation angles arranged in chronological order, with each record corresponding to the truck's pose change at a specific acquisition moment. Extract the pose record corresponding to the last frame from the dynamic displacement trajectory sequence. This record contains the cumulative displacement vector and cumulative rotation angle of the cargo box relative to the starting position in the global coordinate system. Add the cumulative displacement vector to the starting position coordinates to obtain the world coordinates of the cargo box reference point at the truck's final stopping position. The cumulative rotation angle is directly used as the cargo box orientation angle at the final stopping position, thereby determining the final stopping position.

[0040] Obtain the cargo loading boundary line at the final docking position. Using this boundary line as a reference, offset it inwards by a loading area safety distance threshold to obtain the inner loading boundary line. The cargo loading boundary line corresponding to the final docking position is a set of closed line segments connected end-to-end, obtained from the panoramic image taken at that position using cargo edge feature extraction. During the offset process, the loading area safety distance threshold is set to 0.3 meters. For each line segment in the loading boundary line, calculate the unit direction vector of the segment, and determine the inner direction of the segment by the counterclockwise vertex order of the closed polygon. The inner direction is the unit normal vector pointing inwards into the polygon. Translate each line segment 0.3 meters along the corresponding inner direction. After translation, adjacent line segments may separate or intersect at their endpoints, requiring connection processing to maintain closure: For cases where endpoints separate, check the intersection of the lines containing the two translated segments. If the intersection is within a reasonable segment of the extensions of the two segments, trim or extend the segments to the intersection. If the two segments are parallel or their intersection is far apart, add an arc transition or polyline with a width equal to the offset distance between the endpoints. This forms a new closed polygon, serving as the loading inner boundary line.

[0041] The fixed ground boundary of the loading area and the side wall boundary of the cargo box are used together as the outer boundary line of the loading area. The fixed ground boundary is the rectangular ground area marked during the construction of the loading area, determined by the coordinates of the four corner points in the global coordinate system. The side wall boundary of the cargo box is constructed by projecting the side wall point cloud obtained by a binocular camera: the 3D point cloud of the cargo box side reconstructed by the binocular camera is projected vertically onto the ground plane to obtain a 2D scattered point set. The concave hull algorithm is used to extract the outer contour polygon of the 2D scattered point set. The search radius of the concave hull algorithm is set to 0.2 meters to ensure that the contour is extracted. The fixed ground boundary polygon and the side wall projected contour polygon are combined by a Boolean union operation to form the outer boundary line of the loading area.

[0042] The three-dimensional space enclosed by the inner and outer loading boundary lines is defined as the effective loading space. The inner loading boundary line extends upwards to the plane defined by the inner surface of the cargo box top: a horizontal plane is fitted from the top point cloud data inside the cargo box, serving as the plane of the cargo box top inner surface. The outer loading boundary line extends upwards to the plane of the highest allowable stacking height in the loading area, which is a horizontal plane at a fixed height from the ground as set according to safety regulations. The inner boundary stretching body is obtained by vertically stretching the inner loading boundary line to the plane of the cargo box top inner surface; the outer boundary stretching body is obtained by vertically stretching the outer loading boundary line to the plane of the highest stacking height. The intersection of the inner and outer boundary stretching bodies is taken, and the intersection is confined to the area above the cargo box sidewalls and the ground; the resulting closed three-dimensional shape is the effective loading space.

[0043] The method of dividing the effective loading space into multiple candidate loading zones is implemented according to the following steps: Obtain the total quantity of goods to be loaded and the external dimensions of each goods, including length, width, and height. Calculate the average occupied volume of all goods based on their external dimensions. For each goods, multiply its length, width, and height to obtain the volume occupied by a single item. Sum the volumes occupied by all goods to be loaded and divide by the total quantity of goods to be loaded to obtain the average occupied volume.

[0044] The initial number of partitions is calculated based on the average occupied space volume and the total effective loading space volume. The total effective loading space volume is obtained by volume integration on the 3D solid model of the effective loading space, or by using a voxelization method, filling the model with 0.02-meter cube voxels, counting the number of voxels, and multiplying by the volume of a single voxel. The initial number of partitions is calculated as follows: divide the total effective loading space volume by the average occupied space volume, round the result down, and then multiply the rounded value by an adjustment factor of 0.9. The adjustment factor of 0.9 is set to allow approximately 10% operational spacing and stacking error space between goods and between goods and boundaries, preventing partitions from being too tightly arranged to be practically inoperable. After multiplying by 0.9 and rounding down again, the initial number of partitions is obtained.

[0045] The bottom surface of the effective loading space is divided into grids according to the initial number of partitions, with each grid serving as an initial loading partition. The bottom surface of the effective loading space refers to the projected area of ​​the effective loading space on the horizontal ground. This bottom surface is composed of polygons formed by the projection of the outer boundary line and the projection of the area inside the outer boundary line after removing obstacles. The grid division adopts a rectangular element subdivision method: the minimum axially enclosing rectangle of the bottom surface is calculated, and the enclosing rectangle is divided into equal intervals along the length direction. Columns are divided at equal intervals along the width direction. Okay, among them and The value of must satisfy Approximately the initial number of partitions, and and The proportions should be as consistent as possible with the aspect ratio of the enclosing rectangle. After partitioning, the intersection of each rectangular cell grid with the bottom polygon of the effective loading space is calculated, and the grids with an intersection area greater than half the area of ​​the original grid are retained as the initial loading partitions.

[0046] Based on the maximum length and maximum width of each cargo's dimensions, the positions of the dividing lines between adjacent initial loading zones are fine-tuned to ensure that the length and width of each candidate loading zone are not less than the dimensions of the cargo it carries. The fine-tuning process iterates through all dividing line segments between adjacent initial loading zones. For each dividing line segment, the initial loading zones on both sides of the segment are determined. The cargo sets pre-assigned to the two initial loading zones are obtained, and the maximum cargo length and maximum cargo width are extracted from these sets. The current geometric dimensions of the two initial loading zones are checked: for one initial loading zone, its length is taken as the span of the zone's polygon in the length direction, and its width is taken as the span of the zone's polygon in the width direction. If the length of either zone is less than the corresponding maximum cargo length, or its width is less than the corresponding maximum cargo width, the dividing line segment is shifted to the opposite side by a fine-tuning step of 0.05 meters. After the shift, the boundaries of the two zones are reconstructed, and the length and width dimensions of the two zones are recalculated. Repeat the checking and moving steps until the length and width of the initial loading partitions on both sides are not less than the maximum length and width of the goods they need to accommodate, or until the dividing line segment can no longer be moved to the position of the adjacent dividing line. After each adjustment, check whether the adjusted partition is still within the bottom range of the effective loading space. If it is outside the range, return to the previous effective position. After completing the fine-tuning of all dividing line segments, the resulting initial loading partitions are the candidate loading partitions.

[0047] Example 3:

[0048] In practical implementation, the process of generating a sequential loading sequence of goods based on the spatial dimensions of each candidate loading partition and the distance between adjacent candidate loading partitions includes the following implementation methods: Based on the height data in the spatial dimensions of each candidate loading partition, all candidate loading partitions are sorted from low to high to form a partition height sequence. The height data in the spatial dimensions refers to the available vertical height of the candidate loading partition, and the available height value is the minimum vertical distance from the bottom of the candidate loading partition upwards to the top of the effective loading space within that partition. For each candidate loading partition, its corresponding available height value is obtained. Using an ascending sorting algorithm, with the available height value as the sorting key, all candidate loading partitions are arranged in ascending order. The resulting ordered list of candidate loading partitions is the partition height sequence.

[0049] Based on the distance between adjacent candidate loading partitions, the passage difficulty coefficient between each candidate loading partition and the loading area entrance is calculated. The distance between adjacent candidate loading partitions is defined as: the width of the passage opening perpendicular to the common boundary at the geometric midpoint of the common boundary between two candidate loading partitions sharing a common boundary on the bottom surface of the effective loading space. All candidate loading partitions and loading area entrances are mapped to a set of nodes in a graph structure, with a fixed entrance node corresponding to the loading area entrance and a partition node corresponding to each candidate loading partition. The rule for constructing edges between partition nodes is as follows: if two candidate loading partitions are adjacent on the bottom surface of the effective loading space and the distance between them is greater than zero, then an undirected edge is added between the corresponding two partition nodes; similarly, if a loading area entrance is adjacent to a candidate loading partition and the distance between them is greater than zero, then an undirected edge is added between the entrance node and that partition node. The edge weight of each edge is determined by: obtaining the distance between the regions corresponding to the two nodes connected by the edge, taking the reciprocal of the distance, and using the reciprocal as the edge weight. Using Dijkstra's algorithm, with the entry node as the source, the shortest path from the entry node to each partition node is calculated. The path length of the shortest path is defined as the sum of the edge weights of all edges on the path. The calculated path length is used as the travel difficulty coefficient for the corresponding candidate loading partition. For adjacent partitions with larger intervals, the smaller the edge weight, the lower the travel difficulty coefficient of the path containing that edge.

[0050] A priority queue is established based on the partition height sequence and access difficulty coefficient. Each element in the priority queue corresponds to a candidate loading partition and is associated with a priority key value; a smaller priority key value indicates a higher priority. The formula for calculating the priority key value is:

[0051]

[0052] in, The priority key value indicating the candidate load partition; This represents the height normalized value of the candidate load partition, calculated as follows: The method is as follows: obtain the maximum value of all candidate load partition height data from the partition height sequence. and minimum value For the current candidate load partition, its height data is ,but , The numerical range of is the closed interval [0,1]; This represents the normalized value of the passage difficulty coefficient for candidate loading partitions, calculated as follows: The method is to obtain the maximum value from the passage difficulty coefficients of all candidate loading partitions. and minimum value For the current candidate load partition, its passage difficulty coefficient is ,but , The numerical range of is the closed interval [0,1]; This represents the high weighting factor, with a value of 0.5. This represents the difficulty of passage weighting factor, with a value of 0.5. Height weighting factor. and the weighting factor of access difficulty The rationale for setting both to 0.5 is to give equal importance to height and accessibility factors in priority decision-making, thus tilting priority towards candidate loading partitions with lower height and lower accessibility. Each candidate loading partition is then assigned a priority key value based on its calculated priority. Insert into a priority queue implemented with a min-heap structure. The top element of the heap retains the candidate load partition with the smallest priority key value, i.e., the candidate load partition with the highest priority.

[0053] Traverse the priority queue and allocate a cargo with matching dimensions to the highest-priority candidate loading partition. Pop the top element from the priority queue to obtain the highest-priority candidate loading partition. Among all unallocated cargoes to be loaded, perform a dimension matching check: for each cargo, obtain its length, width, and height from its dimension data. Compare the cargo length, width, and height with the candidate loading partition's length, width, and height. If the cargo length, width, and height are all no greater than the candidate loading partition's length, width, and height, then the cargo matches the current candidate loading partition's dimensions. If multiple cargoes meet the dimension matching criteria, calculate the volume occupied by each matching cargo (the product of length, width, and height). Select the cargo with the largest volume from the multiple cargoes as the allocation target. Record the cargo identifier and desired placement coordinates of the selected cargo in the successive loading sequence. The expected placement coordinates refer to the three-dimensional coordinates of the geometric center point of the bottom surface of the current candidate loading partition in the world coordinate system, and its vertical coordinate components are taken as the elevation values ​​of the bottom surface of the candidate loading partition.

[0054] The candidate loading partition with assigned goods is removed from the priority queue. The priority queue is checked for emptiness and whether there are still unassigned goods to be loaded. If the priority queue is not empty and there are still unassigned goods, the process of popping the top element from the priority queue, matching and recording goods for the popped candidate loading partition, and removing the corresponding partition from the priority queue is repeated until all unassigned goods are allocated. If the priority queue is empty during the allocation process but there are still unassigned goods, it indicates that the total capacity of the current candidate loading partitions is insufficient to load all remaining goods. In this case, a signal needs to be triggered to readjust the candidate loading partition division or change the goods loading scheme.

[0055] See Figure 4 In the graph, the horizontal axis represents the sorting index of the candidate loading partitions, sorted from lowest to highest based on their available height values. The left vertical axis corresponds to the "available height value" shown by the blue solid line, in meters; the right vertical axis corresponds to the "passage difficulty coefficient" shown by the red dashed line, which has no unit. The blue solid line curve shows a gradually increasing trend, reflecting that as the sorting index increases, the available height of the candidate loading partitions gradually increases, with the value range rising from approximately 1.2 meters to 3.7 meters, showing the distribution characteristics of partition height. The red dashed line curve shows a periodic fluctuation trend, with the passage difficulty coefficient varying between 0.1 and 1.1, reflecting the change in the distance between different candidate loading partitions and its impact on passage difficulty.

[0056] As can be observed in the figure, the passage difficulty coefficient shows distinct alternations of peaks and troughs multiple times in the candidate loading zone sequence. Peaks correspond to areas with higher passage difficulty, while troughs indicate areas with more relaxed passage. This periodic variation indicates a periodic characteristic in the spatial layout of the zone spacing, affecting the ease of passage during cargo loading. The trends of the blue solid line and the red dashed line show no obvious synchronous relationship, indicating that the height of the candidate loading zone and the passage difficulty are two relatively independent influencing factors.

[0057] Example 4:

[0058] In practical implementation, after each cargo is placed into its corresponding target loading partition according to the sequential loading sequence, a real-time image of the cargo container's interior is reacquired. By comparing the differences between the real-time image and the desired loading image, the actual position and orientation of the placed cargo are identified, and the available dimensions of the remaining candidate loading partitions in the effective loading space are updated. The process of identifying the actual position and orientation of the placed cargo by comparing the differences between the real-time image and the desired loading image includes the following implementation methods.

[0059] Real-time images of the cargo container's interior are acquired by multiple binocular cameras deployed on both sides of the loading area, which are simultaneously triggered to capture images after the cargo is placed. Distortion and stereo corrections are performed on the raw images captured by each binocular camera to generate a distortion-free corrected image. Finally, all distortion-free corrected images generated by the binocular cameras are stitched together in a panoramic view to obtain a single frame of real-time image covering the entire interior of the cargo container.

[0060] The desired loading image is obtained by using 3D rendering. This image is a simulated image generated before the cargo is placed, based on the desired placement coordinates and dimensions of the cargo in the sequential loading sequence. During 3D rendering, an internal space model of the cargo container, consistent with the effective loading space, is constructed in the virtual 3D scene. A cuboid model with the same dimensions as the cargo is placed at the desired placement coordinates, with its orientation aligned with the default direction of the cargo container coordinate system. The virtual camera is configured using the same intrinsic and extrinsic parameters as a stereo camera, and the internal image of the cargo container from the virtual camera's perspective is rendered as the desired loading image.

[0061] Calculate the pixel-level difference image between the real-time image and the desired loading image. Convert both the real-time image and the desired loading image to grayscale images. For each corresponding pixel in the two grayscale images, calculate the absolute value of the difference in pixel grayscale values. Compile the absolute values ​​of the differences at all locations into a new two-dimensional matrix, where each element of the two-dimensional matrix represents the absolute value of the difference at the corresponding pixel location. This two-dimensional matrix is ​​the difference image.

[0062] The difference image is binarized to obtain a change region mask. The Otsu method is used to automatically determine the segmentation threshold during binarization. The Otsu method calculation process is as follows: The gray-level histogram of the difference image is statistically analyzed, dividing the gray-level range into 256 equal intervals; each gray-level is used as a candidate segmentation threshold, and the inter-class variance at that candidate threshold is calculated. The inter-class variance is derived from four statistical measures: the proportion of foreground pixels, the proportion of background pixels, the mean gray-level of foreground pixels, and the mean gray-level of background pixels; the gray-level that maximizes the inter-class variance is selected as the final segmentation threshold. Pixels in the difference image with gray-level values ​​greater than the final segmentation threshold are marked as foreground pixels and assigned the value 255; pixels with gray-level values ​​not greater than the final segmentation threshold are marked as background pixels and assigned the value 0. The resulting binary image is the change region mask.

[0063] In the changing region mask, the contours of all connected regions are extracted to obtain multiple difference regions. Connected region extraction employs a two-pass scanning method: the first pass scans each pixel in the changing region mask, assigns a temporary label to each foreground pixel, and records the equivalence relationships between adjacent pixel labels; the second pass unifies the equivalence labels to the same label, with each label corresponding to a connected region. For each connected region, a boundary tracking algorithm is used to extract its outer contour. Starting from the top left foreground pixel of the connected region, the algorithm searches for adjacent foreground pixels in a clockwise direction, recording the contour point sequence. A preset minimum change area threshold of 200 pixels is set. The area enclosed by the contours of each connected region is calculated, and connected regions corresponding to contours with an area less than 200 pixels are removed, leaving the remaining connected regions as difference regions.

[0064] For each region of difference, extract the minimum bounding rectangle and calculate the coordinates of its center point and principal axis. The process for extracting the minimum bounding rectangle is as follows: obtain the set of contour points of the region of difference and calculate the convex hull of the contour points; use each edge of the convex hull as a candidate direction for one side of the rectangle, calculate the projection span of the point set along and perpendicular to that direction, and construct a bounding rectangle covering all contour points; among all the bounding rectangles corresponding to the candidate directions, select the bounding rectangle with the smallest area as the minimum bounding rectangle. The coordinates of the center point of the minimum bounding rectangle are taken as the arithmetic mean of the coordinates of the four vertices of the rectangle. The principal axis direction is defined as the angle between the line containing the longer side of the minimum bounding rectangle and the horizontal axis of the image coordinate system. The longer side is the longer of the two adjacent sides of the rectangle, and the angle is taken as the inverse cosine of the dot product of the direction vector of the longer side and the direction vector of the horizontal axis. The angle range is... .

[0065] The center point coordinates and the principal axis direction of the rectangle are used as the actual position and actual attitude angle of the placed goods, respectively. The process of transforming the center point coordinates from the image coordinate system to the world coordinate system is as follows: using the intrinsic and extrinsic parameter matrices of the stereo camera, the homogeneous coordinates of the two-dimensional center point in the image coordinate system are transformed by inverse perspective projection to obtain the three-dimensional coordinates in the camera coordinate system. Then, the extrinsic parameter matrix is ​​used to transform it to the world coordinate system to obtain the actual three-dimensional position of the placed goods. The process of transforming the principal axis direction of the rectangle from the image coordinate system to the actual attitude angle in the world coordinate system is as follows: based on the rotation matrix in the camera's extrinsic parameters, the principal axis direction vector of the rectangle is mapped from the image plane to the ground plane of the world coordinate system. The angle between the mapped vector and the preset reference orientation vector of the world coordinate system is calculated; this angle is the actual attitude angle.

[0066] The method for updating the available dimensions of the remaining candidate loading zones in the effective loading space is implemented according to the following steps: Obtain the actual position and actual attitude angle of the placed goods, and calculate the actual cuboid region occupied by the placed goods within the effective loading space based on the external dimensions of the placed goods. Use the actual position as the coordinates of the geometric center of the bottom face of the actual occupied cuboid region, use the actual attitude angle to determine the orientation of the actual occupied cuboid region on the ground plane, use the length and width of the placed goods in the external dimensions as the side length of the bottom face of the actual occupied cuboid region, and use the height of the goods as the height value of the actual occupied cuboid region. Based on the above parameters, construct a cube representing the actual occupied region in the world coordinate system. Each face of the cube has a rotation relationship with the coordinate axes determined by the actual attitude angle. This cube is the actual occupied cuboid region.

[0067] The updated effective load space is obtained by subtracting the actually occupied cuboid region from the effective load space. Both the effective load space and the actually occupied cuboid region are converted into 3D voxel representations, with each 3D voxel having a side length of 0.05 meters. The effective load space and the actually occupied cuboid region are then voxelized and filled, marking all voxel units within the corresponding space as occupied. In the voxel set corresponding to the effective load space, all voxel units belonging to the actually occupied cuboid region are changed from occupied to free, i.e., removed from the voxel set. The connected space formed by the remaining occupied voxels constitutes the updated effective load space.

[0068] The bottom surface of the updated effective loading space is re-meshed to generate a new set of candidate loading partitions. The re-meshing still uses rectangular cell partitioning. The number and layout of partitions on the bottom surface of the updated effective loading space are recalculated according to the number of remaining goods to be loaded and the external dimensions of each goods. This ensures that the length and width of each partition in the newly generated set of candidate loading partitions are not less than the corresponding dimensions in the external dimensions of the goods assigned to that partition.

[0069] Calculate the clearance distance in the height direction for each new candidate loading partition, and use this clearance distance as the available height dimension of the new candidate loading partition. The clearance distance is calculated as follows: within the bottom surface area of ​​the new candidate loading partition, radiate a ray vertically upwards from the geometric center of the bottom surface. Calculate the first intersection point of this ray with the updated effective loading space's upper surface boundary. The distance from the ray's origin to this intersection point is the clearance distance in the height direction. The upper surface boundary is fitted from the top voxel boundary of the updated effective loading space. Assign the calculated clearance distance to the corresponding new candidate loading partition as its available height dimension, and store it along with other dimension data in the candidate loading partition's attributes for subsequent adjustments in the loading sequence.

[0070] See Figure 5 In the graph, the horizontal axis represents gray levels, ranging from 0 to 255; the left vertical axis represents pixel frequency; and the right vertical axis represents the inter-class variance. The blue solid curve represents the gray level distribution of foreground pixels, the gray dashed curve represents the gray level distribution of background pixels, and the red solid curve represents the variation in inter-class variance corresponding to different gray levels. The dashed vertical line in the graph indicates the optimal segmentation threshold position, which is gray level 128.

[0071] This figure illustrates the specific process of automatically determining the segmentation threshold based on the Otsu method in Example 4. The peak values ​​of the foreground pixel grayscale distribution are concentrated in the grayscale range of approximately 160 to 200, and the distribution is relatively concentrated, indicating that the grayscale features of the area where goods are placed are relatively high. The peak values ​​of the background pixel grayscale distribution are located in the grayscale range of approximately 50 to 80, and the distribution is relatively dispersed, reflecting that the grayscale of the area where no goods are placed is relatively low. The red inter-class variance curve shows a clear single-peak shape, reaching its maximum value at grayscale level 128, indicating that this grayscale value can distinguish the foreground and background to the greatest extent, serving as the optimal threshold for binarization of the difference image, and thus accurately extracting the mask of the change area.

[0072] Example 5:

[0073] In practice, multiple binocular cameras are deployed at equal intervals on supports on both sides of the loading area, with the optical axis of each camera pointing towards the central axis of the loading area. The installation height of the supports is set based on the maximum height of the cargo box of the truck to be loaded, setting the support height to 1.5 meters above the maximum height of the cargo box, allowing the binocular cameras to capture the entire side and interior area of ​​the cargo box from a top-down angle. The spacing between adjacent binocular cameras is determined based on the horizontal field of view of the binocular cameras and the required overlap rate between adjacent images, with the overlap rate set at 20%. The spacing between adjacent binocular cameras is calculated as follows: obtain the horizontal field of view value of the binocular cameras, calculate the horizontal coverage width of a single binocular camera on the ground plane, and multiply the horizontal coverage width by 0.8 to obtain the spacing value between adjacent binocular cameras, ensuring that the overlap area of ​​the fields of view of adjacent binocular cameras on one side of the cargo box is not less than 20%.

[0074] All stereo cameras are configured to synchronously trigger acquisition commands. An external trigger signal generator simultaneously outputs pulse signals to all stereo cameras. The external trigger signal generator is a synchronous pulse generator based on a field-programmable gate array (FPGA), with a pulse width set to 100 microseconds and a synchronous trigger error of less than 1 millisecond. At the same time point upon receiving the trigger pulse, all stereo cameras synchronously capture one frame of image. The captured image frames from all stereo cameras constitute the raw data for a panoramic stitched image.

[0075] The synchronous trigger acquisition command is repeatedly executed to continuously acquire raw data of multiple frames of panoramic stitched images at fixed time intervals. The time interval is set to 200 milliseconds. The 200-millisecond interval is based on the fact that the moving speed of the truck during the entry process is usually less than 5 kilometers per hour. At this speed, 200 milliseconds corresponds to a truck displacement of approximately 0.28 meters, which matches the accuracy requirements of the subsequent dynamic displacement trajectory. After each synchronous trigger, the raw data of one frame of panoramic stitched image is appended to the image data sequence in chronological order, resulting in a real-time image sequence composed of raw data from multiple frames of panoramic stitched images.

[0076] In practical implementation, the process of establishing the dynamic displacement trajectory of the cargo box within the loading area based on the positional changes of the loading boundary line in multiple consecutive frames of images includes the following implementation methods: Extracting the depth value change boundary of the corresponding cargo box area from the global depth map of each frame, as the cargo box outline of the current frame image. The global depth map is generated as follows: stereo matching is performed on the left and right eye images acquired by each stereo camera, the disparity map is calculated and converted into a depth map, and then the depth maps of all stereo cameras are stitched together through camera extrinsic calibration relationships to form a global depth map covering the complete visible area of ​​the cargo box. The cargo box area is delineated in the global depth map by using the projection of the extracted cargo box loading boundary line onto the image plane as a mask, retaining only the depth data within the mask. For the depth map within the cargo box area, the depth gradient magnitude is calculated. The depth gradient magnitude is calculated using the Sobel operator to calculate the depth change rate of the depth map in the horizontal and vertical directions respectively, and the square root of the sum of the squares of the depth change rates in the two directions is obtained. Local maxima suppression is performed on the depth gradient magnitude map to extract the local maxima pixels in the gradient direction. These local maxima pixels are then connected to form a continuous curve to obtain the depth value change boundary, which serves as the cargo box outline of the current frame image.

[0077] Obtain the loading boundary line corresponding to the global depth map of the current frame. The loading boundary line has been obtained from the cargo box edge feature extraction process and stored as an attribute corresponding to the image frame. Calculate the world coordinate system coordinates of multiple key feature points of the loading boundary line. The selection method for key feature points is as follows: extract the vertices of the loading boundary line polygon as the first type of key feature points; simultaneously, calculate the curvature at the midpoint of each edge of the loading boundary line. The curvature calculation method is to fit an arc between the two adjacent pixels before and after the midpoint, calculate the reciprocal of the radius of curvature of the arc, and select points with curvature greater than 0.05 per pixel as the second type of key feature points. Combine the first and second types of key feature points to form a key feature point set. For each key feature point in the key feature point set, obtain its image coordinates in the corresponding frame image. Using the intrinsic and extrinsic parameter matrices obtained from the stereo camera calibration, convert the image coordinates into three-dimensional coordinates in the world coordinate system. The transformation process is as follows: First, the image coordinates and the depth value of the point are back-projected to the camera coordinate system by the intrinsic parameter matrix, and then the camera coordinate system coordinates are transformed to the world coordinate system by the extrinsic parameter matrix to obtain the world coordinate system coordinates.

[0078] The coordinates of multiple key feature points in the current frame are registered with those in the previous frame. The displacement vector and rotation angle of the cargo box from the previous frame to the current frame are obtained by calculating the rigid body transformation matrix. Point set registration employs an iterative nearest-point algorithm, whose inputs are the source point set (composed of the coordinates of key feature points in the previous frame) and the target point set (composed of the coordinates of key feature points in the current frame). In each iteration, for each point in the source point set, the point with the closest Euclidean distance in the target point set is searched as its corresponding point, forming a set of corresponding point pairs. Based on this set, an optimal rotation matrix and an optimal translation vector are solved using singular value decomposition (SVD) to minimize the sum of squared distances between all corresponding point pairs after rotation and translation. The source point set is updated with the solved rotation matrix and translation vector, and the next iteration begins. The iteration terminates when the change in the sum of squared distances calculated in two consecutive iterations is less than 0.0001 square meters or the number of iterations reaches 50. The output rotation matrix and translation vector after iteration termination are the rigid body transformation matrix. The displacement vector and rotation angle are obtained from the rigid body transformation matrix: the three components of the translation vector are directly taken from the translation vector; the rotation matrix is ​​decomposed into the rotation angle about the vertical axis (Z-axis) of the world coordinate system and the rotation components about other axes. Since the cargo box mainly performs planar motion within the loading area, only the rotation angle about the vertical axis is taken as the rotation angle of the cargo box from the previous frame to the current frame. The rotation angle is calculated by the inverse trigonometric function of the corresponding element in the rotation matrix.

[0079] The displacement vectors and rotation angles corresponding to all frames are serialized and stored in chronological order to form the dynamic displacement trajectory of the cargo box. In the serialized storage, each record includes a frame number, a timestamp, the X-axis component, Y-axis component, and Z-axis component of the displacement vector, as well as the rotation angle value around the vertical axis. The storage format uses comma-separated text lines, with each line corresponding to the relative transformation relationship from one frame to the next. All lines are arranged in ascending order of frame number, forming the dynamic displacement trajectory of the cargo box from entering the loading area to stopping.

[0080] See Figure 6 The graph uses time (seconds) as the horizontal axis, the left vertical axis as the displacement (meters), and the right vertical axis as the rotation angle (degrees) around the vertical axis. The curves include the displacement vector's X-axis component (blue solid line), Y-axis component (green solid line), Z-axis component (purple dashed line), and rotation angle around the vertical axis (red dotted line).

[0081] As can be seen from the figure, the X-axis component of the displacement vector shows a linear increasing trend, gradually increasing from 0 meters to about 55 meters from 0 seconds, indicating that the cargo box moves continuously along the X-axis direction within the loading area. The Y-axis and Z-axis components of the displacement vector fluctuate relatively little, with the amplitude controlled within ±0.5 meters, indicating that the displacement of the cargo box in the Y-axis and Z-axis directions is small, which is consistent with the expected trajectory of the truck mainly moving along the X-axis direction.

[0082] The rotation angle curve around the vertical axis shows obvious fluctuations. In the initial stage (0-40 seconds), the rotation angle fluctuates positively, reaching a maximum of nearly 1.2 degrees, indicating that the cargo box is rotating clockwise during movement. Subsequently (40-70 seconds), the rotation angle gradually turns negative, reaching a minimum of about -0.8 degrees, indicating that the cargo box has undergone counterclockwise rotation adjustment. In the final stage (70-100 seconds), the rotation angle rises back to near the positive value, with a reduced amplitude, and tends to stabilize.

[0083] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A truck loading management method based on image recognition, characterized in that, Includes the following steps: S1, during the process of the truck entering the loading area, uses multiple binocular cameras deployed on both sides of the loading area to simultaneously collect real-time image sequences of the truck's cargo box. S2, extract the edge features of the cargo box for each frame of the real-time image sequence, determine the loading boundary line of the cargo box, and establish the dynamic displacement trajectory of the cargo box within the loading area based on the position change of the loading boundary line in multiple consecutive frames of images. S3, based on the dynamic displacement trajectory and the preset loading area safety distance threshold, determine the effective loading space in the loading area for placing goods, and divide the effective loading space into multiple candidate loading zones; S4. Obtain the external dimensions of the goods to be put into storage, and generate a sequential loading sequence of goods based on the spatial dimensions of each candidate loading partition and the interval distance between adjacent candidate loading partitions. The sequential loading sequence includes the target loading partition of each goods and the expected placement coordinates within the target loading partition.

2. The truck loading management method based on image recognition according to claim 1, characterized in that, In step S2, the edge features of the cargo box are extracted for each frame of the real-time image sequence to determine the loading boundary line of the cargo box, specifically as follows: Each frame of the image is converted to grayscale and filtered to obtain a preprocessed image; The preprocessed image is subjected to edge point detection based on local gradient magnitude to obtain an initial edge point set; The initial set of edge points is subjected to line segment extraction based on Hough transform to obtain multiple candidate edge segments; Calculate the structural similarity between each candidate edge segment and the edge of the preset cargo box reference template, and select candidate edge segments with structural similarity greater than the preset threshold as cargo box edge segments; Connect and close the endpoints of all selected cargo box edge segments to construct the loading boundary line of the cargo box.

3. The truck loading management method based on image recognition according to claim 1, characterized in that, In step S3, based on the dynamic displacement trajectory and the preset loading area safety distance threshold, the effective loading space for placing goods in the loading area is determined, specifically as follows: Based on the dynamic displacement trajectory, determine the final stopping position of the truck when it stops in the loading area; Obtain the cargo loading boundary line of the final docking position, and use the loading boundary line as a reference to offset a loading area safety distance threshold inwards from the cargo box to obtain the loading inner boundary line. The fixed ground boundary of the loading area and the side wall boundary of the cargo box are used together as the outer boundary line of the loading area; The three-dimensional space enclosed by the inner boundary line and the outer boundary line of the loading is defined as the effective loading space.

4. The truck loading management method based on image recognition according to claim 1, characterized in that, In step S3, multiple candidate loading partitions are divided within the effective loading space, specifically as follows: Obtain the total quantity of goods to be loaded and the external dimensions of each item; Calculate the average space volume occupied by all goods based on the external dimensions of each item; Calculate the initial number of partitions based on the average occupied space volume and the total volume of the effective loading space; The bottom surface of the effective loading space is divided into grids according to the number of initial partitions, with each grid serving as an initial loading partition; Based on the maximum length and maximum width in the external dimensions of each cargo, the position of the dividing line between adjacent initial loading zones is finely adjusted so that the length and width of each candidate loading zone are not less than the external dimensions of the cargo it carries.

5. The truck loading management method based on image recognition according to claim 1, characterized in that, In step S4, a sequential loading sequence of goods is generated based on the spatial dimensions of each candidate loading partition and the distance between adjacent candidate loading partitions, specifically as follows: Based on the height data in the spatial dimensions of each candidate loading partition, all candidate loading partitions are sorted from low to high to form a partition height sequence; Based on the distance between adjacent candidate loading zones, calculate the passage difficulty coefficient between each candidate loading zone and the loading area entrance; A priority queue is established based on the partition height sequence and the passage difficulty coefficient, wherein candidate loading partitions with higher priority have lower height values ​​and lower passage difficulty coefficients; Traverse the priority queue, allocate a cargo that matches its dimensions to the candidate loading partition with the highest current priority, and record the cargo's identifier and corresponding expected placement coordinates into the successive loading sequence; Candidate loading partitions that have been allocated cargo are removed from the priority queue, and the above allocation process is repeated until all cargo has been allocated.

6. The truck loading management method based on image recognition according to claim 1, characterized in that, The method further includes: S5. After each cargo is placed into the corresponding target loading partition according to the sequential loading sequence, a real-time image of the cargo box is reacquired. By comparing the difference between the real-time image and the expected loading image, the actual position and orientation of the placed cargo are identified, and the available size of the remaining candidate loading partitions in the effective loading space is updated.

7. The truck loading management method based on image recognition according to claim 6, characterized in that, In step S5, the actual position and orientation of the placed goods are identified by comparing the difference between the real-time image and the expected loading image. Specifically: Calculate the pixel-level difference image between the real-time image and the desired loaded image; The difference image is binarized to obtain a mask of the changed region; In the mask of the changing region, the contours of all connected regions are extracted to obtain multiple different regions; For each region of difference, extract its minimum bounding rectangle and calculate the coordinates of the center point and the direction of the principal axis of the rectangle; The coordinates of the center point and the direction of the main axis of the rectangle are respectively taken as the actual position and actual attitude angle of the placed goods.

8. The truck loading management method based on image recognition according to claim 7, characterized in that, In step S5, updating the available size of the remaining candidate loading partitions in the effective loading space specifically involves: Obtain the actual position and actual attitude angle of the placed goods, and calculate the actual cuboid area occupied by the goods within the effective loading space based on the external dimensions of the goods. The updated effective loading space is obtained by subtracting the actually occupied cuboid area from the effective loading space. The bottom surface of the updated effective loading space is re-meshed and divided to generate a new set of candidate loading partitions; Calculate the clearance distance in the height direction for each new candidate loading partition, and use this clearance distance as the available height dimension of the new candidate loading partition.

9. The truck loading management method based on image recognition according to claim 1, characterized in that, In step S1, multiple binocular cameras deployed on both sides of the loading area are used to simultaneously acquire real-time image sequences of the truck's cargo box, specifically: Multiple binocular cameras are deployed at equal intervals on brackets on both sides of the loading area, with the optical axis of each binocular camera facing the central axis of the loading area; Set all binocular cameras to trigger acquisition commands synchronously, so that all cameras capture an image at the same time point, forming the raw data of a panoramic stitched image; The synchronous trigger acquisition command is repeatedly executed to continuously acquire the raw data of multiple panoramic stitched images at fixed time intervals, and the raw data of the multiple panoramic stitched images are arranged in chronological order to obtain an instant image sequence.

10. A truck loading management method based on image recognition according to claim 9, characterized in that, In step S2, a dynamic displacement trajectory of the cargo box within the loading area is established based on the positional changes of the loading boundary line in multiple consecutive frames of images. Specifically: Extract the depth value change boundary of the corresponding cargo box area in the global depth map of each frame, and use it as the cargo box outline of that frame; Obtain the loading boundary line corresponding to the global depth map of this frame, and calculate the world coordinate system coordinates of multiple key feature points of the loading boundary line; The coordinates of multiple key feature points in the current frame are registered with the coordinates of multiple key feature points in the previous frame. By calculating the rigid body transformation matrix, the displacement vector and rotation angle of the cargo box from the previous frame to the current frame are obtained. The displacement vectors and rotation angles corresponding to all frames are serialized and stored in chronological order to form the dynamic displacement trajectory of the cargo box.

Images