Multi-sensor fusion stack board recognition method and system based on BEV features

Abstract

The application discloses a kind of based on BEV feature multi-sensor fusion stack board identification method and system, its method steps include: collecting image data containing stack board, point cloud data;Image data is processed by feature network, generates image feature, is projected to BEV space by LSS method, with the sparse depth map projected by point cloud data as explicit supervision of image depth estimation, constructs depth cross-entropy loss function, constrains network depth estimation capability, obtains image BEV feature;Voxelization point cloud data, extract voxel feature by feature network, conversion mapping is to BEV space, obtains point cloud BEV feature;Angle prediction optimization is carried out to BEVFusion model training, introduce 3D-IOU loss function, jointly constrain the angle and scale parameter of stack board three-dimensional detection frame;Fusion image, point cloud BEV feature generates fusion BEV feature, input model after training outputs stack board identification information.To realize the three-dimensional identification detection of stack board.

Description

Technical Field

[0001] This invention relates to the field of robot perception technology, and in particular to a multi-sensor fusion pallet recognition method and system based on BEV (Bird's Eye View) features. Background Technology

[0002] Pallets are commonly used load-bearing devices in the process of cargo storage and transfer. Pallet position detection is a key foundational link in automated warehousing and intelligent handling operations. The detection accuracy and applicability directly affect the stability of cargo gripping, stacking and transportation operations.

[0003] Currently, the mainstream pallet detection methods in the industry are mainly divided into two categories: one is the pure point cloud template matching detection method, and the other is the RGB image recognition combined with point cloud plane fitting detection method.

[0004] Among these methods, the point cloud-based template matching approach requires the pre-construction of dedicated point cloud templates for each pallet of different specifications, resulting in a large amount of initial modeling work, cumbersome on-site deployment and debugging processes, and poor versatility. The detection method using RGB image recognition combined with point cloud-fitted planes relies on fitting the side plane of the pallet to solve for pose parameters. This method has stringent requirements for the regularity of the pallet's side plane, and is prone to pose calculation errors when dealing with pallets with side deformation or substandard flatness, leading to a significant decrease in recognition reliability.

[0005] Therefore, this invention attempts to propose a solution that can stably and accurately achieve 3D recognition and detection of pallets without relying on prefabricated templates or strict requirements on the flatness of the pallet side surface. Summary of the Invention

[0006] Therefore, the main objective of this invention is to provide a multi-sensor fusion pallet recognition method and system based on BEV features, so as to stably achieve three-dimensional recognition and detection of pallets.

[0007] To achieve the above objectives, according to one aspect of the present invention, a multi-sensor fusion stack identification method based on BEV features is provided, comprising the following steps:

[0008] Collect image data and point cloud data of the scene where the stack is located;

[0009] Image data is processed by an image feature extraction network to generate two-dimensional image features, which are then projected into the BEV space using the LSS method. During this process, a sparse depth map generated by the projection of point cloud data is used as explicit supervision for image depth estimation. A depth cross-entropy loss function is constructed to constrain and optimize the network's depth estimation capability and obtain image BEV features.

[0010] Voxelized point cloud data is processed by a point cloud feature extraction network to extract voxel features, which are then transformed and mapped to the BEV space to obtain point cloud BEV features.

[0011] Train the BEVFusion model, optimize angle prediction based on the anti-symmetric structural characteristics of the stack, introduce the 3D-IOU loss function, and jointly constrain the angle and scale parameters of the 3D detection box of the stack to complete the model training.

[0012] The BEV features of the image and the BEV features of the point cloud are fused to generate fused BEV features, which are then input into the BEVFusion model. The detection head outputs stack recognition information through multiple branches.

[0013] The angle prediction optimization step includes:

[0014] To address the anti-symmetric structure of the pallet, the pallet angles are periodically mapped in four directions to obtain directional classification labels. After normalizing the angles, supervised learning is performed using sine and cosine values.

[0015] For symmetrical targets on a pallet, opposite directions are divided into equivalent groups to eliminate ambiguity in the definition of positive and negative directions and optimize angle prediction.

[0016] In a possible preferred embodiment, the step of introducing a 3D-IOU loss function and jointly constraining the angle and scale parameters of the 3D detection box of the stack includes:

[0017] By calculating the height overlap and horizontal projection overlap area between the predicted 3D bounding box and the ground truth 3D bounding box, the 3D intersection-union ratio is solved, and a 3D-IOU loss function is constructed. The angle and scale parameters of the 3D detection box of the stack are jointly constrained.

[0018] In a possible preferred embodiment, the step of constructing the deep cross-entropy loss function includes:

[0019] The point cloud data is projected onto the image coordinate system using a transformation matrix constructed from the intrinsic and extrinsic parameters of multiple sensors, and the corresponding pixel coordinates and depth values ​​are calculated.

[0020] For the case where the same pixel is projected by multiple laser points, the minimum depth value is retained to construct a sparse depth truth map; the pixel's true depth is converted into a discrete depth category truth value according to the preset maximum and minimum depth values ​​and the discrete step size.

[0021] Pixels whose depth falls within a preset effective range are selected to form a valid pixel set; using the valid pixel set, the depth cross-entropy loss is calculated based on the network's predicted probability of the true depth category of each pixel.

[0022] In a possible preferred embodiment, the method further includes the following steps:

[0023] In the initial stage of training the BEVFusion model, L1 loss independent regression was used to determine the target position of the pallet, its length, width and height dimensions, and the basic yaw angle.

[0024] The joint loss function, which combines deep cross-entropy loss, orientation classification loss, normalized angle sine and cosine loss, and 3D-IOU loss, completes the overall training and optimization of the model.

[0025] In a possible preferred embodiment, the image feature extraction network is at least one of ResNet, SwingTransformer, and EfficientNet.

[0026] In a possible preferred embodiment, the BEVFusion model detection head branch includes at least one or more of the following: category classification branch, center position regression branch, length, width and height dimension regression branch, angle regression branch, orientation classification branch, and attribute classification branch.

[0027] To achieve the above objectives, corresponding to the above method, according to another aspect of the present invention, a multi-sensor fusion stack identification system based on BEV features is also provided, comprising:

[0028] The storage unit contains a program that implements the steps of the multi-sensor fusion stack identification method based on BEV features as described above, so that the control unit, processing unit and identification unit can retrieve and execute it as needed.

[0029] The control unit is used to control the multi-sensor unit to collect image data and point cloud data in the scene where the stack is located;

[0030] The processing unit controls the image feature extraction network to process image data, generate two-dimensional image features, and project them into the BEV space using the LSS method. During this process, a sparse depth map generated by projecting point cloud data is used as explicit supervision for image depth estimation. A depth cross-entropy loss function is constructed to constrain and optimize the network's depth estimation ability, thereby obtaining image BEV features. The unit also voxelizes point cloud data, controls the point cloud feature extraction network to extract voxel features, transforms and maps them to the BEV space, and obtains point cloud BEV features. Finally, the unit fuses the image BEV features and the point cloud BEV features to generate fused BEV features.

[0031] The recognition unit is used to input the fused BEV features into the BEVFusion model and output the stack target recognition information through the multi-branch detection head.

[0032] In a possible preferred embodiment, the system further includes:

[0033] The model training unit, connected to the storage unit, is used to retrieve the program of the method steps to form a multi-task joint loss function consisting of joint depth cross-entropy loss, orientation classification loss, normalized angle sine and cosine loss, and 3D-IOU loss, to complete the overall training and optimization of the BEVFusion model and update the BEVFusion model in the recognition unit.

[0034] To achieve the above objectives, in accordance with the above methods, according to another aspect of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, it implements the steps of the multi-sensor fusion stack identification method based on BEV features as described above.

[0035] The multi-sensor fusion stack recognition method and system based on BEV features provided by this invention cleverly introduces laser point cloud projection to generate a sparse depth map as explicit supervision for depth estimation, and adds depth cross-entropy loss to compensate for the shortcomings of traditional models that only rely on implicit supervision of detection boxes. This effectively improves the image pixel depth prediction accuracy, enabling two-dimensional image features to be accurately projected into the BEV space and enhancing the effectiveness of multi-sensor feature fusion. Furthermore, addressing the characteristics of stack anti-symmetry and ambiguity in angle ground truth, this scheme adopts a four-directional periodic mapping and angle normalization design, and sets a direction equivalence grouping strategy for symmetrical targets to eliminate ambiguity in positive and negative direction labeling. This avoids gradient conflicts during BEVFusion model training and significantly improves angle prediction accuracy and model training stability. Simultaneously, this scheme also introduces 3D-IOU loss to jointly constrain the angle, position, and scale of the 3D detection box, solving the problems of stack angle and size parameter coupling and inverted length and width predictions, ensuring the overall geometric rationality of the output 3D detection box. By combining multi-task joint loss collaborative optimization, the goal is to further achieve the high-precision and high-robustness detection and recognition requirements of pallets in industrial scenarios while retaining the advantages of the BEV feature recognition scheme. Attached Figure Description

[0036] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0037] Figure 1 This is a schematic diagram of the logical process of generating fused BEV features in the multi-sensor fusion stack recognition method based on BEV features of the present invention.

[0038] Figure 2 This is a schematic diagram illustrating the logical process of angle prediction and optimization for the anti-symmetrical structural characteristics of the stack in the multi-sensor fusion stack identification method based on BEV features of the present invention.

[0039] Figure 3 This is a schematic diagram of the steps of the multi-sensor fusion stack identification method based on BEV features of the present invention;

[0040] Figure 4 This is a schematic diagram of the architecture of the multi-sensor fusion stack recognition system based on BEV features of the present invention. Detailed Implementation

[0041] To enable those skilled in the art to better understand the technical solutions of the present invention, the specific technical solutions of the present invention will be clearly and completely described below in conjunction with embodiments, so as to help those skilled in the art further understand the present invention. Obviously, the embodiments described in this application are merely some embodiments of the present invention, and not all embodiments. It should be noted that, for those skilled in the art, the embodiments and features in the embodiments of this application can be combined with each other without departing from the concept of the present invention and without conflict. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the disclosure and protection scope of the present invention.

[0042] Furthermore, the terms "first," "second," "S100," "S200," etc., used in the specification, claims, and drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such features can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those described herein. At the same time, the stages described in each step are not necessarily to be implemented in the same step; it should be understood that the implementation order of the contents of each step stage can be adjusted and interchanged without violating the inventive concept, so that embodiments of the invention described herein can be implemented in orders other than those described herein. Additionally, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. Unless otherwise expressly specified and limited, the terms "set," "arrange," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; a mechanical connection or an electrical connection; a direct connection or an indirect connection through an intermediate medium; or a connection within two elements. Those skilled in the art can understand the specific meaning of the above terms in this case based on the specific circumstances and in conjunction with existing technology.

[0043] In order to achieve stable and accurate 3D detection of pallets without relying on prefabricated templates or strict requirements on the flatness of the pallet side, this invention proposes a recognition scheme based on BEV features. This scheme utilizes the recognition features of different sensors to efficiently fuse in the BEV space, thereby directly and accurately completing the 3D recognition and detection of the pallets.

[0044] However, in practice, the inventors discovered that directly using the known BEV scheme for target detection on the stack still suffers from poor stability. The reason for this is as follows:

[0045] Existing BEV models are unstable in recognizing angles of symmetrical targets such as pallets. For symmetrical targets like pallets, the lack of a clear definition of their front and back leads to ambiguity in the ground truth angle values. For example, both 0 and π are correct for the same target. If the model directly regresses the trained angle values, the training will be unstable, affecting the accuracy of angle prediction.

[0046] Furthermore, image features play a limited role in existing BEV models because they lack sufficient spatial scale estimation capabilities, preventing accurate projection of features onto the correct 3D locations.

[0047] Meanwhile, existing BEV models regress the target center position, length, width, height, and angle separately. These loss functions are independent of each other and cannot directly measure the overall spatial overlap between the predicted bounding box and the ground truth bounding box. This leads to an inconsistency between the training objective and the final 3D-IOU evaluation metric, affecting 3D detection performance.

[0048] Therefore, referring to Figures 1 to 3 As shown, this invention provides a multi-sensor fusion stack identification method based on BEV features, which attempts to overcome the aforementioned technical obstacles by optimizing the BEVFusion model. The example steps include:

[0049] Step S100 involves acquiring image data and point cloud data of the scene where the stack is located. In this example, image data can be acquired using a camera and point cloud data can be acquired using a LiDAR. In other optional methods, an RGBD camera can also be used for acquisition.

[0050] In step S200, the image data is processed by the image feature extraction network to generate two-dimensional image features, which are then projected onto the BEV space using the LSS method. During this process, the sparse depth map generated by the projection of point cloud data is used as explicit supervision for image depth estimation. A depth cross-entropy loss function is constructed to constrain and optimize the depth estimation capability of the network and obtain the image BEV features.

[0051] Specifically, the image feature extraction network can employ ResNet, SwinTransformer, EfficientNet, etc., with ResNet50 backbone network preferred as the two-dimensional image feature extraction model in this example. When projecting the acquired two-dimensional image features into the BEV space, this scheme preferably uses a Lift-Splat-Shoot (LSS) structure. First, it predicts the depth probability distribution of each pixel in the image, and then projects the two-dimensional features into the three-dimensional space based on the predicted depth. However, during this process, because the depth estimation part of the existing BEVFusion model lacks additional supervision information and relies solely on the final bounding box to supervise backpropagation and optimize the network's depth estimation capability, the BEVFusion model suffers from insufficient image feature space scale estimation capability.

[0052] To this end, this example proposes to introduce a sparse depth map based on point cloud projection as explicit supervision for depth estimation, so as to enhance the depth estimation capability of the network and enable image features to be projected more accurately into the BEV space, thereby playing a greater role in the fusion.

[0053] For example, first define the predicted image depth output as a discretized probability distribution, and let the prediction range be such that the maximum value is... The minimum value is Let the distance from the walk be... Then the discrete set of depth categories is:

[0054]

[0055] The feature size of a two-dimensional image is The predicted depth probability is:

[0056] .

[0057] (1) Generate depth ground truth using point cloud data

[0058] Take any point in the point cloud data The transformation matrix from the laser to the image coordinate system is obtained based on the extrinsic and intrinsic parameters of the lidar and camera. Projecting the point cloud onto the image coordinate system yields the projection points:

[0059]

[0060] Depend on Calculate the corresponding pixel coordinates and depth value :

[0061]

[0062]

[0063]

[0064] Calculate the ground truth D of the sparse depth map. If multiple points project onto the same pixel, retain the minimum depth value and construct the ground truth of the sparse depth map:

[0065]

[0066] Calculate the discrete depth category ground truth based on the sparse depth map ground truth:

[0067] ;

[0068] (2) Calculate the depth cross-entropy loss function

[0069] Limited set of valid pixel range Calculate the depth cross-entropy loss based on the pixels within set M:

[0070]

[0071] (3) Based on the predicted depth distribution, the two-dimensional image features are projected into the three-dimensional space and compressed into the BEV space to obtain the image BEV features.

[0072] Step S300: Voxelize point cloud data, extract voxel features through a point cloud feature extraction network, transform and map to BEV space, and obtain point cloud BEV features.

[0073] Specifically, in this embodiment, the method for voxelizing point cloud data involves randomly sampling points within a voxel and calculating the average feature of the sampled points as the voxel feature. In this example, the point cloud feature extraction network can use VoxelNet, PointPillars, SparseConv, etc., to compress the 3D features into the BEV space. The SparseEncoder structure of BEVFusion is used to convert the point cloud voxel features into BEV features. Then, the SECOND backbone network and SECOND_FPN structure are used for further feature extraction.

[0074] Step S400 trains the BEVFusion model, optimizes angle prediction based on the anti-symmetric structural characteristics of the stack, introduces the 3D-IOU loss function, and jointly constrains the angle and scale parameters of the stack's 3D detection box to complete the model training.

[0075] Specifically, the BEVFusion model uses L1 loss independent regression for the target position (x, y, z), length, width, height (w, h, l), and angle (yaw). Due to the special nature of the pallet (anti-symmetric, without a clear definition of the positive direction), this introduces two problems: ambiguity of the true angle value and coupling of the angle scale.

[0076] For example, consider a data frame containing a pallet target with a true angle of 95°, a length of 3m, and a width of 2m. Assume the front, back, left, and right sides of the pallet are identical pairwise, and all four sides have forked holes. Due to symmetry, the true angle of this pallet could also be defined as -85°. If the existing BEVFusion model directly regresses the angle, this truth ambiguity will cause conflicting gradient descent directions during model training, leading to training instability and affecting the final angle prediction accuracy.

[0077] Furthermore, due to the ambiguous definition of the positive direction, the pallet could also be represented as having an angle of 40°, a length of 2m, and a width of 3m. It is evident that the existing BEVFusion model regresses the angle, position, length, and width independently, lacking training supervision that jointly reflects the accuracy of the overall 3D bounding box. For targets like the pallet, this could lead to opposite length and width predictions (or, alternatively, a 90° difference in angle).

[0078] Therefore, in order to resolve the aforementioned issues of ambiguity in the true value of the angle and coupling between the angle scale, this invention proposes two optimization designs:

[0079] (1) Target Angle Prediction Optimization Design

[0080] This invention addresses the problem of ambiguity in angle truth values ​​by normalizing the angle and introducing a four-class directional prediction method, as detailed below:

[0081] Set the true value of the original angle Using a four-directional periodic mapping, the normalized angle r is calculated, and a four-class label for one direction is calculated. As a supervisory signal for training:

[0082]

[0083] in , respectively indicating direction ;

[0084] Calculate the normalized angle r:

[0085]

[0086] Since the angle itself has periodicity, this example uses the normalized sine and cosine values ​​of the angle as direct training supervision values. For example, taking 4r makes the period... Normalize the angle to Within the range. Among them, the sine and cosine monitoring values ​​of the normalized angle. for:

[0087]

[0088]

[0089] During model inference, the network directly outputs the following results: And the direction prediction d, based on these values, the final angle prediction can be calculated:

[0090]

[0091] Since the pallet is antisymmetric, the target remains unchanged after a 180° rotation. Therefore, the orientation labels for symmetrical targets can be grouped. When calculating the orientation loss, categories (0,2) and (1,3) are considered as two orientation groups, and the two orientations within a group are equivalent for symmetrical targets. For asymmetrical targets, the orientation is classified into standard four categories.

[0092] Taking the aforementioned data as an example, if the true value of the pallet's angle is 95°, this method can normalize 95° to the range of (-45°, 45°), resulting in a normalized angle value of 5° and a true value label of 1 for the direction classification. For an angle of -85°, the normalized angle value is also 5°, and the direction is 3. When calculating the loss, the normalized angle value is unambiguous for both positive and negative directions; and since the pallet is a symmetrical target, directions 1 and 3 belong to the same direction group, so the direction loss calculation is also consistent.

[0093] In this way, the normalized angle avoids the problem of ambiguity in the true value of the angle; while the direction grouping strategy for symmetrical targets makes training unaffected by the positive or negative direction of the target, avoiding the problem of uncertainty in the definition of the positive direction of symmetrical targets.

[0094] (2) Introducing the 3D-IOU loss function

[0095] The aforementioned target angle prediction optimization design solves the angle prediction problem caused by ambiguity in the ground truth angle, but the coupling problem between angle and scale still exists. Therefore, this example further introduces 3D-IOU loss, which directly calculates the intersection degree between the final 3D predicted bounding box and the ground truth 3D bounding box. It includes both angle and scale information. During the model training and optimization process, angle and scale establish a joint constraint relationship, which can improve the geometric consistency of the overall 3D bounding box recognition.

[0096] The specific method for calculating the 3D-IOU loss function is as follows:

[0097] Let the detection bounding box be B1 and the ground truth bounding box be B2.

[0098] 1) Define the 3D bounding box

[0099] Each 3D bounding box can be represented by the following parameters:

[0100]

[0101] in: Indicates the coordinates of the center of the 3D bounding box; Indicates width, length, and height; This represents the rotation angle around the z-axis.

[0102] 2) Calculate the intersecting volume

[0103] The intersection volume of two 3D bounding boxes is denoted as:

[0104]

[0105] (a) Overlapping in the height direction

[0106]

[0107]

[0108]

[0109] (b) Horizontal overlapping area

[0110] Projecting the two 3D bounding boxes onto the BEV plane yields two 2D rotated rectangles.

[0111] Two-dimensional intersection area It can be calculated using polygon clipping algorithms, the Separating Axis Theorem (SAT), or methods for finding intersections of convex polygons.

[0112] (c) Intersecting volume

[0113] The three-dimensional intersection volume is calculated as follows:

[0114]

[0115] (d) Calculate the 3D crossover ratio loss

[0116]

[0117]

[0118]

[0119] .

[0120] Step S500 fuses the BEV features of the image and the BEV features of the point cloud to generate fused BEV features, which are then input into the BEVFusion model. The detection head outputs stack recognition information through multiple branches.

[0121] Specifically, in this example, it is preferable to fuse image BEV features and point cloud BEV features using a feature stitching method. For example, image BEV features... and point cloud BEV features The feature shapes are (Cimg,fh,fw) and (Clidar,fh,fw). First, the two features are concatenated directly along the channel dimension to obtain a feature with the shape (Cimg+Clidar,fh,fw). Then, after a 3x3 convolution, the feature channels are transformed into the defined output channel Cout. Other fusion methods can also be selected, including cross-attention fusion and multi-scale fusion.

[0122] On the other hand, considering that existing multi-sensor collaborative detection solutions still have problems with insufficient information utilization, such as RGB image data being only roughly used for target area selection, and the image visual features and point cloud spatial features not being effectively integrated and reused, it is impossible to fully perceive the pallet's appearance color and determine the pallet's loading status and other attribute parameters. The detection dimension is single, so it is difficult to meet the actual needs of refined pallet detection in complex warehousing scenarios.

[0123] Therefore, in this example, the BEVFusion detection head model framework transfusion head is preferably adjusted to add the following branches: category classification branch, center position regression branch, length, width and height dimension regression branch, angle regression branch, direction classification branch, and attribute classification branch.

[0124] Taking the addition of an attribute branch as an example, an attribute classification branch is introduced into the detection head to predict the target attribute. In this example, the color and cargo status of the pallet are predicted, where color is defined as a multi-class classification and cargo status as a binary classification. Each attribute uses a CenterHead module as the classification head, with the input being features decoded by a transformer. The module structure is a three-layer one-dimensional MLP network, where the number of output channels in the last layer defines the number of categories for that attribute. For example, if the color prediction head defines four categories (black, blue, white, yellow), then the classification head has four output channels. During prediction, the confidence score for each attribute category is obtained through a Sigmoid function. The index of the maximum confidence score is taken as the attribute prediction result, and the maximum value is taken as the attribute prediction confidence score. For example, if the color prediction output after Sigmoid is {0.8, 0.1, 0.1, 0.2}, then the predicted result is a black pallet (confidence score of 0.8). Cross-entropy loss can be used to train this attribute classification head during training.

[0125] For cargo status prediction, similarly, the example can define two categories (empty, loaded), with two output channels for the cargo status classification head. If the model's cargo status classification output for a certain pallet, after passing through a Sigmoid function, is {0.1, 0.75}, then the model predicts the pallet to be loaded (confidence level 0.75). Other branches can be implemented similarly. Since the detection head branch setting scheme is a known technique, it will not be elaborated here. Furthermore, those skilled in the art can set other branches as needed to achieve refined detection.

[0126] Finally, after fusing BEV features into the pre-set and trained BEVFusion model framework, a query is generated using a densely predicted heatmap. After decoding by a transformer, the calculated pose, size, and attribute information of the stack can be output through the detector head.

[0127] On the other hand, corresponding to the above method examples, such as Figure 4 As shown, the present invention also provides a multi-sensor fusion stack recognition system based on BEV features, examples of which include:

[0128] The storage unit contains a program that implements the steps of the multi-sensor fusion stack identification method based on BEV features as described in any of the above examples, so that the control unit, processing unit and identification unit can retrieve and execute it as needed.

[0129] The control unit is used to control the multi-sensor unit to acquire image data and point cloud data in the scene where the stack is located. Examples of the multi-sensor unit include: LiDAR and camera, or any one of RGBD camera.

[0130] The processing unit controls the image feature extraction network to process image data, generate two-dimensional image features, and project them into the BEV space using the LSS method. During this process, a sparse depth map generated by projecting point cloud data is used as explicit supervision for image depth estimation. A depth cross-entropy loss function is constructed to constrain and optimize the network's depth estimation capability, thereby obtaining image BEV features. The unit also voxelizes point cloud data, controls the point cloud feature extraction network to extract voxel features, transforms and maps them to the BEV space, and obtains point cloud BEV features. Finally, the unit fuses the image BEV features and the point cloud BEV features to generate fused BEV features.

[0131] The recognition unit is used to input the fused BEV features into the BEVFusion model and output the stack target recognition information through the multi-branch detection head.

[0132] Furthermore, in an optional embodiment, the multi-sensor fusion stack recognition system based on BEV features further includes: a model training unit connected to the storage unit, used to retrieve the program of the method steps, to form a multi-task joint loss function by combining depth cross-entropy loss, orientation classification loss, normalized angle sine and cosine loss, and 3D-IOU loss, to complete the overall training and optimization of the BEVFusion model, and to update the BEVFusion model in the recognition unit.

[0133] On the other hand, corresponding to the above method examples, the present invention also provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program, when executed by a processor, implements the steps of the multi-sensor fusion stack recognition method based on BEV features described in any of the above examples.

[0134] In summary, the multi-sensor fusion stack recognition method and system based on BEV features provided by this invention cleverly introduces laser point cloud projection to generate sparse depth maps as explicit supervision for depth estimation, adds depth cross-entropy loss, and makes up for the shortcomings of traditional models that only rely on implicit supervision of detection boxes. This effectively improves the image pixel depth prediction accuracy, enables two-dimensional image features to be accurately projected into the BEV space, and enhances the effectiveness of multi-sensor feature fusion.

[0135] Furthermore, to address the characteristics of antisymmetry of the stack and ambiguity of the true angle value, this solution adopts a four-directional periodic mapping and angle normalization design, and sets a direction equivalence grouping strategy for symmetrical targets to eliminate the ambiguity of positive and negative direction labels. This can avoid gradient conflicts during BEVFusion model training and significantly improve the accuracy of angle prediction and the stability of model training.

[0136] Meanwhile, this solution also introduces 3D-IOU loss to jointly constrain the angle, position and scale of the 3D detection box, which solves the problems of coupling between the pallet angle and size parameters and the inversion of length and width prediction, and ensures the overall geometric rationality of the output 3D detection box.

[0137] By combining multi-task joint loss collaborative optimization, the goal is to further achieve the high-precision and high-robustness detection and recognition requirements of pallets in industrial scenarios while retaining the advantages of the BEV feature recognition scheme.

[0138] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The present invention is limited only by the claims and their full scope and equivalents. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the protection scope of the invention.

[0139] Those skilled in the art will understand that, besides implementing the system, apparatus, unit, and its modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and its modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0140] Furthermore, all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a microcontroller, chip, or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0141] Furthermore, various different implementations of the present invention can be combined arbitrarily, as long as they do not violate the spirit of the present invention, they should also be regarded as the content disclosed in the present invention.

Claims

1. A multi-sensor fusion stack identification method based on BEV features, comprising the following steps: Collect image data and point cloud data of the scene where the stack is located; Image data is processed by an image feature extraction network to generate two-dimensional image features, which are then projected into the BEV space using the LSS method. During this process, a sparse depth map generated by the projection of point cloud data is used as explicit supervision for image depth estimation. A depth cross-entropy loss function is constructed to constrain and optimize the network's depth estimation capability and obtain image BEV features. Voxelized point cloud data is processed by a point cloud feature extraction network to extract voxel features, which are then transformed and mapped to the BEV space to obtain point cloud BEV features. The BEVFusion model is trained to optimize angle prediction based on the anti-symmetric structural characteristics of the stack. The steps include: performing a four-directional periodic mapping on the true value of the stack angle to obtain four-directional classification labels and normalized angles normalized to the (-45°, 45°) angle range; using the sine and cosine values ​​of the normalized angles as the supervision signals for angle regression; during model inference, the stack angles are calculated based on the prediction results of the four-directional classification and the predicted values ​​of the sine and cosine values ​​of the normalized angles. When calculating the orientation classification loss, two orientation categories that differ by 180° in the four orientation classifications are grouped into the same equivalent orientation group. The two orientations within the same equivalent orientation group are equivalent for the symmetrical and antisymmetrical stack targets, thus eliminating ambiguity in the definition of positive and negative orientations. By calculating the overlap area of ​​the predicted 3D detection box and the ground truth 3D detection box in the height direction and the overlap area of ​​the horizontal projection, the 3D intersection-union ratio is solved, and the 3D-IOU loss function is constructed. The angle parameters and scale parameters of the stack 3D detection box are jointly constrained, and the depth cross-entropy loss, orientation classification loss, normalized angle sine and cosine loss, and 3D-IOU loss are combined to form a multi-task joint loss function to complete the overall training and optimization of the model. The BEV features of the image and the BEV features of the point cloud are fused to generate fused BEV features, which are then input into the BEVFusion model. The stack recognition information is output through the multi-branch detection head.

2. The method according to claim 1, wherein the step of constructing the deep cross-entropy loss function comprises: The point cloud data is projected onto the image coordinate system using a transformation matrix constructed from the intrinsic and extrinsic parameters of multiple sensors, and the corresponding pixel coordinates and depth values ​​are calculated. For the case where the same pixel is projected by multiple laser points, the minimum depth value is retained to construct a sparse depth truth map; The pixel's true depth is converted into a discrete depth category true value according to the preset maximum and minimum depth values ​​and the discrete step size; Pixels whose depth falls within a preset effective range are selected to form a valid pixel set; using the valid pixel set, the depth cross-entropy loss is calculated based on the network's predicted probability of the true depth category of each pixel.

3. The method according to claim 1, wherein the step further comprises: In the initial stage of training the BEVFusion model, L1 loss independent regression was used to determine the target position, length, width and height dimensions of the pallet, and the basic yaw angle.

4. The method according to claim 1, wherein the image feature extraction network is at least one of ResNet, SwinTransformer, and EfficientNet.

5. The method according to claim 1, wherein the BEVFusion model detection head branch comprises: At least one or more of the following: category classification branch, center position regression branch, length, width and height dimension regression branch, angle regression branch, direction classification branch, and attribute classification branch.

6. A multi-sensor fusion pallet recognition system based on BEV features, comprising: The storage unit contains a program that implements the steps of the multi-sensor fusion stack identification method based on BEV features as described in any one of claims 1 to 5, so that the control unit, processing unit and identification unit can retrieve and execute it as needed. The control unit is used to control the multi-sensor unit to collect image data and point cloud data in the scene where the stack is located; The processing unit controls the image feature extraction network to process image data, generate two-dimensional image features, and project them into the BEV space using the LSS method. During this process, a sparse depth map generated by projecting point cloud data is used as explicit supervision for image depth estimation. A depth cross-entropy loss function is constructed to constrain and optimize the network's depth estimation ability, thereby obtaining image BEV features. The unit also voxelizes point cloud data, controls the point cloud feature extraction network to extract voxel features, transforms and maps them to the BEV space, and obtains point cloud BEV features. Finally, the unit fuses the image BEV features and the point cloud BEV features to generate fused BEV features. The recognition unit is used to input the fused BEV features into the BEVFusion model and output the stack target recognition information through the multi-branch detection head.

7. The system of claim 6, further comprising: The model training unit, connected to the storage unit, is used to retrieve the program of the method steps to form a multi-task joint loss function consisting of joint depth cross-entropy loss, orientation classification loss, normalized angle sine and cosine loss, and 3D-IOU loss, to complete the overall training and optimization of the BEVFusion model and update the BEVFusion model in the recognition unit.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the multi-sensor fusion stack identification method based on BEV features as described in any one of claims 1 to 5.

Images