Boarding pass identification method

Abstract

The application provides a plate truck identification method, which realizes sub-pixel level segmentation and positioning of a high plate loading surface of a plate truck, a whole vehicle loading surface and a height increasing tool (such as a small stool) by orthogonal projection conversion from a three-dimensional point cloud to a two-dimensional image, construction of an instance segmentation model combining a convolutional neural network and a Transformer architecture, and output of verified three-dimensional corner point coordinates to a downstream intelligent logistics system after multi-dimensional geometric constraint verification. The method can support automatic loading of paper rolls by unmanned forklifts in paper mills, automatic stacking of container goods and other industrial scenes, and provide centimeter-level spatial positioning guarantee for automatic loading and unloading.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of pallets. Specifically, the present application relates to a pallet recognition method. BACKGROUND

[0002] In the pallet unmanned loading operation, the pallet is first modeled, and then the loading plane corner points are positioned, and some key information of the pallet, such as the vehicle type, the small stool and the like, is also extracted. When the robot carries the goods to the pallet for loading operation, the specific coordinate information of the goods to be placed needs to be known, so that the goods can be placed in the appropriate position. In the pallet loading operation scene, by detecting the coordinates of the 4 or 6 corner points of the pallet loading plane, the placement position is calculated in combination with the goods attribute, so as to finally complete the entire loading operation. The accuracy of the pallet corner point extraction will seriously affect the stacking planning, especially in the tight stacking scene. SUMMARY

[0003] Therefore, it is necessary to provide a pallet recognition method aiming at the technical problem of how to improve the accuracy of pallet corner point extraction.

[0004] In a first aspect, the present application provides a pallet recognition method. The method comprises.

[0005] S10, comprehensively scanning the pallet at a global detection site to obtain a global three-dimensional point cloud of the pallet; scanning a specific part of the pallet at a preset detection site to obtain a local three-dimensional point cloud of the pallet.

[0006] S20, orthogonally projecting the global three-dimensional point cloud and the local three-dimensional point cloud along a world coordinate system XOY plane to generate a global projection image and a local projection image respectively; wherein the global projection image and the local projection image are identified by different colors.

[0007] S30, pseudo-color synthesis is performed on the global projection image and the local projection image to generate a pseudo-color projection image.

[0008] S40, inputting the pseudo-color projection image into a trained instance segmentation model to output a pallet structure instance segmentation result; wherein the instance segmentation model comprises a backbone network, an encoder, a decoder and an instance segmentation head, the pallet structure instance segmentation result comprises a pallet structure instance mask and a corresponding category thereof, and the category comprises a whole vehicle loading surface, a high board loading surface and / or a height increasing tool.

[0009] S50, performing pallet type recognition based on the pallet structure instance segmentation result, and extracting two-dimensional pixel coordinates of the pallet corner points in the pseudo-color projection image; inversely projecting the two-dimensional pixel coordinates back to a three-dimensional point cloud coordinate system to determine the three-dimensional space position of the pallet corner points.

[0010] S60, performing multi-dimensional geometric constraint verification on the three-dimensional spatial position of the wagon corner point, and outputting the three-dimensional spatial position after the verification is passed.

[0011] In one of the embodiments, S40 specifically comprises.

[0012] S401, inputting the pseudo-color projection image into the backbone network to extract multi-scale image features.

[0013] S402, inputting the multi-scale image features into the encoder to generate encoded features containing global context information by using a self-attention mechanism.

[0014] S403, inputting the multi-scale image features and the encoded features into the decoder to generate high-resolution pixel-level features and N embedding vectors.

[0015] S404, inputting the pixel-level features and the embedding vectors into the instance segmentation head to generate a wagon structure instance segmentation result.

[0016] In one of the embodiments, the encoder is a Transformer encoder, and the decoder comprises a pixel decoder and a Transformer decoder; S403 specifically comprises.

[0017] S4031, inputting the multi-scale image features into the pixel decoder to generate high-resolution pixel-level features by using a feature pyramid network to upsample and fuse high-level semantic information of the multi-scale image features.

[0018] S4032, inputting the encoded features into the Transformer decoder to generate N embedding vectors by using N learnable object query vectors to interact with the encoded features through a cross-attention mechanism.

[0019] In one of the embodiments, S404 specifically comprises.

[0020] S4041, inputting the N embedding vectors into a multi-layer perceptron to generate N mask embeddings and N predicted categories.

[0021] S4042, performing dot product operation on each mask embedding and the pixel-level features to generate N probability maps.

[0022] S4043, converting the N probability maps into binary masks through an activation function as N predicted instance masks.

[0023] S4044, performing dot product operation on the N predicted instance masks and the N predicted categories to generate a wagon structure instance segmentation result.

[0024] In one of the embodiments, the instance segmentation head employs an instance mask loss function L m As follows.

[0025] .

[0026] .

[0027] .

[0028] wherein A represents the real instance mask, B represents the predicted instance mask, x a represents the real class of the i-th pixel position, x b represents the corresponding predicted class, w i represents the weight, which decays as the distance of the i-th pixel position to the edge of the real instance mask increases.

[0029] In one of the embodiments, the class loss function employed by the instance segmentation head during training is the cross-entropy loss between the predicted class and the real class.

[0030] In one of the embodiments, the training data set construction method of the instance segmentation model comprises.

[0031] S701, based on the real plate car point cloud data, generating plate car enhanced point cloud data through data augmentation operation; based on the real plate car point cloud data and the plate car enhanced point cloud data, constructing a plate car point cloud data set.

[0032] S702, orthogonal projection and pseudo-color synthesis are performed on the plate car point cloud data set to generate pseudo-color projection image samples.

[0033] S703, labeling the pseudo-color projection image samples to obtain a training data set, wherein the labeling rule is.

[0034] For high-low plate cars, label the whole vehicle loading surface and high plate loading surface.

[0035] For high-low plate cars with height increasing tools, label the whole vehicle loading surface, high plate loading surface and height increasing tools.

[0036] For flat cars, only label the whole vehicle loading surface.

[0037] In one of the embodiments, the data augmentation operation specifically comprises.

[0038] Randomly inject 0%-15% noise into the real plate car point cloud data to generate enhanced point cloud data simulating clutter reflection.

[0039] Randomly remove 5%-10% of the points in the real point cloud data of the flatbed truck to generate enhanced point cloud data simulating point cloud missing.

[0040] Remove the point cloud corresponding to the side turnover plate from the real point cloud data of the flatbed truck containing the side turnover plate to generate enhanced point cloud data simulating the case without the side turnover plate.

[0041] Remove the point cloud corresponding to the front bumper from the real point cloud data of the flatbed truck containing the front bumper to generate enhanced point cloud data simulating the case without the front bumper.

[0042] Rotate, translate, and / or horizontally mirror the real point cloud data of the flatbed truck to generate enhanced point cloud data simulating random flatbed truck parking positions.

[0043] In one of the embodiments, the data augmentation operation specifically further includes.

[0044] Randomly add at least one height-increasing tool near the middle bumper in the real point cloud data of the flatbed truck without the height-increasing tool.

[0045] Calculate and fuse the newly added point cloud and the missing point cloud caused by occlusion due to the addition of the height-increasing tool to generate enhanced point cloud data simulating the case with the height-increasing tool.

[0046] In one of the embodiments, S60 specifically includes.

[0047] S601, perform width constraint verification on the three-dimensional spatial position of the corner point of the flatbed truck.

[0048] Calculate the front plate width W f and the rear plate width W b of the flatbed truck or the high-low flatbed truck.

[0049] .

[0050] .

[0051] wherein x1, y1, z1 represent the three-dimensional spatial coordinates of the left front corner point of the flatbed truck plate, x6, y6, z6 represent the three-dimensional spatial coordinates of the right front corner point of the flatbed truck plate, x3, y3, z3 represent the three-dimensional spatial coordinates of the left rear corner point of the flatbed truck plate, and x4, y4, z4 represent the three-dimensional spatial coordinates of the right rear corner point of the flatbed truck plate.

[0052] The front plate width W f and the rear plate width W b of the flatbed truck or the high-low flatbed truck need to meet.

[0053] .

[0054] .

[0055] wherein W min represents the minimum plate width threshold, W max represents the maximum plate width threshold.

[0056] The middle plate width W m .

[0057] .

[0058] wherein x2, y2, z2 represent the three-dimensional space coordinates of the left end corner point of the high plate and low plate connecting part in the plate surface, and x5, y5, z5 represent the three-dimensional space coordinates of the right end corner point of the high plate and low plate connecting part in the plate surface.

[0059] The plate width W m needs to meet.

[0060] .

[0061] S602, the three-dimensional space position of the plate corner point is length-constrained and verified.

[0062] The left plate surface length L l and the right plate surface length L r of the flat plate car or the high-low plate car are calculated.

[0063] .

[0064] .

[0065] The left plate surface length L l and the right plate surface length L r need to meet.

[0066] .

[0067] .

[0068] wherein L min represents the minimum plate length threshold, L max represents the maximum plate length threshold.

[0069] S603, the three-dimensional space position of the plate corner point is angle-constrained and verified.

[0070] The absolute value of the angle between the vector and the vector is less than the angle threshold θ baseline , so as to ensure that the left and right side edges of the plate car are basically parallel.

[0071] The vector and the absolute value of the included angle of vector and vector baffle to ensure that the front and rear fenders of the pallet truck are substantially parallel.

[0072] wherein, vector represents a vector formed by the left front corner point and the left rear corner point of the pallet truck deck; vector represents a vector formed by the right front corner point and the right rear corner point of the pallet truck deck; vector represents a vector formed by the left front corner point and the right front corner point of the pallet truck deck; and vector represents a vector formed by the left rear corner point and the right rear corner point of the pallet truck deck.

[0073] S604, height-constrained verification is performed on the three-dimensional spatial positions of the corner points of the pallet truck.

[0074] .

[0075] wherein, for a flat pallet truck, i = 1, 2, 3, 4, and for a high-low pallet truck, i = 1, 2, 3, 4, 5, 6, z min represents a minimum deck height threshold, and z max represents a maximum deck height threshold.

[0076] The pallet truck recognition method converts a three-dimensional point cloud to a two-dimensional image through orthogonal projection conversion, constructs an instance segmentation model that fuses a convolutional neural network and a Transformer architecture, realizes sub-pixel level segmentation and positioning of a high pallet loading surface, a whole vehicle loading surface, and a height-increasing tool (such as a small stool), and outputs verified three-dimensional corner point coordinates to a downstream intelligent logistics system after multi-dimensional geometric constraint verification. This method can support paper roll unmanned forklift automatic loading in a paper mill, automatic stacking of container goods, and other industrial scenarios, and provide centimeter-level spatial positioning guarantee for automatic loading and unloading. BRIEF DESCRIPTION OF DRAWINGS

[0077] Figure 1 is a schematic diagram of a pallet truck in an embodiment.

[0078] Figure 2 is a schematic diagram of a high-low pallet truck with a side turnover plate in an embodiment.

[0079] Figure 3 is a schematic diagram of a global projection image in an embodiment.

[0080] Figure 4 is a schematic diagram of a preset detection site in an embodiment.

[0081] Figure 5 is a schematic diagram of a local projection image in an embodiment.

[0082] Figure 6 Figure 1 is a schematic diagram of a loading surface of a whole vehicle in an embodiment.

[0083] Figure 7 Figure 3 is a schematic diagram of a height-increasing tool on a small bench in an embodiment.

[0084] Figure 8 Figure 4 is a schematic diagram of a loading surface of a whole vehicle in an embodiment.

[0085] Figure 9 Figure 5 is a structural framework diagram of an instance segmentation model in an embodiment. DETAILED DESCRIPTION

[0086] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0087] In an embodiment, a pallet recognition method is provided, which comprises the following steps.

[0088] S10, comprehensively scanning the pallet at all detection sites to obtain a global three-dimensional point cloud of the pallet; and scanning a specific part of the pallet at a preset detection site to obtain a local three-dimensional point cloud of the pallet.

[0089] Specifically, the three-dimensional point cloud of the pallet can be obtained by any of the following methods.

[0090] Method 1: SLAM of laser radar.

[0091] The point cloud data of the pallet can be collected by various means such as handheld tools, unmanned robots such as unmanned forklifts, and remote control robots. SLAM algorithms such as Lio-SAM, fast-lio2, etc. can be used to complete the modeling work of the pallet.

[0092] Method 2: Fixed position installation of three-dimensional laser radar scanning.

[0093] A three-dimensional laser radar with a large FOV angle, a small ranging error, and a moderate ranging range is selected, and a top-mounted data collection method is used. A specific implementation scheme is provided below.

[0094] A multi-line laser radar with a horizontal field of view of 360 degrees, a vertical field of view greater than 90 degrees, a ranging range greater than 40 m, a ranging error less than 1 cm, and a line bundle greater than 64 lines is selected. The laser radar can be a mechanical radar or a solid-state radar. To increase the field of view, the laser radar can also be loaded on a gimbal so that it can rotate.

[0095] Top installation refers to installing the radar directly above the parking area, and the installation height can be selected within the range of 5-10 m. When installing, it is necessary to ensure that more than 50% of the radar beam is perpendicular to the plate surface or the included angle with the plate surface is less than 70 degrees. The parking area can be selected as the loading parking area, the weighing parking area, etc.

[0096] In actual application, appropriate methods can be selected from the above two methods according to different requirements and application scenarios. The following lists several specific scenarios for reference.

[0097] (1) In scenarios with higher accuracy requirements, such as modeling and plate corner point extraction error less than 2 cm, the laser SLAM combined with top-mounted radar mode can be selected.

[0098] (2) In scenarios where SLAM modeling vehicles such as forklifts cannot operate autonomously, such as weighing parking areas, the top-mounted radar mode is selected.

[0099] (3) In scenarios where top-mounted radar cannot be installed and the site has conditions for unmanned vehicle operation, the laser SLAM mode is selected.

[0100] S20, orthogonally project the global three-dimensional point cloud and the local three-dimensional point cloud along the XOY plane of the world coordinate system to generate a global projection image and a local projection image, respectively.

[0101] Among them, the global projection image and the local projection image are marked with different colors.

[0102] Specifically, the projection scale is adaptively calculated according to the point cloud range, so that a single pixel of the image corresponds to about 1 cm of actual distance.

[0103] The gray value represents a linear mapping of the height, and the mapping parameter is also calculated by adaptive point cloud statistics, which can adapt to different radar installation heights and different plate heights without manual configuration. The projection formula is as follows.

[0104] .

[0105] .

[0106] .

[0107] Among them, (x i ,y i ,z i ) is the three-dimensional coordinates of the i-th point, (u i ,v i ) is the pixel coordinates in the projection image, and I(u i ,v i ) is the gray value of the corresponding pixel, (s xy ,sz respectively, are the projection offsets adaptively obtained according to the point cloud statistics, z min is the statistical minimum value of the height of the point cloud, which is automatically calculated by the algorithm. The global projection image is as shown in Figure 3 .

[0108] Considering that the detection effect of the point cloud collected at different positions on different parts of the flat car is not the same in the laser SLAM process, the following further optimization method is proposed.

[0109] Please refer to Figures 3-5 , the red part is the point cloud collected at the stationary position 1 and the stationary position 2, which can better accurately detect and model the front and rear fenders of the flat car. The gray part is the point cloud collected at all positions, which can comprehensively detect and model the flat surface.

[0110] S30, pseudo-color synthesis is performed on the global projection image and the local projection image to generate a pseudo-color projection image.

[0111] Specifically, pseudo-color synthesis is performed on the global projection image and the local projection image to generate a pseudo-color projection image. The RGB three-channel data of this pseudo-color projection image is used as the input of the subsequent deep learning network.

[0112] S40, input the pseudo-color projection image into the trained instance segmentation model to output the flat car structure instance segmentation result.

[0113] The instance segmentation model includes a backbone network, an encoder, a decoder, and an instance segmentation head. The flat car structure instance segmentation result includes a flat car structure instance mask and its corresponding category. The category includes a whole vehicle loading surface, a high plate loading surface, and / or a height increasing tool.

[0114] As an embodiment of step S40, it specifically includes the following sub-steps.

[0115] S401, input the pseudo-color projection image into the backbone network to extract multi-scale image features.

[0116] S402, input the multi-scale image features into the encoder to generate encoded features containing global context information by using a self-attention mechanism.

[0117] S403, input the multi-scale image features and the encoded features into the decoder to generate high-resolution pixel-level features and N embedded vectors.

[0118] Further, the specific implementation of step S403 is as follows.

[0119] S4031, input the multi-scale image features into a pixel decoder, use a feature pyramid network to up-sample the multi-scale image features and fuse high-level semantic information to generate high-resolution pixel-level features.

[0120] S4032, input the encoded features into a Transformer decoder, use N learnable object query vectors to interact with the encoded features through cross-attention mechanism to generate N embedding vectors.

[0121] S404, input the pixel-level features and the embedding vectors into an instance segmentation head to generate the flatcar structure instance segmentation result.

[0122] Further, the specific implementation of step S404 is as follows.

[0123] S4041, input the N embedding vectors into a multi-layer perceptron to generate N mask embeddings and N predicted categories.

[0124] S4042, perform dot product operation between each mask embedding and the pixel-level features to generate N probability maps.

[0125] S4043, convert the N probability maps into binary masks through an activation function as N predicted instance masks.

[0126] S4044, perform dot product operation between the N predicted instance masks and the N predicted categories to generate the flatcar structure instance segmentation result.

[0127] S50, perform flatcar type recognition based on the flatcar structure instance segmentation result, and extract the two-dimensional pixel coordinates of the flatcar corner points in the pseudo-color projection image; project the two-dimensional pixel coordinates back to the three-dimensional point cloud coordinate system to determine the three-dimensional spatial position of the flatcar corner points.

[0128] Specifically, after obtaining the flatcar structure instance segmentation result, if both the whole vehicle loading surface instance and the high flat loading surface instance are included, it is determined as a high-low flatcar type; if only the whole vehicle loading surface instance is included, it is determined as a flatcar type.

[0129] As shown in Figure 1 , the numbers 1-6 marked in the figure correspond to important corner point information of the flatcar respectively. Among them, the information output of corner point 2 and corner point 5 depends on the type of flatcar: if the flatcar is a flatcar, then corner point 2 and corner point 5 will not output any information; if the flatcar is a high-low flatcar, then corner point 2 and corner point 5 will normally output the corresponding point information.

[0130] After obtaining the corner points on the image, the xyz values of the corner points are obtained using the following formula.

[0131] .

[0132] .

[0133] .

[0134] For the flatbed truck, i = 1, 3, 4, 6, and for the high-low bed truck, i = 1, 2, 3, 4, 5, 6. Among them, the corner point 1 represents the left front end corner point of the bed surface (or the carrying platform), which is used to define the boundary position of the bed surface in the left front; the corner point 6 represents the right front end corner point of the bed surface, which is used to define the boundary position of the bed surface in the right front; the corner point 3 represents the left rear end corner point of the bed surface, which is used to define the boundary position of the bed surface in the left rear; the corner point 4 represents the right rear end corner point of the bed surface, which is used to define the boundary position of the bed surface in the right rear; the corner point 2 represents the left end corner point of the high plate and low plate connecting part of the bed surface, which is used to define the boundary position of the high plate and low plate connecting transition area in the left side; the corner point 5 represents the right end corner point of the high plate and low plate connecting part of the bed surface, which is used to define the boundary position of the high plate and low plate connecting transition area in the right side.

[0135] S60, the three-dimensional space position of the bed corner point is subjected to multi-dimensional geometric constraint verification, and the three-dimensional space position is output after the verification is passed.

[0136] As an embodiment of step S60, it specifically includes the following sub-steps.

[0137] S601, the three-dimensional space position of the bed corner point is subjected to width constraint verification.

[0138] The front bed surface width W f and the rear bed surface width W b of the flatbed truck or the high-low bed truck are calculated.

[0139] .

[0140] .

[0141] Among them, x1, y1, z1 represents the three-dimensional space coordinates of the left front end corner point of the bed surface, x6, y6, z6 represents the three-dimensional space coordinates of the right front end corner point of the bed surface, x3, y3, z3 represents the three-dimensional space coordinates of the left rear end corner point of the bed surface, and x4, y4, z4 represents the three-dimensional space coordinates of the right rear end corner point of the bed surface.

[0142] The front bed surface width W f and the rear bed surface width W b must be satisfied.

[0143] .

[0144] .

[0145] wherein W min represents the minimum plate width threshold, W max represents the maximum plate width threshold.

[0146] The middle plate width W m of the high-low plate truck is calculated.

[0147] .

[0148] wherein x2, y2, z2 represent the three-dimensional space coordinates of the left end corner point of the high plate and low plate connection part in the plate surface, and x5, y5, z5 represent the three-dimensional space coordinates of the right end corner point of the high plate and low plate connection part in the plate surface.

[0149] The plate width W m needs to meet.

[0150] .

[0151] S602, the three-dimensional space position of the plate truck corner point is length-constrained and verified.

[0152] The left plate surface length L l and the right plate surface length L r of the flat plate truck or the high-low plate truck are calculated.

[0153] .

[0154] .

[0155] The left plate surface length L l and the right plate surface length L r need to meet.

[0156] .

[0157] .

[0158] wherein L min represents the minimum plate length threshold, L max represents the maximum plate length threshold.

[0159] S603, the three-dimensional space position of the plate truck corner point is angle-constrained and verified.

[0160] The absolute value of the angle between the vector and the vector is less than the angle threshold θ baseline , so as to ensure that the left and right edges of the plate truck are basically parallel.

[0161] The absolute value of the angle between the vector and the vector The absolute value of the included angle of the two vectors is less than an angle threshold θ baffle To ensure that the front and rear fenders of the board car are substantially parallel.

[0162] wherein the vector represents a vector formed by the left front end corner point 1 and the left rear end corner point 3 of the board surface of the board car; the vector represents a vector formed by the right front end corner point 6 and the right rear end corner point 4 of the board surface of the board car; the vector represents a vector formed by the left front end corner point 1 and the right front end corner point 6 of the board surface of the board car; and the vector represents a vector formed by the left rear end corner point 3 and the right rear end corner point 4 of the board surface of the board car.

[0163] S604, height constraint verification is performed on the three-dimensional space position of the corner point of the board car.

[0164] .

[0165] wherein, for a flat board car, i = 1, 2, 3, 4, and for a high-low board car, i = 1, 2, 3, 4, 5, 6, z min represents a minimum board surface height threshold, z max represents a maximum board surface height threshold.

[0166] In an embodiment, in the training stage of the instance segmentation model, the following auxiliary information needs to be collected.

[0167] Board car type: flat board car, high-low board car; distinguished according to whether the board car has high and low loading planes, and only one height loading plane is a flat board car, and the presence of high and low loading planes is a high-low board car.

[0168] Board car side turnover plate: with side turnover plate, without side turnover plate.

[0169] Whether the board surface has a small board stool: with a small board stool, without a small board stool.

[0170] Board car width value: front board surface width, middle board surface width, rear board surface width.

[0171] Board car length value: left side board surface length, right side board surface length.

[0172] The number of samples of the flat board car and the high-low board car is not less than 4000.

[0173] The number of samples of the board car with a small board stool is not less than 2000.

[0174] The number of samples of the board car with a side turnover plate is not less than 2000; the high-low board car with a side turnover plate is as shown in Figure 2 .

[0175] In one embodiment, the training data set construction method of the instance segmentation model comprises the following steps.

[0176] S701, based on the real point cloud data of the board car, generate enhanced point cloud data of the board car through data augmentation operation; based on the real point cloud data of the board car and the enhanced point cloud data of the board car, construct a board car point cloud data set.

[0177] Specifically, 0%-15% noise is randomly injected into the real point cloud data of the board car to generate enhanced point cloud data simulating clutter reflection, simulating clutter reflection caused by laser radar measurement.

[0178] Randomly remove 5%-10% of the points from the real point cloud data of the board car to generate enhanced point cloud data simulating point cloud loss, simulating point cloud loss caused by rainwater and other conditions on the board.

[0179] Remove the point cloud corresponding to the side turning board from the real point cloud data of the board car containing the side turning board to generate enhanced point cloud data simulating the situation without a side turning board, simulating both with and without a side turning board.

[0180] Remove the point cloud corresponding to the front baffle from the real point cloud data of the board car containing the front baffle to generate enhanced point cloud data simulating the situation without a front baffle, simulating both with and without a front baffle.

[0181] Rotate, translate and / or horizontally mirror the real point cloud data of the board car to generate enhanced point cloud data simulating random board car parking positions, simulating the randomness of board car parking positions.

[0182] In the real point cloud data of the board car without height increasing tools, at least one height increasing tool such as a solid wooden stool or a hollow steel stool is randomly added near the middle baffle. Then, through laser ray tracing algorithm, the newly added point cloud and the missing point cloud caused by occlusion are calculated and fused to generate enhanced point cloud data simulating the situation with height increasing tools.

[0183] S702, orthogonal projection and pseudo-color synthesis are performed on the board car point cloud data set to generate pseudo-color projection image samples.

[0184] S703, label the pseudo-color projection image samples to obtain a training data set, wherein the labeling rule is.

[0185] For high-low board cars, label the whole vehicle loading surface and the high board loading surface; as shown in Figure 6 , the blue color is the whole vehicle loading surface and the green color is the high board loading surface.

[0186] For high-low board cars with height increasing tools (such as stools), label the whole vehicle loading surface, the high board loading surface and the height increasing tools; as shown in Figure 7 , the pink color is the height increasing tool.

[0187] For the flatbed truck, only the whole vehicle loading surface is marked, as shown in Figure 8 The blue color represents the whole vehicle loading surface.

[0188] In an embodiment, the instance segmentation head adopts an instance mask loss function L m as follows during training.

[0189] .

[0190] .

[0191] .

[0192] wherein A represents the real instance mask, B represents the predicted instance mask, x a represents the real class of the i-th pixel position, x b represents the corresponding predicted class, w i represents the weight, which decays as the distance of the i-th pixel position from the edge of the real instance mask increases.

[0193] In an embodiment, the class loss function adopted by the instance segmentation head during training is the cross-entropy loss between the predicted class and the real class.

[0194] Specifically, in order to improve the accuracy of corner extraction, an instance segmentation algorithm with high boundary position extraction accuracy is adopted. Compared with the target detection algorithm, the instance segmentation algorithm can better regress the position of the edge and corner of the flatbed truck.

[0195] A mainstream instance segmentation algorithm based on convolutional neural network and Transformer can be selected, such as MaskRCNN, Mask RT-DETR, etc.

[0196] The specific implementation of an instance segmentation algorithm improvement method based on convolutional neural network + Transformer attention mechanism is given as follows.

[0197] The instance segmentation model is composed of four parts: backbone network, encoder, decoder, and instance segmentation head.

[0198] (I) Backbone network

[0199] The backbone network extracts key image features from the input flatbed truck projection image. The backbone network can select any advanced CNN (convolutional neural network) architecture, such as ResNet. The image features extracted by the backbone network are represented by F i .

[0200] (II) Encoder

[0201] The image feature F i is input into the Transformer as tokens, and the global context information of the image is captured by the self-attention mechanism.

[0202] The decoder

[0203] The pixel decoder

[0204] The image feature F i is obtained in (I) is upsampled to high resolution to generate pixel-level feature F p .

[0205] The Feature Pyramid Network (FPN) is used to gradually upsample and fuse high-level semantic information.

[0206] The Transformer decoder

[0207] N object query vectors are used in the Transformer decoder model to "query" the image feature to locate the mask position and class label of the target. Each object query vector is a learnable parameter randomly generated at model initialization and gradually optimized through training, focusing on the relevant area in the image feature through cross-attention mechanism.

[0208] A set of N embeddings, denoted as E t , is generated using the Transformer decoder.

[0209] The semantic segmentation head

[0210] The E t generated in (III) is passed through an MLP network to generate N mask embeddings. The mask embedding is a set of N-dimensional vectors, each corresponding to a potential segmentation area in the image.

[0211] Each mask embedding is point-multiplied with the F p high-resolution feature map generated in (III) to generate N probability maps. Then the probability map is converted to a binary mask through the Sigmoid function as N mask predictions.

[0212] Optimally, the mask loss function is designed to enhance edge accuracy to ensure the accuracy of the final corner extraction.

[0213] The loss function is divided into two parts.

[0214] The Dice loss measures the proportion of overlapping areas between the predicted mask and the real mask, and the formula is.

[0215] .

[0216] Binary cross-entropy calculates the cross-entropy of the prediction probability and the true label pixel by pixel, and enhances the optimization of edge details.

[0217] In particular, considering that the accuracy of the plate truck edge corner point is most concerned, a weighted binary cross-entropy loss function is adopted.

[0218] .

[0219] x a is the true label of the i-th pixel position, x b is the corresponding predicted label, w i is the weight, and as the distance between the i-th pixel position and the edge of the true mask decays, the decay function can be selected from function types such as exponential decay.

[0220] The total loss of the mask is .

[0221] The above loss function design method for enhancing edge accuracy strengthens the mask edge loss by setting the weight decaying with the distance from the plate truck edge.

[0222] (Three) generated E t After passing through the MLP network, N category predictions are generated, and the category loss function calculates the matching loss of the predicted category and the true category using cross-entropy.

[0223] N mask predictions and N category predictions are multiplied, each mask is assigned a category label, and then the background label is removed and only the foreground target label is kept to generate the final instance segmentation result. The foreground label includes the high plate loading surface, the whole vehicle loading surface, and the small plate stool.

[0224] It should be understood that although each step in the flowchart involved in each of the above embodiments is displayed in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise stated herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in each of the above embodiments can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps.

[0225] The above merely describes specific implementation manners of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical region disclosed by the embodiments of the present application, and all should be covered in the protection scope of the embodiments of the present application. Therefore, the protection scope of the embodiments of the present application should be subject to the protection scope of the claims.

Claims

1. A method for identifying handcarts, characterized in that, include: S10, perform a full scan of the flatbed vehicle at all detection points to obtain the global three-dimensional point cloud of the flatbed vehicle; A specific part of the flatbed truck is scanned at a preset detection point to obtain a local three-dimensional point cloud of the flatbed truck; wherein, the global three-dimensional point cloud can perform comprehensive detection and modeling of the flatbed truck surface, and the local three-dimensional point cloud can perform precise detection and modeling of the front and rear baffles of the flatbed truck. S20, the global 3D point cloud and the local 3D point cloud are orthogonally projected along the world coordinate system XOY plane to generate a global projection image and a local projection image respectively; wherein, the global projection image and the local projection image are marked with different colors; S30, perform pseudo-color synthesis on the global projection image and the local projection image to generate a pseudo-color projection image; S40, the pseudo-color projection image is input into the trained instance segmentation model, and the flatbed structure instance segmentation result is output; wherein, the instance segmentation model includes a backbone network, an encoder, a decoder and an instance segmentation head, and the flatbed structure instance segmentation result includes a flatbed structure instance mask and its corresponding category, the category including the whole vehicle loading surface, the high-bed loading surface and / or the heightening tool; S50, based on the segmentation results of the flatbed vehicle structure instance, identify the flatbed vehicle type and extract the two-dimensional pixel coordinates of the flatbed vehicle corner points in the pseudo-color projection image; inversely project the two-dimensional pixel coordinates back to the three-dimensional point cloud coordinate system to determine the three-dimensional spatial position of the flatbed vehicle corner points. S60, perform multi-dimensional geometric constraint verification on the three-dimensional spatial position of the corner point of the flatbed truck, and output the three-dimensional spatial position after the verification is passed; Wherein, the encoder is a Transformer encoder, and the decoder includes a pixel decoder and a Transformer decoder; S40 specifically includes: S401, Input the pseudo-color projection image into the backbone network and extract multi-scale image features; S402, the multi-scale image features are used as the token input of the Transformer encoder, and a self-attention mechanism is used to generate encoded features containing global context information; S403, input the multi-scale image features and the encoded features into the decoder to generate high-resolution pixel-level features and N embedding vectors; S404, Input the pixel-level features and the embedding vector into the instance segmentation head to generate the flatbed vehicle structure instance segmentation result; Furthermore, S403 specifically includes: S4031, The multi-scale image features are input into the pixel decoder, and the feature pyramid network is used to upsample the multi-scale image features and fuse high-level semantic information to generate high-resolution pixel-level features. S4032, the encoded features are input into the Transformer decoder, and N learnable object query vectors are used to interact with the encoded features through a cross-attention mechanism to generate N embedding vectors.

2. The method according to claim 1, characterized in that, S404 specifically includes: S4041, Input the N embedding vectors into the multilayer perceptron to generate N mask embeddings and N predicted categories; S4042, each mask embedding is multiplied with the pixel-level feature to generate N probability maps; S4043, The N probability maps are converted into binary masks by an activation function, which are used as N prediction instance masks; S4044, Perform a dot product operation between the N predicted instance masks and the N predicted categories to generate a flatbed truck structure instance segmentation result.

3. The method according to claim 2, characterized in that, The instance segmentation head uses the instance mask loss function L during training. m as follows: ； ； ； Where A represents the real instance mask, B represents the predicted instance mask, and x a Let x represent the true class at the i-th pixel position. b w represents the corresponding prediction category. i The weight represents the distance between the i-th pixel position and the edge of the real instance mask, which decreases as the distance increases.

4. The method according to claim 2, characterized in that, The instance segmentation head uses the cross-entropy loss function between the predicted class and the true class during training.

5. The method according to claim 1, characterized in that, The method for constructing the training dataset for the instance segmentation model includes: S701, Based on the real point cloud data of the flatbed truck, generate enhanced point cloud data of the flatbed truck through data augmentation operation; construct a flatbed truck point cloud dataset based on the real point cloud data of the flatbed truck and the enhanced point cloud data of the flatbed truck. S702, orthogonal projection and pseudo-color synthesis are performed on the flatbed truck point cloud dataset to generate pseudo-color projection image samples; S703, the pseudo-color projection image samples are labeled to obtain a training dataset, wherein the labeling rules are as follows: For low-bed trailers, mark the full vehicle loading surface and the high-bed loading surface; For low-bed trailers equipped with lift-up tools, mark the overall vehicle loading surface, the high-bed loading surface, and the lift-up tools. For flatbed trucks, only the loading surface of the entire vehicle is marked.

6. The method according to claim 5, characterized in that, The data augmentation operation specifically includes: Randomly inject 0%-15% noise into the real point cloud data of the flatbed truck to generate enhanced point cloud data that simulates clutter reflection; Randomly remove 5%-10% of the points from the actual point cloud data of the flatbed truck to generate enhanced point cloud data that simulates the missing points in the point cloud; Remove the point cloud corresponding to the side tipper from the real point cloud data of the flatbed truck containing the side tipper, and generate enhanced point cloud data simulating the case without the side tipper; Remove the point cloud corresponding to the front baffle from the real point cloud data of the flatbed truck that includes the front baffle, and generate enhanced point cloud data simulating the case without the front baffle; Rotate, translate, and / or mirror the real point cloud data of the flatbed truck to generate augmented point cloud data simulating random parking positions of the flatbed truck.

7. The method according to claim 6, characterized in that, The data augmentation operation specifically includes: In the real point cloud data of the flatbed truck without any height-increasing tools, at least one height-increasing tool is randomly added in the area near the middle baffle. Using a laser ray tracing algorithm, the newly added point cloud due to the addition of the height-increasing tool and the missing point cloud due to occlusion are calculated and fused to generate enhanced point cloud data simulating the situation with the height-increasing tool.

8. The method according to claim 1, characterized in that, The S60 specifically includes: S601, Perform width constraint verification on the three-dimensional spatial position of the corner point of the flatbed truck: Calculate the width W of the front panel of the flatbed truck or low-bed trailer. f and the width W of the back panel b : ； ； Where x1, y1, z1 represent the three-dimensional spatial coordinates of the left front corner of the flatbed, x6, y6, z6 represent the three-dimensional spatial coordinates of the right front corner of the flatbed, x3, y3, z3 represent the three-dimensional spatial coordinates of the left rear corner of the flatbed, and x4, y4, z4 represent the three-dimensional spatial coordinates of the right rear corner of the flatbed. Front panel width W f and the width W of the back panel b Must meet: ； ； Among them, W min W represents the minimum board width threshold. max This indicates the maximum board width threshold; Calculate the width W of the middle panel of the high and low flatbed truck. m : ； Where x2, y2, and z2 represent the three-dimensional spatial coordinates of the left corner point of the connection between the high and low plates on the flatbed, and x5, y5, and z5 represent the three-dimensional spatial coordinates of the right corner point of the connection between the high and low plates on the flatbed. Board width W m Must meet: ； S602, Perform length constraint verification on the three-dimensional spatial position of the corner point of the flatbed truck: Calculate the length L of the left side panel of the flatbed truck or low-bed truck. l and the length L of the right side panel r : ； ； Left side panel length L l and the length L of the right side panel r Must meet: ； ； Among them, L min L represents the minimum plate length threshold. max This indicates the maximum plate length threshold; S603, Perform angle constraint verification on the three-dimensional spatial position of the corner point of the flatbed truck: vector sum vector The absolute value of the included angle is less than the angle threshold θ baseline To ensure that the straight lines of the left and right edges of the flatbed cart are basically parallel; vector sum vector The absolute value of the included angle is less than the angle threshold θ baffle To ensure that the front and rear baffles of the flatbed cart are basically parallel; Where, vector This represents the vector formed by the left front corner and the left rear corner of the flatbed truck's surface; vector This represents the vector formed by the right front corner and the right rear corner of the flatbed truck's surface; vector This represents the vector formed by the left and right front corners of the flatbed truck's platform; vector This represents the vector formed by the left and right rear corner points of the flatbed truck's surface. S604, Perform height constraint verification on the three-dimensional spatial position of the corner point of the flatbed truck: ； Where, for flatbed trucks, i=1,2,3,4; for high-low flatbed trucks, i=1,2,3,4,5,6; z min z represents the minimum board height threshold. max This indicates the maximum board height threshold.

Images