Cabin box body information delay adaptive alignment method and system

Abstract

The invention provides a cabin box body information delay adaptive alignment method and system, and the method comprises the steps: obtaining a target position-category vector in an image, and constructing a target position time sequence list; adding a time sequence list containing point cloud information on the basis of the target position time sequence list to obtain a target average height time sequence list; obtaining coordinate, speed and acceleration information of the trolley, and obtaining a world position-target position time sequence information list through world coordinate-target position mapping; based on the target position time sequence list, obtaining a target vector time sequence list and a random cutting target vector time sequence list; and providing an inter-frame target position delay adaptive prediction model, aligning the time sequence target of each frame between different sensors through the model, complementing shielded and / or missing target position information, and completing cabin box body information delay adaptive alignment. Alignment of box body information in the cabin operation process is achieved, and the problem that a visual target is shielded and cannot be tracked in a visual mode is solved.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and in particular to a method and system for adaptively aligning cabin box information delay during a quay crane ship scanning operation, and also to a corresponding computer terminal and a computer-readable storage medium. Background Art

[0002] Object detection methods based on deep learning have been widely used in industrial scenarios, but their instability and black box characteristics make it impossible to predict their decision-making process, so they cannot be used alone in scenarios with high-precision detection requirements.

[0003] The current mainstream vision-based target tracking algorithms are highly dependent on the sequence of historical and future detection targets, and will fail when the target detection method fails. For example, visual target tracking methods such as KCF, CMT, Kalman filtering, ByteTrack, Deepsort, etc. can track multi-frame objects captured by non-moving cameras, but when facing the moving camera coordinate system during the quay crane ship scanning operation and the box object is blocked, the target will also be lost. Although the target can be detected when it is not blocked in the subsequent frame, it will be marked as a new object by the tracking method. Because the point cloud information will be blocked by the hoist at the same time, and the radar faces the cabin box in a downward posture, it can only scan the upper surface of the box. The operation time causes the laser radar scanning time to be short and the point cloud to be sparse. Therefore, the 3D target detection and tracking method based on time series point cloud is not applicable in this scenario.

[0004] After searching, we found:

[0005] The Chinese invention patent application with publication number CN117830355A, "A target re-tracking method, device, equipment and computer-readable storage medium", designs the head and shoulders of people in the area of ​​interest for its queuing scenario, and combines the Kalman filter target and the target information of the previous frame to solve the target offset problem caused by occlusion. However, in the quay crane operation ship scanning scenario, it will encounter a scenario where the box is completely blocked by the spreader and partially blocked by the spreader at the beginning. Therefore, this method fails in the quay crane operation ship scanning scenario.

[0006] The Chinese invention patent application with publication number CN115496779A, "Image-based object tracking method, device, electronic device and storage medium", is also improved based on the Deepsort target tracking method. Its tracking target must have appeared in its image frame sequence. This method is also ineffective in the scanning scenario of the quay crane operation ship.

[0007] The Chinese invention patent application with publication number CN117893571A, "Method, system and device for tracking motion of occluded targets based on autoregressive motion model", regressively tracks the motion trajectory of the occluded target based on neighboring targets, but the tracked target must have appeared in its image frame sequence, and this method may have some unknown unstable situations.

[0008] The Chinese invention patent application with publication number CN117036740A, "An Anti-Occlusion Tracking Method for Moving Targets", first extracts different feature vectors for fusion and filtering to resist complex light shadows, and uses the response peak detection mechanism of the target tracker to determine occlusion, and then uses Kalman filtering to predict the position of the occluded target. The target of this method must appear at least once in the input frame sequence, otherwise the method is not available, so it fails in the quay crane operation ship scanning scene. Summary of the invention

[0009] In view of the above-mentioned deficiencies in the prior art, the present invention provides a method and system for adaptively aligning cabin box information delay during a quay crane ship scanning operation, and also relates to a corresponding computer terminal and a computer-readable storage medium.

[0010] According to one aspect of the present invention, a method for adaptively aligning cabin box information delay is provided, comprising:

[0011] Get the target position of the box and the hanger in the camera image - the category vector , build a time series list of target positions , used for coarse positioning of targets;

[0012] At the target location time sequence list Based on the time series list containing point cloud information obtained from the laser radar, the target average height time series list of the box and the spreader is obtained. , where m is the number of targets, which is used to assist in locating targets that may be obscured;

[0013] Get the coordinates, speed and acceleration information of the car , where N is the length of the frame sequence. Through the world coordinate-target position mapping, the world position-target position timing information list of the box and the hanger is obtained. ;

[0014] Based on the target position time sequence list , respectively obtain the target vector time series list containing the complete box and the hanger visual features And a time series list of randomly cropped target vectors containing box and spreader edge features ;

[0015] Provide an inter-frame target position delay adaptive prediction model, , , , , and The inter-frame target position delay adaptive prediction model is input to align the timing targets of each frame between different sensors to complete the target position information of the obscured and / or missing boxes and spreaders, and complete the delayed adaptive alignment of the cabin box information.

[0016] Preferably, the target position-category vector of the box and the sling in the camera image is obtained. , build a time series list of target positions ,include:

[0017] Provides a single-stage target detection model that takes an RGB channel image as input and outputs a target position-category vector ;in, = [[current target category, x-axis coordinate of the current target geometric center on the image, y-axis coordinate of the current target geometric center on the image, pixel length of the current target in the x-axis direction on the image, pixel length of the current target in the y-axis direction on the image], ...];

[0018] Obtain a camera image, use the camera image as the input of the single-stage target detection model, output the target position of the container body and the spreader and the spreader type through the visual positioning of the single-stage target detection model, and obtain the corresponding target position-category vector , build a time series list of target positions , used for coarse positioning of occluded targets.

[0019] Preferably, the time sequence list at the target location Based on the time series list containing point cloud information obtained from the laser radar, the target average height time series list of the box and the hanger is obtained. ,include:

[0020] Provides a pre-calibrated camera intrinsic / extrinsic parameters, including: optical center / rotation matrix / translation vector - , the point cloud , projected to the coordinates of the camera image ,get:

[0021] s∙ u v 1 =K∙[R|t]∙ X Y Z 1

[0022] in, is the scale factor, which indicates the ratio from three-dimensional space to two-dimensional image plane;

[0023] The target position-category vector The coordinates of the four vertices that contain the polygonal area , , , , construct a quadrilateral boundary range of the X and Y axes set on a two-dimensional plane; for the coordinates of any point in the point cloud , determine whether the arbitrary point is located inside the boundary of the quadrilateral, filter the point cloud data, and only retain valid points within the boundary;

[0024] After processing the abnormal values ​​and outliers in the point cloud, the point cloud is averaged to obtain the target average height time series list of the box and the hanger .

[0025] Preferably, the acquisition of the coordinates, speed and acceleration information of the vehicle , where N is the length of the frame sequence. Through the world coordinate-target position mapping, the world position-target position timing information list of the box and the hanger is obtained. ,include:

[0026] The position of the car is collected through the inertial odometer ,speed and acceleration information, and at the same time obtain the hull type information and convert it to the center coordinates in the ship width direction to obtain the hull information;

[0027] The world coordinates of the center of the hull are calculated by using the hull information and the center position of the current camera image field of view. , determine whether the current camera image field of view includes the edge of the cabin and the orientation of the field of view on the hull, and obtain the relative position of the center of the current frame camera image and the ship; where the orientation of the field of view on the hull is determined as:

[0028]

[0029] in, The world coordinates converted from the center of the camera image in the current frame; is the preset threshold;

[0030] When Position is "above", the ranking coordinates Set to 0, if Position is "middle" and "below", then ,in is the number of pixels on the width side of the container;

[0031] Target position timing list based on current frame target , sort the targets in the current frame, and then assign , get the ranking coordinates of all targets in a single frame, and construct the target time series list List of ships with target timings for boxes and spreaders ;

[0032] According to the hull type information, it is converted into the center coordinates in the ship width direction and the position information of the car. and speed information And the target time series list List of ships with target timings for boxes and spreaders , bind the targets in each rank between frames, and get the world position-target position timing information list of the box and the spreader .

[0033] Preferably, the target position time sequence list based on the current frame target , sort the targets in the current frame, and then assign , get the ranking coordinates of all targets in a single frame, and construct the target time series list List of ships with target timings for boxes and spreaders ,include:

[0034] From the target position time sequence list , take out the i-th frame, according to the 0th "category information" of the first dimension, the target location information of the i-th frame includes: all container location information and spreader position information , where q represents the q box targets contained in the i-th frame, Respectively represent the pixel position information of the center point of the 1st / 2nd / ... / qth box target in the i-th frame; using the position time sequence list of the target in the i-th frame , sort the targets in the i-th frame, where the sorting method is: first sort in ascending order according to the y coordinate, then cluster according to the y coordinate value of each target, and set the threshold , traverse and calculate the distance between these y, the distance is less than The final result is that these targets can be divided into multiple rows. In each row, these targets are each taken by x coordinates, and then sorted according to the x coordinates in each row. From then on, the targets of the i-th frame are changed from disordered to ordered; next, the y coordinates of each target are compared with Add them together to get the ranking of all targets in a single frame, that is, the ranking coordinates of the ships where all boxes are located and the coordinates of the ship where the spreader is located , They represent the coordinates of the ship where the 1st / 2nd / ... / qth box is located in the i-th frame respectively; suppose the N-th frame is being processed and the N-1 frames have been processed before, then and Add to as a tuple , get the ship ranking list containing the target timing of all current frames, i.e., the 1st to the Nth frame boxes and spreaders , N is the total number of frames that have been processed before processing the i-th frame information (N-1) plus the current frame information being processed; when processing the initial frame, ; Similarly, and After sorting, the same operation is performed as above to obtain , which represents the target time sequence list containing all current frame boxes and spreaders;

[0035] Preferably, the center coordinates of the ship width direction and the position information of the vehicle are converted according to the hull type information. and speed information And the target time series list List of ships with target timings for boxes and spreaders , bind the targets of each rank between frames, including:

[0036] According to the obtained hull type information, convert it to the center coordinates in the width direction of the ship; at the same time, obtain the position of the car and speed and target time series list for boxes and spreaders List of ships with target timings for boxes and spreaders ;

[0037] Calculate instantaneous velocity for adjacent time frames , as well as , calculate the acceleration of the car and the box in adjacent frames, then update the positions of adjacent frames and predict the world position of the next frame;

[0038] Compare the difference between the relative position change calculated between adjacent frames and the relative position change measured by the sensor. If it is less than the set threshold, , then the targets are considered consistent and binding is performed, otherwise target binding is abandoned and the update The corresponding target ranking information in is the value "-1", indicating that the position target of the i-th frame does not need to be bound to the ranking; then through , ship width information, box width information, calculate the world coordinates of each row of box targets, and then bind the targets in each frame, the rows of corresponding targets, and the world coordinates to obtain the world position-target position timing information list of the box and the spreader , Describes the position of the target in the i-th frame; whether the target in the i-th frame needs to be bound is based on The target ranking information value in is used for judgment; ,in is the world coordinate of the i-th frame.

[0039] Preferably, the time sequence list based on the target position , respectively obtain the target vector time series list containing the complete box and the hanger visual features And a time series list of randomly cropped target vectors containing box and spreader edge features ,include:

[0040] Based on the target position time sequence list , obtain multiple target location-category vectors , using the target position-category vector Crop the image target and get the Target image sequence for row boxes , where 1 to N are the frame numbers; the target image sequence Each frame in is used as the input of an autoencoder AutoEncoder-1 to record the key features of the same target in different frames ; For the target image sequence Randomly crop the images in to get the target image sequence ; The target image sequence As the input of another autoencoder AutoEncoder-2, it outputs the key features of the same target in different frames , used to locate occluded targets; All target positions-category vectors whose categories are boxes The target image sequence is respectively encoded by AutoEncoder-1 and AutoEncoder-2. and the target image sequence Compress it into vectors to get the target vector time series list of the box and the hanger And the random clipping target vector time series list , where d is the dimension of the target vector.

[0041] Preferably, the provision of an inter-frame target position delay adaptive prediction model, , , , , and The inter-frame target position delay adaptive prediction model is input to align the timing targets of each frame between different sensors to complete the target position information of the obscured and / or missing boxes and hoists, including:

[0042] Provided is an inter-frame target position delay adaptive prediction model, the inter-frame target position delay adaptive prediction model includes a multi-modal feature space alignment network and a temporal neural network; wherein the multi-modal feature space alignment network is used for aligning multi-modal feature space information, and the temporal neural network is used for delay adaptation between different modal information;

[0043] The multimodal feature space alignment network is composed of a fully connected neural network. , , , , and Mapped to the same representation space, the temporal neural network is a spatiotemporal convolutional network, which is modeled using a spatiotemporal convolutional network and integrates temporal visual features. ,average height , spreader and box historical world coordinates As well as the historical positions of the spreader targets, the temporal attention mechanism is finally used to adaptively align the boxes and spreader targets that are delayed and / or no longer appear in each frame due to occlusion:

[0044]

[0045]

[0046]

[0047] in, is the feature representation after integration on time frame t; Represents a fully connected neural network and its pre-processing step, i.e., the concatenation operation; is the fusion feature after time series modeling on time frame t; represents a temporal neural network; Indicates; align the tth frame data to the frame.

[0048] According to another aspect of the present invention, a cabin box information delay adaptive alignment system is provided, comprising: a box hoist target visual detector, a target height-position binder, a world coordinate-target position mapper, a target feature recording module, and an inter-frame target position delay adaptive predictor; wherein:

[0049] The box sling target visual detector is used to obtain the target position-category vector of the box and sling in the camera image. , build a time series list of target positions ;

[0050] The target height-position binder is used to list the time sequence at the target position. Based on the time series list containing point cloud information obtained from the laser radar, the target average height time series list of the box and the hanger is obtained. , where m is the number of targets;

[0051] The world coordinate-target position mapper is used to obtain the coordinate, speed and acceleration information of the car , where N is the length of the frame sequence. Through the world coordinate-target position mapping, the world position-target position timing information list of the box and the hanger is obtained. ;

[0052] The target feature recording module is used to record the target position based on the time sequence list , respectively obtain the target vector time series list containing the complete box and the hanger visual features And a time series list of randomly cropped target vectors containing box and spreader edge features ;

[0053] The inter-frame target position delay adaptive predictor is used to input the , , , , and , align the timing targets of each frame between different sensors, complete the target position information of the occluded and / or missing boxes and spreaders, and complete the delayed adaptive alignment of the cabin box information.

[0054] According to a third aspect of the present invention, there is provided a computer terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, can be used to execute any one of the methods described above in the present invention, or to run the system described above in the present invention.

[0055] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to execute any of the methods described above in the present invention, or to run the system described above in the present invention.

[0056] Due to the adoption of the above technical solution, the present invention has at least one of the following beneficial effects compared with the prior art:

[0057] The present invention solves the problem that the target cannot be detected when the target is blocked by the existing target detection method in the quay crane crane operation scene by simulating and aligning the features of the box when it is blocked and using the target position of the spreader for auxiliary positioning.

[0058] The present invention integrates the distance, position, speed and acceleration information of laser radar and inertial odometer, designs an inter-frame target position delay adaptive predictor to align target information that may be delayed, and adds deterministic information to improve the accuracy of target positioning.

[0059] The present invention uses the box sling target visual detection part to obtain 2D images from the camera image sensor, extract single-frame box and sling information, and provide a basic information platform for multi-sensor multi-modal information fusion.

[0060] The present invention provides depth / height information for the target of the single-modality image through the target height-position binding part, so as to cope with the possible misdetection problem of the visual detector box detector.

[0061] The present invention integrates the speed, acceleration and world position information of the inertial odometer through the world coordinate-target position mapping part to solve the problem of predicted target drift derived from the visual occlusion of the target in traditional inter-frame target tracking algorithms such as Kalman filtering.

[0062] The present invention uses the target feature recording part. When the hoist is working, it will block the box. At this time, the box only shows scattered edge information on the image. The edge image information and the complete box image information, after the box hoist target visual detector is used to extract features, the data distribution deviation is large. This module will sample the edge image information and align it with the distribution of the complete box image information; and output the distribution information to the inter-frame target position delay adaptive predictor to provide information about the blocked box for inter-frame data index alignment of the multi-sensor data list.

[0063] The present invention uses the delayed adaptive prediction part of the target position between frames to fuse multi-modal information, aligns the data frames of the visual sensor and the lidar sensor through the time series model and attention mechanism, and improves the efficiency of index alignment of inter-frame data according to image features. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0065] Figure 1 The figure is a working flow chart of a method for adaptively aligning cabin box information delay in one embodiment of the present invention.

[0066] Figure 2 The figure is a schematic diagram of the component modules of the cabin box information delay adaptive alignment system in one embodiment of the present invention.

[0067] Figure 3 It is a schematic diagram of the overall operation of a method and system for adaptively aligning cabin box information delay in a preferred embodiment of the present invention.

[0068] Figure 4 It is a working schematic diagram of the target feature recording module in a preferred embodiment of the present invention.

[0069] Figure 5 Schematic diagram of the operation of an inter-frame target position delay adaptive predictor in a preferred embodiment of the present invention.

[0070] Figure 6 The figure is a schematic diagram of a cabin in a specific application example of the present invention.

[0071] Figure 7 It is a schematic diagram of a box being blocked during an operation process in a specific application example of the present invention.

[0072] Figure 8 The figure is a specific practical effect diagram of the method and system for adaptively aligning cabin box information delay. DETAILED DESCRIPTION

[0073] The following is a detailed description of the embodiments of the present invention: This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation method and a specific operation process are given. It should be pointed out that for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

[0074] In view of the lack of effective target detection and tracking technology in the prior art for quay crane operation ship scanning scenarios, an embodiment of the present invention provides a cabin box information delay adaptive alignment method. This method constructs an information delay adaptive alignment method for the quay crane ship scanning operation process, and realizes the alignment of the box information during the cabin operation process by fusing the information of the lidar, camera and inertial odometer, thereby solving the problem that the visual target is blocked and cannot be tracked visually.

[0075] Specifically, Figure 1 As shown, the method for adaptively aligning cabin box information delay provided by this embodiment may include the following operations:

[0076] S1, obtain the target position-category vector of the box and the sling in the camera image , build a time series list of target positions , used for coarse positioning of targets;

[0077] S2, time sequence list at target position Based on the time series list containing point cloud information obtained from the laser radar, the target average height time series list of the box and the hanger is obtained. , where m is the number of targets, used to assist in locating targets that may be obscured; , represented as a real number set, represented as a tensor of shape N×m×1, whose elements all belong to the real number set;

[0078] S3, obtain the coordinates, speed and acceleration information of the car , where N is the length of the frame sequence. Through the world coordinate-target position mapping, the world position-target position timing information list of the box and the hanger is obtained. ;

[0079] S4, based on the target position time sequence list , respectively obtain the target vector time series list containing the complete box and the hanger visual features And a time series list of randomly cropped target vectors containing box and spreader edge features ;

[0080] S5, provides an inter-frame target position delay adaptive prediction model, , , , , and The input frame target position delay adaptive prediction model is used to align the timing targets of each frame between different sensors to complete the target position information of the occluded and / or missing boxes and spreaders, and complete the delayed adaptive alignment of the cabin box information.

[0081] In some preferred embodiments, the above S1, obtains the target position-category vector of the box and the hanger in the camera image , build a time series list of target positions , and can further include the following operations:

[0082] S11, providing a single-stage target detection model, which is used to input RGB channel images and output target position-category vectors ;in, = [[current target category, x-axis coordinate of the current target geometric center on the image, y-axis coordinate of the current target geometric center on the image, pixel length of the current target in the x-axis direction on the image, pixel length of the current target in the y-axis direction on the image], ...];

[0083] S12, obtaining a camera image, using the camera image as the input of a single-stage target detection model, outputting the target position of the container body and the spreader and the spreader type through the visual positioning of the single-stage target detection model, and obtaining the corresponding target position-category vector , build a time series list of target positions , used for coarse positioning of occluded targets.

[0084] In some preferred embodiments, the above S2, when in the target position, the sequence list Based on the time series list containing point cloud information obtained from the laser radar, the target average height time series list of the box and the hanger is obtained. , and can further include the following operations:

[0085] S21, provide a calibrated camera internal / external parameters, including: optical center / rotation matrix / translation vector - , the point cloud , projected to the coordinates of the camera image ,get:

[0086] s∙ u v 1 =K∙[R|t]∙ X Y Z 1

[0087] in, is the scale factor, which indicates the ratio from three-dimensional space to two-dimensional image plane;

[0088] S22, target location-category vector The coordinates of the four vertices that contain the polygonal area , , , , construct a quadrilateral boundary range of the X and Y axes set on a two-dimensional plane; for the coordinates of any point in the point cloud , determine whether the arbitrary point is located inside the boundary of the quadrilateral, filter the point cloud data, and only retain valid points within the boundary;

[0089] S23, after processing the abnormal values ​​and outliers in the point cloud, the point cloud is averaged to obtain the target average height time series list of the box and the hanger .

[0090] In some preferred implementations, in the above S23, the outliers and abnormal values ​​in the point cloud are processed using the quartile distance cluster point detection method.

[0091] In some preferred embodiments, in the above S23, an average height calculation method is used to average the values ​​in the point cloud.

[0092] In some preferred embodiments, the above S3 obtains the coordinates, speed and acceleration information of the car. , where N is the length of the frame sequence. Through the world coordinate-target position mapping, the world position-target position timing information list of the box and the hanger is obtained. , and can further include the following operations:

[0093] S31, collect the position of the car through the inertial odometer ,speed and acceleration information; at the same time, the hull type information is obtained and converted to the center coordinates in the ship width direction to obtain the hull information;

[0094] S32, calculate the world coordinates of the hull center through the hull information and the center position of the current camera image field of view , determine whether the current camera image field of view includes the edge of the cabin and the orientation of the field of view on the hull, and obtain the relative position of the center of the current frame camera image and the ship; where the orientation of the field of view on the hull is determined as:

[0095]

[0096] in, The world coordinates converted from the center of the camera image in the current frame; is a preset threshold (which can be an empirical value), which is determined by the camera field of view and preferably takes a value of 15 (the optimal value corresponding to the application scenario);

[0097] S33, when Position is "above", the ranking coordinates Set to 0, if Position is "middle" and "below", then ,in is the number of pixels on the container width side; Time series list of target positions based on the current frame target , sort the targets in the current frame, and then assign , get the ranking coordinates of all targets in a single frame, and construct the target time series list List of ships with target timings for boxes and spreaders ;include:

[0098] From the target position time sequence list , take out the i-th frame, according to the 0th "category information" of the first dimension, the target location information of this frame (i-th frame) includes: all container location information and spreader position information , where q represents the q box targets contained in the frame (if there are no q, then it is supplemented to q (q is set to ≥6), and the supplementary content is to fill the four positions of the horizontal and vertical coordinate list of the center point with -1, that is, [-1, -1, -1, -1], and supplement it to q); Respectively represent the pixel position information of the 1st / 2nd / ... / qth box target center point of the i-th frame, including horizontal and vertical pixel coordinates and pixel length and width; the two are collectively referred to as the target position information of the frame. All target information of the i-th frame consists of: the position list of the target of the i-th frame , sort the targets in the i-th frame, [the sorting method is: first sort in ascending order according to the y coordinate, then cluster according to the y coordinate value of each target, and set the threshold , traverse and calculate the distance between these y, the distance is less than The final result is that these targets can be divided into multiple rows. In each row, these targets are each taken by x coordinates, and then sorted according to the x coordinates in each row. Since then, the targets of the i-th frame have changed from disorder to order. This method can improve the efficiency of target position alignment between frames and more conveniently extract the ranking information of the target in the ship. Next, the y coordinates of each target are compared with Add together to get the ranking of all targets in a single frame (i-th frame), that is, the ranking coordinates of the ships where all boxes are located and the coordinates of the ship where the spreader is located ,in Respectively represent the position coordinates of the ship where the 1st / 2nd / ... / qth box is located in the i-th frame; assuming that the Nth frame is being processed and N-1 frames have been processed before, then and Add to as a tuple , get the ship ranking list containing the target timing of the box and the spreader for all current frames (i.e., the 1st to the Nth frame) , N is the total number of frames that have been processed before processing the frame information (N-1) plus the current frame information being processed; when processing the initial frame, ; Similarly, and After sorting, the same operation is performed as above to obtain , which represents the target time sequence list containing all current frame boxes and spreaders;

[0099] S34, converting the center coordinates in the ship width direction and the position information of the car according to the hull type information and speed information And the target time series list List of ships with target timings for boxes and spreaders , bind the targets in each rank between frames, and get the world position-target position timing information list of the box and the spreader ;include:

[0100] S341, first convert the acquired hull type information to the center coordinates in the ship width direction; at the same time, obtain the position of the car and speed and target time series list for boxes and spreaders List of ships with target timings for boxes and spreaders ;

[0101] S342, then calculate the instantaneous speed, for adjacent time frames , as well as , calculate the acceleration of the car and the box in adjacent frames, then update the positions of adjacent frames and predict the world position of the next frame;

[0102] S343, target binding threshold: compare the difference between the relative position change calculated between adjacent frames and the relative position change measured by the sensor. If it is less than the set threshold, , then the targets are considered consistent and binding is performed, otherwise target binding is abandoned and the update The corresponding target ranking information in is the value "-1", indicating that the position target of the i-th frame does not need to be bound to the ranking; then through , ship width information, box width information, calculate the world coordinates of each row of box targets, and then bind the targets in each frame, the rows of corresponding targets, and the world coordinates to obtain the world position-target position timing information list of the box and the spreader , Describes the position of the target in the i-th frame; whether the target in the i-th frame needs to be bound is based on The target ranking information value in is used for judgment; ,in, are the pixel coordinates of all objects (containers and spreaders) in the i-th frame in the current frame; are the world coordinates of all targets (containers and spreaders) in the i-th frame; are the row coordinates of all targets (containers and spreaders) in the hold in the i-th frame. , q is the target number of boxes mentioned above, 1 is the number of hangers, and 4 is the horizontal and vertical coordinates and length and width of the center point of the target.

[0103] In the above S4, the target position information of each frame and Sort and map the world coordinates to get the ranking information in the ship and , and then package the world coordinate position information of each frame into a tuple or list , and Also packaged as Then calculate the world position of each target in each frame (also a list), and and Packaged .

[0104] In some preferred embodiments, the above S4, based on the target position time sequence list , respectively obtain the target vector time series list containing the complete box and the hanger visual features And a time series list of randomly cropped target vectors containing box and spreader edge features , and can further include the following operations:

[0105] S41, based on the target position time sequence list , obtain multiple target location-category vectors , using the target position-category vector Crop the image target and get the Target image sequence for row boxes , where 1 to N are the frame numbers; the target image sequence Each frame in is used as the input of an autoencoder AutoEncoder-1 to record the key features of the same target in different frames ;

[0106] S42, for the target image sequence Randomly crop the images in to get the target image sequence ; The target image sequence As the input of another autoencoder AutoEncoder-2, it outputs the key features of the same target in different frames , used to locate occluded targets;

[0107] S43, all target positions-category vectors with category as box The target image sequence is respectively encoded by AutoEncoder-1 and AutoEncoder-2. and the target image sequence Compress it into vectors to get the target vector time series list of the box and the hanger And the random clipping target vector time series list , where d is the dimension of the target vector.

[0108] In some preferred implementations, the random cropping in S42, preferably using a classification processing cropping method, may further include the following operations:

[0109] According to the possible preset value of the cabin depth (which can be an empirical value), the box distance is r (r layers of height, r can be 6 in a specific application example), and the height value is divided into r intervals according to the camera field of view and the setting value of the spreader occlusion area (which can be an empirical value); according to the different interval distances, the percentage of the area of ​​the unobstructed part of the box when the spreader is blocked to the box pixel area when it is not blocked by the spreader is counted, and an average is taken as the setting value of the clipping area of ​​the interval. The empirical value involved in this method can be obtained through an expert system.

[0110] In some preferred implementations, the autoencoder AutoEncoder-1 and the autoencoder AutoEncoder-2 have the same architecture, and are both AutoEncoders.

[0111] In some preferred embodiments, the training of the above-mentioned autoencoders AutoEncoder-1 and AutoEncoder-2 are performed using training sets divided from target image A and target image B, respectively. The division method is random sampling. The sampled training sets and test sets are required to have no overlapping contents. The ratio of the former to the latter is 7:3, and the loss function adopts KL divergence.

[0112] In some preferred embodiments, the above S5 provides an inter-frame target position delay adaptive prediction model. , , , , and Input the inter-frame target position delay adaptive prediction model to align the timing targets of each frame between different sensors to complete the target position information of the occluded and / or missing boxes and hoists, and further include the following operations:

[0113] S51, providing an inter-frame target position delay adaptive prediction model, the inter-frame target position delay adaptive prediction model includes a multi-modal feature space alignment network and a temporal neural network; wherein the multi-modal feature space alignment network is used for aligning multi-modal feature space information, and the temporal neural network is used for delay adaptation between different modal information;

[0114] S52, the multimodal feature space alignment network is composed of a fully connected neural network. , , , , and Mapped to the same representation space, the temporal neural network is a spatiotemporal convolutional network, which is modeled using a spatiotemporal convolutional network and integrates temporal visual features. ,average height , spreader and box historical world coordinates As well as the historical positions of the spreader targets, the temporal attention mechanism is finally used to adaptively align the boxes and spreader targets that are delayed and / or no longer appear in each frame due to occlusion:

[0115]

[0116]

[0117]

[0118] in, is the feature representation after integration on time frame t; Represents a fully connected neural network and its pre-processing step, i.e., the concatenation operation; is the fusion feature after time series modeling on time frame t; represents a temporal neural network; Indicates the target detected by the target data of the t-th frame image The time-series position map is aligned to the target average height time-series table of the box and the spreader No. Frame, that is , The numerical interval with t is k, k is given by Output.

[0119] In the above S52, the historical world coordinates of the spreader and the box and the historical position of the spreader target, by and Take out the world coordinates of the corresponding target respectively and target height get.

[0120] In the above S52, for example: The tth frame data is aligned to The t+1th frame data; The t+1th frame data is aligned to The alignment scheme mainly prevents the data frame misalignment problem caused by the inconsistent working frequency of the lidar and the camera.

[0121] The specific application steps include:

[0122] 1. FCN

[0123] The FCN is divided into the following steps: , , , , , ,Then The specific working method is as follows:

[0124] First, and ,as well as Expanded into a two-dimensional vector, we get , and Then use channel splicing to The splicing is performed on the first channel to obtain The process is shown as follows:

[0125]

[0126]

[0127]

[0128]

[0129] Represents the concatenation function, which is used to concatenate all data on the first channel. It means that the information of other channels after the first channel will be expanded in sequence. Indicates that vectors are concatenated in the second dimension.

[0130] Then will Use a fully connected neural network to map to the same representation space and obtain the integrated feature representation. If the data of the tth frame is processed, the obtained feature representation is , the input is expressed as , the process is as follows:

[0131]

[0132] is the embedded representation of the data of the tth frame, including the target coordinates, ranking coordinates, world coordinates, target height information, and visual feature information of the target in a single frame: , . Where V represents the embedding dimension. The final composition E={ E i | i ∈ [1, N]} .

[0133] Next, we need to use the target coordinates, ranking coordinates, world coordinates and target height information of each frame to align, because different sensors have different operating frequencies. STCN and Attn modules are used.

[0134] 2. STCN

[0135] Use STCN to process and obtain the embedded representation of each frame, and obtain the fused features after time series modeling. First, create a Window, and is an even number, where Represented as a set of positive integers. Let the padding size be , and use this fill value to input Filling process to obtain . Then Start using Perform convolution, taking the tth moment as an example:

[0136] Where n= .

[0137] 3. Attn

[0138] Attn is a combination of a series of algorithms, including weight distribution calculation, weight normalization, index offset calculation, and Alignment After obtaining the fused time sequence information, the attention weighted processing is performed on each frame feature, and then the index offset list of the current time point t can be obtained, that is, , then you can The index is added and processed, and to align.

[0139] Then, according to the features at time t, an attention mechanism that decreases from the inside to the outside is adopted, that is, the features farther away from the current frame t have less influence on the current frame. Assuming that this attention distribution conforms to the characteristics of a Gaussian distribution, it focuses on the key features related to the current frame.

[0140] The following formula describes the weight distribution of this attention mechanism. represents the weight at time t, and then starting from time t, it decays in the form of an exponential function to both sides, and finally approaches 0. The weight distribution is expressed as follows:

[0141]

[0142] in, is a constant. λ is the decay rate, which controls the speed at which the weight decreases. It is the central time point. is the current time step. represents the distribution at that time t.

[0143] The weights are then normalized:

[0144]

[0145] The center time point is calculated as t through the weighted attention mechanism, and the time step For the current point , and then constitute :

[0146]

[0147] Subsequent use and Add the elements corresponding to index and get t frame alignment The index of , completing the entire alignment step.

[0148] Based on the same inventive concept, an embodiment of the present invention further provides a cabin box information delay adaptive alignment system, which is applied in the quay crane ship scanning operation process.

[0149] Specifically, Figure 2 As shown, the cabin box information delay adaptive alignment system provided by this embodiment may include: a box hoist target visual detector 1, a target height-position binder 2, a world coordinate-target position mapper 3, a target feature recording module 4 and an inter-frame target position delay adaptive predictor 5; wherein:

[0150] Box hoist target visual detector 1, used to obtain the target position of the box and hoist in the camera image - category vector , build a time series list of target positions ;

[0151] Target height-position binder 2, used for timing lists at target positions Based on the time series list containing point cloud information obtained from the laser radar, the target average height time series list of the box and the hanger is obtained. , where m is the number of targets;

[0152] World coordinate-target position mapper 3, used to obtain the coordinates, speed and acceleration information of the car , where N is the length of the frame sequence. Through the world coordinate-target position mapping, the world position-target position timing information list of the box and the hanger is obtained. ;

[0153] Target feature recording module 4, used to list the target position in time sequence , respectively obtain the target vector time series list containing the complete box and the hanger visual features And a time series list of randomly cropped target vectors containing box and spreader edge features ;

[0154] Inter-frame target position delay adaptive predictor 5, used for input , , , , and , align the timing targets of each frame between different sensors, complete the target position information of the occluded and / or missing boxes and spreaders, and complete the delayed adaptive alignment of the cabin box information.

[0155] The technical solution provided by the above-mentioned embodiment of the present invention is further described in detail below in conjunction with the accompanying drawings. It should be noted that the steps in the method provided by the present invention can be implemented using the corresponding components in the system, and those skilled in the art can refer to the technical solution of the system to implement the step flow of the method, and can also refer to the technical solution of the method to implement the composition of the system, that is, the embodiments in the system and the embodiments in the method can be understood as mutually preferred examples, which will not be described in detail here.

[0156] like Figure 3 As shown, the method and system for adaptively aligning cabin box information delay during the quay bridge ship scanning operation provided by the above embodiment of the present invention first inputs the camera image into the box hoist target visual detector 1 to obtain the target position-category vector , get the target position time sequence list of the box and the hanger , which is used to roughly locate the target. On this basis, the time series list containing point cloud information obtained from the laser radar is added and input into the target height-position binder 2 to obtain the target average height time series list of the box and the spreader , used to assist in locating targets that may be obscured; in addition, the collected images form an image time series list, which is passed to the target feature recording module 4 and outputs the target vector time series list of the box and the hanger Random clipping target vector time series list of the box and the spreader ,The former contains the complete visual features of the box and the sling, and the latter contains the edge features of the box; at the same time, the coordinates, speed, and acceleration information of the car are obtained from the inertial odometer The world coordinate-target position mapper 3 is also passed in to obtain the world position-target position time series information list of the box and the hanger ; Finally, , , , , and The input inter-frame target position delay adaptive predictor 5 aligns the timing targets of each frame between different sensors to complete the occluded / missing box target position information.

[0157] Box crane target visual detector 1 is a single-stage target detection model that inputs RGB channel images and outputs target position-category vectors , = [[current target category, x-axis coordinate of the current target geometric center position on the image, y-axis coordinate of the current target geometric center position on the image, pixel length of the current target in the x-axis direction on the image, pixel length of the current target in the y-axis direction on the image], ...]. The container spreader target vision detector 1 is used to visually locate the target position of the container and the spreader. The spreader type target is used to locate the obscured target.

[0158] The target height-position binder 2 includes a point cloud projector, a point cloud filter, an outlier processing module and an average height calculation module, which are used to estimate the distance between the target and the camera and determine the height value of the target.

[0159] Point cloud projector. Uses calibrated camera intrinsic / extrinsic parameters: optical center / rotation matrix / translation vector . , projected to the coordinates of the image acquired by the camera .

[0160] s∙ u v 1 =K∙[R|t]∙ X Y Z 1

[0161] Point cloud filter. First input the frame point cloud and the corresponding image, and (target position-category vector), by setting the boundary range of X and Y axes on the two-dimensional plane, the point cloud data is filtered and only valid points within the boundary range are retained. Contains the coordinates of the 4 vertices of the polygon area , , , , construct a quadrilateral boundary. For the coordinates of any point in the point cloud , determine whether it is inside the quadrilateral.

[0162] Outlier processing module: This module uses the quartile distance cluster point detection method to process outliers and outliers. This method avoids the influence of extreme values ​​and has strong robustness.

[0163] Average height calculation module. After processing abnormal values ​​and outliers, the points in the area are averaged.

[0164] The world coordinate-target position mapper 3 includes three parts: a camera / image world position coordinate perception module, a single-frame target position to world coordinate mapping module, and an inter-frame target alignment module. The camera / image world position coordinate perception module is used for the center field of view ranking and the relative position of the camera and the ship, and the single-frame target position to world coordinate mapping module is used for mapping the targets of the box and the spreader to the world coordinate and the ranking of the ship.

[0165] First, the acceleration and speed of the car, the position of the car, and the hull type information collected by the inertial odometer are input and converted to the center coordinates in the width direction of the ship, as well as the target time series list of the box and the sling, and then input into the camera / image world position coordinate sensor module to output the relative position of the current frame camera / image center and the ship.

[0166] Camera / image world position coordinate perception module: Calculate the world coordinates of the hull center through the hull information (hull center coordinates and hull type) and the current camera field of view image center position , determine whether the current field of view includes the side of the ship (the edge of the cabin), the direction of the hull, etc. The directions include "above", "middle" and "below". Direction determination method:

[0167]

[0168] in The world coordinates converted from the center of the current frame image; It is an empirical value, determined by the camera field of view, and is set to 15 in a preferred embodiment of the present invention.

[0169] Single frame target position to world coordinate mapping module: After getting the output of the camera / image world position coordinate sensor, input it into this module. If Position is "above", the world coordinate is ranked Set to 0, if it is "middle" and "below", then Then the target list of this frame / this frame output by the target visual detector 1 is ranked and assigned , get the world coordinates of all targets in a single frame, including container location information , Lifting device position information After the world coordinates of each frame target are mapped, the target time sequence table 1 of the box and the hanger is obtained.

[0170] Inter-frame target alignment module: After obtaining the target time sequence list 1 of the box and the spreader, convert it to the width center coordinates of the ship and the position of the trolley according to the hull type information and speed , and the target time series list 1 of the box and the spreader, bind the targets between frames. First, according to the acquired hull type information, convert it to the width center coordinates of the ship; at the same time, obtain the position and speed of the car and , and the target timing table of the box and the spreader 1. Then calculate the instantaneous speed, for adjacent time frames , as well as , calculate the acceleration of the car and the box in adjacent frames. Then update the position of the adjacent frames and predict the world position of the next frame. Target binding threshold: compare the difference between the relative position change calculated between adjacent frames and the relative position change measured by the sensor. If it is less than a certain threshold, , the targets are considered consistent and binding is performed, otherwise target binding is abandoned.

[0171] After the above three modules are processed, the world position timing information list of the box and the hanger is finally output.

[0172] like Figure 4 As shown, the target feature recording module 4 includes AutoEncoder-1 and AutoEncoder-2. First, the output of the visual detector 1 is used Crop the image target to get the target image of the i-th row of boxes , 1 to n are the number of frames, each frame is used as the input of AutoEncoder-1, recording the key features of the same target in different frames Then randomly crop the images in list A to get , as input to AutoEncoder-2, output Used to locate occluded objects.

[0173] The predicted box target information is compressed into vectors by using AutoEncoder-1 and AutoEncoder-2 to obtain the time series list of the target vectors of the box and the hoist. Random clipping target vector time series list of the box and the spreader The architecture of AutoEncoder-1 and AutoEncoder-2 is the same, both are AutoEncoder.

[0174] The training of AutoEncoder-1 and AutoEncoder-2 is performed using the "training sets" divided by A and B respectively. The division method is random sampling. The sampled "training set" and "test set" are required to have non-overlapping contents. The ratio of the former to the latter is 7:3. The loss function uses KL divergence.

[0175] The random cropping method is to process and crop by classification. According to the possible empirical values ​​of the cabin depth, there are 6 box distances (roughly 6 layers of height), and the height values ​​are divided into 6 intervals according to the camera field of view and the empirical values ​​of the sling occlusion area. According to the different interval distances, the percentage of the area of ​​the unobstructed part of the box when the sling is blocked to the box pixel area when it is not blocked by the sling is counted, and an average is taken as the empirical value of the cropping area in this interval.

[0176] like Figure 5 As shown, the inter-frame target position delay adaptive predictor 5 includes a multi-modal feature space alignment network module and a temporal neural network module. The former is used for aligning multi-modal feature space information, and the latter is used for delay adaptation between different modal information.

[0177] Combination , , , and First, a fully connected neural network is used to map them to the same representation space, and then a spatiotemporal convolutional network is used to model and fuse temporal visual features. ,average height And the historical world coordinates of the spreader and box , the historical position of the hoist target, and finally use the temporal attention mechanism to adaptively align the box targets that are delayed / no longer appear in each frame due to occlusion.

[0178]

[0179]

[0180]

[0181] in, is the feature representation after integration over time frame t. It is the fusion feature after time series modeling on time frame t. The tth frame data is aligned with the frame.

[0182] Figure 6 is a schematic diagram of a cabin in a specific application example of the present invention, Figure 7FIG. 1 is a schematic diagram of a box being blocked during an operation in a specific application example of the present invention. By using the method and system for adaptively aligning cabin box information delay provided by the above embodiment of the present invention, during the quay bridge ship scanning operation, the following information can be obtained: Figure 8 The effect shown. Figure 8 A cargo ship can be seen in the image, with multiple cargo boxes placed in the center of the field of view. Most of the cargo boxes are framed by gray-white visualization boxes, which are the visualization results of the predicted location of box pixels on the image output by the "Box-Spreader Visual Object Detector". From the box, we can see: “row:0, id: 0, D20f, rolley_pos: 70.03, xy: (1113, 376), (2401, 593), average_distance: 51.43576...”. These parameters are explained as follows: “row: 0” means that the box is in the 0th row of the cabin; “id: 0” means that the target is the 0th predicted target; “D20f” means that the category of the box target is a double 20-foot box; “rolley_pos” indicates the current world position (y-axis direction) of the inertial odometer; “xy: (1113, 376),(2401, 593)” indicates the pixel coordinates of the upper left and lower right corners of the prediction box for the box; “average_distance” is the height of the target box measured by the lidar. From the information display box in the upper left corner, you can see the output results after the display algorithm process is completed, including: "The 3, 7 rows are obstructed" means that the 3rd and 7th rows are obstructed; the 3rd row of boxes is obstructed by the spreader, and the 7th row does not appear in the picture due to FOV reasons, so it is classified as "obstructed"; "Trolley_pos_status: top" is the "top" status output by S32, Position, where "top" means that the center of the lens is above the center of the ship; "First_box_offset: 0" is also the post-processing result after the output of the S32 Position step; "the row is in progress: 3" means that the 3rd row (counting from row 0) is working because the spreader is detected in the 3rd row.

[0183] An embodiment of the present invention further provides a computer terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the processor can be used to execute any one of the above methods of the present invention, or to run any one of the above systems of the present invention.

[0184] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random access memory (English: random-access memory, abbreviated: RAM), such as static random access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above method), computer instructions, etc., and the above computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. And the above computer programs, computer instructions, data, etc. can be called by the processor.

[0185] The above-mentioned computer programs, computer instructions, etc. may be stored in one or more memories in partitions, and the above-mentioned computer programs, computer instructions, data, etc. may be called by a processor.

[0186] The processor is used to execute the computer program stored in the memory to implement the various steps of the method or various modules of the system involved in the above embodiments. For details, please refer to the relevant descriptions in the above method and system embodiments.

[0187] The processor and the memory may be independent structures or integrated structures. When the processor and the memory are independent structures, the memory and the processor may be coupled and connected via a bus.

[0188] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it can be used to execute any of the above methods of the present invention, or to run any of the above systems of the present invention.

[0189] Among them, computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general or special-purpose computer. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a user device. Of course, the processor and the storage medium can also be present in a communication device as discrete components.

[0190] The method and system for adaptively aligning cabin box information delay provided by the above-mentioned embodiment of the present invention are applied to the process of quay crane ship scanning operation. By simulating and aligning the features of the box when it is blocked, and using the target position of the hoist to assist in positioning, the problem that the existing target detection method cannot detect the target when it is blocked in the quay crane hoist operation scene is solved. By adding the distance, position, speed, and acceleration information of the laser radar and inertial odometer for fusion, an inter-frame target position delay adaptive predictor is designed to align the target information that may be delayed, and add deterministic information to improve the accuracy of target positioning.

[0191] All matters not covered in the above embodiments of the present invention are well known in the art.

[0192] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A method for adaptively aligning cabin information delay, characterized in that: include: Get the target position of the box and the hanger in the camera image - the category vector , build a time series list of target positions , used for coarse positioning of targets; At the target location time sequence list Based on the time series list containing point cloud information obtained from the laser radar, the target average height time series list of the box and the spreader is obtained. , where m is the number of targets, which is used to assist in locating targets that may be obscured; Get the coordinates, speed and acceleration information of the car , where N is the length of the frame sequence. Through the world coordinate-target position mapping, the world position-target position timing information list of the box and the hanger is obtained. ; Based on the target position time sequence list , respectively obtain the target vector time series list containing the complete box and the hanger visual features And a time series list of randomly cropped target vectors containing box and spreader edge features ; Provide an inter-frame target position delay adaptive prediction model, , , , , and The inter-frame target position delay adaptive prediction model is input to align the timing targets of each frame between different sensors to complete the target position information of the obscured and / or missing boxes and spreaders, and complete the delayed adaptive alignment of the cabin box information.

2. The method for adaptively aligning cabin information delay according to claim 1 is characterized in that: The target position-category vector of the box and the hanger in the camera image is obtained , build a time series list of target positions ,include: Provides a single-stage target detection model that takes an RGB channel image as input and outputs a target position-category vector ;in, = [[current target category, x-axis coordinate of the current target geometric center on the image, y-axis coordinate of the current target geometric center on the image, pixel length of the current target in the x-axis direction on the image, pixel length of the current target in the y-axis direction on the image], ...]; Obtain a camera image, use the camera image as the input of the single-stage target detection model, output the target position of the container body and the spreader and the spreader type through the visual positioning of the single-stage target detection model, and obtain the corresponding target position-category vector , build a time series list of target positions , used for coarse positioning of occluded targets.

3. The method for adaptively aligning cabin information delay according to claim 1 is characterized in that: The sequence table at the target location Based on the time series list containing point cloud information obtained from the laser radar, the target average height time series list of the box and the hanger is obtained. ,include: Provides a pre-calibrated camera intrinsic / extrinsic parameters, including: optical center / rotation matrix / translation vector - , the point cloud , projected to the coordinates of the camera image ,get: ; in, is the scale factor, which indicates the ratio from three-dimensional space to two-dimensional image plane; The target position-category vector The coordinates of the four vertices that contain the polygonal area , , , , construct a quadrilateral boundary range of the X and Y axes set on a two-dimensional plane; for the coordinates of any point in the point cloud , determine whether the arbitrary point is located inside the boundary of the quadrilateral, filter the point cloud data, and only retain valid points within the boundary; After processing the abnormal values ​​and outliers in the point cloud, the point cloud is averaged to obtain the target average height time series list of the box and the hanger .

4. The method for adaptively aligning cabin information delay according to claim 1, characterized in that: The coordinates, speed and acceleration information of the car are obtained , where N is the length of the frame sequence. Through the world coordinate-target position mapping, the world position-target position timing information list of the box and the hanger is obtained. ,include: The position of the car is collected through the inertial odometer ,speed and acceleration information, and at the same time obtain the hull type information and convert it to the center coordinates in the ship width direction to obtain the hull information; The world coordinates of the center of the hull are calculated by using the hull information and the center position of the current camera image field of view. , determine whether the current camera image field of view includes the edge of the cabin and the orientation of the field of view on the hull, and obtain the relative position of the center of the current frame camera image and the ship; where the orientation of the field of view on the hull is determined as: ; in, The world coordinates converted from the center of the camera image in the current frame; is the preset threshold; When Position is "above", the ranking coordinates Set to 0, if Position is "middle" and "below", then ,in is the number of pixels on the width side of the container; Target position timing list based on current frame target , sort the targets in the current frame, and then assign , get the ranking coordinates of all targets in a single frame, and construct the target time series list List of ships with target timings for boxes and spreaders ; According to the hull type information, it is converted into the center coordinates in the ship width direction and the position information of the car. and speed information And the target time series list List of ships with target timings for boxes and spreaders , bind the targets in each rank between frames, and get the world position-target position timing information list of the box and the spreader .

5. The method for adaptively aligning cabin information delay according to claim 4 is characterized in that: The target position time sequence list based on the current frame target , sort the targets in the current frame, and then assign , get the ranking coordinates of all targets in a single frame, and construct the target time series list List of ships with target timings for boxes and spreaders ,include: From the target position time sequence list , take out the i-th frame, according to the 0th "category information" of the first dimension, the target location information of the i-th frame includes: all container location information and spreader position information , where q represents the q box targets contained in the i-th frame, Respectively represent the pixel position information of the center point of the 1st / 2nd / ... / qth box target in the i-th frame; using the position time sequence list of the target in the i-th frame , sort the targets in the i-th frame, where the sorting method is: first sort in ascending order according to the y coordinate, then cluster according to the y coordinate value of each target, and set the threshold , traverse and calculate the distance between these y, the distance is less than The final result is that these targets can be divided into multiple rows. In each row, these targets are each taken by x coordinates, and then sorted according to the x coordinates in each row. From then on, the targets of the i-th frame are changed from disordered to ordered; next, the y coordinates of each target are compared with Add them together to get the ranking of all targets in a single frame, that is, the ranking coordinates of the ships where all boxes are located and the coordinates of the ship where the spreader is located , They represent the coordinates of the ship where the 1st / 2nd / ... / qth box is located in the i-th frame respectively; suppose the N-th frame is being processed and the N-1 frames have been processed before, then and Add to as a tuple , get the ship ranking list containing the target timing of all current frames, i.e., the 1st to the Nth frame boxes and spreaders , N is the total number of frames that have been processed before processing the i-th frame information (N-1) plus the current frame information being processed; when processing the initial frame, ; Similarly, and After sorting, the same operation is performed as above to obtain , which represents the target time sequence list containing all current frame boxes and spreaders; The center coordinates of the ship width direction and the position information of the car are converted according to the hull type information. and speed information And the target time series list List of ships with target timings for boxes and spreaders , bind the targets of each rank between frames, including: According to the obtained hull type information, convert it to the center coordinates in the width direction of the ship; at the same time, obtain the position of the car and speed and target time series list for boxes and spreaders List of ships with target timings for boxes and spreaders ; Calculate instantaneous velocity for adjacent time frames , as well as , calculate the acceleration of the car and the box in adjacent frames, then update the positions of adjacent frames and predict the world position of the next frame; Compare the difference between the relative position change calculated between adjacent frames and the relative position change measured by the sensor. If it is less than the set threshold, , then the targets are considered consistent and binding is performed, otherwise target binding is abandoned and the update The corresponding target ranking information in is the value "-1", indicating that the position target of the i-th frame does not need to be bound to the ranking; then through , ship width information, box width information, calculate the world coordinates of each row of box targets, and then bind the targets in each frame, the rows of corresponding targets, and the world coordinates to obtain the world position-target position timing information list of the box and the spreader , Describes the position of the target in the i-th frame; whether the target in the i-th frame needs to be bound is based on The target ranking information value in is used for judgment; ,in is the world coordinate of the i-th frame.

6. The method for adaptively aligning cabin information delay according to claim 1, characterized in that: The time sequence list based on the target position , respectively obtain the target vector time series list containing the complete box and the hanger visual features And a time series list of randomly cropped target vectors containing box and spreader edge features ,include: Based on the target position time sequence list , obtain multiple target location-category vectors , using the target position-category vector Crop the image target and get the Target image sequence for row boxes , where 1 to N are the frame numbers; the target image sequence Each frame in is used as the input of an autoencoder AutoEncoder-1 to record the key features of the same target in different frames ; For the target image sequence Randomly crop the images in to get the target image sequence ; The target image sequence As the input of another autoencoder AutoEncoder-2, it outputs the key features of the same target in different frames , used to locate occluded targets; All target positions-category vectors whose categories are boxes The target image sequence is respectively encoded by AutoEncoder-1 and AutoEncoder-2. and the target image sequence Compress it into vectors to get the target vector time series list of the box and the hanger And the random clipping target vector time series list , where d is the dimension of the target vector.

7. The method for adaptively aligning cabin information delay according to claim 1, characterized in that: The method provides an inter-frame target position delay adaptive prediction model, and , , , , and The inter-frame target position delay adaptive prediction model is input to align the timing targets of each frame between different sensors to complete the target position information of the obscured and / or missing boxes and hoists, including: Provided is an inter-frame target position delay adaptive prediction model, the inter-frame target position delay adaptive prediction model includes a multi-modal feature space alignment network and a temporal neural network; wherein the multi-modal feature space alignment network is used for aligning multi-modal feature space information, and the temporal neural network is used for delay adaptation between different modal information; The multimodal feature space alignment network is composed of a fully connected neural network. , , , , and Mapped to the same representation space, the temporal neural network is a spatiotemporal convolutional network, which is modeled using a spatiotemporal convolutional network and integrates temporal visual features. ,average height , spreader and box historical world coordinates As well as the historical positions of the spreader targets, the temporal attention mechanism is finally used to adaptively align the boxes and spreader targets that are delayed and / or no longer appear in each frame due to occlusion: ; in, is the feature representation after integration on time frame t; Represents a fully connected neural network and its pre-processing step, i.e., the concatenation operation; is the fusion feature after time series modeling on time frame t; represents a temporal neural network; Indicates aligning the tth frame data to the frame.

8. A cabin box information delay adaptive alignment system, characterized in that: include: Box hoist target visual detector, target height-position binder, world coordinate-target position mapper, target feature recording module and inter-frame target position delay adaptive predictor; wherein: The box sling target visual detector is used to obtain the target position-category vector of the box and sling in the camera image. , build a time series list of target positions ; The target height-position binder is used to list the time sequence at the target position. Based on the time series list containing point cloud information obtained from the laser radar, the target average height time series list of the box and the hanger is obtained. , where m is the number of targets; The world coordinate-target position mapper is used to obtain the coordinate, speed and acceleration information of the car , where N is the length of the frame sequence. Through the world coordinate-target position mapping, the world position-target position timing information list of the box and the hanger is obtained. ; The target feature recording module is used to record the target position based on the time sequence list , respectively obtain the target vector time series list containing the complete box and the hanger visual features And a time series list of randomly cropped target vectors containing box and spreader edge features ; The inter-frame target position delay adaptive predictor is used to input the , , , , and , align the timing targets of each frame between different sensors, complete the target position information of the occluded and / or missing boxes and spreaders, and complete the delayed adaptive alignment of the cabin box information.

9. A computer terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it can be used to perform the method described in any one of claims 1 to 7, or to run the system described in claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it can be used to perform the method described in any one of claims 1 to 7, or to run the system described in claim 8.

Images