A fastener assembly prediction method and system based on multi-modal visual fusion

Abstract

The application discloses a fastener assembly prediction method and system based on multi-modal visual fusion, and relates to the technical field of intelligent manufacturing assembly. The method comprises the following steps: acquiring multi-modal power battery pack fastener assembly visual data, and preprocessing the power battery pack fastener assembly visual data; the power battery pack fastener assembly visual data comprises visible light image data, point cloud data and infrared image data; fastener recognition and positioning are performed based on a target detection network by using the visible light image data to obtain preliminary recognition features; the multi-modal power battery pack fastener assembly visual data is fused; a motion compensation network based on perception fusion is used to correct the preliminary recognition features according to the fused features, and pose prediction is performed according to the actual fastener position after correction; and compensation calculation is performed in combination with a preset fastener processing sequence. The application can realize accurate dynamic positioning and error-proof assembly of power battery pack bolts in a mobile workpiece complex scene.

Description

Technical Field

[0001] This invention relates to the field of intelligent manufacturing assembly technology, and in particular to a fastener assembly prediction method and system based on multimodal vision fusion. Background Technology

[0002] In modern continuous production lines, especially in scenarios with stringent requirements for assembly sequence and precision, such as power battery packs, precise fastener assembly is crucial. However, existing technologies primarily address screw tightening under static or teach-path conditions. Specifically, this includes the following shortcomings: Some solutions rely on visual positioning of the workpiece when it is stationary and do not have a dynamic pose compensation mechanism.

[0003] Some solutions focus on the tightening and positioning of individual screws, without addressing the logical control of the tightening sequence between multiple fasteners. In scenarios such as battery pack module fastening, an incorrect tightening sequence can lead to uneven structural stress distribution, causing serious quality issues such as seal failure or structural damage.

[0004] In complex industrial environments, the success rate of single-modal recognition drops significantly. Existing technologies lack redundancy and complementarity in multimodal sensing, resulting in insufficient system stability. Furthermore, some solutions do not consider the issue of coordinated control of calibration errors among multiple cameras and sensors. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the purpose of this invention is to provide a fastener assembly prediction method and system based on multimodal vision fusion, which can achieve accurate dynamic positioning and error-proof assembly of power battery pack bolts in complex scenarios with moving workpieces.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solution: The first aspect of this invention provides a fastener assembly prediction method based on multimodal visual fusion, comprising the following steps: The assembly visual data of the fasteners of the power battery pack is acquired in multiple modes and preprocessed. The assembly visual data of the fasteners of the power battery pack includes visible light image data, point cloud data and infrared image data. Fastener identification and localization are performed using visible light image data based on a target detection network to obtain preliminary identification features; The visual data of fastener assembly of multimodal power battery packs are fused together. A motion compensation network based on perception fusion is used to correct the initial identification features according to the fused features, and pose prediction is performed according to the corrected actual fastener position. At the same time, compensation calculation is performed in combination with the preset fastener processing order.

[0007] Furthermore, preprocessing includes data alignment, data cleaning, and data unification operations on visible light image data, point cloud data, and infrared image data, respectively.

[0008] Furthermore, before acquiring visual data of multimodal power battery pack fastener assembly, the data acquisition equipment, assembly tools, and fasteners are calibrated. The data acquisition equipment includes a 3D camera, an infrared camera, and a visible light camera. Visible light image data is acquired through the visible light camera, point cloud data is acquired through the 3D camera, and infrared image data is acquired through the infrared camera.

[0009] Furthermore, the specific steps for fastener identification and localization using visible light image data based on a target detection network are as follows: Multi-scale feature extraction and fusion are performed on visible light image data to obtain dense candidate boxes and the correlation information between category and confidence. The dense candidate boxes are decoded, the relative offsets are converted into the box position and size in the image pixel coordinates, and the final confidence score of each box is calculated. Low-scoring candidate boxes are filtered out based on a pre-set confidence threshold, while high-probability candidate boxes are retained. Non-maximum suppression is applied to the retained candidate boxes to eliminate overlapping boxes, resulting in a unique list of two-dimensional bounding boxes for fasteners. The coordinates, width, height, and visible light confidence level of the fastener's center point are calculated based on the bounding boxes in the two-dimensional bounding box list to form preliminary identification features.

[0010] Furthermore, the specific steps for fusing the multimodal power battery pack fastener assembly visual data are as follows: Spatiotemporal alignment is performed on visible light image data, point cloud data, and infrared image data at the same moment with timestamps and credibility masks attached. The aligned visible light image data, point cloud data, and infrared image data are dimensionally unified and then stitched together to obtain a stitched sequence. By introducing learnable modal confidence weights, a Transformer-style cross-attention network is used to autonomously suppress failed modalities in the spliced ​​sequence, resulting in fused features.

[0011] Furthermore, the specific steps for using a motion compensation network based on perceptual fusion to correct the initially identified features based on the fused features, and then predicting the pose based on the corrected actual fastener position, are as follows: Construct a motion compensation network, which includes a correction subnetwork and a compensation subnetwork connected in sequence; The correction subnetwork uses the initial visible light recognition features as the query and the fused features as the key. It performs correction processing through two layers of cross-attention regression to obtain the corrected fastener 2D frame, 3D center coordinates and joint confidence of the three-modal data. The corrected 3D center coordinates are transformed from the camera coordinate system to the robot base coordinate system to obtain the actual fastener position. Calculate the rigid motion vector of the conveyor belt and fasteners based on the motion state of the conveyor belt, and perform consistency verification. Starting from the actual fastener position, the predicted pose is calculated using the rigid motion vectors of the conveyor belt and the fastener through a quadratic extrapolation method.

[0012] A second aspect of the present invention provides a fastener assembly prediction system based on multimodal vision fusion, comprising: The data acquisition module is configured to acquire multimodal visual data of power battery pack fastener assembly and preprocess the visual data of power battery pack fastener assembly, which includes visible light image data, point cloud data and infrared image data. The preliminary feature recognition module is configured to use visible light image data to identify and locate fasteners based on a target detection network, thereby obtaining preliminary recognition features. The position prediction module is configured to fuse multimodal power battery pack fastener assembly visual data, use a motion compensation network based on perception fusion to correct the initial identification features based on the fused features, and perform pose prediction based on the corrected actual fastener position. At the same time, it performs compensation calculations in conjunction with the preset fastener processing order.

[0013] A third aspect of the present invention provides a computer-readable storage medium storing a computer program adapted to be loaded by a processor and to execute steps in the fastener assembly prediction method based on multimodal vision fusion as described in the first aspect of the present invention.

[0014] A fourth aspect of the present invention provides a computer device comprising: A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the fastener assembly prediction method based on multimodal visual fusion as described in the first aspect of the present invention.

[0015] A fifth aspect of the present invention provides a computer program product or computer program comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps in the fastener assembly prediction method based on multimodal vision fusion as described in the first aspect of the present invention.

[0016] The above one or more technical solutions have the following beneficial effects: This invention discloses a fastener assembly prediction method and system based on multimodal visual fusion. Through redundant perception using visible light, infrared, and point cloud modes, it automatically reduces the weight of any modality affected by changes in illumination, oil contamination, or reflections, and activates other modalities for complementarity, thus maintaining stable recognition results and avoiding overall positioning failure caused by the failure of a single modality. Furthermore, this invention incorporates a dynamic pose compensation mechanism for continuous workpiece movement scenarios. It immediately calculates the rigid motion vector of the conveyor belt at the moment of image acquisition and performs reverse correction for communication delays and mechanical response lags, enabling the robot to synchronously obtain the compensated target coordinates at the instant of exposure. This avoids coordinate failure problems caused by the static assumption, achieving real-time tracking and precise alignment throughout the entire process.

[0017] This invention employs a joint calibration and error coordination control method to uniformly optimize the extrinsic parameters of multiple cameras and sensors. It can automatically recalculate the extrinsic parameter matrix during each vehicle model change or maintenance cycle and compress and compensate for accumulated errors, preventing errors from amplifying step by step between units, thereby continuously improving the overall positioning accuracy.

[0018] Beyond the positioning level, this invention introduces a preset fastener processing sequence matrix and embeds sequence verification logic in the motion compensation network. The predicted pose is only output when the current fastener number matches the process requirements; otherwise, the enable is locked and an alarm is triggered, fundamentally eliminating the risk of uneven structural stress, sealing failure, or structural damage caused by sequence errors.

[0019] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart of the fastener assembly prediction method based on multimodal visual fusion in Embodiment 1 of the present invention. Detailed Implementation

[0022] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0023] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof. The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0024] Example 1: Embodiment 1 of this invention provides a fastener assembly prediction method based on multimodal vision fusion. The fastener is described in detail using bolts as an example. Figure 1 As shown, it includes the following steps: S1: Acquire multimodal visual data of power battery pack fastener assembly and preprocess the visual data of power battery pack fastener assembly, which includes visible light image data, point cloud data and infrared image data.

[0025] S1.1: Perform calibration.

[0026] In one specific implementation, before acquiring the multimodal power battery pack fastener assembly visual data, the data acquisition equipment, assembly tools, and fasteners are calibrated. The data acquisition equipment includes a 3D camera, an infrared camera, and a visible light camera.

[0027] S1.1.1: Intrinsic parameter calibration. In this embodiment, multi-view shooting with a planar target is used to obtain the camera intrinsic parameter matrix by minimizing the reprojection error.

[0028] S1.1.2: External parameter calibration. The external pose transformation from the camera to the robot's base coordinates is obtained using hand-eye calibration. Then, the direction of the conveyor belt movement is calibrated, and the conveyor belt coordinate system is established.

[0029] S1.2: Acquire visual data of multimodal power battery pack fastener assembly.

[0030] In one specific implementation, visible light image data I, infrared image data T, and point cloud data P are acquired synchronously, and timestamps are recorded; the time difference between the three frames is controlled within a preset range. Visible light image data is acquired through a visible light camera, point cloud data is acquired through a 3D camera, and infrared image data is acquired through an infrared camera.

[0031] S1.3: Preprocess the visual data of the power battery pack fastener assembly.

[0032] In one specific implementation, preprocessing includes data alignment, data cleaning, and data unification operations on visible light image data, point cloud data, and infrared image data, respectively.

[0033] Specifically, the point cloud data is projected into a depth map, ensuring a one-to-one correspondence between pixels and visible light image data. Infrared image data is aligned to visible light pixels using homography, achieving spatial alignment. Then, the visible light image data is normalized, the infrared image data undergoes temperature truncation and outlier filtering, and the point cloud data is removed from outliers. Finally, the visible light image data, point cloud data, and infrared image data are scaled to the same resolution to construct a five-channel tensor for RGB, temperature, and depth.

[0034] S2: Fastener identification and localization are performed using visible light image data based on a target detection network to obtain preliminary identification features.

[0035] S2.1: Perform multi-scale feature extraction and fusion on visible light image data to obtain dense candidate boxes and the correlation information between category and confidence.

[0036] In one specific implementation, this embodiment uses preprocessed visible light image data as input, and forms a five-level feature map through a convolutional stack with a stride of 2. , The hierarchical index represents the feature level, corresponding to downsampling rates of 2, 4, 8, 16, and 32. Residual connections and channel attention are introduced at each level, with the number of output channels always being C. The top-down path uses nearest-neighbor upsampling to propagate deep semantics to shallower layers, and lateral connections use 1×1 convolutions to compress channels, forming fused multi-scale features. .

[0037] S2.2: Decode the dense candidate boxes, convert the relative offsets into box positions and sizes in image pixel coordinates, and calculate the final confidence score for each box.

[0038] In one specific implementation, for Apply a rotating frame detection head. The classification branch outputs a single-channel confidence plot. , and These represent the height and width of level l, respectively. The regression branch outputs a five-channel offset plot. Including relative offset of grid center Logarithmic offset of width and height Angular deviation After activation and exponential operations, pixel-level predictions are obtained: , , , , .

[0039] in, The current layer step size, This indicates the pixel coordinates of the center of the rotated bounding box on the input image. and These represent the pixel width and pixel height of the rotating frame, respectively. This indicates the rotation angle of the longer side of the frame relative to the horizontal axis of the image. , , They are respectively , and The predefined anchor points. After decoding, the final confidence score for each box is obtained. .

[0040] S2.3: Filter out low-scoring candidate boxes based on a pre-set confidence threshold, and retain high-probability candidate boxes.

[0041] In one specific implementation, pixel-by-pixel comparison With global threshold ,reserve The candidate box set is obtained by performing activation and exponential operations on the candidate box set and then performing pixel-level prediction to obtain the final candidate box set. .

[0042] S2.4: Perform non-maximum suppression on the retained candidate boxes to eliminate overlapping boxes and obtain a unique list of two-dimensional bounding boxes for fasteners.

[0043] In one specific implementation, Sort by confidence level in descending order; iteratively select the highest-scoring box b, and calculate the scores of the remaining boxes. The ratio of the intersection area to the union area (IoU) of the rotated frames, discarding Repeat this process until no one remains, to obtain a unique list of boxes. ,in The threshold for overlap set for the nonmaximum suppression process.

[0044] S2.5: Calculate the coordinates of the fastener's center point, width, height, and visible light confidence level based on the bounding boxes in the two-dimensional bounding box list to form preliminary identification features.

[0045] In one specific implementation, for each frame Take the center pixel , Index the bounding box, read the synchronous depth map, and obtain the depth map. After inverse projection and Transformation to obtain the robot's base coordinate system in three dimensions , This is a rigid extrinsic homogeneous transformation matrix (4×4) from the camera coordinate system to the robot base coordinate system. It fully describes rotation and translation and is used to map camera measurements to the robot's motion space. and Preliminary characteristics , Indicates the first The pixel width of each candidate rotation box Indicates the first The pixel height of each candidate rotation box. Indicates the first The rotation angle of the long side of each candidate rotation box relative to the horizontal axis of the image.

[0046] S3: The visual data of the assembly of fasteners for the multimodal power battery pack are fused together. The motion compensation network based on perception fusion is used to correct the initial identification features according to the fused features, and the pose is predicted according to the actual position of the fastener after correction. At the same time, compensation calculation is performed in combination with the preset fastener processing order.

[0047] S3.1: Fusion of visual data on the assembly of fasteners for multimodal power battery packs.

[0048] S3.1.1: Perform spatiotemporal alignment on visible light image data, point cloud data, and infrared image data at the same moment with timestamps and credibility masks.

[0049] S3.1.2: After unifying the dimensions of the aligned visible light image data, point cloud data, and infrared image data, the spliced ​​sequence is obtained.

[0050] In one specific implementation, the three modal data are each reduced to a unified channel through 1×1 convolution. ,get ,in The features of visible light image data, infrared image data, and point cloud data are unified in dimensionality, and the three features are concatenated along the channels to obtain a spliced ​​sequence. .

[0051] S3.1.3: Introduce learnable modal confidence weights and use a Transformer-style cross-attention network to autonomously suppress failed modalities in the spliced ​​sequence to obtain fused features.

[0052] In one specific implementation, for Global average pooling is used to obtain modal tokens, which are then processed by a fully connected layer and Softmax to generate weight vectors. ;by For coefficient pairs Perform weighted summation and output the fused features. .

[0053] S3.2: Use a motion compensation network based on perception fusion to correct the preliminary identification features according to the fused features, and predict the pose based on the corrected actual fastener position.

[0054] S3.2.1: Construct a motion compensation network, which includes a correction subnetwork and a compensation subnetwork connected in sequence.

[0055] In one specific implementation, the correction subnetwork is characterized by preliminary features. For querying, feature fusion As the key value, the offsets of the two layers of cross-attention regression are set to... and , frame geometric offset ,in, All of these were acquired through learning. This is the pixel-level center correction amount during the correction process. To correct the width and height offset of the initial decoding frame during the calibration process, This is a confidence correction factor used to adjust the confidence level originally obtained solely from visible light, ensuring that the fused confidence level is corrected. It better reflects the true reliability of trimodal consistency, based on angular offset. right Make corrections to obtain The corrected 3D center can be obtained through the correction subnetwork. With conveyor belt speed The compensation subnetwork takes the corrected 3D center and conveyor belt speed as input and outputs the predicted pose. Among them, the compensation subnetwork... The spliced ​​result is the delay compensation result in the x-direction output after passing through two layers of FC. y, z Direct pass-through to obtain the predicted pose .

[0056] S3.2.2: The correction subnetwork uses the preliminary visible light recognition features as the query and the fused features as the key value. It performs correction processing through two layers of cross-attention regression to obtain the corrected fastener 2D frame, 3D center coordinates and joint confidence of the three-modal data.

[0057] S3.2.3: Transform the corrected 3D center coordinates from the camera coordinate system to the robot base coordinate system to obtain the actual fastener position.

[0058] In one specific implementation, the joint confidence level is calculated. : .

[0059] in, For the Sigmoid function, It is the inverse Sigmoid function.

[0060] Center the camera system in three dimensions through Transform to the robot's base coordinate system to obtain the actual center. .

[0061] S3.2.4: Calculate the rigid motion vector of the conveyor belt and fasteners based on the motion state of the conveyor belt, and perform consistency verification.

[0062] In one specific implementation, the conveyor belt speed v is obtained by the encoder differentiation; if the L2 norm difference between the visually estimated speed and v is greater than a set tolerance, an anomaly is marked and compensation is skipped.

[0063] S3.2.5: Starting from the actual fastener position, the predicted pose is calculated using the rigid motion vector of the conveyor belt and the fastener through a quadratic extrapolation method.

[0064] In one specific implementation, a constant velocity model and acceleration correction are used to... Extrapolate the delay time along the x-direction : .

[0065] in, To predict the x-coordinate of the pose, To correct the x-coordinate of the 3D center in the base coordinate system, The delay compensation result of the motion compensation network, The acceleration is obtained by linear regression of the conveyor belt linear velocity.

[0066] Then predict the y-axis coordinates of the pose. The z-axis coordinates of the predicted pose The rotation angle of the predicted pose remains unchanged. The predicted pose is obtained. .

[0067] S3.3: Perform compensation calculations based on the preset fastener processing sequence.

[0068] In one specific implementation, each is calculated according to a preset index function. Based on priority scores, the highest scorer receives the reward first. If If so, skip it and mark it for manual re-inspection.

[0069] S3.3.1: Generate a sequential index.

[0070] Specifically, for the M detected bolts, a unique integer index is assigned according to the Z-shaped scanning rule of "from left to right, from front to back". A smaller index indicates that it will be processed earlier.

[0071] S3.3.2: Calculate the distance term . .

[0072] in, For the first The predicted pose coordinates of each bolt (horizontal and vertical coordinates). The xy plane coordinates of the current robot tool in the base coordinate system.

[0073] S3.3.3: Synthesis Priority Score . .

[0074] in, To preset positive weights, satisfying Prioritize order over distance. Obtain the priority score sequence { }

[0075] S3.3.4: Sorting and Distribution.

[0076] Will{ Sort in descending order and take the first element. The next target pose is sent to the motion controller. ρ is a preset confidence threshold; if... This means that the consistency of information in the visible light, infrared, and point cloud modes is low. At this time, the system will skip the queue and enter the manual review area to take the next highest score, until the queue is empty or the system enters the manual review area.

[0077] Example 2: Embodiment 2 of the present invention provides a fastener assembly prediction system based on multimodal vision fusion, comprising: The data acquisition module is configured to acquire multimodal visual data of the power battery pack fastener assembly and preprocess the visual data of the power battery pack fastener assembly, which includes visible light image data, point cloud data and infrared image data.

[0078] The data acquisition module is also configured as follows: Perform calibration work.

[0079] In one specific implementation, before acquiring the multimodal power battery pack fastener assembly visual data, the data acquisition equipment, assembly tools, and fasteners are calibrated. The data acquisition equipment includes a 3D camera, an infrared camera, and a visible light camera.

[0080] Intrinsic parameter calibration. In this embodiment, multi-view shooting with a planar target is used to obtain the camera intrinsic parameter matrix by minimizing the reprojection error.

[0081] External parameter calibration. The external pose transformation from the camera to the robot's base coordinates is obtained using hand-eye calibration. Then, the direction of the conveyor belt movement is calibrated, and the conveyor belt coordinate system is established.

[0082] Acquire visual data of multimodal power battery pack fastener assembly.

[0083] In one specific implementation, visible light image data I, infrared image data T, and point cloud data P are acquired synchronously, and timestamps are recorded; the time difference between the three frames is controlled within a preset range. Visible light image data is acquired through a visible light camera, point cloud data is acquired through a 3D camera, and infrared image data is acquired through an infrared camera.

[0084] Preprocess the visual data of the power battery pack fastener assembly.

[0085] In one specific implementation, preprocessing includes data alignment, data cleaning, and data unification operations on visible light image data, point cloud data, and infrared image data, respectively.

[0086] Specifically, the point cloud data is projected into a depth map, ensuring a one-to-one correspondence between pixels and visible light image data. Infrared image data is aligned to visible light pixels using homography, achieving spatial alignment. Then, the visible light image data is normalized, the infrared image data undergoes temperature truncation and outlier filtering, and the point cloud data is removed from outliers. Finally, the visible light image data, point cloud data, and infrared image data are scaled to the same resolution to construct a five-channel tensor for RGB, temperature, and depth.

[0087] The preliminary feature recognition module is configured to use visible light image data to identify and locate fasteners based on a target detection network, thereby obtaining preliminary recognition features.

[0088] The preliminary feature recognition module is also configured as follows: Multi-scale feature extraction and fusion are performed on visible light image data to obtain dense candidate boxes and the correlation information between category and confidence.

[0089] In one specific implementation, this embodiment uses preprocessed visible light image data as input, and forms a five-level feature map through a convolutional stack with a stride of 2. , The hierarchical index represents the feature level, corresponding to downsampling rates of 2, 4, 8, 16, and 32. Residual connections and channel attention are introduced at each level, with the number of output channels always being C. The top-down path uses nearest-neighbor upsampling to propagate deep semantics to shallower layers, and lateral connections use 1×1 convolutions to compress channels, forming fused multi-scale features. .

[0090] The dense candidate boxes are decoded, the relative offsets are converted into box positions and sizes in image pixel coordinates, and the final confidence score of each box is calculated.

[0091] In one specific implementation, for Apply a rotating frame detection head. The classification branch outputs a single-channel confidence plot. , and These represent the height and width of level l, respectively. The regression branch outputs a five-channel offset plot. Including relative offset of grid center Logarithmic offset of width and height Angular deviation After activation and exponential operations, pixel-level predictions are obtained: , , , , .

[0092] in, The current layer step size, This indicates the pixel coordinates of the center of the rotated bounding box on the input image. and These represent the pixel width and pixel height of the rotating frame, respectively. This indicates the rotation angle of the longer side of the frame relative to the horizontal axis of the image. , , They are respectively , and The predefined anchor points. After decoding, the final confidence score for each box is obtained. .

[0093] Low-scoring candidate boxes are filtered out based on a pre-set confidence threshold, while high-probability candidate boxes are retained.

[0094] In one specific implementation, pixel-by-pixel comparison With global threshold ,reserve The candidate box set is obtained by performing activation and exponential operations on the candidate box set and then performing pixel-level prediction to obtain the final candidate box set. .

[0095] Non-maximum suppression is applied to the retained candidate boxes to eliminate overlapping boxes, resulting in a unique list of two-dimensional bounding boxes for fasteners.

[0096] In one specific implementation, Sort by confidence level in descending order; iteratively select the highest-scoring box b, and calculate the scores of the remaining boxes. The ratio of the intersection area to the union area (IoU) of the rotated frames, discarding Repeat this process until no one remains, to obtain a unique list of boxes. ,in The threshold for overlap set for the nonmaximum suppression process.

[0097] The coordinates, width, height, and visible light confidence level of the fastener's center point are calculated based on the bounding boxes in the two-dimensional bounding box list to form preliminary identification features.

[0098] In one specific implementation, for each frame Take the center pixel , Index the bounding box, read the synchronous depth map, and obtain the depth map. After inverse projection and Transformation to obtain the robot's base coordinate system in three dimensions , This is a rigid extrinsic homogeneous transformation matrix (4×4) from the camera coordinate system to the robot base coordinate system. It fully describes rotation and translation and is used to map camera measurements to the robot's motion space. and Preliminary characteristics , Indicates the first The pixel width of each candidate rotation box Indicates the first The pixel height of each candidate rotation box. Indicates the first The rotation angle of the long side of each candidate rotation box relative to the horizontal axis of the image.

[0099] The position prediction module is configured to fuse multimodal power battery pack fastener assembly visual data, use a motion compensation network based on perception fusion to correct the initial identification features based on the fused features, and perform pose prediction based on the corrected actual fastener position. At the same time, it performs compensation calculations in conjunction with the preset fastener processing order.

[0100] The location prediction module is also configured as follows: The visual data of multimodal power battery pack fastener assembly is fused.

[0101] Spatiotemporal alignment is performed on visible light image data, point cloud data, and infrared image data at the same moment with timestamps and credibility masks.

[0102] The aligned visible light image data, point cloud data, and infrared image data are dimensionally unified and then stitched together to obtain a stitched sequence.

[0103] In one specific implementation, the three modal data are each reduced to a unified channel through 1×1 convolution. ,get ,in These are features from visible light image data, infrared image data, and point cloud data after dimensional unification. The three features are then concatenated along the channels to obtain a stitched sequence. .

[0104] By introducing learnable modal confidence weights, a Transformer-style cross-attention network is used to autonomously suppress failed modalities in the spliced ​​sequence, resulting in fused features.

[0105] In one specific implementation, for Global average pooling is used to obtain modal tokens, which are then processed by a fully connected layer and Softmax to generate weight vectors. ;by For coefficient pairs Perform weighted summation and output the fused features. .

[0106] A motion compensation network based on perception fusion is used to correct the initially identified features based on the fused features, and pose prediction is performed based on the corrected actual fastener position.

[0107] Construct a motion compensation network, which consists of a correction subnetwork and a compensation subnetwork connected in sequence.

[0108] In one specific implementation, the correction subnetwork is characterized by preliminary features. For querying, feature fusion As the key value, the offsets of the two layers of cross-attention regression are set to... and , frame geometric offset ,in, All of these were acquired through learning. This is the pixel-level center correction amount during the correction process. To correct the width and height offset of the initial decoding frame during the calibration process, This is a confidence correction factor used to adjust the confidence level originally obtained solely from visible light, ensuring that the fused confidence level is corrected. It better reflects the true reliability of trimodal consistency, based on angular offset. right Make corrections to obtain The corrected 3D center can be obtained through the correction subnetwork. With conveyor belt speed The compensation subnetwork takes the corrected 3D center and conveyor belt speed as input and outputs the predicted pose. Among them, the compensation subnetwork... The spliced ​​result is the delay compensation result in the x-direction output after passing through two layers of FC. y, z Direct pass-through to obtain the predicted pose .

[0109] The correction subnetwork uses the preliminary visible light identification features as the query and the fused features as the key. It performs correction processing through two layers of cross-attention regression to obtain the corrected fastener 2D frame, 3D center coordinates and joint confidence of the three-modal data.

[0110] The corrected 3D center coordinates are transformed from the camera coordinate system to the robot base coordinate system to obtain the actual fastener position.

[0111] In one specific implementation, the joint confidence level is calculated. : .

[0112] in, For the Sigmoid function, It is the inverse Sigmoid function.

[0113] Center the camera system in three dimensions through Transform to the robot's base coordinate system to obtain the actual center. .

[0114] The rigid motion vectors of the conveyor belt and fasteners are calculated based on the motion state of the conveyor belt, and a consistency check is performed.

[0115] In one specific implementation, the conveyor belt speed v is obtained by the encoder differentiation; if the L2 norm difference between the visually estimated speed and v is greater than a set tolerance, an anomaly is marked and compensation is skipped.

[0116] Starting from the actual fastener position, the predicted pose is calculated using the rigid motion vectors of the conveyor belt and the fastener through a quadratic extrapolation method.

[0117] In one specific implementation, a constant velocity model and acceleration correction are used to... Extrapolate the delay time along the x-direction : .

[0118] in, To predict the x-coordinate of the pose, To correct the x-coordinate of the 3D center in the base coordinate system, The delay compensation result of the motion compensation network, The acceleration is obtained by linear regression of the conveyor belt linear velocity.

[0119] Then predict the y-axis coordinates of the pose. The z-axis coordinates of the predicted pose The rotation angle of the predicted pose remains unchanged. The predicted pose is obtained. .

[0120] Compensation calculations are performed in conjunction with the preset fastener processing sequence.

[0121] In one specific implementation, each is calculated according to a preset index function. Based on priority scores, the highest scorer receives the reward first. If If so, skip it and mark it for manual re-inspection.

[0122] Generate a sequential index.

[0123] Specifically, for the M detected bolts, a unique integer index is assigned according to the Z-shaped scanning rule of "from left to right, from front to back". A smaller index indicates that it will be processed earlier.

[0124] Calculate distance term . .

[0125] in, For the first The predicted pose coordinates of each bolt (horizontal and vertical coordinates). The xy plane coordinates of the current robot tool in the base coordinate system.

[0126] Synthesis Priority Score . .

[0127] in, To preset positive weights, satisfying Prioritize order over distance. Obtain the priority score sequence { }

[0128] Sorting and distribution.

[0129] Will{ Sort in descending order and take the first element. The next target pose is sent to the motion controller. ρ is a preset confidence threshold; if... This means that the consistency of information in the visible light, infrared, and point cloud modes is low. At this time, the system will skip the queue and enter the manual review area to take the next highest score, until the queue is empty or the system enters the manual review area.

[0130] Example 3: Embodiment 3 of the present invention provides a computer-readable storage medium storing a computer program adapted for loading by a processor and executing steps in the fastener assembly prediction method based on multimodal visual fusion as described in Embodiment 1 of the present invention, wherein the steps are: The assembly visual data of the fasteners of the power battery pack is acquired in multiple modes and preprocessed. The assembly visual data of the fasteners of the power battery pack includes visible light image data, point cloud data and infrared image data. Fastener identification and localization are performed using visible light image data based on a target detection network to obtain preliminary identification features; The visual data of fastener assembly of multimodal power battery packs are fused together. A motion compensation network based on perception fusion is used to correct the initial identification features according to the fused features, and pose prediction is performed according to the corrected actual fastener position. At the same time, compensation calculation is performed in combination with the preset fastener processing order.

[0131] The detailed steps are the same as those of the fastener assembly prediction method with multimodal vision fusion provided in Example 1, and will not be repeated here.

[0132] Example 4: Embodiment 4 of the present invention provides a computer device, the device comprising: A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the steps in the fastener assembly prediction method based on multimodal visual fusion as described in Embodiment 1 of the present invention, wherein the steps are: The assembly visual data of the fasteners of the power battery pack is acquired in multiple modes and preprocessed. The assembly visual data of the fasteners of the power battery pack includes visible light image data, point cloud data and infrared image data. Fastener identification and localization are performed using visible light image data based on a target detection network to obtain preliminary identification features; The visual data of fastener assembly of multimodal power battery packs are fused together. A motion compensation network based on perception fusion is used to correct the initial identification features according to the fused features, and pose prediction is performed according to the corrected actual fastener position. At the same time, compensation calculation is performed in combination with the preset fastener processing order.

[0133] The detailed steps are the same as those of the fastener assembly prediction method with multimodal vision fusion provided in Example 1, and will not be repeated here.

[0134] Example 5: Embodiment 5 of the present invention provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps in the fastener assembly prediction method based on multimodal visual fusion as described in Embodiment 1 of the present invention. The steps are as follows: The assembly visual data of the fasteners of the power battery pack is acquired in multiple modes and preprocessed. The assembly visual data of the fasteners of the power battery pack includes visible light image data, point cloud data and infrared image data. Fastener identification and localization are performed using visible light image data based on a target detection network to obtain preliminary identification features; The visual data of fastener assembly of multimodal power battery packs are fused together. A motion compensation network based on perception fusion is used to correct the initial identification features according to the fused features, and pose prediction is performed according to the corrected actual fastener position. At the same time, compensation calculation is performed in combination with the preset fastener processing order.

[0135] The detailed steps are the same as those of the fastener assembly prediction method with multimodal vision fusion provided in Example 1, and will not be repeated here.

[0136] The steps and methods involved in Examples 2, 3, 4 and 5 above correspond to those in Example 1. For specific implementation methods, please refer to the relevant description section of Example 1.

[0137] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data processing device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, an optical medium, or a semiconductor medium, etc. The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A fastener assembly prediction method based on multimodal visual fusion, characterized in that, Includes the following steps: The assembly visual data of the fasteners of the power battery pack is acquired in multiple modes and preprocessed. The assembly visual data of the fasteners of the power battery pack includes visible light image data, point cloud data and infrared image data. Fastener identification and localization are performed using visible light image data based on a target detection network to obtain preliminary identification features; The visual data of fastener assembly of multimodal power battery packs are fused together. A motion compensation network based on perception fusion is used to correct the initial identification features according to the fused features, and pose prediction is performed according to the corrected actual fastener position. At the same time, compensation calculation is performed in combination with the preset fastener processing order.

2. The fastener assembly prediction method based on multimodal vision fusion as described in claim 1, characterized in that, Preprocessing includes data alignment, data cleaning, and data unification operations for visible light image data, point cloud data, and infrared image data, respectively.

3. The fastener assembly prediction method based on multimodal vision fusion as described in claim 1, characterized in that, Before acquiring visual data of multimodal power battery pack fastener assembly, the data acquisition equipment, assembly tools and fasteners are calibrated. The data acquisition equipment includes a 3D camera, an infrared camera and a visible light camera. Visible light image data is acquired through the visible light camera, point cloud data is acquired through the 3D camera and infrared image data is acquired through the infrared camera.

4. The fastener assembly prediction method based on multimodal vision fusion as described in claim 1, characterized in that, The specific steps for fastener identification and localization using visible light image data based on a target detection network are as follows: Multi-scale feature extraction and fusion are performed on visible light image data to obtain dense candidate boxes and the correlation information between category and confidence. The dense candidate boxes are decoded, the relative offsets are converted into the box position and size in the image pixel coordinates, and the final confidence score of each box is calculated. Low-scoring candidate boxes are filtered out based on a pre-set confidence threshold, while high-probability candidate boxes are retained. Non-maximum suppression is applied to the retained candidate boxes to eliminate overlapping boxes, resulting in a unique list of two-dimensional bounding boxes for fasteners. The coordinates, width, height, and visible light confidence level of the fastener's center point are calculated based on the bounding boxes in the two-dimensional bounding box list to form preliminary identification features.

5. The fastener assembly prediction method based on multimodal vision fusion as described in claim 1, characterized in that, The specific steps for fusing visual data of multimodal power battery pack fastener assembly are as follows: Spatiotemporal alignment is performed on visible light image data, point cloud data, and infrared image data at the same moment with timestamps and credibility masks attached. The aligned visible light image data, point cloud data, and infrared image data are dimensionally unified and then stitched together to obtain a stitched sequence. By introducing learnable modal confidence weights, a Transformer-style cross-attention network is used to autonomously suppress failed modalities in the spliced ​​sequence, resulting in fused features.

6. The fastener assembly prediction method based on multimodal vision fusion as described in claim 1, characterized in that, The specific steps for using a motion compensation network based on perceptual fusion to correct the initially identified features based on the fused features, and then predicting the pose based on the corrected actual fastener position, are as follows: Construct a motion compensation network, which includes a correction subnetwork and a compensation subnetwork connected in sequence; The correction subnetwork uses the initial visible light recognition features as the query and the fused features as the key. It performs correction processing through two layers of cross-attention regression to obtain the corrected fastener 2D frame, 3D center coordinates and joint confidence of the three-modal data. The corrected 3D center coordinates are transformed from the camera coordinate system to the robot base coordinate system to obtain the actual fastener position. Calculate the rigid motion vector of the conveyor belt and fasteners based on the motion state of the conveyor belt, and perform consistency verification. Starting from the actual fastener position, the predicted pose is calculated using the rigid motion vectors of the conveyor belt and the fastener through a quadratic extrapolation method.

7. A fastener assembly prediction system based on multimodal vision fusion, characterized in that, include: The data acquisition module is configured to acquire multimodal visual data of power battery pack fastener assembly and preprocess the visual data of power battery pack fastener assembly, which includes visible light image data, point cloud data and infrared image data. The preliminary feature recognition module is configured to use visible light image data to identify and locate fasteners based on a target detection network, thereby obtaining preliminary recognition features. The position prediction module is configured to fuse multimodal power battery pack fastener assembly visual data, use a motion compensation network based on perception fusion to correct the initial identification features based on the fused features, and perform pose prediction based on the corrected actual fastener position. At the same time, it performs compensation calculations in conjunction with the preset fastener processing order.

8. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the fastener assembly prediction method based on multimodal visual fusion as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and executed as described in any one of claims 1-6: a fastener assembly prediction method based on multimodal visual fusion.

10. A computer device, characterized in that, include: A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program that, when executed by the processor, implements the fastener assembly prediction method based on multimodal visual fusion as described in any one of claims 1-6.

Images