Workpiece gap recognition tracking method and device based on visual cue segmentation

Abstract

The application discloses a workpiece gap recognition and tracking method and device based on visual prompt segmentation, and relates to the technical field of computer vision and image processing. The method comprises the following steps: detecting the workpiece boundary by using an infrared photoelectric sensor group for rough positioning; collecting a preposed local image and marking a pixel point visual prompt set; inputting the image and the visual prompt set into a visual prompt segmentation model to obtain a gap mask; analyzing the gap position and shape, controlling the device to move along the gap and outputting an operation instruction; collecting a local image frame sequence, dynamically tracking the gap, estimating the next time trajectory through Kalman filtering, and continuously moving along the gap; collecting a postposed image by a camera to obtain a coverage mask, calculating the gap coverage degree for coverage verification. The application realizes gap recognition by using visual prompt segmentation, does not need to preset a trajectory and match a template, can adapt to various non-standard workpieces, and performs closed-loop coverage verification through front and rear cameras, so that the automatic gluing degree and production efficiency are greatly improved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image processing technology, specifically relating to a method and apparatus for workpiece gap recognition and tracking based on visual cue segmentation. Background Technology

[0002] With the rapid development of intelligent manufacturing and industrial automation technologies, industrial product assembly processes are gradually upgrading towards flexibility, intelligence, and high precision. Sealing and adhesive application, as a crucial step in industrial product assembly, is a key link in ensuring the waterproof, dustproof, airtight, and structural bonding performance of products such as electronic housings, automotive parts, building material profiles, and new energy battery housings. It directly determines the product's lifespan and operational reliability, and its efficiency and adhesive application quality have a significant impact on the overall production line's efficiency.

[0003] Currently, adhesive application technologies in the industrial field can be divided into three categories, each with its own application limitations:

[0004] The first type is manual glue application, which relies on operators holding a glue gun and manually applying glue along the gaps in the workpiece. The quality of the work depends entirely on the operator's experience and skill. This method has low production efficiency, poor glue application consistency, and is prone to defects such as uneven glue width, glue breaks, and glue overflow. Moreover, long-term operation can easily lead to fatigue, resulting in large fluctuations in quality and high labor costs, making it unable to meet the needs of large-scale, high-precision mass production.

[0005] The second category is traditional automated gluing equipment, which mostly adopts a teach-and-playback or template matching working mode. A fixed gluing trajectory is preset through manual teaching, or after positioning is completed based on workpiece template matching, gluing is performed according to the preset trajectory and fixed dispensing parameters. Although this type of solution improves efficiency and consistency compared to manual gluing, it can only be adapted to the mass production of workpieces with a single specification and regular shape gaps. When changing the type of workpiece or dealing with non-standard irregular workpieces, it is necessary to re-teach programming and template debugging, resulting in long production changeover cycles and weak flexibility. Moreover, when there are clamping deviations or dimensional tolerances in the workpiece, the fixed gluing trajectory is prone to misalignment and displacement, which may lead to the risk of sealing failure.

[0006] The third category involves assisted gluing solutions that incorporate traditional machine vision. These solutions acquire workpiece images using industrial cameras and employ traditional image processing algorithms such as edge detection, contour extraction, and template matching to identify gap locations, thereby guiding the gluing mechanism. While this approach improves adaptability to workpiece deviations to some extent, it still relies on closed-set recognition logic, limiting its application to pre-defined workpiece types and gap shapes. When faced with untrained non-standard workpieces or irregularly shaped gaps, recognition accuracy drops significantly or even fails to detect them. Furthermore, traditional image processing algorithms are less robust to changes in lighting, workpiece surface texture, and dirt interference, leading to false positives and false negatives. In addition, most existing vision-based gluing solutions only provide pre-application trajectory guidance, lacking online quality verification and closed-loop error correction mechanisms after application. Defects such as glue leakage, uneven glue application, and glue breakage cannot be detected and corrected in real time during operation, requiring separate manual re-inspection procedures. This not only increases labor costs and processing time but also hinders the automation of the entire gluing process, limiting further improvements in production efficiency.

[0007] In summary, existing adhesive application technologies generally rely on preset trajectories and template matching, which cannot effectively adapt to the flexible production needs of various non-standard workpieces. At the same time, they lack dynamic tracking of the adhesive application process and closed-loop verification and error correction capabilities after adhesive application, resulting in delayed defect detection, high reliance on manual labor, and an overall level of automation that is difficult to meet the current development needs of intelligent manufacturing. Summary of the Invention

[0008] To solve the above-mentioned technical problems, the present invention adopts the following solution:

[0009] On the one hand, the present invention provides a workpiece gap recognition and tracking method based on visual cue segmentation, comprising the following steps:

[0010] S1. The workpiece boundary is detected by the infrared photoelectric sensor group, and the working device equipped with the front camera and the rear camera is moved to the detection area of ​​the workpiece edge to complete the coarse positioning of the gap area.

[0011] S2. The front-facing camera captures a local image of the area to be detected and labels a set of visual cues for each pixel.

[0012] S3. Input the front local image and pixel visual cue set into the visual cue segmentation model, and output the front gap mask image.

[0013] S4. The position and shape information of the gap are obtained by parsing the front gap mask image, and the working device is controlled to move along the gap and output working instructions.

[0014] S5. During the movement of the working device, the continuously acquired sequence of previous local image frames is input into the visual cue segmentation model to dynamically track the gap position of each frame. The trajectory of the gap at the next moment is predicted by Kalman filtering. Based on the prediction result, the memory of the visual cue segmentation model is written with consistency gating to control the working device to move continuously along the gap.

[0015] S6. The rear camera captures a local image of the rear area after the operation, inputs it into the visual cueing segmentation model to obtain the rear cover mask image, and calculates the gap coverage to verify the adhesive layer coverage.

[0016] In step S1, the infrared photoelectric sensor group includes a vertical infrared photoelectric sensor and multiple horizontal infrared photoelectric sensors.

[0017] Step S1 further includes the following steps:

[0018] S11. Using a vertical infrared photoelectric sensor, control the working device equipped with a front camera and a rear camera to move downwards until the horizontal surface of the workpiece is detected.

[0019] S12. The horizontal infrared photoelectric sensor controls the horizontal movement of the working device until the vertical boundary of the workpiece is detected.

[0020] S13. The moving work device makes all infrared photoelectric sensors in a non-triggered state. At this time, the work device is located above the gap between the workpieces, and completes the coarse positioning of the gap area.

[0021] In step S2, the metadata of the front-facing local image includes the calibration parameters of the front-facing camera.

[0022] In step S2, the set of pixel visual cues includes positive pixel cues representing gap regions and negative pixel cues representing background regions.

[0023] In step S3, the visual cue segmentation model supports pixel-type visual cue input and adopts an open-domain segmentation architecture with streaming memory mechanism. Internally, it includes an image encoder, a cue encoder, a mask decoder, a memory encoder, a memory bank, and a memory attention module. The memory bank is configured to be written with gating control based on the consistency verification result of the Kalman filter predicted trajectory. When the consistency is lower than a preset threshold, the current frame is prohibited from being written to the memory bank.

[0024] Step S4 further includes the following steps:

[0025] S41. Extract the gap outline and gap center line from the front gap mask image, and calculate the gap position coordinates using the gap pixel coordinates and the front camera calibration parameters.

[0026] S42. Control the working device equipped with a front camera and a rear camera to move to the starting coordinate of the gap.

[0027] S43. Control the working device to move along the trajectory of the center line of the gap;

[0028] S44. Adjust the discharge flow rate and moving speed parameters according to the gap width and curvature in the gap shape parameters at the current position.

[0029] Step S5 further includes the following steps:

[0030] S51. During the movement of the working device, the front-facing camera continuously captures front-facing local images of the area to be detected, forming a continuous image frame sequence.

[0031] S52. Input the image frame sequence frame by frame into the visual cue segmentation model. Through the memory encoder, memory bank and memory attention module in the model, use the gap mask output from the previous frame as a cue, reuse the gap features of the historical frame, and dynamically track the gap position in each frame of the image.

[0032] S53. Perform a spatial consistency check between the centerline extracted from the gap mask of the current frame output and the predicted trajectory of the Kalman filter at the same time. If the consistency is lower than the preset threshold, the memory encoder is prohibited from writing its mask and features into the memory bank; otherwise, writing is allowed.

[0033] S54. If the spatial consistency of the current frame is greater than or equal to the preset threshold, the coordinates of the gap centerline obtained frame by frame are used as valid observations and input into the Kalman filter; if the consistency is lower than the preset threshold, the current observation is deemed invalid and discarded.

[0034] S55. If valid observations exist, the system state is updated based on the valid observations and the trajectory position of the gap centerline is predicted at the next moment. If no valid observations exist, the observation update step is skipped and the trajectory position at the next moment is predicted directly based on the Kalman filter.

[0035] S56. Based on the estimated trajectory position and the current position deviation of the working device, output a travel control command to control the working device to continuously move along the gap path.

[0036] Step S6 further includes the following steps:

[0037] S61. Acquire a partial rear image after the operation using the rear camera;

[0038] S62. A set of visual cues for marking colloidal regions on a rear local image;

[0039] S63. Input the rear local image and the set of visual cues for the pixels of the colloidal region into the visual cues segmentation model to obtain the rear overlay mask image;

[0040] S64. Calculate the gap coverage of the back cover mask image and the front gap mask image. The gap coverage is the ratio of the intersection area of ​​the two masks to the area of ​​the front gap mask.

[0041] S65. If the gap coverage is greater than or equal to the preset threshold, the adhesive layer coverage verification is passed; if it is less than the preset threshold, the verification is failed, and the operating device is controlled to retract and perform the adhesive replenishment action.

[0042] On the other hand, the present invention provides a workpiece gap recognition and tracking device based on visual cue segmentation, comprising:

[0043] The sensor sensing module includes an infrared photoelectric sensor group, a front camera, and a rear camera, used to detect workpiece boundaries and acquire local images;

[0044] The central control and storage module is used to run the visual cue segmentation model and store the set of pixel visual cues;

[0045] The job execution interface module receives job instructions from the central control and storage module and parses them into control signals to drive the job unit in order to perform the colloidal material output operation.

[0046] The walking mechanism interface module receives walking commands from the central control and storage module and parses them into control signals to drive the displacement unit, thereby moving the sensor perception module and the work execution interface module.

[0047] This invention proposes a workpiece gap recognition and tracking method and device based on visual cue segmentation. Through a four-level, fully closed-loop architecture of "coarse positioning - fine recognition - dynamic execution - closed-loop verification," it significantly improves the automation level and production efficiency of adhesive application. Compared with existing technologies, it has outstanding substantive features and significant progress, with the following specific beneficial effects:

[0048] This invention employs an open-domain segmentation model that supports pixel-level visual cues to achieve gap recognition. It eliminates the need for creating matching templates for specific workpieces. By marking positive and negative pixels, it can accurately segment non-standard workpieces and irregularly shaped gaps even with small sample sizes, enabling rapid adaptation to flexible production scenarios with diverse product types and small batches. Furthermore, this invention adds a Kalman filter trajectory consistency gating writing mechanism to the memory of the visual cue segmentation model, avoiding tracking drift caused by the accumulation of erroneous features and continuously ensuring the stability of long-distance dynamic gap tracking. Finally, this invention uses a dual-camera scheme with front-end gap recognition and rear-end adhesive layer detection, calculating the adhesive layer's coverage of the gap to verify work quality.

[0049] This invention automates the entire process from coarse positioning of workpiece boundaries, precise identification of gaps, dynamic tracking of trajectories to verification and error correction of adhesive layers, reducing manual intervention and effectively improving overall production efficiency, thus possessing high industrial application value. Attached Figure Description

[0050] Figure 1 This is a flowchart of the overall method steps in Embodiment 1 of the present invention;

[0051] Figure 2 This is a schematic diagram of the internal structure of the visual cue segmentation model in Embodiment 1 of the present invention;

[0052] Figure 3 This is a flowchart of step S4 in Embodiment 1 of the present invention;

[0053] Figure 4 This is a flowchart of step S6 in Embodiment 1 of the present invention;

[0054] Figure 5 This is a block diagram of the device components in Embodiment 2 of the present invention. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] Furthermore, for clarity and brevity, descriptions of well-known structures, functions, and configurations may have been omitted. Those skilled in the art will recognize that various changes and modifications can be made to the examples described herein without departing from the spirit and scope of this disclosure.

[0057] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0058] Example 1

[0059] This embodiment provides a workpiece gap recognition and tracking method based on visual cue segmentation.

[0060] like Figure 1 As shown, the workpiece gap recognition and tracking method based on visual cue segmentation includes the following steps:

[0061] S1. The workpiece boundary is detected by the infrared photoelectric sensor group, and the working device equipped with the front camera and the rear camera is moved to the detection area of ​​the workpiece edge to complete the coarse positioning of the gap area.

[0062] S2. The front-facing camera captures a partial image of the area to be detected and labels a set of visual cues for each pixel.

[0063] S3. Input the front local image and pixel visual cue set into the visual cue segmentation model, and output the front gap mask image.

[0064] S4. The position and shape information of the gap are obtained by parsing the front gap mask image, and the working device is controlled to move along the gap and output working instructions.

[0065] S5. During the movement of the working device, the continuously acquired sequence of previous local image frames is input into the visual cue segmentation model to dynamically track the position of the gap in each frame. The trajectory of the gap at the next moment is predicted by Kalman filtering. Based on the prediction results, the memory of the visual cue segmentation model is written with consistency gating to control the working device to move continuously along the gap.

[0066] S6. The rear camera captures a local image of the rear area after the operation, inputs it into the visual cueing segmentation model to obtain the rear cover mask image, and calculates the gap coverage to verify the adhesive layer coverage.

[0067] In this embodiment, the infrared photoelectric sensor group in S1 includes a vertical infrared photoelectric sensor and multiple horizontal infrared photoelectric sensors.

[0068] Specifically: the vertical sensor is coaxially mounted directly above the workpiece, with the detection direction vertically downwards; multiple horizontal sensors are evenly arranged circumferentially, with the detection direction horizontal. All infrared photoelectric sensors use reflective infrared photoelectric switches.

[0069] In this embodiment, step S1 further includes the following steps:

[0070] S11. Using a vertical infrared photoelectric sensor, control the working device equipped with a front camera and a rear camera to move downwards until the horizontal surface of the workpiece is detected.

[0071] S12. The horizontal infrared photoelectric sensor controls the horizontal movement of the working device until the vertical boundary of the workpiece is detected.

[0072] S13. The moving work device makes all infrared photoelectric sensors in a non-triggered state. At this time, the work device is located above the gap between the workpieces, and completes the coarse positioning of the gap area.

[0073] Specifically, step S11 involves the working device descending at a set speed along the Z-axis, while the vertical sensor collects the reflected light intensity in real time. When the light intensity reaches a preset trigger threshold, it is determined that the surface directly below is the horizontal surface of the workpiece, and a stop command is immediately issued to lock the current Z-axis height coordinate, thus completing the vertical positioning of the workpiece.

[0074] Specifically, step S12 involves the working device feeding horizontally at a constant speed along the X / Y axes, with each horizontal sensor synchronously detecting side signals. When any sensor is triggered, the corresponding side boundary coordinates are recorded; the movement continues until all horizontal sensors have been triggered, acquiring boundary position data for multiple sides of the workpiece.

[0075] Specifically, step S13 involves controlling the working device to slightly retract towards the inside of the workpiece based on the collected boundary coordinates. The output status of all sensors is monitored in real time; when all sensors change from triggered to non-triggered, the workpiece splicing gap directly below the working device is determined, completing the coarse positioning of the gap.

[0076] In this embodiment, the metadata of the front-facing local image in S2 includes the calibration parameters of the front-facing camera.

[0077] It should be noted that the calibration parameters include the camera's intrinsic parameter matrix, distortion coefficients, and extrinsic calibration parameters. These parameters are pre-calibrated using a checkerboard calibration method and transmitted synchronously with the image metadata to convert pixel coordinates into three-dimensional physical coordinates in the mechanical coordinate system.

[0078] In this embodiment, the pixel visual cue set in S2 includes positive pixel cue representing the gap region and negative pixel cue representing the background region.

[0079] Specifically, positive examples indicate the target segmentation area by selecting pixels in the gap region; negative examples indicate pixels in non-gap locations such as the workpiece surface and background to exclude interference areas.

[0080] In this embodiment, as Figure 2 As shown, the visual cue segmentation model in S3 supports pixel-type visual cue input and adopts an open-domain segmentation architecture with a streaming memory mechanism. Internally, it includes an image encoder, a cue encoder, a mask decoder, a memory encoder, a memory bank, and a memory attention module. The memory bank is configured to be written with gating based on the consistency verification result of the Kalman filter predicted trajectory. When the consistency is lower than a preset threshold, the current frame is prohibited from being written to the memory bank.

[0081] Specifically, the visual cue segmentation model is based on the SAM2 architecture, forming an open-domain segmentation network with streaming memory. The model structure includes: an image encoder (using a ViT-B backbone network), a cue encoder (encoding the positional coordinates of positive / negative cue points in the input pixels), a mask decoder (generating segmentation masks based on a Transformer decoder), and a memory encoder, a memory (first-in, first-out queue), and a memory attention module. The connections between these modules are as follows: the image encoder extracts the features of the current frame image, concatenates them with the cue features output by the cue encoder, and then sends the concatenation to the mask decoder; simultaneously, the memory encoder compresses the masks and image features of historical frames and writes them into the memory, and the mask decoder retrieves relevant historical features from the memory through the memory attention module to assist in the segmentation of the current frame. The training data comes from various workpiece gap images collected from the production line, with manually labeled gap masks used as ground truth. The training strategy employs freezing the image encoder, fine-tuning the cue encoder, mask decoder, and memory module, jointly optimizing Dice Loss and cross-entropy loss, using the AdamW optimizer, with an initial learning rate of 1e-4 and a batch size of 8. The input is a local RGB image captured by the front-facing camera and the corresponding set of visual cues for each pixel. The output is a slit binary mask image with the same resolution as the input image.

[0082] The memory-gated writing method for the visual cue segmentation model is as follows: After outputting the slit mask for each frame, its single-pixel centerline is extracted. At the same time as the Kalman filter, the Hausdorff distance is calculated based on the slit trajectory centerline estimated from historical states. If the distance exceeds a preset threshold of 5 pixels, the memory encoder interrupts the encoding and writing operation of the mask and low-resolution features for that frame, retaining only the mask output by the mask decoder for current job control. If the distance does not exceed the threshold, the memory encoder is allowed to compress the mask and corresponding image region cropping features and write them into the memory.

[0083] In this embodiment, as Figure 3 As shown, step S4 further includes the following steps:

[0084] S41. Extract the gap outline and gap center line from the front gap mask image, and calculate the gap position coordinates using the gap pixel coordinates and the front camera calibration parameters.

[0085] S42. Control the working device equipped with a front camera and a rear camera to move to the starting coordinate of the gap.

[0086] S43. Control the working device to move along the trajectory of the center line of the gap;

[0087] S44. Adjust the discharge flow rate and moving speed parameters according to the gap width and curvature in the gap shape parameters at the current position.

[0088] Specifically, step S41 involves: performing morphological opening operations on the gap mask to remove noise and extracting the inner and outer contours of the gap; using a skeleton extraction algorithm to calculate the gap centerline and obtain the pixel coordinates of each point; combining calibration parameters and the hand-eye calibration matrix to convert to mechanical three-dimensional coordinates, and simultaneously calculating the gap width and curvature parameters at each point. Step S42 involves: calculating the motion parameters of each axis based on the three-dimensional coordinates of the gap's starting point using inverse kinematics; controlling multi-axis servo linkage to precisely move the working device to directly above the gap's starting position, maintaining a preset standard working height to prepare the position for subsequent operations. Step S43 involves: using the gap centerline as the reference path, generating a smooth and continuous motion trajectory using a spline interpolation algorithm; planning a speed curve to control the working device's uniform movement; providing real-time feedback on the working device's position during movement and correcting for deviations from the target trajectory to ensure the working device is always aligned with the gap centerline. Specifically, step S44 involves: presetting the mapping relationship between the gap width and valve opening, curvature and moving speed; acquiring the current position parameters in real time; increasing the valve opening to increase the output when the width increases, and reducing the moving speed when the curvature increases, to ensure uniform glue layer width and avoid glue accumulation and glue breakage defects.

[0089] In this embodiment, step S5 further includes the following steps:

[0090] S51. During the movement of the working device, the front-facing camera continuously captures front-facing local images of the area to be detected, forming a continuous image frame sequence.

[0091] S52. Input the image frame sequence frame by frame into the visual cue segmentation model. Through the memory encoder, memory bank and memory attention module in the model, use the gap mask output from the previous frame as a cue, reuse the gap features of the historical frame, and dynamically track the gap position in each frame of the image.

[0092] S53. Perform a spatial consistency check between the centerline extracted from the gap mask of the current frame output and the predicted trajectory of the Kalman filter at the same time. If the consistency is lower than the preset threshold, the memory encoder is prohibited from writing its mask and features into the memory bank; otherwise, writing is allowed.

[0093] S54. If the spatial consistency of the current frame is greater than or equal to the preset threshold, the coordinates of the gap centerline obtained frame by frame are used as valid observations and input into the Kalman filter; if the consistency is lower than the preset threshold, the current observation is deemed invalid and discarded.

[0094] S55. If valid observations exist, the system state is updated based on the valid observations and the trajectory position of the gap centerline is predicted at the next moment. If no valid observations exist, the observation update step is skipped and the trajectory position at the next moment is predicted directly based on the Kalman filter.

[0095] S56. Based on the estimated trajectory position and the current position deviation of the working device, output a travel control command to control the working device to continuously move along the gap path.

[0096] In step S52, the memory encoder stores the gap mask features of historical frames into a memory bank. The current frame uses the memory attention module to retrieve historical features, thus achieving continuous gap tracking and segmentation without repeated annotation. In step S53, the centerline is extracted from the gap mask segmented from each frame, and the current point coordinates are obtained and converted to values ​​in the machine coordinate system via hand-eye calibration. Simultaneously, the centerline and the Kalman filter are used to perform a spatial consistency check on the predicted trajectory at the current moment. In step S54, if the spatial consistency check passes, the memory encoder is allowed to write the current frame features into the memory bank, and the centerline coordinates are input as valid observations into the Kalman filter to fuse the observations and complete the state update. If the check fails, writing to the memory bank is prohibited, and the invalid observation is discarded. In step S55, based on the updated system state, the Kalman filter calculates the gap position at the next moment and outputs a smooth, noise-resistant trajectory prediction result. In step S56: the deviation between the estimated trajectory position and the current actual position of the working device is calculated, and a speed control command is generated through a PID algorithm to drive the walking mechanism to correct the deviation in real time, so as to ensure that the working device always travels along the center line of the gap.

[0097] In this embodiment, as Figure 4 As shown, step S6 further includes the following steps:

[0098] S61. Acquire a partial rear image after the operation using the rear camera;

[0099] S62. A set of visual cues for marking colloidal regions on a rear local image;

[0100] S63. Input the rear local image and the set of visual cues for the pixels of the colloidal region into the visual cues segmentation model to obtain the rear overlay mask image;

[0101] S64. Calculate the gap coverage of the back cover mask image and the front gap mask image. The gap coverage is the ratio of the intersection area of ​​the two masks to the area of ​​the front gap mask.

[0102] S65. If the gap coverage is greater than or equal to the preset threshold, the adhesive layer coverage verification is passed; if it is less than the preset threshold, the verification is failed, and the operating device is controlled to retract and perform the adhesive replenishment action.

[0103] In step S61, the rear camera is fixedly installed behind the working device, maintaining a constant distance from the front camera, and captures real-time images of the area where the work has just been completed for quality inspection. In step S62, pixels in the colloid region of the rear image are selected as positive examples, and the workpiece surface and uncovered gap areas are selected as negative examples, guiding the model to accurately segment the actual contour of the colloid. In step S63, the same visual cue segmentation model is reused, inputting the rear image and the colloid cue set, and outputting a binary mask image of the corresponding colloid region to accurately represent the actual coverage area of ​​the colloid. In step S64, after pixel-level registration and alignment of the front and rear mask images, the ratio of the intersection pixel area to the total pixel area of ​​the front gap mask is calculated to obtain a quantitative index of the colloid layer's coverage of the gap. In step S65, if the gap coverage meets the standard, the verification is passed, and the device continues along the trajectory; if it does not meet the standard, the defect location and gap size are recorded, the working device is controlled to retract to the corresponding position, and the output rate is adjusted according to the gap width.

[0104] Example 2

[0105] like Figure 5 As shown, this embodiment provides a workpiece gap recognition and tracking device based on visual cue segmentation, including:

[0106] The sensor sensing module includes an infrared photoelectric sensor group, a front camera, and a rear camera, used to detect workpiece boundaries and acquire local images;

[0107] The central control and storage module is used to run the visual cue segmentation model and store the set of pixel visual cues;

[0108] The job execution interface module receives job instructions from the central control and storage module and parses them into control signals to drive the job unit in order to perform the colloidal material output operation.

[0109] The walking mechanism interface module receives walking commands from the central control and storage module and parses them into control signals to drive the displacement unit, thereby moving the sensor perception module and the work execution interface module.

[0110] In this embodiment, the sensor sensing module is integrated and installed on the actuator end bracket of the walking mechanism, moving synchronously with the mechanism. Specifically, the infrared sensor group is responsible for coarse positioning of the workpiece boundary, the front and rear dual cameras are responsible for gap identification and adhesive layer quality inspection, respectively, and each device is electrically connected to the central control and storage module to transmit data.

[0111] In this embodiment, the central control and storage module has a built-in storage unit that stores model weights, preset parameters, and a set of prompts; the central control and storage module has a built-in computing unit that is responsible for running the segmentation algorithm, trajectory calculation, Kalman filtering, and the entire process scheduling.

[0112] In this embodiment, the job execution interface module is responsible for parsing the job instructions (such as flow rate setpoint, valve opening, and start / stop logic) issued by the central control and storage module into execution signals that can directly drive the motor or valve. It integrates a PWM waveform generator, which can precisely control the PWM frequency and pulse width.

[0113] In this embodiment, the walking mechanism interface module serves as a motion control hub, receiving the three-dimensional trajectory coordinates and speed planning parameters of the gap centerline from the central control and storage module.

[0114] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Based on the technical essence of the present invention, any simple modifications, equivalent substitutions, and improvements made to the above embodiments within the spirit and principles of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method for workpiece gap recognition and tracking based on visual cue segmentation, characterized in that, Includes the following steps: S1. The workpiece boundary is detected by the infrared photoelectric sensor group, and the working device equipped with the front camera and the rear camera is moved to the detection area of ​​the workpiece edge to complete the coarse positioning of the gap area. S2. The front-facing camera captures a partial image of the area to be detected and labels a set of visual cues for each pixel. S3. Input the front local image and pixel visual cue set into the visual cue segmentation model, and output the front gap mask image. The visual cue segmentation model in S3 supports pixel-type visual cue input and adopts an open-domain segmentation architecture with streaming memory mechanism. Internally, it includes an image encoder, a cue encoder, a mask decoder, a memory encoder, a memory bank, and a memory attention module. The memory bank is configured to be written with gating based on the consistency check result of the Kalman filter predicted trajectory. When the consistency is lower than a preset threshold, writing the current frame to the memory bank is prohibited. S4. The position and shape information of the gap are obtained by parsing the front gap mask image. The operating device is controlled to move along the gap and output colloidal material output operation command. S5. During the movement of the working device, the continuously acquired sequence of previous local image frames is input into the visual cue segmentation model to dynamically track the position of the gap in each frame. The trajectory of the gap at the next moment is predicted by Kalman filtering. Based on the prediction results, the memory of the visual cue segmentation model is written with consistency gating to control the working device to move continuously along the gap. S5 further includes: S51. During the movement of the working device, the front-facing camera continuously captures front-facing local images of the area to be detected, forming a continuous image frame sequence. S52. Input the image frame sequence frame by frame into the visual cue segmentation model. Through the memory encoder, memory bank and memory attention module in the model, use the gap mask output from the previous frame as a cue, reuse the gap features of the historical frame, and dynamically track the gap position in each frame of the image. S53. Perform a spatial consistency check between the centerline extracted from the gap mask of the current frame output and the predicted trajectory of the Kalman filter at the same time. If the consistency is lower than the preset threshold, the memory encoder is prohibited from writing its mask and features into the memory bank; otherwise, writing is allowed. S54. If the spatial consistency of the current frame is greater than or equal to the preset threshold, the coordinates of the gap centerline obtained frame by frame are used as valid observations and input into the Kalman filter; if the consistency is lower than the preset threshold, the current observation is deemed invalid and discarded. S55. If valid observations exist, the system state is updated based on the valid observations and the trajectory position of the gap centerline is predicted at the next moment. If no valid observations exist, the observation update step is skipped and the trajectory position at the next moment is predicted directly based on the Kalman filter. S56. Based on the estimated trajectory position and the current position deviation of the working device, output a travel control command to control the working device to continuously move along the gap path. S6. The rear camera captures a local image of the rear area after the operation, inputs it into the visual cueing segmentation model to obtain the rear cover mask image, and calculates the gap coverage to verify the adhesive layer coverage.

2. The workpiece gap recognition and tracking method based on visual cue segmentation according to claim 1, characterized in that, The infrared photoelectric sensor group in S1 includes a vertical infrared photoelectric sensor and multiple horizontal infrared photoelectric sensors.

3. The workpiece gap recognition and tracking method based on visual cue segmentation according to claim 2, characterized in that, S1 further includes the following steps: S11. Using a vertical infrared photoelectric sensor, control the working device equipped with a front camera and a rear camera to move downwards until the horizontal surface of the workpiece is detected. S12. The horizontal infrared photoelectric sensor controls the horizontal movement of the working device until the vertical boundary of the workpiece is detected. S13. The moving work device makes all infrared photoelectric sensors in a non-triggered state. At this time, the work device is located above the gap between the workpieces, and completes the coarse positioning of the gap area.

4. The workpiece gap recognition and tracking method based on visual cue segmentation according to claim 1, characterized in that, The metadata of the front-facing local image in S2 includes the calibration parameters of the front-facing camera.

5. The workpiece gap recognition and tracking method based on visual cue segmentation according to claim 1, characterized in that, The set of pixel visual cues in S2 includes positive pixel cues representing gap regions and negative pixel cues representing background regions.

6. The workpiece gap recognition and tracking method based on visual cue segmentation according to claim 1, characterized in that, S4 further includes the following steps: S41. Extract the gap outline and gap center line from the front gap mask image, and calculate the gap position coordinates using the gap pixel coordinates and the front camera calibration parameters. S42. Control the working device equipped with a front camera and a rear camera to move to the starting coordinate of the gap. S43. Control the working device to move along the trajectory of the center line of the gap; S44. Adjust the discharge flow rate and moving speed parameters according to the gap width and curvature in the gap shape parameters at the current position.

7. The workpiece gap recognition and tracking method based on visual cue segmentation according to claim 1, characterized in that, S6 further includes the following steps: S61. Acquire a partial rear image after the operation using the rear camera; S62. A set of visual cues for marking colloidal regions on a rear local image; S63. Input the rear local image and the set of visual cues for the pixels of the colloidal region into the visual cues segmentation model to obtain the rear overlay mask image; S64. Calculate the gap coverage of the back cover mask image and the front gap mask image. The gap coverage is the ratio of the intersection area of ​​the two masks to the area of ​​the front gap mask. S65. If the gap coverage is greater than or equal to the preset threshold, the adhesive layer coverage verification is passed; if it is less than the preset threshold, the verification is failed, and the operating device is controlled to retract and perform the adhesive replenishment action.

8. A workpiece gap recognition and tracking device based on visual cue segmentation, characterized in that, The method for performing the workpiece gap recognition and tracking method based on visual cue segmentation as described in any one of claims 1 to 7 includes: The sensor sensing module includes an infrared photoelectric sensor group, a front camera, and a rear camera, used to detect workpiece boundaries and acquire local images; The central control and storage module is used to run the visual cue segmentation model and store the set of pixel visual cues; The job execution interface module receives job instructions from the central control and storage module and parses them into control signals to drive the job unit in order to perform the colloidal material output operation. The walking mechanism interface module receives walking commands from the central control and storage module and parses them into control signals to drive the displacement unit, thereby moving the sensor perception module and the work execution interface module.

Images