Chip mounter automatic control method and system

Abstract

The invention relates to the field of chip mounter control, and particularly discloses an automatic control method and system for a chip mounter, which aims at accidental errors possibly existing in a single model by simultaneously executing regression prediction of suction nozzle center coordinates and semantic segmentation of suction nozzle contours. Through carrying out consistency verification and weighted fusion on prediction results of two tasks of regression and segmentation, high precision and high reliability of a final output coordinate are ensured, and a root problem of unstable vision measurement is effectively solved. Furthermore, in order to improve the feature extraction quality of the double-task model, feature alignment based on spatial distribution symmetric fractions is carried out by utilizing the physical prior of central symmetry of a suction nozzle, so that a model encoder can generate shared features which are more friendly to regression and segmentation tasks, and the positioning precision of the model is fundamentally improved. And finally, inputting an accurate visual positioning result into a PID compensation control system to realize closed-loop accurate compensation of the suction nozzle attitude error.

Description

Technical Field

[0001] This application relates to the field of pick and place machine control, and more specifically, to an automatic control method and system for a pick and place machine. Background Technology

[0002] Pick-and-place machines, also known as surface mount systems, are core automated equipment in the electronics manufacturing industry, used to precisely place surface mount components onto the pads of printed circuit boards. With the rapid development of electronic products towards higher density, miniaturization, and higher performance, higher demands are placed on the placement accuracy and speed of pick-and-place machines. During high-speed placement, the posture of the pick-and-place machine nozzle can experience slight deviations due to mechanical movement, vibration, and long-term wear. If these deviations are not corrected in time, they will directly lead to component placement misalignment, seriously affecting product yield and reliability.

[0003] To address the aforementioned issues, existing technologies typically employ machine vision-based solutions to detect the center position of the nozzle. Traditional methods largely rely on image processing algorithms, such as Hough circle transform and template matching. However, these methods face numerous challenges in practical applications. For instance, due to the diverse models and sizes of nozzles, and the significant variations in image contrast and reflectivity under different lighting conditions, traditional algorithms exhibit poor robustness. Particularly when the nozzle is contaminated, worn, or has a complex background, the algorithm is prone to false detections or decreased positioning accuracy, failing to meet the sub-pixel accuracy requirements of modern pick-and-place machines. Furthermore, these algorithms often require tedious manual parameter adjustments for different nozzle models, resulting in low flexibility and automation. Moreover, errors in visual inspection are coupled with inherent rotational eccentricity errors and mechanical vibrations within the nozzle itself. If the visual measurement itself is unstable, the subsequent compensation and control system loses accurate input, potentially introducing even greater errors and leading to negative optimization.

[0004] Therefore, an optimized automatic control scheme for the pick-and-place machine is desired. Summary of the Invention

[0005] To address the aforementioned technical problems, this application is proposed. Embodiments of this application provide an automatic control method and system for a chip mounter.

[0006] According to one aspect of this application, an automatic control method for a pick-and-place machine is provided, comprising: The camera exposure is triggered when the nozzle moves below the camera's field of view to obtain the original nozzle image; The original nozzle image is cropped and pixel values ​​are normalized to obtain the nozzle ROI image; Perform dual-task joint model inference on the nozzle ROI image to obtain the nozzle regression prediction coordinates and nozzle segmentation mask; Based on the nozzle segmentation mask, semantic segmentation and centroid calculation are performed on the nozzle ROI image to obtain the nozzle segmentation prediction coordinates; The results of nozzle regression prediction coordinates and nozzle segmentation prediction coordinates are fused and consistency verified to obtain the final nozzle coordinates; Transform the final coordinates of the nozzle to the world coordinate system to obtain the final world coordinates of the nozzle; The deviation between the final world coordinates of the nozzle and the world coordinates of the theoretical center point of the camera is calculated to obtain the attitude error vector; The attitude error vector is input into a PID-based compensation control system to obtain an operating compensation signal.

[0007] According to another aspect of this application, an automatic control system for a pick-and-place machine is provided, comprising: The nozzle image acquisition module is used to trigger camera exposure to obtain the original nozzle image when the nozzle moves below the camera's field of view. The nozzle image preprocessing module is used to crop and normalize the pixel values ​​of the original nozzle image to obtain the nozzle ROI image; The nozzle segmentation mask module is used to perform dual-task joint model inference on the nozzle ROI image to obtain the nozzle regression prediction coordinates and the nozzle segmentation mask. The nozzle segmentation coordinate calculation module is used to perform semantic segmentation and centroid calculation on the nozzle ROI image based on the nozzle segmentation mask to obtain the nozzle segmentation prediction coordinates; The nozzle coordinate fusion module is used to fuse and verify the consistency of the nozzle regression prediction coordinates and the nozzle segmentation prediction coordinates to obtain the final nozzle coordinates. A coordinate system transformation module is used to transform the final coordinates of the suction nozzle to the world coordinate system to obtain the final world coordinates of the suction nozzle. The attitude deviation calculation module is used to calculate the deviation between the final world coordinates of the nozzle and the world coordinates of the theoretical center point of the camera to obtain the attitude error vector. The compensation signal generation module is used to input the attitude error vector into the PID-based compensation control system to obtain the operation compensation signal.

[0008] Compared with existing technologies, this application provides an automatic control method and system for a pick-and-place machine. It constructs a dual-task joint learning framework, simultaneously performing regression prediction of the nozzle center coordinates and semantic segmentation of the nozzle contour. The regression task enables rapid localization, while the segmentation task accurately depicts the geometric contour, creating a complementary advantage. Secondly, addressing the potential for random errors in a single model, this solution performs consistency verification and weighted fusion of the prediction results from both regression and segmentation tasks. This dual verification mechanism ensures high accuracy and reliability of the final output coordinates, effectively solving the root cause of visual measurement instability. Furthermore, to improve the feature extraction quality of the dual-task model, a feature alignment strategy based on spatially distributed symmetry fractions is proposed, utilizing the physical prior of the nozzle's central symmetry. This ensures the model encoder generates shared features that are more compatible with both regression and segmentation tasks, fundamentally improving the model's localization accuracy. Finally, the accurate visual localization results are input into a PID compensation control system to achieve closed-loop precise compensation for nozzle posture errors, effectively addressing error coupling caused by mechanical vibration and rotational eccentricity, thus ensuring the placement accuracy and yield of the pick-and-place machine under high-speed operation. Attached Figure Description

[0009] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0010] Figure 1 This is a flowchart of an automatic control method for a pick-and-place machine according to an embodiment of this application; Figure 2 This is a schematic diagram of the data flow in the automatic control method for a pick-and-place machine according to an embodiment of this application; Figure 3 This is a flowchart of an embodiment of the automatic control method for a pick-and-place machine according to an embodiment of this application, which performs dual-task joint model inference on the nozzle ROI image to obtain nozzle regression prediction coordinates and nozzle segmentation mask; Figure 4 This is a flowchart of another embodiment of the automatic control method for a pick-and-place machine according to the present application, which performs dual-task joint model inference on the nozzle ROI image to obtain nozzle regression prediction coordinates and nozzle segmentation mask; Figure 5 This is a flowchart illustrating the automatic control method for a pick-and-place machine according to an embodiment of this application, which performs semantic segmentation and centroid calculation on the nozzle ROI image to obtain the nozzle segmentation prediction coordinates. Figure 6This is a block diagram of an automatic control system for a pick-and-place machine according to an embodiment of this application. Detailed Implementation

[0011] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0012] As indicated in this application and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not specifically singular and may include plural forms. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0013] While this application makes various references to certain modules of the systems according to embodiments of this application, any number of different modules can be used and run on user terminals and / or servers. The modules described are merely illustrative, and different aspects of the systems and methods may use different modules.

[0014] Flowcharts are used in this application to illustrate the operations performed by the system according to embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, various steps can be processed in reverse order or simultaneously as needed. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0015] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0016] In the existing technology, the visual positioning of the pick-and-place machine nozzle is easily affected by factors such as changes in lighting, nozzle contamination, and model differences. This results in insufficient positioning accuracy and poor stability of traditional image algorithms or even single deep learning models. Such inaccurate visual measurement results are coupled with errors such as mechanical vibration, making subsequent compensation control difficult to achieve and may even introduce new errors.

[0017] To address the aforementioned technical problems, this application proposes an automatic control method for a pick-and-place machine. The core of this method lies in obtaining high-precision nozzle center coordinates through a cleverly designed dual-task joint deep learning model. Specifically, when the nozzle moves below the camera, the system first captures and preprocesses the nozzle image to obtain a standardized Region of Interest (ROI) image. Subsequently, this ROI image is input into the dual-task joint model, which includes a shared encoder and two parallel task heads: a regression head and a segmentation head. The encoder extracts deep, robust visual features from the image; the regression head directly predicts the nozzle center coordinates based on these features, i.e., the nozzle regression prediction coordinates; simultaneously, the segmentation head performs pixel-level classification of the image, generating an accurate nozzle segmentation mask. Then, the system uses this mask for morphological processing and geometric moment calculation to obtain another high-precision center coordinate, i.e., the nozzle segmentation prediction coordinates. To ensure the ultimate reliability of the final result, a result fusion and consistency verification mechanism is introduced: by comparing the differences between the regression coordinates and the segmentation coordinates, if they are sufficiently close, it proves that both predictions are valid, and the system performs a weighted fusion to obtain a double-verified final nozzle coordinate. Finally, this high-precision coordinate is transformed to the world coordinate system and compared with the theoretical center point of the camera to calculate the accurate attitude error vector, which is then fed into the PID compensation control system to generate a real-time operation compensation signal, thereby achieving precise closed-loop control of the nozzle attitude and effectively ensuring mounting accuracy and production yield.

[0018] Figure 1 This is a flowchart of an automatic control method for a chip mounter according to an embodiment of this application. Figure 2 This is a schematic diagram of the data flow in an automatic control method for a pick-and-place machine according to an embodiment of this application. Figure 1 and Figure 2As shown, the automatic control method for a pick-and-place machine according to an embodiment of this application includes the following steps: S100, triggering camera exposure when the nozzle moves below the camera's field of view to obtain an original nozzle image; S200, cropping and normalizing the pixel values ​​of the original nozzle image to obtain a nozzle ROI image; S300, performing dual-task joint model inference on the nozzle ROI image to obtain nozzle regression prediction coordinates and a nozzle segmentation mask; S400, performing semantic segmentation and centroid calculation on the nozzle ROI image based on the nozzle segmentation mask to obtain nozzle segmentation prediction coordinates; S500, performing result fusion and consistency verification on the nozzle regression prediction coordinates and nozzle segmentation prediction coordinates to obtain the final nozzle coordinates; S600, converting the final nozzle coordinates to the world coordinate system to obtain the final world coordinates of the nozzle; S700, calculating the deviation between the final world coordinates of the nozzle and the world coordinates of the theoretical center point of the camera to obtain an attitude error vector; S800, inputting the attitude error vector into a PID-based compensation control system to obtain a running compensation signal.

[0019] Specifically, in step S100, camera exposure is triggered when the nozzle moves below the camera's field of view to obtain an original nozzle image. It should be understood that, as the pick-and-place machine operates at high speed, the nozzle, being a moving component, experiences real-time position changes. If continuous exposure or asynchronous shooting methods are used, motion blur can easily lead to image quality degradation, or improper shooting timing may prevent the capture of a complete nozzle image, thus affecting the accuracy of subsequent visual positioning. Therefore, in the technical solution of this application, camera exposure is triggered only when the nozzle moves below the camera's field of view, ensuring accurate imaging at the moment the nozzle is in the optimal observation position. This effectively avoids motion blur, obtaining a clear and complete original nozzle image, providing a high-quality data foundation for subsequent high-precision coordinate extraction and attitude error calculation.

[0020] More specifically, in a concrete example of this application, this step is implemented as follows: First, the motion control system of the pick-and-place machine drives the nozzle to move along a predetermined trajectory according to a preset placement program. The motion control system monitors the nozzle's position in the world coordinate system in real time. Second, the motion control system pre-calibrates and stores the camera's mounting position and the coordinates of its field of view center in the world coordinate system. Next, during execution, the motion control system continuously monitors the nozzle's real-time coordinates and compares them with the coordinates of the camera's field of view center. Finally, when the system determines that the nozzle's center coordinates are about to enter or have already entered the predetermined trigger area of ​​the camera's field of view, the motion control system immediately sends a hardware or software trigger signal to the camera control module. This signal drives the camera to complete an instantaneous exposure and image acquisition, thereby obtaining a clear original image of the nozzle, which is then transmitted for subsequent analysis.

[0021] Specifically, in step S200, the original nozzle image is cropped and its pixel values ​​are normalized to obtain the nozzle ROI image. It should be understood that since the original nozzle image captured by the camera is relatively large, it contains a large amount of background areas unrelated to the nozzle itself. This redundant information not only increases the computational burden on the subsequent deep learning model but may also introduce noise interference, affecting the model's extraction of key nozzle features. Furthermore, different lighting conditions or camera parameter settings can lead to differences in the range and distribution of pixel values ​​in the image. This inconsistency reduces the model's convergence speed and generalization ability. Therefore, in the technical solution of this application, the original nozzle image is further cropped and its pixel values ​​are normalized to remove irrelevant background and unify the data scale. This reduces computational load, improves processing efficiency, and provides the deep learning model with a fixed-size, uniformly distributed input, thereby enhancing the stability of model training and the accuracy of final localization.

[0022] More specifically, in a specific example of this application, this step is implemented as follows: First, a cropping operation is performed. Based on the approximate position of the nozzle in the camera's field of view, a rectangular area is cropped with that position as the center and a preset fixed size (e.g., 224x224 pixels). This size setting must ensure that it can completely contain various types of nozzles and the limited contextual information around them, while maximizing the exclusion of irrelevant background. The cropped image is the region of interest. Second, pixel value normalization processing is performed. The system performs a numerical transformation on each pixel in the cropped nozzle ROI image. In particular, in a specific example of this application, the normalization method is to linearly map the pixel value from the original integer range of [0, 255] to the floating-point range of [0, 1] or [-1, 1]. For example, min-max normalization is used, and its calculation formula is: new pixel value = (original pixel value - 0) / (255 - 0). After these two steps, a nozzle ROI image with uniform size and standardized pixel value range is finally obtained, which can be directly used as input for the subsequent dual-task joint model.

[0023] Specifically, in step S300, a dual-task joint model inference is performed on the nozzle ROI image to obtain the nozzle regression prediction coordinates and the nozzle segmentation mask. It should be understood that single visual localization tasks have inherent limitations. For example, simple coordinate regression methods are sensitive to the global position of an image but may ignore fine contour details, while simple semantic segmentation methods, although accurately depicting contours, may be insensitive to global position. Both methods may produce deviations when facing complex conditions such as nozzle contamination or glare. Therefore, in the technical solution of this application, a dual-task joint model is further designed and adopted for inference. This model can execute regression and segmentation tasks in parallel, thereby simultaneously parsing nozzle image information from two different but complementary dimensions. This allows for the comprehensive utilization of the regression task's direct perception of position and the segmentation task's precise shape recognition, providing two independent and mutually verifiable localization results for subsequent result fusion and verification, thereby significantly improving the robustness and final accuracy of localization.

[0024] Figure 3 This is a flowchart illustrating an embodiment of the automatic control method for a pick-and-place machine according to an embodiment of this application, which involves performing dual-task joint model inference on the nozzle ROI image to obtain nozzle regression prediction coordinates and a nozzle segmentation mask. Figure 3 As shown, step S300 includes: S310, inputting the nozzle ROI image into the encoder of the dual-task joint learning model to obtain a visual shared feature map of the nozzle; S320, inputting the visual shared feature map of the nozzle into the regression head of the dual-task joint learning model to obtain the regression prediction coordinates of the nozzle; S330, inputting the visual shared feature map of the nozzle into the segmentation head of the dual-task joint learning model to obtain the nozzle segmentation mask.

[0025] Specifically, in this embodiment, step S310 involves inputting the nozzle ROI image into the encoder of the dual-task joint learning model to obtain a shared visual feature map of the nozzle. It should be understood that although the regression task (predicting coordinates) and the segmentation task (depicting contours) have different objectives, they both rely on a deep understanding of the same nozzle ROI image, such as identifying the nozzle's edges, holes, and overall structure. Designing a separate feature extraction network for each task would result in a large amount of repetitive computation, significantly increasing the model's complexity and inference time, which does not meet the high-efficiency industrial application requirements of pick-and-place machines. Therefore, in the technical solution of this application, a shared encoder structure is used, inputting the nozzle ROI image into this encoder to extract a common and efficient visual feature representation for the subsequent regression and segmentation tasks. This allows for deep mining of the core image information through a single forward propagation, greatly improving computational efficiency and utilizing the regularization effect of multi-task learning, enabling the encoder to learn features that are more beneficial and robust to both tasks, thereby improving the overall model performance.

[0026] More specifically, in a concrete example of this application, the encoding process of the nozzle ROI image includes: First, a pre-processed nozzle ROI image (e.g., a 224x224x3 tensor) is used as input data. Second, this input tensor is fed into a pre-trained MobileNetV3 model as an encoder. The MobileNetV3 model contains a series of carefully designed inverted residual blocks and attention mechanisms. The input image propagates layer by layer in the network, undergoing calculations through depthwise separable convolutions, linear bottleneck layers, and non-linear activation functions (such as h-swish), gradually reducing its spatial size while gradually increasing the number of channels (feature dimensions). This process essentially involves continuously compressing and refining image information, gradually abstracting from simple features such as edges and colors at the bottom layer to high-level semantic features at the top layer that can characterize the geometric structure and position of the nozzle. Finally, at the end of the MobileNetV3 network, the output of a certain intermediate or last layer is extracted; this output is a high-dimensional tensor, which is the visual shared feature map of the nozzle. This feature map condenses all the information in the original image that is crucial for locating the nozzle center and is distributed to subsequent regression and segmentation heads for task processing.

[0027] Specifically, in this embodiment, step S320 involves inputting the nozzle visual shared feature map into the regression head of the dual-task joint learning model to obtain the nozzle regression prediction coordinates. It should be understood that the nozzle visual shared feature map output by the encoder is a high-dimensional abstract data representation containing rich spatial and semantic information; it is not itself a direct coordinate value. To parse the precise location of the nozzle center point from this complex feature map, a mapping from feature space to coordinate space is required. Therefore, in this technical solution, the nozzle visual shared feature map is further input into a regression head to decode and transform the features, directly predicting the two-dimensional coordinates of the nozzle center. This enables end-to-end, rapid image-to-coordinate localization, providing the system with a direct, quantitative prediction result regarding the nozzle position, meeting the high real-time requirements of the pick-and-place machine.

[0028] More specifically, in a concrete example of this application: First, a shared visual feature map of the nozzle, generated by the encoder of a dual-task joint learning model, is received. This feature map is a three-dimensional tensor (e.g., 7x7x960). Second, the feature map is flattened. This operation rearranges the three-dimensional feature map tensor into a one-dimensional feature vector. Next, this long vector is fed as input into a regression head consisting of one or more fully connected layers. Each fully connected layer performs a linear transformation and non-linear activation on the input feature vector to further integrate global information and learn the complex non-linear relationship between features and coordinates. Finally, the last layer of the regression head is a fully connected layer without an activation function, with its output nodes set to 2. These two output values ​​correspond to the x and y coordinates of the nozzle center in the ROI image coordinate system, i.e., the nozzle regression prediction coordinates.

[0029] Specifically, in this embodiment, step S330 involves inputting the nozzle visual shared feature map into the segmentation head of the dual-task joint learning model to obtain the nozzle segmentation mask. It should be understood that since the nozzle visual shared feature map output by the encoder is a downsampled, highly abstract feature representation, its spatial resolution is far lower than the original input image, and it cannot be directly used for pixel-level fine-grained classification. To obtain a segmentation result that accurately depicts the nozzle outline, these low-resolution feature maps containing high-level semantic information must be restored to the original image size, and each pixel must be assigned a specific category. Therefore, in this technical solution, the nozzle visual shared feature map is further input into a segmentation head to perform upsampling and pixel-level classification, generating a nozzle segmentation mask of the same size as the input ROI image. This fully utilizes the high-level semantic information in the shared feature map, accurately identifying each pixel in the image belonging to the nozzle, providing a solid geometric basis for subsequently obtaining high-precision coordinates through centroid calculation.

[0030] More specifically, in a concrete example of this application: First, the nozzle visual shared feature map (e.g., a 7x7x960 tensor) output by the shared encoder is fed into the U-Net upsampling decoder, which serves as the segmentation head. This decoder consists of a series of upsampling modules. Second, in each upsampling module of the decoder, the spatial size of the feature map is progressively enlarged (e.g., from 7x7 to 14x14, then to 28x28, until it is restored to 224x224) through operations such as transposed convolution or bilinear interpolation. To compensate for the spatial detail lost during downsampling, a key structure of the U-Net decoder is a skip connection, which concatenates the feature map obtained during upsampling with a feature map of the corresponding layer of the encoder that has the same spatial size but retains more detail. This process integrates deep semantic information and shallow detail information. Finally, after multiple levels of upsampling and skip connections, the decoder outputs a feature map of the same size as the original ROI image. This map is then passed through a 1x1 convolutional layer to reduce the number of channels to 1, and a sigmoid activation function is applied to generate a single-channel probability map. The value of each pixel in this map ranges from 0 to 1, representing the probability that the pixel belongs to the nozzle region. This probability map is the nozzle segmentation mask.

[0031] Figure 4 This is a flowchart illustrating another embodiment of the automatic control method for a pick-and-place machine according to an embodiment of this application, which involves performing dual-task joint model inference on the nozzle ROI image to obtain nozzle regression prediction coordinates and a nozzle segmentation mask. Figure 4 As shown, step S300 includes: S301, passing the nozzle ROI image through an image feature encoder based on a pyramid network to obtain a shallow feature map, a mid-level feature map, and a deep feature map of the nozzle ROI; S302, aligning the shallow feature map, the mid-level feature map, and the deep feature map of the nozzle ROI to obtain aligned shallow feature map, aligned mid-level feature map, and aligned deep feature map of the nozzle ROI; S303, calculating the positional summation between the aligned shallow feature map, the aligned mid-level feature map, and the aligned deep feature map of the nozzle ROI to obtain the visually shared feature map of the nozzle.

[0032] Accordingly, in step S302, the shallow feature map, the middle feature map, and the deep feature map of the nozzle ROI are aligned to obtain the aligned shallow feature map, the aligned middle feature map, and the aligned deep feature map of the nozzle ROI. Specifically, in this embodiment of the application, it includes: calculating the spatial distribution symmetry scores of the shallow feature map, the middle feature map, and the deep feature map of the nozzle ROI to obtain the shallow spatial distribution matrix, the middle spatial distribution matrix, and the deep spatial distribution matrix of the nozzle ROI; and performing centroid alignment based on the spatial distribution symmetry scores of the shallow feature map, the middle feature map, and the deep feature map of the nozzle ROI to obtain the aligned shallow feature map, the aligned middle feature map, and the aligned deep feature map of the nozzle ROI.

[0033] More specifically, for the encoder of the dual-task joint learning model, if a sequential encoder (such as ResNet or MobileNet) is used, the receptive field can be increased by pooling or stride convolution to extract high-level semantic information during the layer-by-layer feature extraction process. This is friendly to the regression head, but lossy downsampling will lose the spatial resolution and details of the feature map, affecting the accuracy of the segmentation head at the pixel level.

[0034] Based on this, the encoder of the dual-task joint learning model can employ a feature pyramid network to obtain feature maps of different depths. Here, the shallow feature map of the nozzle ROI is used as an example. ), nozzle ROI mid-layer feature map ( ) and deep feature map of nozzle ROI ( For example, for feature maps at different depths... , and It can be based on Convolutional channel dimensionality reduction and size enlargement based on upsampling (e.g., bilinear interpolation) align the spatial scale. However, considering that the task of the dual-task joint learning model described here is essentially to understand the geometric characteristics of the nozzle orifice, and the most crucial physical (geometric) characteristic of the nozzle orifice lies in its central symmetry, it is desirable to approach the feature map from this perspective. , and Align them.

[0035] Specifically, for feature maps , and It is argued that if candidate points in the feature map along the feature distribution dimension of the feature matrix space are taken as ideal centers, then feature vectors centered at these points and at different radii and angles should exhibit a certain degree of symmetry. Therefore, a spatial distribution symmetry fraction is introduced during feature alignment. ;in These are the coordinates of each point on the feature matrix of the feature map, i.e., the coordinates of the candidate center; It is the first A sampling radius, for example , and It is the number of radii. It is the first Each sampling angle, for example, at Evenly spaced between That is, corresponding to coordinates The feature vectors along the channel dimension of the feature map, and This represents the cosine similarity.

[0036] That is, targeting the candidate center On both sides, along the same diameter on the matrix plane, at a distance of The symmetry of the channel feature vectors of two points is measured. Clearly, if the candidate point is the center of the nozzle hole, then the features on both sides (e.g., both inside the hole or both on the edge of the hole wall) are semantically consistent, and therefore their feature vector directions should also be similar, i.e., the cosine similarity approaches 1. Conversely, if the candidate point is in an unstructured background region, then the features on both sides are random, and the cosine similarity will be very low. Therefore, the spatial distribution symmetry score... By using average pooling across multiple radii and multiple directions of symmetry metrics, a comprehensive evaluation of the point was achieved. The reliability of the center of symmetry.

[0037] Thus, for feature maps , and In essence, this yields the shallow spatial distribution matrix of the nozzle ROI, represented by spatially distributed symmetric fractions. ), Spatial distribution matrix of nozzle ROI in the middle layer ( ) and the deep spatial distribution matrix of the nozzle ROI ( Furthermore, the eigenvalues ​​of the matrix are all represented in probabilistic form. Considering the consistency with centroid calculation based on the nozzle segmentation mask, the spatial distribution matrix can be targeted... , and Calculate its centroid and base it on the width of the eigenmatrix. Alignment in the spatial dimension of height.

[0038] Specifically, firstly, the geometric centroids of the generated shallow, middle, and deep spatial distribution matrices of the nozzle ROI are calculated. Taking the shallow spatial distribution matrix as an example, the system treats it as a two-dimensional probability distribution map. By calculating its zeroth moment (i.e., the sum of all element values) and first moment (i.e., the sum of the products of each element value and its coordinates), and based on the centroid formula: centroid coordinates = (first moment in the x-direction / zeroth moment, first moment in the y-direction / zeroth moment), the centroid coordinates representing the symmetry center of the shallow feature are accurately calculated. Similarly, the centroid coordinates of the mid- and deep features are calculated. and Next, the system spatially aligns the original shallow, mid, and deep feature maps of the nozzle ROI based on these centroids. This alignment operation uses a unified target point (e.g., the geometric center of the feature map canvas) as a reference, applying a translation transformation to each feature map to align its respective centroid (…). , , The feature map is precisely moved to the target point. After translation, due to the movement of the feature map, its edges will produce regions with empty feature values. Here, if the feature map... , and If the channel vectors at the aligned spatial location are empty, bilinear interpolation can be used for feature value filling. This involves a weighted average of the four nearest valid features around the empty location to generate smooth and continuous feature values. The final alignment and feature value filling process results in a shallow feature map of the aligned nozzle ROI. ), and the mid-layer feature map of the aligned nozzle ROI ( ) and the deep feature map of the aligned nozzle ROI ( A point-to-point fusion method is used based on a unified spatial scale to obtain the visual shared feature map of the suction nozzle. In this way, the centroid alignment based on the spatial distribution symmetry fraction, with physical meaning as a strong prior, not only provides an alignment benchmark for feature maps of different depths, but also provides a common anchor point for regression and segmentation tasks. This helps the feature distribution to obtain better localization based on spatial segmentation and centroid calculation, and this localization can in turn help the regression task through fine alignment of spatial distribution.

[0039] Specifically, in step S400, based on the nozzle segmentation mask, semantic segmentation and centroid calculation are performed on the nozzle ROI image to obtain the nozzle segmentation prediction coordinates. It should be understood that the nozzle segmentation mask output by the deep learning model is a probability map containing the probability that each pixel belongs to the nozzle region, rather than a definite geometric shape or coordinate point, and its edges may contain some noise or discontinuous isolated points. To extract a stable and accurate center coordinate from this probability map, it needs to be converted into a definite binary image and optimized, and finally its geometric center is calculated mathematically. Therefore, in the technical solution of this application, semantic segmentation and centroid calculation are further performed based on the nozzle segmentation mask to transform the model's pixel-level classification result into a quantified coordinate representing the physical center of the nozzle. This yields a localization result based on complete contour information, insensitive to local noise and reflections, providing a high-precision, source-independent reference coordinate for subsequent result fusion.

[0040] Figure 5 This is a flowchart illustrating the process of semantic segmentation and centroid calculation of the nozzle ROI image to obtain nozzle segmentation prediction coordinates in the automatic control method for a pick-and-place machine according to an embodiment of this application. Figure 5 As shown, step S400 includes: S410, binarizing the nozzle segmentation mask based on a preset threshold to obtain a nozzle binary mask; S420, performing morphological operations on the nozzle binary mask to obtain a smoothed nozzle binary mask; S430, calculating the geometric moments of the smoothed nozzle binary mask to obtain the zeroth-order moment and the first-order moment; S440, calculating the centroid coordinates as the nozzle segmentation prediction coordinates based on the zeroth-order moment and the first-order moment.

[0041] Accordingly, in step S410, the nozzle segmentation mask is binarized based on a preset threshold to obtain a nozzle binary mask. It should be understood that since the nozzle segmentation mask output by the segmentation head is a continuous probability map, where each pixel value represents the probability of belonging to the nozzle region, rather than a definitive yes or no judgment, this probabilistic representation cannot be directly used for subsequent morphological operations and geometric moment calculations, because these operations require a binary image that clearly defines the foreground (nozzle) and background. Therefore, in the technical solution of this application, the nozzle segmentation mask is further binarized based on a preset threshold to transform the probabilistic segmentation result into a deterministic, discrete region division. This provides a clear, analyzable geometric shape for subsequent centroid calculation, and is a key intermediate step in realizing the transformation from pixel-level probability to precise coordinates.

[0042] More specifically, this step is implemented as follows: First, a single-channel nozzle segmentation mask, the same size as the nozzle ROI image generated by the segmentation head, is read from memory. The value range of each pixel in this mask is [0, 1]. Second, a preset fixed threshold is loaded. This threshold is a floating-point number between 0 and 1, for example, 0.5. This threshold represents the confidence standard for determining whether a pixel belongs to the nozzle. Next, the system creates a blank binary image of the same size as the segmentation mask, with all initial pixel values ​​set to 0. Then, each pixel (i,j) of the segmentation mask is traversed, its probability value P(i,j) is read, and compared with the preset threshold. Finally, the binary image is filled according to the comparison result: if P(i,j) is greater than or equal to the preset threshold, the pixel value at the corresponding position (i,j) in the binary image is set to 1 (representing the foreground); if P(i,j) is less than the threshold, the pixel value at that position remains 0 (representing the background). After the traversal is complete, the resulting binary image is the nozzle binary mask.

[0043] Accordingly, in step S420, morphological operations are performed on the nozzle binary mask to obtain a smoothed nozzle binary mask. It should be understood that although the nozzle binary mask obtained after binarization clarifies the foreground and background, it may still contain some imperfections caused by model prediction uncertainties or image noise, such as isolated noise points or burrs on the nozzle outline edge, or tiny holes inside the nozzle body. These imperfect geometric shapes directly affect the accuracy of subsequent centroid calculations, introducing unnecessary errors. Therefore, in the technical solution of this application, morphological operations are further performed on the nozzle binary mask to filter and optimize the binary image, repairing its geometric defects. This generates a nozzle mask with a smoother outline and a more complete region, thereby providing high-quality input for calculating a stable and accurate geometric center, improving the reliability of the final segmentation prediction coordinates.

[0044] More specifically, this step is implemented as follows: First, the system performs an opening operation on the nozzle binary mask. This operation consists of two basic operations: erosion followed by dilation. The erosion operation removes all bright pixel areas smaller than the structuring element in the image, effectively eliminating isolated noise and small burrs outside the binary mask. The subsequent dilation operation restores the remaining foreground area to near its original size. Second, after performing the opening operation, the system performs a closing operation on the resulting image. This operation is the reverse of the opening operation, consisting of dilation followed by erosion. The dilation operation fills small holes and gaps inside the foreground area, while the subsequent erosion operation restores the slightly thickened contours caused by dilation to their original state. By sequentially performing the opening and closing operations, the system effectively removes external noise and fills internal holes, resulting in the smoothed nozzle binary mask, with a complete nozzle area contour and smooth edges.

[0045] Accordingly, in step S430, the geometric moments of the smoothed nozzle binary mask are calculated to obtain the zeroth-order moment and the first-order moment. It should be understood that since the smoothed nozzle binary mask obtained through morphological operations is a set of pixels representing the nozzle outline, it does not directly provide the quantified information of the center coordinates. In order to accurately extract the physical center from this geometry, the technical solution of this application further calculates the geometric moments of the smoothed nozzle binary mask to quantify the spatial distribution of foreground pixels in the binary image. In this way, the geometric morphological information of the image can be transformed into a set of numerical values ​​with clear physical meaning (i.e., the zeroth-order moment and the first-order moment), laying a mathematical foundation for the subsequent direct calculation of the centroid coordinates using formulas, and serving as a key bridge connecting image processing and coordinate calculation.

[0046] More specifically, this step is implemented as follows: First, a smoothed nozzle binary mask is loaded into the computation unit. This mask is a two-dimensional matrix, where the value of the foreground pixel (nozzle region) is 1, and the value of the background pixel is 0. Second, the system initializes three accumulator variables: , The initial values ​​are all 0. Next, the system iterates through each pixel of the binary mask using a double loop, setting the coordinates of the current pixel as (…). x, y ), its pixel value I ( x, y In each step of the loop, the system performs the following accumulation operation: increments the pixel value... I ( x, y Accumulated to the zeroth moment In the middle; the x-coordinate and pixel value I ( x, y The product of ) (i.e. I ( x, y Accumulated to the first moment In the middle; the y-coordinate and pixel value I ( x, y The product of ) (i.e. I ( x, y Accumulated to the first moment In the middle. Finally, after traversing all pixels, the accumulator... The value is the zeroth moment, representing the total area of ​​the nozzle region; accumulator and The value of is the first moment, which represents the mass distribution of the nozzle region on the x-axis and y-axis, respectively.

[0047] Accordingly, in step S440, the centroid coordinates are calculated as the predicted segmentation coordinates of the nozzle based on the zeroth and first moments. It should be understood that since the zeroth and first moments calculated in the previous stage are integral quantities describing the overall distribution of the nozzle's binary mask, they are not directly usable coordinate values. To transform these abstract mathematical quantities into a specific two-dimensional coordinate point that can represent the geometric center of the nozzle, the technical solution of this application further calculates the centroid coordinates based on the zeroth and first moments, thereby completing the final transformation from the statistical characteristics of the image region to its physical center position. In this way, a precise, stable coordinate point with clear physical meaning can be obtained, which integrates all the information of the complete nozzle contour, serving as the final localization output of the segmentation task branch.

[0048] Specifically, in this embodiment of the application, step S440 includes: calculating the centroid coordinates based on the zeroth moment and the first moment using the following formula: ;in, It is the zeroth moment. Let be the first moment in the x-direction and Let the first moment be in the y-direction. The centroid coordinates represent the geometric center of the smooth nozzle binary mask. The x-coordinate of the centroid. The y-coordinate is the coordinate of the centroid.

[0049] Specifically, in step S500, the nozzle regression prediction coordinates and nozzle segmentation prediction coordinates are fused and their consistency verified to obtain the final nozzle coordinates. It should be understood that since the regression prediction coordinates and segmentation prediction coordinates are results from two different positioning methods—the regression method focuses on global position awareness, while the segmentation method focuses on fine contour analysis—each has its advantages and disadvantages, and may produce different deviations when facing complex working conditions. If only one is accepted, the full information of the dual-task model cannot be fully utilized, and positioning failure may occur when a single method fails. Therefore, in the technical solution of this application, the two independent prediction coordinates are further fused and their consistency verified to combine the advantages of both methods. Unreliable predictions are eliminated through mutual verification, thereby obtaining a more accurate and robust positioning result. This improves the positioning accuracy and stability of the pick-and-place machine under various interferences, ensuring the high reliability of the final output coordinates.

[0050] More specifically, in this embodiment, the process of fusing and verifying the results of the nozzle regression prediction coordinates and the nozzle segmentation prediction coordinates to obtain the final nozzle coordinates includes: First, calculating the Euclidean distance between the nozzle regression prediction coordinates and the nozzle segmentation prediction coordinates. This distance quantifies the degree of consistency between the nozzle regression prediction coordinates and the nozzle segmentation prediction coordinates obtained by the two methods regarding the prediction results. Then, comparing the calculated Euclidean distance with a preset consistency threshold (e.g., 5 pixels). This threshold is an acceptable maximum deviation range calibrated based on experimental data. Finally, the system makes a decision based on the comparison result: If the Euclidean distance is less than the consistency threshold, it indicates that the two prediction results corroborate each other and have high credibility. In this case, the weighted sum of the nozzle regression prediction coordinates and the nozzle segmentation prediction coordinates is calculated to obtain the final nozzle coordinates. If the Euclidean distance is greater than the threshold, it indicates that at least one prediction result may have a large deviation. The system can trigger an exception handling mechanism, such as discarding the measurement or using the one with higher confidence as the result.

[0051] Specifically, in step S600, the final coordinates of the nozzle are transformed to the world coordinate system to obtain the final world coordinates of the nozzle. It should be understood that the final coordinates of the nozzle obtained in the previous steps are defined in a two-dimensional image pixel coordinate system, with the unit being pixels and the origin located at the upper left corner of the image. This coordinate system is only valid for the frame image captured by the camera; it cannot be directly used to describe the actual position of the nozzle in the entire physical workspace of the pick-and-place machine, nor can it be unified with the coordinate reference of the pick-and-place machine's motion control system. Therefore, in the technical solution of this application, the final coordinates of the nozzle are further transformed to the world coordinate system to map the visual measurement results from pixel units to millimeter or micrometer units with actual physical meaning, and to unify them under the global coordinate reference of the pick-and-place machine. This allows the visual positioning results and the motion control commands of the robotic arm to be calculated and compared in the same coordinate system, which is a necessary prerequisite for realizing visual feedback closed-loop control.

[0052] More specifically, in a specific example of this application, this step is implemented as follows: First, camera calibration is pre-completed. This calibration process involves photographing a calibration board with known dimensions and world coordinates, and calculating the camera's intrinsic and extrinsic parameter matrices. The intrinsic parameter matrix describes the camera's own imaging characteristics, such as focal length and principal point position; the extrinsic parameter matrix describes the rotation and translation relationship of the camera coordinate system relative to the world coordinate system (i.e., the pick-and-place machine's mechanical coordinate system). Second, the system uses the final coordinates of the nozzle obtained in the previous stage as input. Next, the system uses the relationships obtained from camera calibration to establish a transformation equation from pixel coordinates to world coordinates. Since the nozzle is below the camera's field of view... The axis coordinates (i.e., the distance to the camera) are known or can be determined by an altitude sensor; let its depth in the camera coordinate system be... The system, based on the inverse process of camera imaging, uses the inverse of the intrinsic parameter matrix to convert pixel coordinates into three-dimensional coordinates in the camera coordinate system. , , Finally, the system uses the inverse transformation of the extrinsic parameter matrix to convert the coordinates in the camera coordinate system (…). , , Transform to the world coordinate system to obtain the final world coordinates of the suction nozzle. , , This world coordinate system precisely describes the three-dimensional physical position of the nozzle center within the pick-and-place machine's workspace.

[0053] Specifically, in step S700, the deviation between the final world coordinates of the nozzle and the world coordinates of the theoretical center point of the camera is calculated to obtain an attitude error vector. It should be understood that since the motion control system of the pick-and-place machine drives the nozzle movement based on an ideal model, it expects the nozzle to move precisely to the center of the camera's field of view. However, in actual high-speed operation, factors such as robotic arm vibration, transmission backlash, thermal expansion, and wear of the nozzle itself can cause the actual position of the nozzle to deviate from this theoretical target position. To accurately compensate for this deviation, it must first be quantified. Therefore, in the technical solution of this application, the deviation between the final world coordinates of the nozzle and the world coordinates of the theoretical center point of the camera is further calculated to obtain an attitude error vector that accurately describes the difference between the current actual position and the target position. In this way, an abstract, invisible attitude deviation can be transformed into a concrete, quantifiable instruction that can be used for calculation by the closed-loop control system, providing a clear target and basis for subsequent accurate compensation.

[0054] More specifically, in a particular example of this application: First, the final world coordinates of the nozzle are obtained from the previous calculation step. , , Secondly, the coordinates of the theoretical center point of the camera's field of view in the same world coordinate system are read from its configuration parameters or calibration data. These coordinates are fixed and represent the ideal target position of the nozzle under error-free conditions. Next, a vector subtraction operation is performed, that is, the world coordinates of the theoretical center point are subtracted from the actual world coordinates of the nozzle. Finally, the result of this subtraction operation is output as a three-dimensional vector. P=( , , The components of this vector represent the offset and direction of the nozzle relative to the ideal center position in the X, Y, and Z axes of the world coordinate system, respectively. It precisely quantifies the attitude error that needs to be corrected.

[0055] Specifically, in step S800, the attitude error vector is input into a PID-based compensation control system to obtain an operational compensation signal. It should be understood that since the attitude error vector calculated in the previous step is merely a static value describing the current state deviation, it cannot directly drive the placement machine's motor or actuators to adjust their motion. To transform this error information into dynamic control commands that can be understood and executed by the mechanical system, a mature control algorithm is needed to generate smooth and effective adjustment quantities based on the magnitude, trend, and accumulation of the error. Therefore, in the technical solution of this application, the attitude error vector is further input into a PID-based compensation control system to dynamically process the error and calculate a precise operational compensation signal for correcting the attitude deviation. This enables a fast, stable, and overshoot-free closed-loop feedback control, ensuring that the nozzle attitude can be accurately corrected to the ideal position, thereby fundamentally guaranteeing the accuracy and success rate of subsequent placement operations.

[0056] More specifically, in a particular example of this application: First, the attitude error vector... P=( , , Each component of the error (e.g., ) is treated as an independent input and fed into three parallel PID controllers, each responsible for compensation of one coordinate axis. Secondly, within each PID controller, the system processes the input error components (e.g., ) Perform three-way parallel computation: the proportional (P) stage directly applies the current error. The system multiplies the error by a proportional coefficient Kp to generate an instantaneous response proportional to the error. The integral (I) stage accumulates historical errors and multiplies them by an integral coefficient Ki to eliminate potential steady-state errors. The derivative (D) stage calculates the rate of change of the current error and multiplies it by a derivative coefficient Kd to predict the future trend of the error, thereby suppressing oscillations and accelerating the system response. Next, the system adds the calculation results (P, I, and D) of these three stages to obtain the total control output for that coordinate axis. Finally, the system combines the total control outputs of the X, Y, and Z axes from the three PID controllers to form a final operational compensation signal. This signal is an instruction containing the specific direction and amplitude of motion, which is sent to the underlying motion controller of the pick-and-place machine to drive the robotic arm actuator to make fine adjustments, thereby eliminating attitude errors.

[0057] In summary, the automatic control method for a pick-and-place machine according to the embodiments of this application is explained. It constructs a dual-task joint learning framework, simultaneously performing regression prediction of the nozzle center coordinates and semantic segmentation of the nozzle contour. The regression task enables rapid localization, while the segmentation task accurately depicts the geometric contour, with the two complementing each other. Secondly, addressing the potential for random errors in a single model, this solution performs consistency verification and weighted fusion of the prediction results from both regression and segmentation tasks. This dual verification mechanism ensures high accuracy and reliability of the final output coordinates, effectively solving the root cause of visual measurement instability. Furthermore, to improve the feature extraction quality of the dual-task model, a feature alignment strategy based on spatially distributed symmetry fractions is proposed, utilizing the physical prior of the nozzle's central symmetry. This ensures that the model encoder generates shared features that are more compatible with both regression and segmentation tasks, fundamentally improving the model's localization accuracy. Finally, the accurate visual localization results are input into the PID compensation control system to achieve closed-loop precise compensation for nozzle posture errors, effectively addressing error coupling such as mechanical vibration and rotational eccentricity, and ensuring the placement accuracy and yield of the pick-and-place machine under high-speed operation.

[0058] Furthermore, an automatic control system for a chip mounter is also provided.

[0059] Figure 6 This is a block diagram of an automatic control system for a pick-and-place machine according to an embodiment of this application. Figure 6 As shown, the automatic control system 100 for a pick-and-place machine according to an embodiment of this application includes: a nozzle image acquisition module 110, used to trigger camera exposure to obtain an original nozzle image when the nozzle moves below the camera's field of view; a nozzle image preprocessing module 120, used to crop and normalize the pixel values ​​of the original nozzle image to obtain a nozzle ROI image; a nozzle segmentation mask module 130, used to perform dual-task joint model inference on the nozzle ROI image to obtain nozzle regression prediction coordinates and a nozzle segmentation mask; and a nozzle segmentation coordinate calculation module 140, used to perform semantic segmentation on the nozzle ROI image based on the nozzle segmentation mask. The system performs segmentation and centroid calculations to obtain the nozzle segmentation prediction coordinates; a nozzle coordinate fusion module 150 is used to fuse and verify the consistency of the nozzle regression prediction coordinates and the nozzle segmentation prediction coordinates to obtain the final nozzle coordinates; a coordinate system transformation module 160 is used to transform the final nozzle coordinates to the world coordinate system to obtain the final world coordinates of the nozzle; an attitude deviation calculation module 170 is used to calculate the deviation between the final world coordinates of the nozzle and the world coordinates of the theoretical center point of the camera to obtain the attitude error vector; and a compensation signal generation module 180 is used to input the attitude error vector into a PID-based compensation control system to obtain an operating compensation signal.

[0060] As described above, the automatic control system 100 for a pick-and-place machine according to embodiments of this application can be implemented in various wireless terminals, such as servers with automatic control algorithms for pick-and-place machines. In one possible implementation, the automatic control system 100 for a pick-and-place machine according to embodiments of this application can be integrated into a wireless terminal as a software module and / or a hardware module. For example, the automatic control system 100 for a pick-and-place machine can be a software module in the operating system of the wireless terminal, or it can be an application developed for the wireless terminal; of course, the automatic control system 100 for a pick-and-place machine can also be one of many hardware modules of the wireless terminal.

[0061] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An automatic control method for a pick-and-place machine, characterized in that, include: The camera exposure is triggered when the nozzle moves below the camera's field of view to obtain the original nozzle image; The original nozzle image is cropped and pixel values ​​are normalized to obtain the nozzle ROI image; Perform dual-task joint model inference on the nozzle ROI image to obtain the nozzle regression prediction coordinates and nozzle segmentation mask; Based on the nozzle segmentation mask, semantic segmentation and centroid calculation are performed on the nozzle ROI image to obtain the nozzle segmentation prediction coordinates; The results of nozzle regression prediction coordinates and nozzle segmentation prediction coordinates are fused and consistency verified to obtain the final nozzle coordinates; Transform the final coordinates of the nozzle to the world coordinate system to obtain the final world coordinates of the nozzle; The deviation between the final world coordinates of the nozzle and the world coordinates of the theoretical center point of the camera is calculated to obtain the attitude error vector; The attitude error vector is input into a PID-based compensation control system to obtain an operating compensation signal.

2. The automatic control method for a pick-and-place machine according to claim 1, characterized in that, Performing dual-task joint model inference on the nozzle ROI image to obtain nozzle regression prediction coordinates and nozzle segmentation mask includes: The nozzle ROI image is input into the encoder of the dual-task joint learning model to obtain the nozzle visual shared feature map; The visual shared feature map of the suction nozzle is input into the regression head of the dual-task joint learning model to obtain the regression prediction coordinates of the suction nozzle; The visual shared feature map of the suction nozzle is input into the segmentation head of the dual-task joint learning model to obtain the suction nozzle segmentation mask.

3. The automatic control method for a pick-and-place machine according to claim 2, characterized in that, The encoder is a MobileNetV3 model, the regression head is a fully connected network, and the segmentation head is a U-Net upsampling decoder.

4. The automatic control method for a pick-and-place machine according to claim 2, characterized in that, Based on the nozzle segmentation mask, the nozzle ROI image is input into the encoder of the dual-task joint learning model to obtain the nozzle visual shared feature map, including: The nozzle ROI image is processed by an image feature encoder based on a pyramid network to obtain a shallow feature map, a middle feature map, and a deep feature map of the nozzle ROI. The shallow feature map, the middle feature map, and the deep feature map of the nozzle ROI are aligned to obtain the aligned shallow feature map, the aligned middle feature map, and the aligned deep feature map of the nozzle ROI. The positional summation of the shallow feature map, the middle feature map, and the deep feature map of the aligned nozzle ROI is calculated to obtain the visual shared feature map of the nozzle.

5. The automatic control method for a pick-and-place machine according to claim 4, characterized in that, Based on the nozzle segmentation mask, the shallow feature map, the mid-layer feature map, and the deep feature map of the nozzle ROI are aligned to obtain aligned shallow feature maps, mid-layer feature maps, and deep feature maps of the nozzle ROI, including: The spatial distribution symmetry fractions of the shallow feature map, the middle feature map, and the deep feature map of the nozzle ROI are calculated respectively to obtain the shallow spatial distribution matrix, the middle spatial distribution matrix, and the deep spatial distribution matrix of the nozzle ROI. Based on the shallow spatial distribution matrix, the middle spatial distribution matrix, and the deep spatial distribution matrix of the nozzle ROI, the shallow feature map, the middle feature map, and the deep feature map of the nozzle ROI are aligned with centroids based on spatial distribution symmetry fractions to obtain aligned shallow feature map, aligned middle feature map, and aligned deep feature map of the nozzle ROI.

6. The automatic control method for a pick-and-place machine according to claim 1, characterized in that, Based on the nozzle segmentation mask, semantic segmentation and centroid calculation are performed on the nozzle ROI image to obtain the nozzle segmentation prediction coordinates, including: The nozzle segmentation mask is binarized based on a preset threshold to obtain a nozzle binary mask; Perform morphological operations on the nozzle binary mask to obtain a smoothed nozzle binary mask; Calculate the geometric moments of the smooth nozzle binary mask to obtain the zeroth-order moment and the first-order moment; Based on the zeroth and first moments, the centroid coordinates are calculated as the predicted nozzle segmentation coordinates.

7. The automatic control method for a pick-and-place machine according to claim 6, characterized in that, Based on the zeroth and first moments, the centroid coordinates are calculated as the predicted nozzle segmentation coordinates, including: calculating the centroid coordinates based on the zeroth and first moments using the following formula, where the formula is: ;in, It is the zeroth moment. Let be the first moment in the x-direction and Let y be the first moment in the y direction.

8. The automatic control method for a pick-and-place machine according to claim 1, characterized in that, The nozzle regression prediction coordinates and nozzle segmentation prediction coordinates are fused and their consistency is verified to obtain the final nozzle coordinates, including: Calculate the Euclidean distance between the nozzle regression prediction coordinates and the nozzle segmentation prediction coordinates; In response to the Euclidean distance being less than the consistency threshold, a weighted sum of the nozzle regression prediction coordinates and the nozzle segmentation prediction coordinates is calculated to obtain the final nozzle coordinates.

9. An automatic control system for a chip mounter, characterized in that, include: The nozzle image acquisition module is used to trigger camera exposure to obtain the original nozzle image when the nozzle moves below the camera's field of view. The nozzle image preprocessing module is used to crop and normalize the pixel values ​​of the original nozzle image to obtain the nozzle ROI image; The nozzle segmentation mask module is used to perform dual-task joint model inference on the nozzle ROI image to obtain the nozzle regression prediction coordinates and the nozzle segmentation mask. The nozzle segmentation coordinate calculation module is used to perform semantic segmentation and centroid calculation on the nozzle ROI image based on the nozzle segmentation mask to obtain the nozzle segmentation prediction coordinates; The nozzle coordinate fusion module is used to fuse and verify the consistency of the nozzle regression prediction coordinates and the nozzle segmentation prediction coordinates to obtain the final nozzle coordinates. A coordinate system transformation module is used to transform the final coordinates of the suction nozzle to the world coordinate system to obtain the final world coordinates of the suction nozzle. The attitude deviation calculation module is used to calculate the deviation between the final world coordinates of the nozzle and the world coordinates of the theoretical center point of the camera to obtain the attitude error vector. The compensation signal generation module is used to input the attitude error vector into the PID-based compensation control system to obtain the operation compensation signal.

Images