An image recognition-based SMT chip mounter fault prediction method and system

Abstract

The application belongs to the technical field of fault identification, and discloses a kind of SMT chip mounter fault prediction method and system based on image recognition, including through the industrial camera of nozzle overhead position, movement side view position and PCB local overhead position set in the synchronous trigger of chip mounter main shaft encoder signal, the collection of each view angle image;Based on the coordinate mapping of each view angle image obtained based on the preset PCB reference positioning point, a physical coordinate canvas is constructed;The spatial position relationship between the nozzle, the component and the PCB pad is extracted in the physical coordinate canvas, a three-segment geometric correlation chain is constructed, and the deviation representation of each link of mounting is formed;Realize the visualization, structure and intelligent analysis of the whole process of SMT chip mounter mounting, not only improve the defect detection precision, but also enhance the fault tracing ability and prediction ability, so as to effectively improve the mounting quality and equipment reliability.

Description

Technical Field

[0001] This invention relates to the field of fault identification technology, and more specifically, to a fault prediction method and system for SMT pick-and-place machines based on image recognition. Background Technology

[0002] In existing technologies, fault detection and quality assessment of SMT placement machines typically rely on visual inspection from a single perspective or indirect judgment based on equipment parameters. However, the overall technical solution still has the following shortcomings:

[0003] First, existing methods mostly use single cameras or distributed visual acquisition methods, lacking a unified coordinate reference and multi-view data fusion mechanism. Data acquired from different perspectives and at different times are difficult to align with high precision, easily causing coordinate drift and error accumulation. This leads to inconsistent expression of the spatial relationship between the nozzle, components and PCB pads, thus affecting positioning accuracy and detection stability.

[0004] Secondly, traditional testing methods usually focus on the final inspection and analysis of the placement results, only judging the deviation of the final placement position of the components. They lack the ability to continuously model the entire process of "sucking-movement-placement", and cannot effectively distinguish the source of error, resulting in insufficient fault tracing ability and difficulty in supporting refined process optimization.

[0005] Furthermore, existing inspection methods based on top-down vision mainly acquire two-dimensional position information, making it difficult to perceive the three-dimensional spatial posture of components on the nozzle. They lack effective means to identify hidden defects such as component tilting, flipping, or improper adsorption, and problems often only become apparent after the mounting is completed, increasing the defect rate and rework costs.

[0006] In addition, existing technologies rely on preset parameters or empirical models to estimate the critical stress state during the mounting process. They lack direct means of observing physical responses such as solder paste deformation during the actual mounting process, which cannot truly reflect the uniformity of mounting pressure, resulting in deviations between the quality assessment results and the actual situation.

[0007] Finally, existing systems generally lack a dynamic determination mechanism for the existence of components at each stage of the process, making it difficult to achieve continuous monitoring throughout the entire process. They rely heavily on post-event detection for anomaly identification, which not only results in detection delays but also makes it difficult to accurately locate the specific stage at which the anomaly occurs, affecting the efficiency and accuracy of fault diagnosis.

[0008] In view of this, the present invention proposes an image recognition-based SMT pick-and-place machine fault prediction method and system to solve the above problems. Summary of the Invention

[0009] To overcome the aforementioned deficiencies of the prior art and to achieve the above objectives, the present invention provides the following technical solution: a fault prediction method for an SMT pick-and-place machine based on image recognition, comprising:

[0010] S1. The industrial camera, located at the top view position of the nozzle, the side view position of the motion, and the top view position of the PCB, is synchronously triggered by the encoder signal of the pick-and-place machine spindle to acquire images from various perspectives; the acquired images from various perspectives are mapped to coordinates based on the preset PCB reference positioning points to construct a physical coordinate canvas.

[0011] S2. Extract the spatial relationship between the nozzle, components and PCB pads in the physical coordinate canvas, construct three geometric association chains, and form a deviation representation of each stage of the mounting process.

[0012] S3. Based on the perspective image of the motion side view position, the spatial posture of the component on the nozzle is inverted to obtain the out-of-plane tilt angle and in-plane rotation angle. Based on the continuous perspective image of the local top view position of the PCB during the placement process, differential analysis is performed to extract the solder paste deformation characteristics and construct pressure uniformity parameters characterizing the placement state.

[0013] S4. Based on deviation characterization, out-of-plane tilt angle and in-plane rotation angle, the existence of the component in the absorption stage, flight stage and landing stage is determined to form a multi-stage existence state combination.

[0014] S5. Using multi-stage existence state combinations and pressure uniformity parameters as the criteria, logically match them with preset fault modes to identify the fault types of the pick-and-place machine and predict the fault development trend.

[0015] Preferably, the method for acquiring images from various viewpoints includes:

[0016] The pulse signal output by the spindle encoder of the pick-and-place machine during spindle rotation is acquired, and the pulse signal is accumulated and counted in real time during the placement process; based on the preset number of pulses per revolution, a mapping relationship between the accumulated count value and the spindle rotation angle is established, and the accumulated count value is converted into an angular position quantity measured in angular values;

[0017] Based on the pick-up stage, movement stage, and placement stage in the mounting process, corresponding angle trigger windows are set respectively; where the angle trigger window is the value range of the angle position quantity;

[0018] When the angle position measurement first enters the angle trigger window corresponding to the suction stage, the first trigger signal is generated, which drives the industrial camera at the top-view position of the suction nozzle to perform perspective image acquisition.

[0019] When the angular position quantity first enters the angular trigger window corresponding to the motion phase, a second trigger signal is generated to drive the industrial camera at the motion side-view position to perform angular image acquisition.

[0020] When the angle position quantity first enters the angle trigger window corresponding to the placement stage, a continuous trigger signal is generated according to the preset angle step size to drive the industrial camera at the top view position of the PCB to acquire the view image.

[0021] Preferably, the method for constructing the physical coordinate canvas includes:

[0022] Before the placement process begins, at least three non-collinear fixed positions are marked on the PCB pads as preset PCB reference positioning points; the machine coordinate system is constructed with the machine origin as the coordinate system origin, and the physical coordinates of each preset PCB reference positioning point in the machine coordinate system are read.

[0023] The acquired images from each viewpoint are preprocessed to obtain feature images from each viewpoint; a reference positioning point recognition operation is performed on each feature image from each viewpoint to extract the pixel coordinates of each preset PCB reference positioning point in the corresponding view image; specifically, the preset PCB reference positioning point is used as the recognition target to locate the position of the preset PCB reference positioning point in each feature image from each viewpoint and extract the corresponding pixel coordinates.

[0024] Based on the pixel coordinates and corresponding physical coordinates of each preset PCB reference positioning point, the least squares method is used to solve the affine transformation matrix for each view image, and the mapping relationship from the pixel coordinate system to the physical coordinate system for each view image is established.

[0025] Using the coordinate system of the pick-and-place machine as the framework of the physical coordinate canvas, the pixel coordinates in the images from each viewpoint are converted into physical coordinates according to the corresponding affine transformation matrix, and then projected onto the corresponding positions within the physical coordinate canvas framework to obtain the constructed physical coordinate canvas.

[0026] Preferably, the method for constructing the three-segment geometric association chain includes:

[0027] Based on a physical coordinate canvas, target detection is performed on images acquired from various perspectives, including the top-view position of the nozzle, the side-view position during movement, and the partial top-view position of the PCB, to identify the center coordinates of the nozzle, components, and PCB pads, respectively. Based on the center coordinates of the nozzle, components, and PCB pads... Based on the spatial relationship between them, construct three geometric association chains;

[0028] The three geometric association chains are all presented in vector form. The first geometric association chain takes the center coordinates of the nozzle as the reference point and the center coordinates of the adsorbed component as the target point, forming the first positional association vector. The second geometric association chain takes the center coordinates of the adsorbed component as the starting point and the center coordinates of the PCB pad as the ending point, forming the second positional association vector. The third geometric association chain takes the center coordinates of the nozzle as the starting point and the center coordinates of the corresponding PCB pad as the ending point, forming the third positional association vector.

[0029] Preferably, the method for obtaining the deviation characterization of each stage of the mounting process includes:

[0030] Based on the positional correlation vectors corresponding to the three geometric relationship chains, spatial direction decomposition and magnitude calculation are performed on each positional correlation vector in the coordinate system of the pick-and-place machine to obtain the corresponding component deviation and distance deviation. The component deviation and distance deviation corresponding to each positional correlation vector are combined to form the deviation representation of each stage of the placement process.

[0031] Preferably, the method for obtaining the out-of-plane tilt angle and the in-plane rotation angle includes:

[0032] Edge detection is performed on the viewpoint image obtained from the motion side view position to extract the contour boundary of the component, and the main direction of the component extending in the viewpoint image plane is obtained by using the minimum bounding rectangle fitting method; with the central axis of the nozzle as the reference direction, the in-plane rotation angle is obtained by calculating the angle between the main direction of the component and the reference direction.

[0033] Based on the component's contour boundary, the upper and lower boundary positions of the component in the motion side-view image are obtained. The projection height of the component is determined according to the distance between the upper and lower boundary positions. A preset standard projection height is introduced, and the difference between the component's projection height and the preset standard projection height is calculated. The difference is substituted into the projection height equation for inverse solution to obtain the out-of-plane tilt angle.

[0034] Preferably, the method for constructing pressure uniformity parameters characterizing the mounting state includes:

[0035] The top-view images of the PCB during the mounting process are arranged in chronological order and then time-aligned to form a continuous sequence of view images. Differential operations are performed on two adjacent view images in the sequence to obtain differential images that characterize the deformation of the solder paste under pressure.

[0036] Region constraints are applied to the differential image based on the PCB pad area to obtain the solder paste effect area; the pixel grayscale change gradient is calculated within the solder paste effect area to form the local deformation intensity distribution of the solder paste.

[0037] The deformation intensity distribution of local solder paste in the PCB pad area is accumulated and statistically analyzed to obtain the deformation energy parameter characterizing the overall pressure of the solder paste; the spatial variance of the local deformation intensity distribution of the solder paste is calculated to obtain the deformation uniformity parameter; based on the deformation energy parameter and the deformation uniformity parameter, the pressure uniformity parameter characterizing the mounting state is constructed.

[0038] Preferably, the method for obtaining the multi-stage existence state combination includes:

[0039] Existence determination rules are constructed for the pick-up, motion, and placement stages of the mounting process to determine whether the components in the corresponding stages are in a valid existence state, and then the existence determination results of each stage are obtained. The existence determination results of each stage are combined according to the corresponding time sequence in the mounting process to form a multi-stage existence state combination that represents the components in the entire mounting process.

[0040] For any stage, the existence determination rule is as follows: when the deviation representation of the corresponding stage is less than or equal to the preset deviation representation range threshold, the absolute value of the out-of-plane tilt angle is less than or equal to the preset out-of-plane tilt angle threshold, and the absolute value of the in-plane rotation angle is also less than or equal to the preset in-plane rotation angle threshold, when all three conditions are met, the element in the corresponding stage is determined to be in a valid existence state, and the existence determination result is 1; otherwise, the element in the corresponding stage is in an invalid existence state, and the existence determination result is 0.

[0041] Preferably, the method for identifying the fault type of the pick-and-place machine and predicting the fault development trend includes:

[0042] A fault mode library is pre-built, which stores different fault types of pick and place machines, and corresponding fault judgment rules are established for each fault type. The fault judgment rules include multi-stage existence state combination characteristics and pressure uniformity parameter change characteristics.

[0043] The acquired multi-stage existence state combinations and pressure uniformity parameters are integrated to form a joint discriminant feature expression for fault type matching. Based on the joint discriminant feature expression, logical matching is performed with all fault determination rules to determine the fault mode corresponding to the current placement state, thereby identifying the fault type of the pick-and-place machine. Time series analysis is performed on the pressure uniformity parameters, and the fault development trend of the corresponding fault type is determined by calculating the slope of the change of the pressure uniformity parameters.

[0044] A fault prediction system for SMT pick-and-place machines based on image recognition, comprising:

[0045] The image acquisition module is used to synchronously trigger industrial cameras set at the top view position of the nozzle, the side view position of the motion, and the top view position of a part of the PCB through the encoder signal of the pick-and-place machine spindle to acquire images from various perspectives; and to construct a physical coordinate canvas by mapping the acquired images from various perspectives based on the preset PCB reference positioning points.

[0046] The deviation decomposition module is used to extract the spatial positional relationship between the nozzle, components and PCB pads in the physical coordinate canvas, construct three geometric association chains, and form a deviation representation of each stage of the mounting process.

[0047] The attitude inversion module inverts the spatial attitude of the component on the nozzle based on the perspective image of the motion side view position to obtain the out-of-plane tilt angle and in-plane rotation angle. It also performs differential analysis based on the continuous perspective image of the local top view position of the PCB during the placement process to extract the solder paste deformation characteristics and construct pressure uniformity parameters characterizing the placement state.

[0048] The existence determination module, based on deviation characterization, out-of-plane tilt angle and in-plane rotation angle, determines the existence of the component in the absorption phase, flight phase and landing phase, forming a multi-stage existence state combination.

[0049] The fault prediction module is used to identify the fault type of the pick-and-place machine and predict the fault development trend by logically matching it with the multi-stage existence state combination and pressure uniformity parameters as the discrimination criteria and preset fault modes.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] This invention synchronously triggers a multi-station industrial camera through a spindle encoder and completes coordinate mapping based on PCB reference positioning points to construct a unified physical coordinate canvas. This allows the nozzle, components, and PCB pads to be represented in the same coordinate system, eliminating coordinate drift and error accumulation caused by acquisition from different angles and at different times. Compared with traditional single-camera or local vision inspection, it significantly improves spatial positioning accuracy and system stability.

[0052] By extracting the spatial relationship between the three in the physical coordinate canvas and establishing a continuous geometric association chain, the error in the mounting process is transformed from "point deviation" to "chain relationship deviation". This allows for the identification of error sources and fault tracing. Compared with the traditional method of only detecting the final mounting position, this method improves the process-level error analysis capability.

[0053] By inverting the spatial posture of the component on the nozzle through the motion side view image, the limitation of traditional top-view inspection in obtaining three-dimensional posture information is overcome. It can identify hidden defects such as component tilting, flipping, and improper adsorption in advance, improve the sensitivity to small posture deviations, and thus reduce the subsequent placement defect rate.

[0054] By performing differential analysis on continuous top-view images of a PCB, solder paste deformation features are extracted and quantified into pressure uniformity parameters, transforming "invisible mounting pressure" into "observable image features," thus achieving indirect measurement. Compared to traditional methods that rely on equipment parameter settings, this enables quality assessment based on actual process feedback.

[0055] Based on deviation characterization and attitude parameters, the existence of components is determined during the absorption, flight, and placement phases, forming a multi-stage state combination. This enables a shift from "result detection" to "full-process monitoring," allowing for precise location of anomaly occurrence stages and improving the timeliness and accuracy of fault detection. Attached Figure Description

[0056] Figure 1 This is a schematic diagram of the process of a fault prediction method for an SMT pick-and-place machine based on image recognition according to the present invention;

[0057] Figure 2 This is a schematic diagram of the structure of an SMT pick-and-place machine fault prediction system based on image recognition according to the present invention. Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] Example 1

[0060] Please see Figure 1 As shown, this embodiment provides a fault prediction method for SMT pick-and-place machines based on image recognition, specifically including the following steps:

[0061] S1. The industrial camera, located at the top view position of the nozzle, the side view position of the motion, and the top view position of the PCB, is synchronously triggered by the encoder signal of the pick-and-place machine spindle to acquire images from various perspectives; the acquired images from various perspectives are mapped to coordinates based on the preset PCB reference positioning points to construct a physical coordinate canvas.

[0062] S2. Extract the spatial relationship between the nozzle, components and PCB pads in the physical coordinate canvas, construct three geometric association chains, and form a deviation representation of each stage of the mounting process.

[0063] S3. Based on the perspective image of the motion side view position, the spatial posture of the component on the nozzle is inverted to obtain the out-of-plane tilt angle and in-plane rotation angle. Based on the continuous perspective image of the local top view position of the PCB during the placement process, differential analysis is performed to extract the solder paste deformation characteristics and construct pressure uniformity parameters characterizing the placement state.

[0064] S4. Based on deviation characterization, out-of-plane tilt angle and in-plane rotation angle, the existence of the component in the absorption stage, flight stage and landing stage is determined to form a multi-stage existence state combination.

[0065] S5. Using multi-stage existence state combinations and pressure uniformity parameters as the criteria, logically match them with preset fault modes to identify the fault types of the pick-and-place machine and predict the fault development trend.

[0066] Methods for acquiring images from various perspectives include:

[0067] The pulse signal output by the spindle encoder of the pick-and-place machine during spindle rotation is acquired, and the pulse signal is accumulated and counted in real time during the placement process; based on the preset number of pulses per revolution, a mapping relationship between the accumulated count value and the spindle rotation angle is established, and the accumulated count value is converted into an angular position quantity measured in angular values;

[0068] Based on the pick-up stage, movement stage, and placement stage in the mounting process, corresponding angle trigger windows are set respectively; where the angle trigger window is the value range of the angle position quantity;

[0069] When the angle position measurement first enters the angle trigger window corresponding to the suction stage, the first trigger signal is generated, which drives the industrial camera at the top-view position of the suction nozzle to perform perspective image acquisition.

[0070] When the angular position quantity first enters the angular trigger window corresponding to the motion phase, a second trigger signal is generated to drive the industrial camera at the motion side-view position to perform angular image acquisition.

[0071] When the angle position quantity first enters the angle trigger window corresponding to the placement stage, a continuous trigger signal is generated according to the preset angle step size to drive the industrial camera at the top view position of the PCB to acquire the view image.

[0072] It should be noted that the spindle encoder of the SMT pick and place machine is an incremental encoder, which is installed on the spindle drive end. During the operation of the SMT pick and place machine, the spindle encoder will continuously output pulse signals corresponding to the changes in spindle rotation angle as the spindle rotates. There is a direct proportional relationship between the cumulative count value of the pulse signal and the spindle rotation angle. Specifically, the larger the spindle rotation angle, the more cumulative count values ​​of the pulse signal output by the spindle encoder.

[0073] After the placement operation begins, the pulse signal is counted in real time to obtain the cumulative count value of the pulse signal; based on the preset number of pulses per revolution. Establish a mapping relationship between the cumulative count value and the spindle rotation angle; where the number of pulses per revolution refers to the number of pulses per revolution of the spindle encoder as the spindle rotates one revolution ( The total number of pulse signals output during the initialization phase is obtained by directly reading the number of pulses per revolution from the technical specifications of the adopted spindle encoder during the system initialization phase, thereby obtaining the specific value of the preset number of pulses per revolution.

[0074] Specifically, the spindle encoder outputs a value corresponding to each complete rotation of the spindle. Therefore, the number of pulse signals corresponding to a unit angle of spindle rotation is 1. Furthermore, the cumulative count value is converted into an angular position quantity measured in angular values. ;in, This indicates an angular position, expressed in degrees. This indicates the current cumulative count value; This indicates the preset number of pulses per revolution of the spindle encoder;

[0075] For example: the preset number of pulses per revolution of the spindle encoder. Current cumulative count value The corresponding angular position quantity That is, the spindle has rotated at this time. .

[0076] After completing the angle position conversion, corresponding angle trigger windows are set according to the pick-up, motion and placement stages of the SMT pick-and-place machine in the placement process. The angle trigger window is defined as the value range of the angle position, that is, a range of angles that can be triggered is defined by the preset lower limit angle position and the preset upper limit angle position. Only when the angle position falls within the angle trigger window is the corresponding trigger signal allowed to be generated.

[0077] The SMT pick and place machine is calibrated to obtain the angle range of the trigger window for each angle. Specifically, the SMT pick and place machine is made to actually execute the complete placement process, and the spindle rotation angle corresponding to each stage of the placement process is counted to obtain the angle range of the trigger window corresponding to the pick-up stage, motion stage and placement stage.

[0078] Set the angle trigger window corresponding to the absorption phase to ,and Angle Trigger Window The angle trigger window corresponds to the angle range within which the spindle drives the nozzle to complete the downward suction action. Preset lower limit angle position quantity Sets the spindle rotation angle when the nozzle begins to contact the component, with a preset upper limit angle position. Set the spindle rotation angle to allow the nozzle to pick up components;

[0079] Set the angle trigger window corresponding to the motion phase to ,and Angle Trigger Window The angle range during the flight transport process of the corresponding component as it moves from the pick-up position to the PCB pad with the nozzle, at which point the angle trigger window is located. Preset lower limit angle position quantity Sets the starting spindle rotation angle after the spindle leaves the pick-up position and enters the flight transport process, with a preset upper limit angle position. Sets the final spindle rotation angle before the spindle reaches the mounting position;

[0080] Set the angle trigger window corresponding to the placement stage to ,and Angle Trigger Window The angle trigger window corresponds to the entire angle range during which the spindle drives the nozzle to press down and place the component onto the PCB pad. Preset lower limit angle position quantity Sets the spindle rotation angle at the start of the application and pressing action, with a preset upper limit angle position. Sets the spindle rotation angle when placement is complete;

[0081] For example, in this embodiment, the angle trigger window corresponding to the absorption phase. Set as The angle trigger window corresponding to the movement phase Set as The angle trigger window corresponding to the placement stage Set as ;

[0082] Industrial cameras are deployed at the nozzle's top-view position, the movement side-view position, and a partial top-view position on the PCB. The pick-up phase of the placement process involves the spindle driving the nozzle downwards, contacting the component, and completing vacuum adsorption. During the pick-up phase, before the component is picked up, the nozzle is directly above the component. The industrial camera at the nozzle's top-view position captures the view between the nozzle tip and the initial position of the component. When the system detects that the angular position measurement first enters the angle trigger window corresponding to the pick-up phase... At that time, a first trigger signal is generated, which drives the industrial camera at the top-view position of the suction nozzle to perform perspective image acquisition;

[0083] The motion phase of the placement process is characterized by the spindle moving the component from the pick-up position to the placement position (i.e., the PCB pad) after the nozzle completes its pick-up action. During this motion phase, the component moves synchronously with the nozzle, and an industrial camera at the side view position needs to capture the side profile image of the component as it is carried by the nozzle during its flight. When the system detects that the angular position value first enters the angle trigger window corresponding to the motion phase... At that time, a second trigger signal is generated to drive the industrial camera in the moving side-view position to perform lateral image acquisition;

[0084] The placement stage of the mounting process is characterized by the spindle driving the nozzle to press down the adsorbed components, thereby placing the components onto the PCB pads. The placement stage requires continuous acquisition of perspective images at fixed angle steps to record the complete dynamic process of the components from approaching the PCB pads to completing the placement.

[0085] In this embodiment, the preset angle step size is set to 2, meaning that a continuous trigger signal is generated every 2° of spindle rotation. Therefore, the trigger signal is generated throughout the entire angle trigger window. The number of times the internal continuous trigger signal is generated is Second-rate;

[0086] When the angle position measurement first enters the corresponding angle trigger window during the placement stage At that time, a continuous trigger signal is generated according to the preset angle step size to drive the industrial camera at the top view position of the PCB to acquire the view image.

[0087] To form complete time information for image acquisition from each viewpoint, in this embodiment, the corresponding timestamp information is recorded synchronously each time a trigger signal is generated, forming a trigger event record, which is then bound to the corresponding viewpoint image to form a viewpoint image dataset.

[0088] Taking a single mounting process as an example, the event logs include:

[0089] Record 1: θ=15°, timestamp Absorption phase; Record 2: θ=90°, timestamp Motion phase; Record 3: θ=200°, timestamp Placement stage... Record 42: θ=280°, timestamp The placement stage;

[0090] After each placement process is completed, the system needs to stop the cumulative counting of pulse signals and clear the cumulative count value to ensure that the cumulative count for the next placement process starts from zero, thus avoiding cumulative count errors between multiple placement processes.

[0091] In this embodiment, the determination of the end of the placement process is based on: the angle position exceeds the upper boundary of the angle trigger window during the placement stage (i.e., The system will only execute the cumulative count value clearing operation after the above conditions are met simultaneously, and the spindle encoder sends a mounting completion signal through the communication interface.

[0092] Methods for constructing a physical coordinate canvas include:

[0093] Before the placement process begins, at least three non-collinear fixed positions are marked on the PCB pads as preset PCB reference positioning points; the machine coordinate system is constructed with the machine origin as the coordinate system origin, and the physical coordinates of each preset PCB reference positioning point in the machine coordinate system are read.

[0094] The acquired images from various perspectives are preprocessed, including grayscale conversion, filtering and noise reduction, edge extraction and morphological processing, to highlight the pre-set PCB reference positioning points, nozzle center, PCB pads and other feature areas in the images from various perspectives, thereby obtaining the feature images from various perspectives after the processing is cleared.

[0095] The reference positioning point recognition operation is performed on the feature images of each viewpoint, and the pixel coordinates of each preset PCB reference positioning point in the corresponding viewpoint image are extracted. Specifically, the preset PCB reference positioning point is used as the recognition target, and the position of the preset PCB reference positioning point in the feature images of each viewpoint is located by similarity matching, and the corresponding pixel coordinates are extracted.

[0096] Based on the pixel coordinates and corresponding physical coordinates of each preset PCB reference positioning point, the least squares method is used to solve the affine transformation matrix for each view image, and the mapping relationship from the pixel coordinate system to the physical coordinate system for each view image is established.

[0097] Taking an image from any given location as an example, suppose the first... The pixel coordinates of the preset PCB reference positioning points are The corresponding physical coordinates are Using the pixel coordinates and physical coordinates of all preset PCB reference points, the affine transformation matrix is ​​solved. : ;

[0098] Wherein, the affine transformation matrix The least squares method is used for fitting and solving, and the affine transformation matrix is ​​obtained by minimizing the sum of squared residuals of the mapping of the preset PCB reference positioning points. The optimal solution; the affine transformation matrix obtained by solving. This refers to the coordinate mapping relationship between the viewpoint image pixel coordinate system and the physical coordinate system at the corresponding position;

[0099] Using the coordinate system of the pick-and-place machine as the framework of the physical coordinate canvas, the pixel coordinates in the images from each viewpoint are converted into physical coordinates according to the corresponding affine transformation matrix, and then projected onto the corresponding positions within the physical coordinate canvas framework to obtain the constructed physical coordinate canvas.

[0100] It should be noted that in this embodiment, the physical coordinate canvas includes viewpoint image information of the nozzle top view position, the movement side view position, and the PCB partial top view position, and has a unified physical coordinate reference; the coordinate origin, coordinate axis direction, and size range of the physical coordinate canvas are consistent with the coordinate system of the entire SMT placement machine, and each coordinate position in the physical coordinate canvas corresponds to a unique physical space position on the SMT placement machine.

[0101] The methods for constructing three-segment geometric association chains include:

[0102] Based on a physical coordinate canvas, target detection is performed on images acquired from various perspectives, including the top-view position of the nozzle, the side-view position during movement, and the partial top-view position of the PCB, to identify the center coordinates of the nozzle, components, and PCB pads, respectively. Based on the center coordinates of the nozzle, components, and PCB pads... The spatial relationship between them is used to construct a three-segment geometric association chain based on the positional transfer logic of the entire mounting process;

[0103] It should be noted that the three completed geometric association chains, with the positional transmission of the placement action as the main line, connect the three targets—the nozzle, the component, and the PCB pad—to form a complete positional association system. All three geometric association chains are presented in vector form, clearly defining the positional transmission logic and deviation direction of each stage of the placement process, as detailed below:

[0104] The first geometric association chain (nozzle-component association): based on the coordinates of the nozzle's center position. Using the reference point as the coordinate of the center position of the adsorbed element For the target point, form the first positional correlation vector. It is used to characterize the nozzle's pick-up and clamping deviation of components;

[0105] The second geometric association chain (component-PCB pad association): based on the coordinates of the center position of the attracted component. Starting from the center coordinates of the PCB pads The endpoint forms the second positional correlation vector. This is used to characterize the displacement deviation of the component center relative to the PCB pad center;

[0106] The third geometric association chain (nozzle-PCB pad association): based on the coordinates of the nozzle center position. Starting from the coordinates of the center position of the corresponding PCB pad, The endpoint forms the third positional correlation vector. It is used to characterize the overall displacement relationship from the nozzle center to the PCB pad center, forming the comprehensive deviation in the overall mounting path;

[0107] The first, second, and third geometric association chains are combined according to the mounting process to form three geometric association chains that describe the spatial transfer relationship between the nozzle, components, and PCB pads: ,and ;

[0108] By establishing a vector closed-loop relationship between three geometric links, a complete spatial constraint expression for the entire placement process is achieved. This enables the formation of a resolvable continuous geometric link between the nozzle pickup process, component transfer process, and placement alignment process, thereby fully characterizing the spatial transfer mechanism in the placement process and improving the ability to express the spatial evolution law of the placement process.

[0109] Methods for obtaining deviation characterization at each stage of the mounting process include:

[0110] Based on the positional correlation vectors corresponding to the three geometric relationship chains, spatial direction decomposition and magnitude calculation are performed on each positional correlation vector in the coordinate system of the pick-and-place machine to obtain the corresponding component deviation and distance deviation. The component deviation and distance deviation corresponding to each positional correlation vector are combined to form the deviation representation of each stage of the placement process.

[0111] Specifically: In the coordinate system of the pick-and-place machine, the first position association vector corresponding to the first geometric association chain is decomposed into spatial vectors to obtain the component deviations of the first position association vector in each coordinate axis direction; at the same time, the magnitude of the first position association vector is calculated to obtain the distance deviation between the nozzle center position and the component center position; the component deviations and distance deviations corresponding to the first position association vector are combined to form the deviation characterization of the pick-up stage.

[0112] In the coordinate system of the pick-and-place machine, the second position association vector corresponding to the second geometric association chain is decomposed into spatial vectors to obtain the component deviations of the second position association vector in each coordinate axis direction; at the same time, the magnitude of the second position association vector is calculated to obtain the distance deviation between the component center position and the PCB pad center position; the component deviations and distance deviations corresponding to the second position association vector are combined to form the deviation characterization of the motion stage.

[0113] In the coordinate system of the pick-and-place machine, the spatial vector decomposition of the third position association vector corresponding to the third geometric association chain is performed to obtain the component deviation of the third position association vector in each coordinate axis direction; at the same time, the magnitude of the third position association vector is calculated to obtain the distance deviation between the nozzle center position and the PCB pad center position; the component deviation and distance deviation corresponding to the third position association vector are combined to form the deviation characterization of the placement stage.

[0114] Taking the first geometric association chain as an example, firstly, the spatial direction decomposition of the first position association vector is performed in the coordinate system of the pick-and-place machine. The component deviation of the first position association vector in the X-axis direction is defined as: , used to indicate the horizontal offset between the nozzle and the component;

[0115] The component deviation of the first position association vector in the Y-axis direction is defined as follows: , used to represent the offset between the nozzle and the component in the front-to-back direction; the component deviation of the first position association vector in the Z-axis direction is defined as: This is used to indicate the deviation between the nozzle and the component in the vertical direction;

[0116] The magnitude of the third position correlation vector is calculated to obtain the distance deviation between the nozzle center position and the PCB pad center position. ;

[0117] For example: Suppose during a certain placement process, the coordinates of the nozzle center position... Coordinates of the center position of the adsorbed element Construct the first positional association vector ;

[0118] Then, the component deviation of the first positional correlation vector in the X-axis direction. The component deviation of the first positional correlation vector in the Y-axis direction. ; Component deviation of the first positional correlation vector in the Z-axis direction Distance deviation between the center position of the nozzle and the center position of the PCB pad ;

[0119] The component deviations and distance deviations corresponding to the first position association vector are combined to form the deviation characterization of the suction stage: the nozzle is 3 units to the right of the component in the X-axis direction, 3 units behind the component in the Y-axis direction, and 2 units below the component in the Z-axis direction. The distance deviation between the center position of the nozzle and the center position of the component is approximately 4.69 units. Similarly, the deviation characterizations of the motion stage and the placement stage can be obtained.

[0120] Methods for obtaining out-of-plane tilt angle and in-plane rotation angle include:

[0121] Edge detection is performed on the viewpoint image obtained from the motion side view position to extract the contour boundary of the component, and the main direction of the component's extension in the viewpoint image plane is obtained by the minimum bounding rectangle fitting method.

[0122] Using the central axis of the nozzle as a reference direction, the in-plane rotation angle is obtained by calculating the angle between the main direction of the element and the reference direction.

[0123] Based on the component's contour boundary, the upper and lower boundary positions of the component in the motion side view image are obtained, and the projection height of the component is determined according to the distance between the upper and lower boundary positions.

[0124] A preset standard projection height is introduced, and the difference between the component's projection height and the preset standard projection height is calculated. The difference is substituted into the projection height equation and the inverse solution is performed to obtain the out-of-plane tilt angle, which characterizes the degree of spatial tilt of the component relative to the nozzle's central axis.

[0125] In this embodiment, to address the potential posture shift issue that may occur during component pick-up and handling in an SMT pick-and-place machine, the spatial posture of the component while it is being picked up by the nozzle is analyzed based on the viewpoint image obtained from the motion side view position to obtain the in-plane rotation angle and out-of-plane tilt angle. The specific implementation process is as follows:

[0126] First, the viewpoint images acquired from the motion side-view position are preprocessed, and the Canny edge detection algorithm is used to process the component region to extract the component's contour boundary. Based on this, the geometric features of the component are fitted according to the contour boundary, and the minimum bounding rectangle fitting method is used to determine the spatial extension direction of the component in the image plane, thereby obtaining the main direction representing the extension of the component.

[0127] The method of using the minimum bounding rectangle fitting to determine the spatial extension direction of a component in the image plane is essentially: using the minimum area of ​​the bounding rectangle of the component's outline to analyze the main axis direction of the component in the image, that is, to determine which direction the component extends the longest in the image plane.

[0128] Specifically: After extracting the outline of the component, the component is fitted with the minimum bounding rectangle. The minimum bounding rectangle is the rotating rectangle that can completely enclose the outline of the component and has the smallest area. The long side of this minimum bounding rectangle is used as the main direction of the component's extension in the image plane.

[0129] Furthermore, the central axis of the suction nozzle is used as a reference datum, where the central axis of the suction nozzle is a straight line extending along the structural axis of the suction nozzle, used to characterize the standard direction of the suction nozzle under ideal suction conditions; under motion side-view imaging conditions, the central axis of the suction nozzle is used as a reference direction, and the angle between the two is calculated based on the geometric relationship between the main direction of the component and the reference direction, which is the in-plane rotation angle;

[0130] The in-plane rotation angle is used to characterize the degree of in-plane rotational offset of a component relative to the central axis of the nozzle when it is being picked up by the nozzle. When the in-plane rotation angle deviates from zero, it indicates that the component has an in-plane attitude deflection during the picking up or transporting process.

[0131] Furthermore, spatial position analysis is performed on the contour boundary of the component to extract the upper and lower boundary positions of the component in the motion side view image, and the distance between the upper and lower boundary positions in the view image coordinate system is calculated to characterize the projection height of the component in the side view direction.

[0132] A preset standard projection height is introduced. This preset standard projection height is obtained by calibrating the component under ideal vertical absorption conditions and is used to characterize the theoretical projection height of the component in a non-tilted state. The projection height of the component is then calculated. The difference between the projected height and the preset standard projection height ,Will Substituting into the projection height equation and solving inversely, we obtain the out-of-plane tilt angle. ;in, The approximate initial estimate of the out-of-plane tilt angle is derived from the simplified formula. The calculated value serves as the starting point for iteratively solving the outward tilt angle, and is used to provide the initial input value for the correction term in the denominator.

[0133] Methods for constructing pressure uniformity parameters characterizing the mounting state include:

[0134] The top-view images of the PCB during the mounting process are arranged in chronological order and then time-aligned to form a continuous sequence of view images. Differential operations are performed on two adjacent view images in the sequence to obtain differential images that characterize the deformation of the solder paste under pressure.

[0135] Region constraints are applied to the differential image based on the PCB pad area to obtain the solder paste effect area; the pixel grayscale change gradient is calculated within the solder paste effect area to form the local deformation intensity distribution of the solder paste.

[0136] By summing and statistically analyzing the local deformation intensity distribution of solder paste within the PCB pad area, a deformation energy parameter characterizing the overall compressive stress of the solder paste can be obtained. Spatial variance calculation of local deformation intensity distribution in solder paste is performed to obtain deformation uniformity parameters. ;

[0137] It should be noted that in this embodiment, because the solder paste undergoes compression, expansion, and thickness changes during the compression process, resulting in changes in the local grayscale values ​​in the viewing image, the pixel grayscale change gradient is used as a characterization of the local deformation intensity of the solder paste. Specifically, it is first necessary to solve for the pixel grayscale change gradient of all pixels within the solder paste's effective area: for any pixel within the solder paste's effective area... The grayscale gradients of the pixel in the horizontal and vertical directions are calculated respectively, and the gradient magnitude is used to characterize the actual deformation degree of the pixel, thereby obtaining the local deformation intensity of the solder paste at the location of the pixel: ;

[0138] in, Represents pixels The local deformation strength value of the solder paste at the location; Represents pixels Gray-scale gradient in the horizontal direction; Represents pixels Gray-scale gradient in the vertical direction; Indicates the area where the solder paste is applied; Represents the horizontal coordinate of a pixel; Represents the vertical coordinate of a pixel; collects the local deformation intensity values ​​of the solder paste for all pixels within the entire solder paste application area, forming the local deformation intensity distribution of the solder paste.

[0139] Based on this, the local deformation intensity values ​​of each solder paste included in the obtained local deformation intensity distribution are accumulated and statistically analyzed to obtain the deformation energy parameter characterizing the overall compressive strength of the solder paste. ;in, This represents the set of pixels formed by all pixels within the area where the solder paste is applied.

[0140] Furthermore, spatial variance calculation is performed on the local deformation intensity distribution of the solder paste to obtain deformation uniformity parameters. Specifically, the local deformation intensity values ​​of the solder paste corresponding to each pixel within the solder paste application area are used as the analysis object, and the deviation from the average deformation intensity value within the solder paste application area is calculated to obtain deformation uniformity parameters. ;in, This indicates the total number of pixels within the area affected by the solder paste; This represents the average deformation intensity value within the solder paste application area, which is obtained by calculating the average of the local deformation intensity values ​​of the solder paste at all pixels within the solder paste application area.

[0141] Based on deformation energy parameters and deformation uniformity parameters, pressure uniformity parameters characterizing the mounting state are constructed. The pressure uniformity parameter is used to characterize the pressure strength and distribution uniformity of the solder paste during the mounting process. A higher pressure uniformity parameter indicates that the solder paste is sufficiently compressed and evenly distributed. A lower pressure uniformity parameter indicates that there is a risk of uneven local pressure on the solder paste or abnormal mounting.

[0142] Methods for obtaining multi-stage existential state combinations include:

[0143] Existence determination rules are constructed for the pick-up, motion, and placement stages of the mounting process to determine whether the components in the corresponding stages are in a valid existence state, thereby obtaining the existence determination results for each stage. The existence determination results of each stage are combined according to the corresponding time sequence in the mounting process to form a multi-stage existence state combination that represents the components in the entire mounting process.

[0144] For any stage, the existence determination rule is as follows: when the deviation representation of the corresponding stage is less than or equal to the preset deviation representation range threshold, the absolute value of the out-of-plane tilt angle is less than or equal to the preset out-of-plane tilt angle threshold, and the absolute value of the in-plane rotation angle is also less than or equal to the preset in-plane rotation angle threshold, when all three conditions are met, the element in the corresponding stage is determined to be in a valid existence state, and the existence determination result is 1; otherwise, the element in the corresponding stage is in an invalid existence state, and the existence determination result is 0.

[0145] In this embodiment, the preset deviation characterization range threshold, preset out-of-plane tilt angle threshold, and preset in-plane rotation angle threshold are all set based on the component assembly accuracy requirements specified in the design documents or process specifications. Specifically, the preset deviation characterization range threshold is: based on the assembly accuracy requirements of the component at the target position, the maximum allowable positional deviation (e.g., the tolerance range specified in the design drawings or process specifications) is obtained as the preset deviation characterization range threshold; the preset out-of-plane tilt angle threshold is: based on the accuracy requirements of the tilt degree of the component in the assembled state, the maximum allowable tilt angle limited by the process specifications (e.g., angle restrictions to ensure bonding, force, or optical performance) is obtained as the preset out-of-plane tilt angle threshold; the preset in-plane rotation angle threshold is: based on the alignment accuracy requirements of the component in the plane, the maximum allowable rotation angle deviation of the process specifications (e.g., determined by the angle requirements to ensure hole alignment and graphic matching) is obtained as the preset in-plane rotation angle threshold; in another embodiment, the preset deviation characterization range threshold, preset out-of-plane tilt angle threshold, and preset in-plane rotation angle threshold can be determined through actual mounting process testing or calibration experiments.

[0146] After obtaining the existence determination results for the absorption phase, flight phase, and landing phase, the determination results for each phase are combined in chronological order to form a multi-phase existence state combination. This is used to describe the continuity of the component's state throughout the entire mounting process;

[0147] Furthermore, different combinations of multi-stage existence states correspond to different placement information. For example: (1,1,1): indicates that the component exists normally in each stage and the placement process is normal; (1,0,1): indicates that the component is in an invalid existence state in the movement stage and may fall off midway; (1,1,0): indicates that the component is in an invalid existence state in the placement stage and may fail to place or be offset; (0,0,0): indicates that the component is in an invalid existence state in the pick-up stage, movement stage and placement stage, and there is a problem of component pick-up failure in the pick-up stage.

[0148] Methods for identifying pick-and-place machine fault types and predicting fault development trends include:

[0149] A fault mode library is pre-built, which stores different fault types of pick and place machines, and corresponding fault judgment rules are established for each fault type. The fault judgment rules include multi-stage existence state combination characteristics and pressure uniformity parameter change characteristics.

[0150] The acquired multi-stage existence state combinations and pressure uniformity parameters are integrated to form a joint discriminant feature expression for fault type matching. Based on the joint discriminant feature expression, logical matching is performed with all fault determination rules to determine the fault mode corresponding to the current placement state, thereby identifying the fault type of the pick-and-place machine. Time series analysis is performed on the pressure uniformity parameters, and the fault development trend of the corresponding fault type is determined by calculating the slope of the change of the pressure uniformity parameters.

[0151] It should be noted that pre-building a fault mode library first requires collecting operational data of the SMT pick-and-place machine under different operating conditions during its historical operation. Combined with known fault type samples, statistical analysis and feature summarization are performed on multi-stage presence state combinations and pressure uniformity parameters to extract typical feature patterns corresponding to different fault types. Based on this, corresponding judgment rules are established for each fault type. These rules consist of two parts: a state pattern and a parameter range. The state pattern describes the typical sequence structure of multi-stage presence state combinations, such as "abnormal presence" in the pick-up stage, "missing" in the motion stage, and "invalid" in the placement stage. The parameter range limits the normal fluctuation range of pressure uniformity parameters, obtained through statistical analysis of historical samples (such as mean, variance, or quantile analysis), and upper and lower thresholds are set based on process experience.

[0152] In the actual judgment process, the current joint discrimination feature expression is input into the fault mode library and matched one by one with the fault judgment rules corresponding to each fault mode. If the current multi-stage existence state combination satisfies the typical sequence structure corresponding to a certain fault mode and the pressure uniformity parameter exceeds the normal fluctuation range, the chip mounter is judged to meet the current fault type.

[0153] In this embodiment, the normal fluctuation range of the pressure uniformity parameter is obtained by statistically analyzing historical sample data of the patch machine under normal operating conditions. The normal fluctuation range of the pressure uniformity parameter is set based on the mean and dispersion of the pressure uniformity parameter in the historical sample data.

[0154] While completing the fault type identification, the fault development trend is further predicted based on the time evolution trend of the pressure uniformity parameter. Specifically, the pressure uniformity parameters corresponding to each time moment are arranged into a parameter sequence according to the time order, and the slope of the pressure uniformity parameter change is obtained by calculating the ratio of the change of the pressure uniformity parameter at adjacent time moments to the corresponding time interval.

[0155] By analyzing the slope of the pressure uniformity parameter change, the fault development trend can be determined. When the slope of the pressure uniformity parameter change is negative and the absolute value is large, it indicates that the pressure state of the solder paste during the placement process is gradually deteriorating, and the corresponding fault type is showing an aggravating trend. When the slope of the pressure uniformity parameter change is 0, it indicates that the pick-and-place machine is operating normally. When the slope of the pressure uniformity parameter change is positive and the absolute value is gradually decreasing, it indicates that the pressure uniformity parameter is tending to converge and stabilize, and the pressure state of the solder paste during the placement process is gradually recovering, and the equipment is operating normally.

[0156] In this embodiment, the multi-station industrial camera is synchronously triggered by the spindle encoder, and coordinate mapping is completed based on the PCB reference positioning point to construct a unified physical coordinate canvas, so that the nozzle, components and PCB pads can be expressed in the same coordinate system. This eliminates the coordinate drift and error accumulation problems caused by different perspectives and different time acquisitions. Compared with traditional single camera or local vision inspection, it significantly improves spatial positioning accuracy and system stability.

[0157] By extracting the spatial relationship between the three in the physical coordinate canvas and establishing a continuous geometric association chain, the error in the mounting process is transformed from "point deviation" to "chain relationship deviation". This allows for the identification of error sources and fault tracing. Compared with the traditional method of only detecting the final mounting position, this method improves the process-level error analysis capability.

[0158] By inverting the spatial posture of the component on the nozzle through the motion side view image, the limitation of traditional top-view inspection in obtaining three-dimensional posture information is overcome. It can identify hidden defects such as component tilting, flipping, and improper adsorption in advance, improve the sensitivity to small posture deviations, and thus reduce the subsequent placement defect rate.

[0159] By performing differential analysis on continuous top-view images of a PCB, solder paste deformation features are extracted and quantified into pressure uniformity parameters, transforming "invisible mounting pressure" into "observable image features," thus achieving indirect measurement. Compared to traditional methods that rely on equipment parameter settings, this enables quality assessment based on actual process feedback.

[0160] Based on deviation characterization and attitude parameters, the existence of components is determined during the absorption, flight, and placement phases, forming a multi-stage state combination. This enables a shift from "result detection" to "full-process monitoring," allowing for precise location of anomaly occurrence stages and improving the timeliness and accuracy of fault detection.

[0161] Example 2

[0162] Please see Figure 2 As shown, parts not described in detail in this embodiment are described in Embodiment 1. An image recognition-based SMT pick-and-place machine fault prediction system is provided, comprising:

[0163] The image acquisition module is used to synchronously trigger industrial cameras set at the top view position of the nozzle, the side view position of the motion, and the top view position of a part of the PCB through the encoder signal of the pick-and-place machine spindle to acquire images from various perspectives; and to construct a physical coordinate canvas by mapping the acquired images from various perspectives based on the preset PCB reference positioning points.

[0164] The deviation decomposition module is used to extract the spatial positional relationship between the nozzle, components and PCB pads in the physical coordinate canvas, construct three geometric association chains, and form a deviation representation of each stage of the mounting process.

[0165] The attitude inversion module inverts the spatial attitude of the component on the nozzle based on the perspective image of the motion side view position to obtain the out-of-plane tilt angle and in-plane rotation angle. It also performs differential analysis based on the continuous perspective image of the local top view position of the PCB during the placement process to extract the solder paste deformation characteristics and construct pressure uniformity parameters characterizing the placement state.

[0166] The existence determination module, based on deviation characterization, out-of-plane tilt angle and in-plane rotation angle, determines the existence of the component in the absorption phase, flight phase and landing phase, forming a multi-stage existence state combination.

[0167] The fault prediction module is used to identify the fault type of the pick-and-place machine and predict the fault development trend by logically matching it with the multi-stage existence state combination and pressure uniformity parameters as the discrimination criteria and preset fault modes.

[0168] Example 3

[0169] This embodiment discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the operation mode of the above-described image recognition-based SMT pick-and-place machine fault prediction method.

[0170] Since the electronic device described in this embodiment is the electronic device used to implement the SMT pick-and-place machine fault prediction method and system based on image recognition in the embodiments of this application, those skilled in the art can understand the specific implementation methods and various variations of the electronic device in this embodiment based on the SMT pick-and-place machine fault prediction method and system based on image recognition described in the embodiments of this application. Therefore, how the electronic device implements the method in the embodiments of this application will not be described in detail here. As long as those skilled in the art implement the electronic device used in the SMT pick-and-place machine fault prediction method and system based on image recognition in the embodiments of this application, it falls within the protection scope of this application.

[0171] It should be noted that all formulas in this manual are calculated by removing dimensions and taking their numerical values. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters and thresholds in the formulas are set by those skilled in the art according to the actual situation.

[0172] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. An image recognition-based SMT mounter failure prediction method, characterized by, include: S1. The industrial camera, located at the top view position of the nozzle, the side view position of the motion, and the top view position of the PCB, is synchronously triggered by the encoder signal of the pick-and-place machine spindle to acquire images from various perspectives; the acquired images from various perspectives are mapped to coordinates based on the preset PCB reference positioning points to construct a physical coordinate canvas. S2. Extract the spatial relationship between the nozzle, components and PCB pads in the physical coordinate canvas, construct three geometric association chains, and form a deviation representation of each stage of the mounting process. The method for constructing the three-segment geometric association chain includes: Based on a physical coordinate canvas, target detection is performed on images acquired from various perspectives, including the top-view position of the nozzle, the side-view position during movement, and the partial top-view position of the PCB, to identify the center coordinates of the nozzle, components, and PCB pads, respectively. Based on the center coordinates of the nozzle, components, and PCB pads... Based on the spatial relationship between them, construct three geometric association chains; The three geometric association chains are all presented in vector form. The first geometric association chain takes the center coordinates of the nozzle as the reference point and the center coordinates of the adsorbed component as the target point, forming the first positional association vector. The second geometric association chain takes the center coordinates of the adsorbed component as the starting point and the center coordinates of the PCB pad as the ending point, forming the second positional association vector. The third geometric association chain takes the center coordinates of the nozzle as the starting point and the center coordinates of the corresponding PCB pad as the ending point, forming the third positional association vector. The methods for obtaining the deviation characterization of each stage of the mounting process include: Based on the positional correlation vectors corresponding to the three geometric relationship chains, spatial direction decomposition and magnitude calculation are performed on each positional correlation vector in the coordinate system of the pick-and-place machine to obtain the corresponding component deviation and distance deviation; the component deviation and distance deviation corresponding to each positional correlation vector are combined to form the deviation characterization of each stage of the placement process. S3. Based on the perspective image of the motion side view position, the spatial posture of the component on the nozzle is inverted to obtain the out-of-plane tilt angle and in-plane rotation angle. Based on the continuous perspective image of the local top view position of the PCB during the placement process, differential analysis is performed to extract the solder paste deformation characteristics and construct pressure uniformity parameters characterizing the placement state. The method for constructing pressure uniformity parameters characterizing the mounting state includes: The top-view images of the PCB during the mounting process are arranged in chronological order and then time-aligned to form a continuous sequence of view images. Differential operations are performed on two adjacent view images in the sequence to obtain differential images that characterize the deformation of the solder paste under pressure. Region constraints are applied to the differential image based on the PCB pad area to obtain the solder paste effect area; the pixel grayscale change gradient is calculated within the solder paste effect area to form the local deformation intensity distribution of the solder paste. The deformation intensity distribution of local solder paste in the PCB pad area is accumulated and statistically analyzed to obtain the deformation energy parameter characterizing the overall pressure of the solder paste; the spatial variance of the local deformation intensity distribution of the solder paste is calculated to obtain the deformation uniformity parameter; based on the deformation energy parameter and the deformation uniformity parameter, the pressure uniformity parameter characterizing the mounting state is constructed. S4. Based on deviation characterization, out-of-plane tilt angle and in-plane rotation angle, the existence of the component in the absorption stage, flight stage and landing stage is determined to form a multi-stage existence state combination. The method for obtaining the multi-stage existence state combination includes: Existence determination rules are constructed for the pick-up, motion, and placement stages of the mounting process to determine whether the components in the corresponding stages are in a valid existence state, and then the existence determination results of each stage are obtained. The existence determination results of each stage are combined according to the corresponding time sequence in the mounting process to form a multi-stage existence state combination that represents the components in the entire mounting process. For any stage, the existence determination rule is as follows: when the deviation representation of the corresponding stage is less than or equal to the preset deviation representation range threshold, the absolute value of the out-of-plane tilt angle is less than or equal to the preset out-of-plane tilt angle threshold, and the absolute value of the in-plane rotation angle is also less than or equal to the preset in-plane rotation angle threshold, when all three conditions are met, the element in the corresponding stage is determined to be in a valid existence state, and the existence determination result is 1; otherwise, the element in the corresponding stage is in an invalid existence state, and the existence determination result is 0. S5. Using multi-stage existence state combinations and pressure uniformity parameters as the criteria, logically match them with preset fault modes to identify the fault types of the pick-and-place machine and predict the fault development trend.

2. The image recognition-based SMT mounter failure prediction method according to claim 1, characterized by, The method for acquiring images from various viewpoints includes: The pulse signal output by the spindle encoder of the pick-and-place machine during spindle rotation is acquired, and the pulse signal is accumulated and counted in real time during the placement process; based on the preset number of pulses per revolution, a mapping relationship between the accumulated count value and the spindle rotation angle is established, and the accumulated count value is converted into an angular position quantity measured in angular values; Based on the pick-up stage, movement stage, and placement stage in the mounting process, corresponding angle trigger windows are set respectively; where the angle trigger window is the value range of the angle position quantity; When the angle position measurement first enters the angle trigger window corresponding to the suction stage, the first trigger signal is generated, which drives the industrial camera at the top-view position of the suction nozzle to perform perspective image acquisition. When the angular position quantity first enters the angular trigger window corresponding to the motion phase, a second trigger signal is generated to drive the industrial camera at the motion side-view position to perform angular image acquisition. When the angle position quantity first enters the angle trigger window corresponding to the placement stage, a continuous trigger signal is generated according to the preset angle step size to drive the industrial camera at the top view position of the PCB to acquire the view image.

3. The image recognition-based SMT mounter failure prediction method according to claim 2, characterized by, The method for constructing the physical coordinate canvas includes: Before the placement process begins, at least three non-collinear fixed positions are marked on the PCB pads as preset PCB reference positioning points; the machine coordinate system is constructed with the machine origin as the coordinate system origin, and the physical coordinates of each preset PCB reference positioning point in the machine coordinate system are read. The acquired images from each viewpoint are preprocessed to obtain feature images from each viewpoint; a reference positioning point recognition operation is performed on each feature image from each viewpoint to extract the pixel coordinates of each preset PCB reference positioning point in the corresponding view image; specifically, the preset PCB reference positioning point is used as the recognition target to locate the position of the preset PCB reference positioning point in each feature image from each viewpoint and extract the corresponding pixel coordinates. Based on the pixel coordinates and corresponding physical coordinates of each preset PCB reference positioning point, the least squares method is used to solve the affine transformation matrix for each view image, and the mapping relationship from the pixel coordinate system to the physical coordinate system for each view image is established. Using the coordinate system of the pick-and-place machine as the framework of the physical coordinate canvas, the pixel coordinates in the images from each viewpoint are converted into physical coordinates according to the corresponding affine transformation matrix, and then projected onto the corresponding positions within the physical coordinate canvas framework to obtain the constructed physical coordinate canvas.

4. The image recognition-based SMT mounter failure prediction method according to claim 3, characterized by, The methods for obtaining the out-of-plane tilt angle and the in-plane rotation angle include: Edge detection is performed on the viewpoint image obtained from the motion side view position to extract the contour boundary of the component, and the main direction of the component extending in the viewpoint image plane is obtained by using the minimum bounding rectangle fitting method; with the central axis of the nozzle as the reference direction, the in-plane rotation angle is obtained by calculating the angle between the main direction of the component and the reference direction. Based on the component's contour boundary, the upper and lower boundary positions of the component in the motion side-view image are obtained. The projection height of the component is determined according to the distance between the upper and lower boundary positions. A preset standard projection height is introduced, and the difference between the component's projection height and the preset standard projection height is calculated. The difference is substituted into the projection height equation for inverse solution to obtain the out-of-plane tilt angle.

5. The image recognition-based SMT mounter failure prediction method according to claim 4, characterized by, The method for identifying the fault types of the pick-and-place machine and predicting the fault development trend includes: A fault mode library is pre-built, which stores different fault types of pick and place machines, and corresponding fault judgment rules are established for each fault type. The fault judgment rules include multi-stage existence state combination characteristics and pressure uniformity parameter change characteristics. The acquired multi-stage existence state combinations and pressure uniformity parameters are integrated to form a joint discriminant feature expression for fault type matching. Based on the joint discriminant feature expression, logical matching is performed with all fault determination rules to determine the fault mode corresponding to the current placement state, thereby identifying the fault type of the pick-and-place machine. Time series analysis is performed on the pressure uniformity parameters, and the fault development trend of the corresponding fault type is determined by calculating the slope of the change of the pressure uniformity parameters.

6. A fault prediction system for an SMT pick-and-place machine based on image recognition, used to implement the fault prediction method for an SMT pick-and-place machine based on image recognition as described in any one of claims 1 to 5, characterized in that, include: The image acquisition module is used to synchronously trigger industrial cameras set at the top view position of the nozzle, the side view position of the motion, and the top view position of a part of the PCB through the encoder signal of the pick-and-place machine spindle to acquire images from various perspectives; and to construct a physical coordinate canvas by mapping the acquired images from various perspectives based on the preset PCB reference positioning points. The deviation decomposition module is used to extract the spatial positional relationship between the nozzle, components and PCB pads in the physical coordinate canvas, construct three geometric association chains, and form a deviation representation of each stage of the mounting process. The method for constructing the three-segment geometric association chain includes: Based on a physical coordinate canvas, target detection is performed on images acquired from various perspectives, including the top-view position of the nozzle, the side-view position during movement, and the partial top-view position of the PCB, to identify the center coordinates of the nozzle, components, and PCB pads, respectively. Based on the center coordinates of the nozzle, components, and PCB pads... Based on the spatial relationship between them, construct three geometric association chains; The three geometric association chains are all presented in vector form. The first geometric association chain takes the center coordinates of the nozzle as the reference point and the center coordinates of the adsorbed component as the target point, forming the first positional association vector. The second geometric association chain takes the center coordinates of the adsorbed component as the starting point and the center coordinates of the PCB pad as the ending point, forming the second positional association vector. The third geometric association chain takes the center coordinates of the nozzle as the starting point and the center coordinates of the corresponding PCB pad as the ending point, forming the third positional association vector. The methods for obtaining the deviation characterization of each stage of the mounting process include: Based on the positional correlation vectors corresponding to the three geometric relationship chains, spatial direction decomposition and magnitude calculation are performed on each positional correlation vector in the coordinate system of the pick-and-place machine to obtain the corresponding component deviation and distance deviation; the component deviation and distance deviation corresponding to each positional correlation vector are combined to form the deviation characterization of each stage of the placement process. The attitude inversion module inverts the spatial attitude of the component on the nozzle based on the perspective image of the motion side view position to obtain the out-of-plane tilt angle and in-plane rotation angle. It also performs differential analysis based on the continuous perspective image of the local top view position of the PCB during the placement process to extract the solder paste deformation characteristics and construct pressure uniformity parameters characterizing the placement state. The method for constructing pressure uniformity parameters characterizing the mounting state includes: The top-view images of the PCB during the mounting process are arranged in chronological order and then time-aligned to form a continuous sequence of view images. Differential operations are performed on two adjacent view images in the sequence to obtain differential images that characterize the deformation of the solder paste under pressure. Region constraints are applied to the differential image based on the PCB pad area to obtain the solder paste effect area; the pixel grayscale change gradient is calculated within the solder paste effect area to form the local deformation intensity distribution of the solder paste. The deformation intensity distribution of local solder paste in the PCB pad area is accumulated and statistically analyzed to obtain the deformation energy parameter characterizing the overall pressure of the solder paste; the spatial variance of the local deformation intensity distribution of the solder paste is calculated to obtain the deformation uniformity parameter; based on the deformation energy parameter and the deformation uniformity parameter, the pressure uniformity parameter characterizing the mounting state is constructed. The existence determination module, based on deviation characterization, out-of-plane tilt angle and in-plane rotation angle, determines the existence of the component in the absorption phase, flight phase and landing phase, forming a multi-stage existence state combination. The method for obtaining the multi-stage existence state combination includes: Existence determination rules are constructed for the pick-up, motion, and placement stages of the mounting process to determine whether the components in the corresponding stages are in a valid existence state, and then the existence determination results of each stage are obtained. The existence determination results of each stage are combined according to the corresponding time sequence in the mounting process to form a multi-stage existence state combination that represents the components in the entire mounting process. For any stage, the existence determination rule is as follows: when the deviation representation of the corresponding stage is less than or equal to the preset deviation representation range threshold, the absolute value of the out-of-plane tilt angle is less than or equal to the preset out-of-plane tilt angle threshold, and the absolute value of the in-plane rotation angle is also less than or equal to the preset in-plane rotation angle threshold, when all three conditions are met, the element in the corresponding stage is determined to be in a valid existence state, and the existence determination result is 1; otherwise, the element in the corresponding stage is in an invalid existence state, and the existence determination result is 0. The fault prediction module is used to identify the fault type of the pick-and-place machine and predict the fault development trend by logically matching it with the multi-stage existence state combination and pressure uniformity parameters as the discrimination criteria and preset fault modes.

Images