Miniature image sensor positioning and assembling method and system based on visual compensation

Abstract

The invention relates to the technical field of micro part assembly, in particular to a micro image sensor positioning assembly method and system based on visual compensation, and the method comprises the steps: respectively obtaining image feature coordinates of a micro image sensor module and a to-be-assembled part through a non-orthogonal two-way microscopic visual system; controlling the non-orthogonal double-path microscopic vision system to perform focusing motion, collecting the variation of the image feature coordinates, and calibrating the offset mapping relation of the image coordinates; driving the to-be-assembled part to move in the three-dimensional working space, and calculating image offset space position error compensation of the focusing amount of the visual system based on the image coordinate offset mapping relation; calibrating a position mapping relation between the image feature space and the three-dimensional working space based on image offset space position error compensation; and converting the deviation of the image feature coordinates into the position deviation of the three-dimensional working space according to the position mapping relation. The problem that cross-depth-of-field position detection is not accurate when the miniature image sensor is assembled is solved, and the assembly precision is improved.

Description

Technical Field

[0001] This invention relates to the field of micro-part assembly technology, and more specifically to a method and system for positioning and assembling micro-image sensors based on vision compensation. Background Technology

[0002] Automated precision assembly of miniature image sensor modules is a key process in the manufacturing of electronic endoscopes. These modules are formed by welding together two types of micro-components: a micro-camera and a data cable. This assembly process is a typical example of high-precision assembly at the micro-nano scale. To achieve automated assembly of this module, existing technologies typically employ a servo control system based on multi-channel microscopic vision. Specifically, the vision system identifies the relative positions of the solder balls on the micro-camera pads and the data cable sub-lines. Subsequently, a position-based vision servo control strategy guides a micromanipulator to adjust the spatial pose of the data cable, aligning it with the pads. Finally, laser welding completes the fixing process. In this process, the precise detection and positioning of the micro-components by the vision system is fundamental to the entire assembly process and directly determines the final assembly accuracy.

[0003] However, existing technologies rely on vision systems to obtain clear images at a fixed focal length. Due to the inherent small field of view and shallow depth of field of microscopic optical systems, when there is an initial height difference between the micro-camera pads and the data cable, or when relative displacement occurs during alignment, target features are prone to deviating from the system's depth of field, resulting in out-of-focus and blurred images. This cross-depth-of-field condition introduces significant image feature point positioning errors, distorting the mapping relationship between image coordinates and spatial coordinates established based on the pinhole imaging model, thus causing inaccurate spatial position detection and affecting assembly accuracy. Summary of the Invention

[0004] To address the issue of inaccurate position detection across depth of field during the assembly of miniature image sensors and improve assembly accuracy, this invention provides a method and system for positioning and assembling miniature image sensors based on vision compensation. The specific technical solution adopted is as follows: The first aspect of the present invention provides a method for positioning and assembling a miniature image sensor based on vision compensation, the method comprising: The image feature coordinates of the miniature image sensor module and the component to be assembled are obtained separately using a non-orthogonal dual-channel microscopic vision system. Control the non-orthogonal dual-path microscopic vision system to perform focusing motion and acquire changes in image feature coordinates, and calibrate the image coordinate offset mapping relationship; Drive the component to be assembled to move in the three-dimensional workspace, and calculate the image offset spatial position error compensation of the focusing amount of the vision system based on the image coordinate offset mapping relationship; Based on image offset spatial position error compensation, the positional mapping relationship between the image feature space and the three-dimensional workspace is calibrated; Based on the position mapping relationship, the deviation of image feature coordinates is converted into position deviation in three-dimensional workspace, and the assembly of the component to be assembled is controlled to complete the assembly with the micro image sensor module.

[0005] Furthermore, controlling the non-orthogonal dual-path microscopic vision system to perform focusing motion and acquire changes in image feature coordinates, and calibrating the image coordinate offset mapping relationship, includes: The focusing axis of the non-orthogonal dual-path microscopic vision system is controlled to perform multiple equidistant reciprocating movements along the optical axis; During each reciprocating motion, the changes in the image feature coordinates are collected; Multiple sets of calibration data are formed by combining the focal axis motion generated by multiple reciprocating motions with the corresponding changes in image feature coordinates. Based on multiple sets of calibration data, the least squares method is used for fitting to calculate the image coordinate offset mapping relationship that describes the linear relationship between focusing motion and image coordinate changes.

[0006] Furthermore, image feature coordinates are compensated based on the image coordinate offset mapping relationship, including: During the process of driving the movement of the parts to be assembled, the image clarity is continuously monitored by a non-orthogonal dual-channel microscopic vision system; When the image of the part to be assembled is detected to be out of focus, the non-orthogonal dual-channel microscopic vision system is controlled to perform an automatic focusing operation. The amount of focus axis motion generated by the autofocus operation is input into the image coordinate offset mapping relationship to calculate the amount of image feature coordinate change that needs to be compensated. The compensated image feature coordinates are obtained by subtracting the change in image feature coordinates to be compensated from the image feature coordinates of the currently acquired miniature image sensor module and the component to be assembled.

[0007] Furthermore, based on the compensated image feature coordinates, the positional mapping relationship between the image feature space and the three-dimensional workspace is determined, including: Drive the component to be assembled to move along multiple directions with known displacements; Record the spatial position change corresponding to each known displacement in the three-dimensional workspace; Repeatedly run the focusing vision system and simultaneously record the focusing amount at each location point; Simultaneously acquire images of the current location and detect changes in the feature coordinates of the current image; Based on the image coordinate offset mapping relationship, the change in image feature coordinates after compensation is calculated; multiple sets of spatial position changes and multiple sets of compensated image feature coordinate changes are used as calibration samples. Based on the calibration samples, the positional mapping relationship between the image feature space and the three-dimensional workspace is obtained by least squares fitting.

[0008] Furthermore, the positional mapping relationship is obtained by combining the mapping relationships of the two microscopic vision paths, including: Based on the changes in the coordinates of the image features acquired by each microscopic vision and the corresponding changes in their spatial positions, and combined with the preset fixed transformation matrix of a single vision path, the mapping matrix of each microscopic vision path is derived. The mapping matrices of the two microscopic vision paths are combined row by row into an augmented matrix; A pseudo-inverse operation is performed on the augmented matrix to obtain a unified position mapping relationship.

[0009] Furthermore, the non-orthogonal dual-path microscopic vision system alters the optical path via a mirror, and the transformation between its camera coordinate system and the part's spatial coordinate system is achieved through a fixed transformation matrix, including: Calculate the refraction angles of the light path in the vertical and horizontal planes based on the installation angle of the reflector. Based on the refraction angle, calculate the rotation matrix from the camera coordinate system to the mirror coordinate system; The transformation matrix is ​​calculated by combining the rotation matrix, the origin offset between the camera coordinate system and the part space coordinate system, and the spatial positional relationship between the mirror coordinate system and the part space coordinate system.

[0010] Furthermore, after defining the positional mapping relationship between the image feature space and the three-dimensional workspace, the method further includes: Select multiple verification points in the three-dimensional workspace; The components to be assembled are moved sequentially to each verification point; Based on the aforementioned position mapping relationship, the real-time detected image feature coordinate deviations are converted into spatial position detection values; The spatial location detection value is compared with the actual spatial location value of the verification point.

[0011] Furthermore, the calibration process of the image coordinate offset mapping relationship and the position mapping relationship constitutes a self-calibration closed-loop process, including: The focusing motion and data acquisition are automatically performed during the calibration process; Automatically trigger a refocusing operation when out-of-focus is detected; Automatic coordinate compensation is performed based on the image coordinate offset mapping relationship.

[0012] Furthermore, based on the position mapping relationship, the deviation of the image feature coordinates is converted into a position deviation in the three-dimensional workspace, controlling the assembly of the component to be assembled with the miniature image sensor module, including: Adjust the motion trajectory of the parts to be assembled based on the positional deviation in the three-dimensional workspace; After aligning the component to be assembled with the miniature image sensor module, a four-point assembly welding operation is performed.

[0013] The second aspect of the present invention provides a vision-compensated micro-image sensor positioning and assembly system for performing the vision-compensated micro-image sensor positioning and assembly method described in the first aspect of the present invention. The system includes: The image acquisition module is configured to acquire the image feature coordinates of the micro image sensor module and the component to be assembled respectively through a non-orthogonal dual-channel microscopic vision system; The offset mapping calibration module is configured to control the non-orthogonal dual-channel microscopic vision system to perform focusing motion and acquire changes in image feature coordinates, and to calibrate the image coordinate offset mapping relationship. The motion compensation module is configured to drive the part to be assembled to move in the three-dimensional workspace and calculate the image offset spatial position error compensation of the focusing amount of the vision system based on the image coordinate offset mapping relationship. The position mapping calibration module is configured to calibrate the position mapping relationship between the image feature space and the three-dimensional workspace based on image offset spatial position error compensation; The motion control module is configured to convert the deviation of image feature coordinates into position deviation in a three-dimensional workspace based on the position mapping relationship, and control the component to be assembled to complete the assembly with the miniature image sensor module.

[0014] The present invention has the following beneficial effects: The present invention provides a micro image sensor positioning and assembly method based on visual compensation. By controlling the focusing motion of the control system and collecting the changes in image feature coordinates to calibrate the image coordinate offset mapping relationship, the method uses the image coordinate offset mapping relationship to compensate for visual detection errors under cross-depth-of-field conditions in real time. The micro image sensor positioning and assembly method based on visual compensation solves the problem of pose detection inaccuracy caused by depth-of-field limitations in microscopic vision systems. By establishing and utilizing the position mapping relationship between the compensated image feature coordinates and the three-dimensional workspace, visual perception and spatial positioning are coupled, thereby improving the accuracy of micro image sensor modules in the automated assembly process and achieving micron-level high-reliability assembly. Attached Figure Description

[0015] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart illustrating a method for positioning and assembling a miniature image sensor based on visual compensation, according to an embodiment of the present invention. Figure 2 This is a schematic diagram of a vision-compensated micro image sensor positioning and assembly system provided in one embodiment of the present invention. Detailed Implementation

[0017] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a vision-compensated micro-image sensor positioning and assembly method and system proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0019] The following description, in conjunction with the accompanying drawings, details the specific scheme of a vision-compensated micro image sensor positioning and assembly method and system provided by the present invention.

[0020] Please see Figure 1 The diagram illustrates a flowchart of a vision-compensated micro-image sensor positioning and assembly method according to an embodiment of the present invention, the method comprising: Step S100: The image feature coordinates of the miniature image sensor module and the component to be assembled are acquired using a non-orthogonal dual-channel microscopic vision system. This embodiment achieves this through a dual-channel microscopic vision system. The system comprises two camera-lens combinations arranged at a non-orthogonal angle above the assembly work area. A reflector changes the direction of light propagation, allowing simultaneous acquisition of image information from the miniature image sensor module and the component to be assembled from two complementary perspectives. The core principle relies on pinhole imaging to establish a precise mapping between spatial coordinates and image coordinates. The specific implementation process is as follows: The coordinates of the data cable's position in the workspace are: Coordinates in the feature space of the data cable image Image Jacobian matrix The relationship between the two has been established, and can be represented as:

[0021] In the formula, The number of groups of measurement data; This is a matrix composed of N sets of workspace position changes; It is a matrix composed of N sets of image feature changes; The optical imaging principle of microscopic vision is the pinhole imaging model, which means that the center point of the optical axis of the camera is equivalent to a pinhole, and the light from the target object is projected onto the imaging plane through the pinhole. For a single-path vision system, the relationship between the image feature space coordinates and the camera coordinate system can be expressed as:

[0022] In the formula, Represents the horizontal coordinates of the image feature space, corresponding to the pixel position in the column direction of the imaging plane; Represents the vertical coordinates in the image feature space, corresponding to the pixel position in the row direction of the imaging plane; This represents the horizontal coordinates of the target object in the camera coordinate system. This represents the vertical coordinates of the target object in the camera coordinate system. It represents the optical axis direction coordinates of the target object in the camera coordinate system and determines the image scaling ratio; This represents the camera intrinsic parameter matrix, which is a fixed 3×3 matrix obtained from the camera hardware parameter calibration. It represents the horizontal scaling factor of the camera, reflecting the conversion relationship between the horizontal pixel size and the physical space size; It represents the vertical scaling factor of the camera, reflecting the conversion relationship between the vertical pixel size and the physical space size; This represents the horizontal coordinate of the principal point of the imaging plane, i.e., the horizontal pixel position of the intersection of the optical axis and the imaging plane; This represents the vertical coordinates of the principal point on the imaging plane, i.e., the vertical pixel position of the intersection of the optical axis and the imaging plane; this formula first uses the camera intrinsic parameter matrix. A linear transformation is performed on the 3D coordinate representation in the camera coordinate system, and then the result is divided by the optical axis direction coordinate to complete the scaling correction of perspective projection. This maps the 3D spatial coordinates to 2D image feature coordinates, achieving a precise conversion from spatial position to image pixel.

[0023] In some embodiments, considering the deflection effect of the reflector on the light path, the transformation relationship between the camera coordinate system and the part space coordinate system of the assembly datum can be established as follows:

[0024]

[0025]

[0026] in, This represents the horizontal coordinate of the object in the part's spatial coordinate system. Represents the vertical coordinates of the object in the part's spatial coordinate system; This represents the height coordinate of the object in the part's spatial coordinate system. The rotation matrix between the camera coordinate system and the mirror coordinate system is a 3×3 orthogonal matrix that describes the change in the direction of light after passing through the mirror. The spatial position matrix representing the coordinate system of the reflector and the spatial coordinate system of the component is a 3×3 non-zero coefficient matrix, which is determined by the physical positional relationship during system installation. The matrix representing the offset of the origins of the camera coordinate system and the part space coordinate system is a 3×1 column vector, used to compensate for positional deviations caused by the non-coincidence of the origins of the two coordinate systems; angle. This represents the pitch angle of light refraction in the YOZ plane (i.e., the side-view plane) of the camera coordinate system under the influence of the mirror. Its value typically ranges from -90° to 90°. Positive values ​​usually indicate upward deflection of the light path, while negative values ​​indicate downward deflection. This represents the angle of refraction in the XOZ plane (i.e., the front view plane) of the camera coordinate system caused by the reflection of the light path. Its value ranges from -90° to 90°, and a positive value usually indicates that the light path is deflected to one side around a specific axis.

[0027] This allows us to derive the coordinate transformation matrix for single-channel microscopic vision. For a system with a fixed physical structure, the matrix is ​​a 3×3 constant matrix; it is used to describe the linear relationship between changes in image features and changes in spatial location. The single-path coordinate transformation formula is:

[0028]

[0029] In the formula, This represents the image feature coordinate vector for single-path vision. , The horizontal coordinates of the image acquired by a single camera. The vertical coordinates of the image acquired by a single camera; Indicates the target object's first The horizontal coordinates of each feature point in the part's spatial coordinate system; Indicates the target object's first The vertical coordinates of each feature point in the part's spatial coordinate system; Indicates the target object's first The height coordinates of each feature point in the part's spatial coordinate system; For a non-orthogonal dual-channel microscopic vision system, the image Jacobian matrices of the two cameras are respectively... and ,in Let be the Jacobian matrix of the image from the first camera, and be... The Jacobian matrix of the second camera image, and the overall image Jacobian matrix of the system, are formed by row-wise combination of the two matrices. Images of the miniature image sensor module and the component to be assembled are simultaneously acquired by two cameras. Key feature points of the two types of targets (such as the center of the pads of the miniature image sensor module and the contour feature points of the component to be assembled) are extracted respectively. Substituted into the single-channel coordinate transformation formula mentioned above, the image feature coordinates under the two cameras are obtained respectively. Then, through complementary fusion of dual-channel data, the accurate coordinates of the two types of targets in the image feature space are finally obtained, thus completing the implementation of step S100.

[0030] Step S100, through the multi-view collaborative acquisition design of a non-orthogonal dual-channel microscopic vision system, combined with a pinhole imaging model and precise multi-coordinate system transformation, achieves efficient and accurate acquisition of image feature coordinates of the miniature image sensor module and the component to be assembled. The dual-channel non-orthogonal arrangement overcomes the limitations of single-channel vision, enabling the acquisition of more comprehensive target feature information and avoiding feature occlusion or information loss issues under a single viewpoint. Simultaneously, the coordinate system transformation process in step S100, through the synergistic effect of intrinsic parameter matrices, rotation matrices, position matrices, and offset matrices, accurately establishes a linear mapping relationship between the three-dimensional spatial position and the two-dimensional image coordinates, ensuring the consistency and accuracy of coordinate transformation and eliminating coordinate deviations caused by system installation and optical path deflection. Finally, through the complementary fusion of dual-channel data, the acquisition accuracy and robustness of image feature coordinates are improved, providing high-quality raw data support for subsequent cross-depth-of-field error compensation and position mapping calibration.

[0031] Step S200: Control the non-orthogonal dual-path microscopic vision system to perform focusing motion and acquire the changes in image feature coordinates, and calibrate the image coordinate offset mapping relationship; Step S200 specifically includes: Step S210: Control the focusing axis of the non-orthogonal dual-path microscopic vision system to perform multiple equidistant reciprocating movements along the optical axis; In some embodiments, the number of reciprocating motions is greater than or equal to 5 to ensure sufficient calibration data is obtained; each reciprocating motion adopts a small step displacement design to ensure the sampling density of the focus position and cover the main focus range in cross-depth-of-field scenes; the motion trajectory starts from the initial focus position, moves along the positive optical axis with a fixed step size to the preset maximum travel, and then returns to the initial focus position along the negative optical axis with the same step size, so as to achieve full coverage of the focus plane in the cross-depth-of-field range; Step S220: During each reciprocating motion, the changes in image feature coordinates are collected; the micro image sensor is positioned and assembled based on visual compensation; specifically, at each step position of the focusing axis reciprocating motion, images of the micro image sensor module and the component to be assembled are simultaneously collected using a non-orthogonal dual-path microscopic vision system. Using the image feature coordinates collected at the initial focusing position as the reference coordinates, for the images collected at each step position, the image feature coordinates of the micro image sensor module and the component to be assembled are extracted, and the changes in the image feature coordinates of the two types of targets relative to the reference coordinates at each step position are calculated. The changes in the horizontal and vertical coordinates of the micro image sensor module and the horizontal and vertical coordinates of the component to be assembled are obtained respectively, and the focusing axis motion corresponding to each step position is recorded simultaneously.

[0032] Step S230: The focusing axis motion generated by multiple reciprocating movements is combined with the corresponding changes in image feature coordinates to form multiple sets of calibration data. Specifically, the focusing axis motion generated during multiple reciprocating movements is correlated and matched with the corresponding changes in image feature coordinates to form multiple sets of calibration data. Each set of calibration data includes three core parameters: the displacement of the focusing axis relative to its initial position, the change in image feature coordinates of the miniature image sensor module, and the change in image feature coordinates of the component to be assembled. Through multiple reciprocating movements, a high-density calibration dataset covering different depths of focus is formed, ensuring that the data samples can comprehensively reflect the correlation between focusing axis motion and changes in image feature coordinates across depth-of-field scenes.

[0033] Step S240: Based on multiple sets of calibration data, the least squares method is used for fitting to calculate the image coordinate offset mapping relationship that describes the linear relationship between focusing motion and image coordinate changes; For spatial pose estimation in microscopic vision, which is characterized by a small field of view and shallow depth of field, when the depth of the micro-part is within the depth of field, the change in three-dimensional spatial position and the change in image feature coordinates satisfy a linear mapping relationship:

[0034] In the formula, The image Jacobian matrix is ​​a 2×3 matrix that describes the linear mapping relationship between changes in three-dimensional spatial position and changes in two-dimensional image coordinates. The matrix elements are determined by the system structure. It represents the image space vector, that is, the change in image feature coordinates. It is a two-dimensional column vector, including the change in horizontal coordinates and the change in vertical coordinates. This represents a three-dimensional spatial vector, i.e., the change in position of the target object in the spatial coordinate system of the part, and is a three-dimensional column vector; this formula uses the image Jacobian matrix J to represent the three-dimensional spatial position change vector. Coordinate transformation vector converted to a two-dimensional image Establish a basic linear relationship between spatial motion and changes in image features to provide a benchmark model for subsequent error compensation; In some embodiments, for pose detection across depth of field where the depth is greater than the depth of field, the camera needs to refocus to ensure clear features. During the focusing motion, there is an angular deviation between the focusing axis and the camera optical axis, introducing position detection errors. This embodiment addresses the cross-depth-of-field detection scenario by performing image detection through camera focusing axis movement, calibrating the image offset Jacobian matrix, and compensating for the detection error, which can be expressed as:

[0035] In the formula, This represents the image offset vector across the depth of field, which is a two-dimensional column vector that includes the horizontal and vertical image offsets caused by installation deviations. The image offset Jacobian matrix is ​​a 2×3 matrix used to describe the linear mapping relationship between the focus axis motion and the image offset. The motion of the focus axis is represented by a three-dimensional column vector, including the horizontal displacement, vertical displacement, and optical axis displacement of the focus axis. This formula establishes a linear correlation model between the displacement and the motion of the focus axis by quantifying the additional image offset caused by the installation deviation of the focus axis, thus clarifying the source and quantification relationship of the error in cross-depth-of-field scenes. In some embodiments, by combining the basic mapping model and the cross-depth-of-field offset mapping model, a complete mapping formula including cross-depth-of-field error compensation is obtained:

[0036] This formula will account for the image offset caused by crossing the depth of field. As an error term, it is added to the basic mapping model. In this process, a complete model that considers both spatial location mapping and cross-depth-of-field error compensation is formed, achieving joint correction of installation deviations and depth-of-field effects, and improving the accuracy of the mapping relationship; then, using the least squares method, the image offset matrix of the i-th microscopic vision is calculated based on the calibration dataset, which can be expressed as:

[0037] In the formula, The set of cross-depth image offset vectors for the i-th camera is a 2×N matrix, where each row corresponds to the set of offsets in one direction and each column corresponds to the offset of a set of calibration data. The transpose matrix representing the motion vector of the focusing axis; The autocorrelation matrix represents the focus axis motion vector, which reflects the correlation between the components of the focus axis motion. This formula converts the focus axis motion vector into the appropriate dimension through matrix transpose, and then calculates the autocorrelation matrix and its inverse matrix through matrix multiplication, finally obtaining the image offset Jacobian matrix. This process ensures the optimal fitting accuracy of the offset mapping model by minimizing the sum of squared errors between the predicted and actual measured values ​​of the image offset.

[0038] In some embodiments, by combining the motion data of the micromanipulator with the defocus compensation result, the final cross-depth-of-field image Jacobian matrix is ​​obtained, which can be expressed as:

[0039] In the formula, To compensate for the set of changes in image feature coordinates after the shift across the depth of field, that is, the effective change after subtracting the shift across the depth of field from the change in the original image coordinates; It is the transpose of the vector of changes in position in three-dimensional space; The autocorrelation matrix of the three-dimensional spatial position change vector is used. The cross-depth offset is calculated by using the solved image offset Jacobian matrix and subtracted from the original image change to obtain the effective image change caused only by spatial position change. Then, the base image Jacobian matrix J is solved by the least squares method. Finally, a complete image coordinate offset mapping relationship that takes into account both spatial position mapping and cross-depth error compensation is established.

[0040] Step S200 constructs a high-density calibration dataset through multiple equidistant reciprocating movements of the focusing axis. Relying on the least squares method, it accurately calibrates the image coordinate offset mapping relationship, effectively solving the position detection error problem in cross-depth-of-field scenes. The reciprocating motion design of the focusing axis ensures that the calibration data covers different focusing depths, providing comprehensive sample support for the mapping model and avoiding fitting bias caused by local data. The least squares method achieves optimal estimation of the mapping parameters by minimizing the sum of squared errors, resulting in a linear mapping relationship with extremely high fitting accuracy, accurately quantifying the correlation between focusing axis movement and image offset. By separating the basic spatial mapping and the cross-depth-of-field offset mapping, it specifically compensates for image offset caused by focusing axis installation deviation, significantly reducing the position detection error in cross-depth-of-field scenes. This provides a reliable error correction model for subsequent position mapping relationship calibration and precise assembly, directly improving the position detection accuracy and stability of the miniature image sensor module assembly.

[0041] Step S300: Drive the part to be assembled to move in the three-dimensional workspace, and calculate the image offset spatial position error compensation of the focusing amount of the vision system based on the image coordinate offset mapping relationship; Step S300 specifically includes: Step S310: During the motion of the component to be assembled, the image clarity is continuously monitored by a non-orthogonal dual-channel microscopic vision system. While the component to be assembled is driven to move in the three-dimensional workspace by a micro-manipulator, the non-orthogonal dual-channel microscopic vision system remains in continuous operation, synchronously acquiring real-time images of the micro-image sensor module and the component to be assembled. The user manually judges whether the target image is in a clear state, providing a quantitative basis for subsequent defocusing judgment. Step S320: When the image of the part to be assembled is detected to be out of focus, the non-orthogonal dual-channel microscopic vision system is controlled to perform an autofocus operation. When the non-orthogonal dual-channel microscopic vision system detects that the sharpness quantification value of the image of the part to be assembled acquired by any one camera is lower than a preset threshold, it is determined that the part to be assembled is in a state of out of focus across the depth of field. At this time, the system triggers an autofocus control command, controlling the two focusing axes of the non-orthogonal dual-channel microscopic vision system to synchronously adjust in a stepwise manner along their respective optical axes. That is, the focusing axis moves gradually in a preset small step size. For each step size, the system acquires an image of the part to be assembled and calculates the sharpness index, until the sharpness quantification value of both images is higher than the preset threshold, and the focusing axis movement stops. During this process, the system records the total displacement of the focusing axis from the out-of-focus position to the sharp position in real time. This displacement is the amount of focusing axis movement generated by the autofocus operation. Step S330: Input the focus axis motion generated by the autofocus operation into the image coordinate offset mapping relationship, and calculate the change in image feature coordinates that needs to be compensated; specifically, based on the image coordinate offset mapping relationship calibrated in step S200, i.e., the image offset Jacobian matrix. This mapping relationship, through fitting the reciprocating motion of the focus axis with the least squares method, accurately establishes a linear correlation between the amount of focus axis motion and the change in image feature coordinates; the amount of focus axis motion recorded in step S320 is then used to... As input parameters, we substitute them into the linear formula corresponding to the image coordinate offset mapping relationship. The calculation is performed, and the result is... This is a two-dimensional vector, representing the change in image feature coordinates of the component to be assembled due to focusing across depth of field, requiring compensation. It includes both horizontal and vertical compensation amounts. Simultaneously, based on the collaborative characteristics of dual-path vision, the compensation amounts for both cameras are calculated separately and synchronized to ensure consistency in dual-path image compensation.

[0042] Step S340: Subtract the image feature coordinate change to be compensated from the currently acquired image feature coordinates of the miniature image sensor module and the component to be assembled to obtain the compensated image feature coordinates. Specifically, the orthogonal dual-channel microscopic vision system acquires the original image feature coordinates of the miniature image sensor module and the component to be assembled in real time, including the coordinates of the miniature image sensor module and the component to be assembled. The image feature coordinate change to be compensated calculated in step S330 is applied to the original image feature coordinates of the two types of targets respectively. Through coordinate difference calculation, the corresponding compensation amount is subtracted from the original coordinates. This difference calculation essentially cancels the image offset error caused by the deviation of the focusing axis installation angle, so that the compensated image feature coordinates can truly reflect the actual spatial position relationship of the two types of targets. After the dual cameras complete the coordinate compensation under their respective fields of view, the system performs fusion verification on the coordinate data after dual-channel compensation. After ensuring data consistency, it is stored as valid input data for subsequent position mapping calibration.

[0043] This embodiment achieves dynamic compensation of image feature coordinates across depth-of-field scenes through a closed-loop process of real-time image sharpness monitoring, automatic focus triggering, precise compensation calculation, and coordinate correction. Relying on the continuous monitoring capability of the non-orthogonal dual-channel microscopic vision system, it can quickly respond to defocusing states during the movement of the component to be assembled. Automatic focusing ensures clear acquisition of image features, while compensation calculations based on the previously calibrated image coordinate offset mapping accurately offset image offset errors caused by focusing axis installation deviations and depth-of-field effects. This allows the compensated image feature coordinates to truly and accurately reflect the actual spatial positional relationship between the miniature image sensor module and the component to be assembled, effectively solving the problem of inaccurate position detection caused by movement across depth of field and ensuring the stability and accuracy of position detection during assembly.

[0044] Step S400: Based on image offset spatial position error compensation, calibrate the positional mapping relationship between the image feature space and the three-dimensional workspace; Step S400 specifically includes: Step S410: Drive the component to be assembled to move along multiple directions with known displacements. Specifically, using the image feature coordinates compensated in step S300 as a reference, the component to be assembled is held by the end effector of the micro-manipulator and driven to actively move along multiple preset directions with known displacements in the three-dimensional workspace. The movement directions must cover the X-axis, Y-axis, Z-axis, and at least two oblique composite directions of the three-dimensional workspace to ensure that the movement data comprehensively reflects the correlation between spatial position changes and image feature changes. The known displacements use micrometer-level steps, and the displacement for each movement is a preset fixed value. After movement, the position of the component to be assembled remains stable. During movement, if the non-orthogonal dual-channel microscopic vision system detects that the image of the component to be assembled is out of focus, the movement is immediately paused, and the automatic focusing and coordinate compensation operation of step S300 is executed. Movement resumes after compensation is completed.

[0045] Step S420: Record the spatial position change corresponding to each known displacement in the three-dimensional workspace; based on the part's spatial coordinate system, the micro-manipulator motion controller records the spatial position coordinates of the part to be assembled before and after each known displacement in real time. Calculate the difference between the coordinates after movement and the coordinates before movement to obtain the spatial position change corresponding to each known displacement. This change includes displacement change information in the X, Y, and Z axes. After each known displacement is completed, the spatial position change and the corresponding movement direction identifier are stored synchronously to form a spatial position change dataset.

[0046] Step S430: Repeatedly focus the vision system to ensure image clarity and simultaneously record the focus level of each position point; simultaneously acquire the image at the current position and detect the change in the feature coordinates of the current image; calculate the compensated change in the feature coordinates of the image based on the image coordinate offset mapping relationship; Specifically, after the component to be assembled completes each known displacement movement and stabilizes, the system executes this step to obtain the precisely corresponding image feature data.

[0047] First, the system controls a non-orthogonal dual-channel microscopic vision system to automatically focus on the current target position. By calculating the image sharpness evaluation value in real time, the system drives the focusing axes of each camera to fine-tune until the images of the acquired miniature image sensor module and the part to be assembled are both in optimal sharpness, and records the adjustment amount of each focusing axis during this process. Next, with the image sharpness stable, the system synchronously acquires images taken by the dual cameras at the current displacement point. Using image processing algorithms, the precise pixel coordinates of feature points on the miniature image sensor module and feature points on the part to be assembled are identified and extracted in the two images, i.e., the original image feature coordinates at the current moment. Then, the system calls a pre-calibrated image coordinate offset mapping relationship. Based on the focusing axis adjustment amount recorded in the previous steps, the theoretical image coordinate offset value introduced by this focusing operation is calculated through this mapping relationship. Subsequently, this theoretical offset value is subtracted from the original image feature coordinates, thereby obtaining the compensated and corrected image feature coordinates that more accurately reflect the true spatial position relationship.

[0048] Finally, for each known displacement segment, the difference between the feature coordinates of the endpoint image and the feature coordinates of the starting image after the above compensation processing is calculated, thus obtaining the change in image feature coordinates that precisely corresponds to the spatial displacement segment. This change data will serve as a key calibration sample for subsequent calibration of the positional mapping relationship between the image feature space and the three-dimensional workspace.

[0049] Step S440: Use multiple sets of spatial position changes and multiple sets of compensated image feature coordinate changes as calibration samples; match multiple sets of spatial position changes with corresponding multiple sets of image feature coordinate changes to form a calibration sample set. The number of samples should be no less than 10 sets, covering different motion directions and including duplicate samples, to ensure that the samples can cover the main motion scenes in the 3D workspace. At the same time, verify the validity of the calibration samples, remove abnormal samples with no obvious correlation, and retain valid samples for subsequent fitting.

[0050] Step S450: Based on the calibration samples, a positional mapping relationship describing the image feature space and the three-dimensional workspace is obtained through least squares fitting. A linear correlation is established between the image feature space and the three-dimensional workspace based on the effective calibration sample set. By using least squares fitting, the error between the predicted and actual measured values ​​of image feature changes is minimized, resulting in a positional mapping relationship describing the correlation between the two. This mapping relationship is determined by the system's physical structure and is a fixed correlation matrix. After fitting, at least five verification points are selected in the three-dimensional workspace, the true spatial positions of each verification point are recorded, the corresponding image feature coordinates are collected and compensated, the predicted spatial position change is calculated through the mapping relationship, and the predicted value is compared with the true value. If the error meets the preset requirements, the positional mapping relationship calibration is deemed valid.

[0051] Step S400 establishes a positional mapping relationship between the image feature space and the three-dimensional workspace. Specifically, relying on the high-precision motion control of the micro-manipulator and the compensated data from non-orthogonal dual-path vision, it ensures that the calibration samples can truly reflect the linear correlation between changes in spatial position and changes in image features, eliminating system interference such as cross-depth-of-field errors and focus shifts. The image Jacobian matrix obtained by least-squares fitting quantifies the mapping law between two-dimensional image features and three-dimensional spatial position, and the combined fitting of dual-path vision further improves the mapping accuracy and robustness. Finally, the established positional mapping relationship provides a mathematical basis for the subsequent conversion of image deviation and spatial deviation, directly solving the problem of connecting visual inspection and spatial control in the assembly of micro image sensors. In some embodiments, the position mapping relationship is obtained by combining the mapping relationships of the two microscopic vision systems, including: deriving the mapping matrix of each microscopic vision system based on the changes in image feature coordinates and corresponding spatial position changes acquired by each system, combined with a preset fixed transformation matrix of a single vision system; combining the mapping matrices of the two vision systems row-wise to form an augmented matrix; and performing a pseudo-inverse operation on the augmented matrix to obtain a unified position mapping relationship. First, for each vision system, based on the changes in image feature coordinates and corresponding spatial position changes acquired individually, and combined with a fixed transformation matrix determined by the system's physical structure, a dedicated mapping matrix for that vision system is independently calculated. Then, the two mapping matrices are combined row-wise to construct an augmented matrix, and finally, a pseudo-inverse operation is performed on the augmented matrix to obtain a unified position mapping relationship that comprehensively reflects the information from both vision systems. This embodiment, by fusing the observation information from the two vision systems, fully utilizes the multi-view advantage brought by the non-orthogonal configuration, improves the robustness and accuracy of the position mapping relationship, and provides a more reliable spatial positioning basis for precision assembly.

[0052] In some embodiments, the non-orthogonal dual-path microscopic vision system alters the optical path via a mirror. The transformation between the camera coordinate system and the part space coordinate system is achieved through a fixed transformation matrix, including: calculating the refraction angles of the optical path in the vertical and horizontal planes based on the mirror mounting angle; calculating the rotation matrix from the camera coordinate system to the mirror coordinate system based on the refraction angles; and calculating the transformation matrix by combining the rotation matrix, the origin offset between the camera coordinate system and the part space coordinate system, and the spatial positional relationship between the mirror coordinate system and the part space coordinate system. This embodiment establishes an accurate coordinate system transformation model, ensuring the consistency between visual measurement results and physical spatial positions, and providing an accurate geometric basis for the entire visual positioning system.

[0053] In some embodiments, after calibrating the positional mapping relationship between the image feature space and the three-dimensional workspace, the method further includes: selecting multiple verification points in the three-dimensional workspace; driving the component to be assembled to move sequentially to each verification point; converting the real-time detected image feature coordinate deviation into a spatial position detection value based on the positional mapping relationship; comparing the spatial position detection value with the actual spatial position value of the verification point, and determining whether the difference between the two meets the assembly accuracy requirements. This embodiment, through the detection and comparison of verification points in multiple regions, can effectively evaluate the accuracy and reliability of the calibrated positional mapping relationship, promptly detect deviations in the mapping relationship, and avoid positional errors in subsequent assembly due to inaccurate mapping relationships, thus providing quality assurance for high-precision assembly.

[0054] In some embodiments, the calibration process of the image coordinate offset mapping relationship and the position mapping relationship constitutes a self-calibration closed-loop process, including: automatically executing focusing motion and data acquisition during the calibration process; automatically triggering a refocusing operation when defocusing is detected; and automatically performing coordinate compensation based on the image coordinate offset mapping relationship. Specifically, after calibration starts, the system automatically controls the focusing axis of the non-orthogonal dual-channel microscopic vision system to perform reciprocating motion, while simultaneously acquiring the amount of focusing axis motion and the corresponding changes in image feature coordinates. During the acquisition process, the system continuously monitors the image sharpness. When defocusing is detected in the image of the part to be assembled, a refocusing operation is automatically triggered to adjust the position of the focusing axis until the image is clear again. Afterward, the system calls the initially calibrated image coordinate offset mapping relationship and automatically compensates and corrects the currently acquired image feature coordinates to ensure the accuracy of subsequent calibration data. This embodiment realizes the automated closed-loop operation of the calibration process, reduces errors and efficiency losses caused by manual operation, and ensures the accuracy and stability of calibration data through real-time defocus detection and automatic compensation, thereby improving the efficiency and reliability of the entire calibration process.

[0055] Step S500: Based on the position mapping relationship, the deviation of the image feature coordinates is converted into the position deviation in the three-dimensional workspace, and the assembly of the component to be assembled is controlled to complete the assembly with the micro image sensor module. Step S500 specifically includes: Step S510: Based on the positional deviation in the three-dimensional workspace, adjust the motion trajectory of the component to be assembled. The system calculates the image feature coordinate deviation between the micro image sensor module and the component to be assembled in real time, and converts this two-dimensional image deviation into a positional deviation in the three-dimensional workspace through the position mapping relationship calibrated in step S400. Based on this positional deviation vector, a position-based visual servo control strategy is adopted to generate motion control commands for the micro-manipulator. Through a multi-degree-of-freedom precision motion platform, the motion trajectory of the component to be assembled is adjusted in real time, causing it to converge towards the target position along the optimal path.

[0056] Step S520: After aligning the component to be assembled with the miniature image sensor module, perform a four-point assembly welding operation. When the system detects that the positional deviation between the component to be assembled and the miniature image sensor module is less than a preset threshold, the alignment is considered complete. The four-point assembly welding program is then triggered, and laser welding is performed on the four weld points sequentially according to a predetermined welding sequence. During the welding process, the welding quality is monitored by a follow-up vision system to ensure the welding strength and consistency of each weld point, completing the final assembly and connection of the miniature image sensor module and the data cable.

[0057] Step S500, based on the previously calibrated position mapping relationship, achieves a precise conversion of image feature deviation to three-dimensional spatial deviation. Combined with the real-time trajectory adjustment of the micro-manipulator, it solves the micron-level alignment problem between the micro image sensor module and the component to be assembled in cross-depth-of-field scenarios. Based on this method, the high-precision and high-stability automated assembly task of the micro image sensor module can be completed.

[0058] In summary, the vision-compensated micro-image sensor positioning and assembly method provided by this invention controls the focusing motion of the system and collects the changes in image feature coordinates to calibrate the image coordinate offset mapping relationship. Then, the image coordinate offset mapping relationship is used to compensate for visual detection errors in cross-depth-of-field conditions in real time. The vision-compensated micro-image sensor positioning and assembly method solves the problem of pose detection inaccuracy caused by depth-of-field limitations in microscopic vision systems. By establishing and utilizing the position mapping relationship between the compensated image feature coordinates and the three-dimensional workspace, visual perception and spatial positioning are coupled, thereby improving the accuracy of micro-image sensor modules in automated assembly and achieving micron-level high-reliability assembly.

[0059] Please see Figure 2 The diagram illustrates a structural schematic of a vision-compensated miniature image sensor positioning and assembly system according to an embodiment of the present invention, the system comprising: The image acquisition module is configured to acquire the image feature coordinates of the micro image sensor module and the component to be assembled respectively through a non-orthogonal dual-channel microscopic vision system; The offset mapping calibration module is configured to control the non-orthogonal dual-channel microscopic vision system to perform focusing motion and acquire changes in image feature coordinates, and to calibrate the image coordinate offset mapping relationship. The motion compensation module is configured to drive the part to be assembled to move in the three-dimensional workspace and calculate the image offset spatial position error compensation of the focusing amount of the vision system based on the image coordinate offset mapping relationship. The position mapping calibration module is configured to calibrate the position mapping relationship between the image feature space and the three-dimensional workspace based on image offset spatial position error compensation; The motion control module is configured to convert the deviation of image feature coordinates into position deviation in a three-dimensional workspace based on the position mapping relationship, and control the component to be assembled to complete the assembly with the miniature image sensor module.

[0060] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0061] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

Claims

1. A method for positioning and assembling a miniature image sensor based on vision compensation, characterized in that, The method includes: The image feature coordinates of the miniature image sensor module and the component to be assembled are obtained separately using a non-orthogonal dual-channel microscopic vision system. Control the non-orthogonal dual-path microscopic vision system to perform focusing motion and acquire changes in image feature coordinates, and calibrate the image coordinate offset mapping relationship; Drive the component to be assembled to move in the three-dimensional workspace, and calculate the image offset spatial position error compensation of the focusing amount of the vision system based on the image coordinate offset mapping relationship; Based on image offset spatial position error compensation, the positional mapping relationship between the image feature space and the three-dimensional workspace is calibrated; Based on the position mapping relationship, the deviation of image feature coordinates is converted into position deviation in three-dimensional workspace, and the assembly of the component to be assembled is controlled to complete the assembly with the micro image sensor module.

2. The method for positioning and assembling a miniature image sensor based on visual compensation as described in claim 1, characterized in that, Controlling the non-orthogonal dual-path microscopic vision system to perform focusing motion and acquire changes in image feature coordinates, calibrating the image coordinate offset mapping relationship, including: The focusing axis of the non-orthogonal dual-path microscopic vision system is controlled to perform multiple equidistant reciprocating movements along the optical axis; During each reciprocating motion, the changes in the image feature coordinates are collected; Multiple sets of calibration data are formed by combining the focal axis motion generated by multiple reciprocating motions with the corresponding changes in image feature coordinates. Based on multiple sets of calibration data, the least squares method is used for fitting to calculate the image coordinate offset mapping relationship that describes the linear relationship between focusing motion and image coordinate changes.

3. The method for positioning and assembling a miniature image sensor based on visual compensation as described in claim 2, characterized in that, Image feature coordinates are compensated based on image coordinate offset mapping, including: During the process of driving the movement of the parts to be assembled, the image clarity is continuously monitored by a non-orthogonal dual-channel microscopic vision system; When the image of the part to be assembled is detected to be out of focus, the non-orthogonal dual-channel microscopic vision system is controlled to perform an automatic focusing operation. The amount of focus axis motion generated by the autofocus operation is input into the image coordinate offset mapping relationship to calculate the amount of image feature coordinate change that needs to be compensated. The compensated image feature coordinates are obtained by subtracting the change in image feature coordinates to be compensated from the image feature coordinates of the currently acquired miniature image sensor module and the component to be assembled.

4. The method for positioning and assembling a miniature image sensor based on visual compensation as described in claim 1, characterized in that, Based on the compensated image feature coordinates, the positional mapping relationship between the image feature space and the three-dimensional workspace is determined, including: Drive the component to be assembled to move along multiple directions with known displacements; Record the spatial position change corresponding to each known displacement in the three-dimensional workspace; Repeatedly run the focusing vision system and simultaneously record the focusing amount at each location point; Simultaneously acquire images of the current location and detect changes in the feature coordinates of the current image; Based on the image coordinate offset mapping relationship, the change in image feature coordinates after compensation is calculated; multiple sets of spatial position changes and multiple sets of compensated image feature coordinate changes are used as calibration samples. Based on the calibration samples, the positional mapping relationship between the image feature space and the three-dimensional workspace is obtained by least squares fitting.

5. The method for positioning and assembling a miniature image sensor based on visual compensation as described in claim 4, characterized in that, The positional mapping relationship is obtained by combining the mapping relationships of the two microscopic vision paths, including: Based on the changes in the coordinates of the image features acquired by each microscopic vision and the corresponding changes in their spatial positions, and combined with the preset fixed transformation matrix of a single vision path, the mapping matrix of each microscopic vision path is derived. The mapping matrices of the two microscopic vision paths are combined row by row into an augmented matrix; A pseudo-inverse operation is performed on the augmented matrix to obtain a unified position mapping relationship.

6. The method for positioning and assembling a miniature image sensor based on visual compensation as described in claim 1, characterized in that, The non-orthogonal dual-path microscopic vision system alters the optical path via a mirror, and the transformation between its camera coordinate system and the part's spatial coordinate system is achieved through a fixed transformation matrix, including: Calculate the refraction angles of the light path in the vertical and horizontal planes based on the installation angle of the reflector. Based on the refraction angle, calculate the rotation matrix from the camera coordinate system to the mirror coordinate system; The transformation matrix is ​​calculated by combining the rotation matrix, the origin offset between the camera coordinate system and the part space coordinate system, and the spatial positional relationship between the mirror coordinate system and the part space coordinate system.

7. The method for positioning and assembling a miniature image sensor based on visual compensation as described in claim 1, characterized in that, After determining the positional mapping relationship between the image feature space and the three-dimensional workspace, the method further includes: Select multiple verification points in the three-dimensional workspace; The components to be assembled are moved sequentially to each verification point; Based on the aforementioned position mapping relationship, the real-time detected image feature coordinate deviations are converted into spatial position detection values; The spatial location detection value is compared with the actual spatial location value of the verification point.

8. The method for positioning and assembling a miniature image sensor based on visual compensation as described in any one of claims 1 to 7, characterized in that, The calibration process for the image coordinate offset mapping relationship and position mapping relationship constitutes a self-calibration closed-loop process, including: The focusing motion and data acquisition are automatically performed during the calibration process; Automatically trigger a refocusing operation when out-of-focus is detected; Automatic coordinate compensation is performed based on the image coordinate offset mapping relationship.

9. The method for positioning and assembling a miniature image sensor based on visual compensation as described in claim 8, characterized in that, Based on the position mapping relationship, the deviation of image feature coordinates is converted into position deviation in three-dimensional workspace, controlling the assembly of the component to be assembled with the miniature image sensor module, including: Adjust the motion trajectory of the parts to be assembled based on the positional deviation in the three-dimensional workspace; After aligning the component to be assembled with the miniature image sensor module, a four-point assembly welding operation is performed.

10. A positioning and assembly system for a miniature image sensor based on vision compensation, characterized in that, The system is used to perform the vision-compensated micro-image sensor positioning and assembly method according to any one of claims 1 to 9, the system comprising: The image acquisition module is configured to acquire the image feature coordinates of the micro image sensor module and the component to be assembled respectively through a non-orthogonal dual-channel microscopic vision system; The offset mapping calibration module is configured to control the non-orthogonal dual-channel microscopic vision system to perform focusing motion and acquire changes in image feature coordinates, and to calibrate the image coordinate offset mapping relationship. The motion compensation module is configured to drive the part to be assembled to move in the three-dimensional workspace and calculate the image offset spatial position error compensation of the focusing amount of the vision system based on the image coordinate offset mapping relationship. The position mapping calibration module is configured to calibrate the position mapping relationship between the image feature space and the three-dimensional workspace based on image offset spatial position error compensation; The motion control module is configured to convert the deviation of image feature coordinates into position deviation in a three-dimensional workspace based on the position mapping relationship, and control the component to be assembled to complete the assembly with the miniature image sensor module.

Images