A method and device for detecting a screen micro gesture, and a storage medium

Abstract

This application discloses a method, apparatus, and storage medium for detecting the microscopic posture of a screen, aimed at improving the detection accuracy of screens. The screen to be tested is placed on the motion platform of a microscopic optical system. An image of the screen is acquired using a microscopic camera within the system, generating a detection image. A clear band region is located within the detection image. Clear band features, including clear band direction and width information, are extracted from the clear band region. A plane normal vector of the screen to be tested is calculated based on the clear band features. After Z-axis displacement of the microscopic camera, the intersection lines between the screen plane and the camera's focusing plane before and after displacement are defined as the first and second intersection lines, respectively. The ambiguity of the plane normal vector is eliminated based on the changing trend between the first and second intersection lines. The posture of the screen to be tested is detected based on the eliminated plane normal vector, generating screen posture data.

Description

Technical Field

[0001] This application relates to the field of screen posture detection, and more particularly to a method, apparatus and storage medium for detecting the microscopic posture of a screen. Background Technology

[0002] In the field of microdisplay, most scenarios require the orientation measurement and adjustment of the produced screen to keep the screen plane parallel to the camera image plane. This ensures that the image captured by the camera is clear in all parts, thereby improving the accuracy of subsequent measurements and defect detection.

[0003] Traditional screen pose measurement methods require a large depth of field in the camera to clearly capture the detection image of the screen even when it is tilted. Then, based on the point-to-point relationship between spatial feature points and image feature points, combined with camera internal parameters, the current screen pose is calculated. Finally, based on the pose calculation, hand-eye calibration of the camera and motion platform system is performed, thereby aligning the screen with the image plane.

[0004] However, as the integration level of screen pixels continues to increase and the internal hierarchical structure of screens becomes increasingly sophisticated, current screens require high-precision microscopic optical systems to meet detection accuracy requirements. The depth of field of optical systems in the micro-display field is typically at the micrometer level, which differs significantly from the optical systems used in traditional screen attitude measurement methods. This means that even a slight positional difference between the motion platform on which the screen is located and the camera will result in a certain tilt angle between the screen and the camera, leading to large blurred areas in the image captured by the camera. In this situation, it is impossible to accurately obtain the coordinates of feature points in the detection image, nor can the screen attitude be determined under the microscopic optical system. Consequently, the leveling process of the screen under test cannot be achieved, resulting in blurred images in subsequent detection processes and reducing the detection accuracy of the screen. Summary of the Invention

[0005] This application addresses the problem that conventional visual servoing methods for aligning the product plane with the image plane in microscopic systems suffer from the inability to determine the screen pose due to the extremely small depth of field. This application discloses a method, apparatus, and storage medium for detecting the microscopic pose of a screen, which improves the detection accuracy of the screen.

[0006] In a first aspect, embodiments of this application provide a method for detecting the microscopic pose of a screen, comprising: The screen to be tested is placed on the motion platform of the micro-optical system, and the screen to be tested is captured by the micro-camera of the micro-optical system to generate a test image. Locate and detect clear band regions in the image; Extract the sharp band features from the sharp band region. The sharp band features include sharp band orientation information and sharp band width information. Calculate the plane normal vector of the screen under test based on the clear band features; After the microscope camera is displaced along the Z-axis, the intersection lines between the plane of the screen to be measured and the focal plane of the camera before and after the displacement are defined as the first intersection line and the second intersection line, respectively. Eliminate the ambiguity of the plane normal vector based on the changing trend between the first and second intersection lines; The screen pose is detected based on the unambiguous plane normal vector, and screen pose data is generated.

[0007] Optionally, the steps for extracting sharp band features from the sharp band region specifically include: After dividing the clear band region into partitions and calculating the focus value, threshold segmentation is then performed to generate clear band region features. The direction and width of the clear band in the detected image are obtained by analyzing the clear band features.

[0008] Optionally, the step of calculating the planar normal vector of the screen under test based on the clear band features specifically includes: The X-axis and Y-axis components of the plane normal vector are determined based on the clear directional information. The Z-axis component of the plane normal vector is determined based on the clear band width information, and the X-axis component, Y-axis component and Z-axis component are integrated to generate the plane normal vector.

[0009] Optionally, after the step of performing pose detection on the screen to be tested based on the unambiguous plane normal vector to generate screen pose data, the detection method further includes: Adjust the motion platform to perform hand-eye calibration and visual servoing of the microscopic optical system based on screen posture data under different poses.

[0010] Optionally, adjusting the motion platform and performing hand-eye calibration and visual servoing of the microscopic optical system based on screen posture data under different poses specifically includes: The position of the end of the motion platform is repeatedly adjusted so that the screen under test is in different poses while within the field of view of the microscope camera, thereby acquiring the motion data of the motion platform. Calculate the end-effector pose data of the motion platform based on the platform's motion data; Images are acquired using a microscope camera, and screen pose data under different poses are obtained. The hand-eye matrix data of the micro-optical system is obtained by solving the screen pose data, platform end pose data and hand-eye model under different poses. The adjustment motion of the motion platform when the screen under test reaches the target posture is calculated based on the target posture matrix and hand-eye matrix data of the screen under test. The pose of the screen under test is adjusted by controlling the amount of exercise on the motion platform.

[0011] Optionally, after adjusting the motion platform and performing hand-eye calibration and visual servoing of the microscopic optical system based on screen pose data under different poses, the detection method further includes: The microscope camera that controls the microscopic optical system performs automatic focusing.

[0012] Optionally, the steps for locating and detecting clear band regions in the image specifically include: Edge detection of the effective screen region in an image; Perform initial sharpness analysis within the effective area of ​​the screen to generate an initial sharp area; Obtain a sharpness threshold, and merge the remaining areas with sharpness greater than the sharpness threshold into the initial sharp area to generate a sharp band area.

[0013] Optionally, before the steps of placing the screen under test on the motion platform of the microscopic optical system, acquiring an image of the screen under test through the microscopic camera of the microscopic optical system, and generating a detection image, the detection method further includes: Place the screen under test on the motion platform of the micro-optical system and input a preset dot matrix image into the screen under test; Acquire a planar image of the screen and extract the coordinates of the dot matrix calibration points in the planar image; Based on the consistency relationship between the orthogonal components of the screen planar image and the line connecting the principal point of the image and the feature point of the product planar image in the direction, the external parameters other than the translation component of the optical axis direction are obtained. The orthogonal component is the orthogonal component of the projection of the line connecting the feature point of the screen planar image and the optical center onto the optical axis. Determine the internal parameters based on the external parameters; The microscopic optical system is calibrated based on external parameters, internal parameters, and coordinates of the lattice calibration points.

[0014] Secondly, embodiments of this application provide a device for detecting the microscopic posture of a screen, comprising: The detection image generation unit is used to place the screen under test on the motion platform of the microscopic optical system, and to acquire images of the screen under test through the microscopic camera of the microscopic optical system to generate a detection image. The clear band region localization unit is used to locate clear band regions in the detection image; The sharp band feature extraction unit is used to extract sharp band features in the sharp band region. The sharp band features include sharp band orientation information and sharp band width information. The planar normal vector calculation unit is used to calculate the planar normal vector of the screen under test based on the clear band features; The intersection line determination unit is used to define the intersection lines of the microscope camera with the screen plane before and after displacement as the first intersection line and the camera focusing plane, respectively, after the microscope camera is displaced along the Z-axis. Ambiguity elimination unit, used to eliminate ambiguity of plane normal vectors based on the changing trend between the first intersection line and the second intersection line; The screen pose data generation unit is used to perform pose detection on the screen under test based on the unambiguous plane normal vector and generate screen pose data.

[0015] Optionally, the clear band feature extraction unit specifically includes: After dividing the clear band region into partitions and calculating the focus value, threshold segmentation is then performed to generate clear band region features. The direction and width of the clear band in the detected image are obtained by analyzing the clear band features.

[0016] Optionally, the plane normal vector calculation unit specifically includes: The X-axis and Y-axis components of the plane normal vector are determined based on the clear directional information. The Z-axis component of the plane normal vector is determined based on the clear band width information, and the X-axis component, Y-axis component and Z-axis component are integrated to generate the plane normal vector.

[0017] Optionally, after the screen pose data generation unit, the detection device further includes: The hand-eye calibration and visual servo unit is used to adjust the motion platform and perform hand-eye calibration and visual servoing of the microscopic optical system based on screen posture data under different poses.

[0018] Optionally, the hand-eye calibration and vision servoing unit specifically includes: The position of the end of the motion platform is repeatedly adjusted so that the screen under test is in different poses while within the field of view of the microscope camera, thereby acquiring the motion data of the motion platform. Calculate the end-effector pose data of the motion platform based on the platform's motion data; Images are acquired using a microscope camera, and screen pose data under different poses are obtained. The hand-eye matrix data of the micro-optical system is obtained by solving the screen pose data, platform end pose data and hand-eye model under different poses. The adjustment motion of the motion platform when the screen under test reaches the target posture is calculated based on the target posture matrix and hand-eye matrix data of the screen under test. The pose of the screen under test is adjusted by controlling the amount of exercise on the motion platform.

[0019] Optionally, after the hand-eye calibration and visual servoing unit, the detection device also includes: The autofocus processing unit is used to control the microscope camera of the microscope optical system to perform autofocus processing.

[0020] Optionally, the clear band area positioning unit specifically includes: Edge detection of the effective screen region in an image; Perform initial sharpness analysis within the effective area of ​​the screen to generate an initial sharp area; Obtain a sharpness threshold, and merge the remaining areas with sharpness greater than the sharpness threshold into the initial sharp area to generate a sharp band area.

[0021] Optionally, before detecting the image generation unit, the detection device further includes: The dot matrix image input unit is used to place the screen under test on the motion platform of the microscopic optical system and input a preset dot matrix image into the screen under test. The dot matrix calibration point coordinate extraction unit is used to acquire screen planar images and extract the coordinates of dot matrix calibration points in the screen planar images; The external parameter acquisition unit is used to obtain external parameters other than the optical axis translation component based on the consistency relationship between the orthogonal component of the screen planar image and the line connecting the image principal point and the product planar feature point in the direction. The orthogonal component is the orthogonal component of the projection of the line connecting the screen planar feature point and the optical center onto the optical axis. The internal parameter acquisition unit is used to obtain internal parameters based on external parameters. The calibration processing unit is used to calibrate the microscopic optical system based on external parameters, internal parameters, and coordinates of the dot matrix calibration points.

[0022] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: In this application, the screen under test is first placed on the motion platform of a microscopic optical system. The microscopic camera of the system acquires images of the screen under test, generating a detection image. The clear band region in the detection image is located. Clear band features, including clear band direction and width information, are extracted from the clear band region. The plane normal vector of the screen under test is calculated based on the clear band features. After Z-axis displacement of the microscopic camera, the intersection lines of the screen plane and the camera's focusing plane before and after displacement are defined as the first and second intersection lines, respectively. The ambiguity of the plane normal vector is eliminated based on the changing trend between the first and second intersection lines. The screen's attitude is then detected based on the eliminated plane normal vector, generating screen attitude data.

[0023] By analyzing the width and direction of the clear band region appearing in the detection image of the tilted screen under test, the plane normal vector of the screen under test is generated. Then, the changing trend of the clear band region with the axial movement of the microscope camera is analyzed to eliminate the ambiguity of the plane normal vector. Based on this, the attitude of the screen under test under the microscope optical system is solved on the basis of the calibration of the microscope optical system. At the same time, adaptive visual alignment between the screen plane and the camera image plane under the microscope optical system is realized, adjusting the screen under test to a fully focused state, so that clearer and more accurate detection images can be obtained in subsequent detection stages, thereby improving the detection accuracy of the screen. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram of the first embodiment of the local strip-shaped defect compensation method for the display screen of this application; Figure 2 This is a schematic diagram of the first embodiment of the method for extracting clear features according to this application; Figure 3 This is a schematic diagram of a first embodiment of the method for calculating the normal vector of the screen plane to be tested according to this application; Figure 4 This is a schematic diagram of the first embodiment of the hand-eye calibration and visual servoing method of this application; Figure 5 This is a schematic diagram of a first embodiment of the method for autofocusing a microscope camera according to this application. Figure 6 This is a schematic diagram of the first embodiment of the preprocessing method for the microscopic optical system of this application; Figure 7 This is a schematic diagram of the first embodiment of the preprocessing method for the microscopic optical system of this application; Figure 8 This is a schematic diagram of the first embodiment of the screen microscopic posture detection device of this application; Figure 9 This is a schematic diagram of one embodiment of the dot matrix display of this application; Figure 10 This is a partially enlarged schematic diagram of one embodiment of the dot matrix of this application; Figure 11 This is a schematic diagram of an embodiment of the present application that has a point defect; Figure 12This is a schematic diagram of an embodiment of the present application for extracting lattice points with point defects. Detailed Implementation

[0026] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0027] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0028] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0029] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0030] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0031] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0032] The technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0033] The method described in this application can be applied to servers, devices, terminals, or other devices with logical processing capabilities; therefore, this application does not limit its application. For ease of description, the following description uses a terminal as the executing entity.

[0034] Please see Figure 1 This application provides an embodiment of a method for compensating for localized band-shaped defects in a display screen, comprising: 101. Place the screen to be tested on the motion platform of the microscopic optical system, and use the microscopic camera of the microscopic optical system to acquire images of the screen to be tested and generate a test image; In this embodiment, the microscopic optical system includes an industrial microscopic imaging component (microscope camera and its accessories), which has a small field of view and depth of field and is mainly used for measuring and inspecting products. It also includes a pressing fixture, a single-axis motion module (used to control the microscope camera's axial movement along its Z-axis), a motion control board, and a six-axis precision displacement control device. The six-axis precision displacement control device controls the movement of the motion platform to adjust the screen's posture. An industrial control computer is used to control the operation of each component in the system and to collect parameters from each system component. The industrial microscopic imaging component consists of a microscope camera, microscope objectives, a microscope tube, and related wiring. The six-axis precision displacement control device consists of a high-precision XYZ-αβγ platform and its controller. The industrial control computer includes a dedicated image acquisition card, an image processing module, a multi-axis motion control card, a display, and related wiring and peripherals. After the industrial camera acquires an image, it is transmitted to the image processing module via the image acquisition card and processed by image processing algorithms.

[0035] In this embodiment, since the depth of field of a microscopic optical system is generally extremely small, the imaging object distance is... This can be approximated as a fixed value D. Imaging object distance For conventional macro cameras with a large depth of field, the object distance from the camera to the subject is uncertain, but for microscopic optical systems... It is essentially a constant, but this constant is unknown and needs to be obtained during the calibration of the microscopic optical system.

[0036] The terminal first places the screen to be tested on the motion platform of the microscopic optical system, and then uses the microscopic camera of the microscopic optical system to acquire images of the screen to be tested and generate a test image.

[0037] 102. Locate and detect clear areas in the image; When the plane of the screen under test is not parallel to the plane of the microscope camera, a clear band will be formed in the detection image, while the area around the clear band will be relatively blurry. In this embodiment, it is necessary to perform clarity detection on the screen area of ​​the detection image according to the characteristics of the screen under test and the microscope camera. The specific method for detecting the clear band area will be described in detail in subsequent embodiments.

[0038] 103. Extract the sharp band features from the sharp band region. The sharp band features include the sharp band direction information and the sharp band width information. In this embodiment, after the clear band region is determined, the pixels in the clear band region need to be analyzed to extract the clear band features. In this embodiment, the clear band features include clear band direction information and clear band width information.

[0039] 104. Calculate the plane normal vector of the screen under test based on the clear band features; The terminal calculates the plane normal vector of the screen under test based on the clear band characteristics. For solving the screen's pose, after calibrating the microscopic optical system, this embodiment utilizes the small depth of field of the microscopic system, ensuring clear imaging of the area where the screen plane intersects the focusing plane, forming a narrow clear band. By analyzing this clear band, the normal vector of the screen plane under test can be derived, thus solving for the pose of the plane under test. The coordinate system of the microscopic camera can be set as follows: The origin The optical center of the microscope camera. The axis is parallel to the image plane. The axis points towards the object along the optical axis, and the equation of the plane of the product in the camera coordinate system is: ,in It is the unit normal vector of the product plane. Let be the coordinates of a spatial point on the product plane. The equation of the focusing plane can be obtained from the calibrated system object distance D. If the system depth of field at this time is The clear imaging area is the intersection of the product plane and the system focusing area, and the focusing volume can be approximated as the area between the two parallel planes. In the image, the intersection is projected as a narrow band.

[0040] Let the normal vector of the product plane be... With optical axis The included angle between them is Then the following relationship exists:

[0041] Consider points A and B on two planes parallel to the screen plane, which are separated by a distance in the direction of the plane's normal. :

[0042] Their Z-coordinate difference is:

[0043] If A and B are points located on the upper and lower boundaries of the focal region, respectively... Then the following relationship exists:

[0044] in It is the thickness of the clear band region in three-dimensional space along the normal direction of the plane of the screen being measured. For small field-of-view microscopy optical systems, the three-dimensional thickness is... Projection width on the X and Y planes of the image for:

[0045] Pixel width measured on the image and The relationship between them is Where p and w represent the pixel size and magnification, respectively, these parameters have been obtained from the calibration of the microscopic optical system. This establishes the relationship between the sharpness band width on the image and the Z-axis component of the product plane normal vector. Relationship:

[0046] Furthermore, the normal vector components can be further determined based on the clear band-shaped direction vectors in the image. and The clear zone is the area between the plane to be measured and the focal plane. The projection of the intersection line onto the image. The equation of the intersection line. The equation corresponding to this line on the graph is: Where u and v are pixel coordinates. Based on perspective projection, the image linear coefficient... Parallel to , where K is the intrinsic parameter matrix of the microscope camera. (Line direction vector) The projection of the perpendicular component of the normal vector n onto the detection image plane. Therefore, the direction of the horizontal component of the plane normal vector of the screen under test. Perpendicular to the straight line in the image. Therefore, the direction of the horizontal component of the normal vector is... Perpendicular to the straight line in the image, we can obtain the normalized result as follows:

[0047] 105. After the microscope camera is displaced along the Z-axis, the intersection lines of the plane of the screen to be measured and the focal plane of the camera before and after the displacement are defined as the first intersection line and the second intersection line, respectively. 106. Eliminate the ambiguity of the plane normal vector based on the changing trend between the first and second intersection lines; 107. Based on the unambiguous plane normal vector, perform attitude detection on the screen under test and generate screen attitude data.

[0048] As can be seen from the normalized plane normal vector in step 104, the signs of the x and y components of the plane normal vector in the microscope camera coordinate system are ambiguous and cannot be uniquely determined by the normal vector of the clear zone in a single image. Therefore, this embodiment proposes adding a hill-climbing method for single-point focusing before the process, while simultaneously processing the images during the process to determine the direction of the clear zone movement. The symbol is . Its principle is as follows: Assume that at the initial moment, the microscope camera is at the origin of the world coordinate system, and the equation of the screen plane is: When the microscope camera moves along the positive direction of its optical axis Afterwards, the coordinates of the origin of the new microscope camera in the world coordinate system are: Then, the plane equation of the product plane in the new microscope camera coordinate system is:

[0049] At this moment, the focusing plane in the coordinate systems of the old and new cameras... The lines of intersection with the product plane are as follows:

[0050]

[0051] The projections of the two spatial lines onto the image plane represent the changing trend of the sharpness band after camera movement. According to the camera projection model, the projections of the two spatial lines onto the image plane are as follows:

[0052]

[0053] The two images above have the same normal vector, and their movement is along the direction of the normal vector. The specific direction depends on the change of the constant term:

[0054] From the above formula, it can be seen that when At that time, the straight line along Directional movement, and vice versa At that time, the straight line along Directional motion. Summarizing the above relationships, we can see that when the camera moves closer to the camera along the Z-axis, the direction of motion of the clear zone is given by the following formula:

[0055] in For the sign function, It is the unit direction of the horizontal component of the normal vector. In general scenarios, the new Z-axis of the product coordinates can be defined to point towards the camera, i.e. Therefore, in this scenario, when the microscope camera approaches the object, the sharp band moves along the horizontal component of the plane normal vector of the screen. Thus, the horizontal component of the normal vector can be calculated based on the movement of the sharp band in the image.

[0056] The above steps complete the calculation of the product plane normal vector. Its form is as follows:

[0057] in , , For clear pixel width, For pixel size, This is the system amplification factor. For system depth of field, To clearly show the parameters for straight lines.

[0058] By calibrating the internal parameters of the microscope camera offline, the center point of the clear band in the detected image can be determined as the origin of the screen coordinate system. To solve for the X and Y components of the camera coordinate system for feature points in an image, it can be set as follows: Because this point is clearly focused, its Z-axis component in the camera coordinate system is D. Therefore, the plane equation of the screen plane in the camera coordinate system can be solved. The X and Y axes of the product coordinate system can be defined by the user. Since the subsequent posture data will be used for hand-eye calibration, the center point of the screen area can be defined as the origin of the product coordinate system, and the long side of the screen area as the X-axis of the product coordinate system. Let the coordinates of a point on the long side of the screen area be... The X and Y components of the corresponding spatial point coordinates are obtained based on the intrinsic parameter matrix K. Substituting the coordinates into the product equation will solve for its Z-axis components. Therefore, the X and Y axis direction vectors of its product are respectively:

[0059]

[0060] At this point, the product's pose in the camera coordinate system can be determined as follows:

[0061] In this embodiment, the screen under test is first placed on the motion platform of the microscopic optical system. The microscopic camera of the system acquires images of the screen under test, generating a detection image. The clear band region in the detection image is located. Clear band features, including clear band direction and width information, are extracted from the clear band region. The plane normal vector of the screen under test is calculated based on the clear band features. After Z-axis displacement of the microscopic camera, the intersection lines of the screen plane and the camera's focusing plane before and after displacement are defined as the first intersection line and the second intersection line. The ambiguity of the plane normal vector is eliminated based on the changing trend between the first and second intersection lines. The screen's attitude is then detected based on the unambiguous plane normal vector, generating screen attitude data.

[0062] By analyzing the width and direction of the clear band region appearing in the detection image of the tilted screen under test, the plane normal vector of the screen under test is generated. Then, the changing trend of the clear band region with the axial movement of the microscope camera is analyzed to eliminate the ambiguity of the plane normal vector. Based on this, the attitude of the screen under test under the microscope optical system is solved on the basis of the calibration of the microscope optical system. At the same time, adaptive visual alignment between the screen plane and the camera image plane under the microscope optical system is realized, adjusting the screen under test to a fully focused state, so that clearer and more accurate detection images can be obtained in subsequent detection stages, thereby improving the detection accuracy of the screen.

[0063] Secondly, since conventional calibration boards cannot emit light themselves, and the camera's field of view is relatively small, making it impossible to image using side lighting, the calibration board is usually adhered to a high-brightness surface light source and then fixed to a stage. Calibration is then performed by adjusting the camera height to focus on the checkerboard calibration board. However, this operation is not only costly but also limited by the available space and requires a very thick light source, leading to a sharp increase in calibration costs. This embodiment utilizes the high precision of Micro LED products and proposes using Micro LED products for calibration. Through testing, this method can meet the calibration error requirement of 10µm.

[0064] Please see Figure 2 This application provides an embodiment of a method for extracting sharp band features, comprising: 201. After dividing the clear band region into partitions and calculating the focus value, threshold segmentation is performed to generate clear band region features; 202. The direction and width of the clear band in the detected image are obtained by analyzing the clear band features.

[0065] The direction and width of the clear band are obtained through image processing. Specifically, the focus value is calculated by dividing the detection image into regions, and then thresholding is performed to obtain the clear band region features. Further analysis of the clear band features yields the direction and width information of the clear band in the detection image.

[0066] Please see Figure 3 This application provides an embodiment of a method for calculating the normal vector of a screen plane to be measured, comprising: 301. Determine the X-axis and Y-axis components of the plane normal vector based on the clear directional information; 302. Determine the Z-axis component of the plane normal vector based on the clear band width information, and integrate the X-axis component, Y-axis component and Z-axis component to generate the plane normal vector.

[0067] The attitude determination part aims to find the representation of the screen normal vector in the coordinate system of the microscope camera: The width of the sharp band in the sharp band feature determines the Z-axis component of the plane normal vector. The clear directional information in the detected image determines the horizontal plane components, namely the X-axis and Y-axis components. .

[0068] The terminal determines the X-axis and Y-axis components of the plane normal vector based on the clear band direction information, and then determines the Z-axis component of the plane normal vector based on the clear band width information. Finally, it integrates the X-axis, Y-axis, and Z-axis components to generate the plane normal vector.

[0069] Please see Figure 4 This application provides an embodiment of a method for hand-eye calibration and visual servoing, comprising: 401. Repeatedly adjust the position of the end of the motion platform so that the screen under test is in different poses while within the field of view of the microscope camera, and obtain the motion data of the motion platform. 402. Calculate the end-effector pose data of the motion platform based on the platform's motion data; 403. Acquire images using a microscope camera and solve for screen pose data under different poses; 404. Based on the screen pose data, platform end pose data and hand-eye model under different poses, the hand-eye matrix data of the microscopic optical system is obtained; 405. Based on the target posture matrix and hand-eye matrix data of the screen under test, calculate the adjustment motion of the motion platform when the screen under test reaches the target posture; 406. Adjust the pose of the screen under test by controlling the motion platform by adjusting the amount of motion.

[0070] Hand-eye calibration and visual servoing in a microscopic optical system are crucial steps in screen pose adjustment. In this embodiment, the terminal repeatedly changes the position of the end effector of the motion platform to keep the screen under test within the field of view of the microscope camera. The aforementioned pose calculation process is repeated to obtain the screen pose under different motion platform poses. Where i = 1, 2, 3... N. The platform end-effector pose data of the motion platform are obtained from the platform motion data. Based on this, the system's hand-eye matrix can be obtained by solving the hand-eye model with the eye on the hand. During alignment operations in a microscopic optical system, let the target attitude matrix of the screen under test be... ,

[0071] Therefore, combining the hand-eye matrix It can be seen that the transformation matrix between the current stage end position and the target stage end position in the base coordinate system is:

[0072] Please see Figure 5 This application provides an embodiment of a method for autofocusing a microscope camera, comprising: 501. Control the microscope camera of the microscope optical system to perform automatic focusing.

[0073] After the motion platform has adjusted the posture of the screen under test, an additional autofocus process is performed. This is because the depth of field of the microscopic vision system is very small, and its Z-axis resolution is much higher than that of the X and Y axes. After the alignment process is completed, its Z-axis error may be greater than the system depth of field. At this time, the overall image will be blurred, which will affect the subsequent measurement and detection process.

[0074] Please see Figure 6 This application provides an embodiment of a method for locating and detecting clear band regions in an image, comprising: 601. Edge detection of the effective screen area in the image; 602. Perform initial sharpness analysis within the effective area of ​​the screen to generate an initial sharp area; 603. Obtain the sharpness threshold, and merge the remaining areas with sharpness greater than the sharpness threshold into the initial sharp area to generate a sharp band area.

[0075] In this embodiment, the terminal first detects the effective area in the detection image, that is, obtains the effective screen area of ​​the detection image through AA region detection. Next, it analyzes the sharpness of each pixel, specifically by analyzing the features of the pixel. After obtaining the sharpness of each pixel, these pixels are merged to generate an initial sharp area. Then, based on the pixel integration degree and structure of the screen under test, a sharpness threshold is obtained, and the remaining areas with sharpness greater than the sharpness threshold are merged into the initial sharp area to generate a sharp band area.

[0076] Please see Figure 7 This application provides an embodiment of a preprocessing method for a microscopic optical system, comprising: 701. Place the screen to be tested on the motion platform of the micro-optical system and input the preset dot matrix image into the screen to be tested; 712. Acquire a planar image of the screen and extract the coordinates of the dot matrix calibration points in the planar image of the screen; 703. Based on the consistency relationship between the orthogonal components of the screen planar image and the line connecting the principal point of the image and the feature point of the product planar image in the direction, find the external parameters other than the translation component in the optical axis direction. The orthogonal component is the orthogonal component of the projection of the line connecting the feature point of the screen planar image and the optical center onto the optical axis. 704. Obtain the internal parameters based on the external parameters; 705. The microscopic optical system is calibrated based on external parameters, internal parameters, and coordinates of the lattice calibration points.

[0077] Before measuring the object's pose, the internal parameters of the perspective model of the microscopic optical system need to be precisely calibrated. However, due to the small depth of field of the microscopic optical system, camera calibration methods for macroscopic large depth-of-field scenes cannot be used. In this embodiment, the terminal acquires a screen plane image that is nearly parallel to the image plane based on the perspective projection model. First, based on the orthogonal component of the projection of the screen plane feature points and the optical center line onto the optical axis, and the consistency relationship between the line connecting the image principal point and the product plane feature points in the direction, all external parameters except for the optical axis direction translation component are calculated. T represents the imaging object distance in the corresponding direction, and R includes the 3 elements in the following matrix M. The r parameter of 3, because the depth of field of the microscopic optical system is very small, its object distance can be approximated as a fixed value. The object distance of the system's perspective model can be set as... Then the system extrinsic parameter matrix M is:

[0078] If the image-side focal length is F, then the following relationship exists for the microscopic optical system:

[0079] in , Represents the image coordinates containing distortion, and has . This represents the first-order radial distortion coefficient of the lens. Adding the two terms in the above formula, while setting... , We can obtain:

[0080] The above formula contains unknown parameters. The equations for k and φ can be solved by establishing a system of equations based on the feature points of the screen plane to obtain the aforementioned unknown parameters. Since the nominal focal length of the microscope objective is very close to its actual focal length, let the focal lengths of the microscope objective and the tube lens be respectively... , Then it can be made At this point, we can obtain Because microscopes are manufactured with high precision, their principal point and pixel size can be assumed to be equal to their nominal values. Therefore, the above process yields initial values ​​for all internal parameters of the microscopic optical system. Based on these initial values, the calibration of the microscopic optical system can be completed by performing nonlinear optimization on the model parameters with the objective function of minimizing the reprojection error of the calibration point. Please refer to [reference needed]. Figure 9 , Figure 9 This is a schematic diagram of a dot matrix display. In this embodiment, a dot matrix display is designed according to the screen resolution. The display contains a 13x11 dot matrix, with each dot consisting of 3x3 pixels. The center of the screen is taken as the origin, and the center coordinates of all dots are calculated based on the physical size of a single pixel. This set of coordinates is the coordinate in the world coordinate system, such as... Figure 10 As shown, Figure 10 This is a magnified view of a portion of the dot matrix (mark is the center coordinate). After manually adjusting the screen plane to be nearly parallel to the camera image plane using a six-axis motion platform, the image was acquired and the coordinates of the dot matrix feature points in the image were extracted. Please refer to the diagram of image dot matrix and coordinate extraction. Figure 11 and Figure 12 Because some LEDs may fail to light up, the center point coordinates after threshold segmentation may deviate from the actual coordinates. Therefore, this embodiment uses the minimum bounding rectangle method to obtain the image coordinates of the dot matrix feature center points after threshold segmentation. ,like Figure 12 As shown. Finally, based on the coordinate points... , The microscopic camera calibration can be completed by combining the above derivation.

[0081] Please see Figure 8 This application provides an embodiment of a screen microscopic posture detection device, comprising: The detection image generation unit 801 is used to place the screen to be tested on the motion platform of the microscopic optical system, and to acquire images of the screen to be tested through the microscopic camera of the microscopic optical system to generate a detection image. The clear band region localization unit 802 is used to locate the clear band region in the detection image; The clear band feature extraction unit 803 is used to extract clear band features in the clear band region. The clear band features include clear band orientation information and clear band width information. Plane normal vector calculation unit 804 is used to calculate the plane normal vector of the screen under test based on the clear band features; The intersection line determination unit 805 is used to define the intersection lines of the microscope camera with the screen plane to be measured and the camera focusing plane before and after the microscope camera is displaced along the Z-axis as the first intersection line and the second intersection line, respectively. Ambiguity elimination unit 806 is used to eliminate the ambiguity of the plane normal vector based on the changing trend between the first intersection line and the second intersection line; The screen pose data generation unit 807 is used to perform pose detection on the screen under test based on the unambiguous plane normal vector and generate screen pose data.

[0082] Optionally, the clear band feature extraction unit 803 specifically includes: After dividing the clear band region into partitions and calculating the focus value, threshold segmentation is then performed to generate clear band region features. The direction and width of the clear band in the detected image are obtained by analyzing the clear band features.

[0083] Optionally, the plane normal vector calculation unit 804 specifically includes: The X-axis and Y-axis components of the plane normal vector are determined based on the clear directional information. The Z-axis component of the plane normal vector is determined based on the clear band width information, and the X-axis component, Y-axis component and Z-axis component are integrated to generate the plane normal vector.

[0084] Optionally, after the screen pose data generation unit 807, the detection device further includes: The hand-eye calibration and visual servo unit is used to adjust the motion platform and perform hand-eye calibration and visual servoing of the microscopic optical system based on screen posture data under different poses.

[0085] Optionally, the hand-eye calibration and vision servoing unit specifically includes: The position of the end of the motion platform is repeatedly adjusted so that the screen under test is in different poses while within the field of view of the microscope camera, thereby acquiring the motion data of the motion platform. Calculate the end-effector pose data of the motion platform based on the platform's motion data; Images are acquired using a microscope camera, and screen pose data under different poses are obtained. The hand-eye matrix data of the micro-optical system is obtained by solving the screen pose data, platform end pose data and hand-eye model under different poses. The adjustment motion of the motion platform when the screen under test reaches the target posture is calculated based on the target posture matrix and hand-eye matrix data of the screen under test. The pose of the screen under test is adjusted by controlling the amount of exercise on the motion platform.

[0086] Optionally, after the hand-eye calibration and visual servoing unit, the detection device also includes: The autofocus processing unit is used to control the microscope camera of the microscope optical system to perform autofocus processing.

[0087] Optionally, the clear area positioning unit 802 specifically includes: Edge detection of the effective screen region in an image; Perform initial sharpness analysis within the effective area of ​​the screen to generate an initial sharp area; Obtain a sharpness threshold, and merge the remaining areas with sharpness greater than the sharpness threshold into the initial sharp area to generate a sharp band area.

[0088] Optionally, before the detection image generation unit 801, the detection device further includes: The dot matrix image input unit is used to place the screen under test on the motion platform of the microscopic optical system and input a preset dot matrix image into the screen under test. The dot matrix calibration point coordinate extraction unit is used to acquire screen planar images and extract the coordinates of dot matrix calibration points in the screen planar images; The external parameter acquisition unit is used to obtain external parameters other than the optical axis translation component based on the consistency relationship between the orthogonal component of the screen planar image and the line connecting the image principal point and the product planar feature point in the direction. The orthogonal component is the orthogonal component of the projection of the line connecting the screen planar feature point and the optical center onto the optical axis. The internal parameter acquisition unit is used to obtain internal parameters based on external parameters. The calibration processing unit is used to calibrate the microscopic optical system based on external parameters, internal parameters, and coordinates of the dot matrix calibration points.

[0089] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0090] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0091] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0092] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0093] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method of detecting micro gestures on a screen, the method comprising: include: The screen to be tested is placed on the motion platform of the micro-optical system, and the screen to be tested is captured by the micro-camera of the micro-optical system to generate a test image. Locate the clear band region in the detected image; Extract the sharp band features from the sharp band region, the sharp band features including sharp band direction information and sharp band width information; Calculate the plane normal vector of the screen under test based on the clear band features; After the microscope camera is displaced along the Z-axis, the intersection lines of the microscope camera with the plane of the screen to be measured and the camera focusing plane before and after the displacement are defined as the first intersection line and the second intersection line, respectively. Eliminate the ambiguity of the plane normal vector based on the changing trend between the first intersection line and the second intersection line; The screen under test is subjected to attitude detection based on the plane normal vector that eliminates ambiguity, and screen attitude data is generated.

2. The detection method according to claim 1, characterized in that, The step of extracting the sharp band features in the sharp band region specifically includes: The sharp band region is partitioned, the focus value is calculated, and then threshold segmentation is performed to generate sharp band region features. The direction and width of the clear band in the detected image are obtained by analyzing the clear band features.

3. The detection method according to claim 2, characterized in that, The step of calculating the planar normal vector of the screen under test based on the clear band features specifically includes: The X-axis and Y-axis components of the plane normal vector are determined based on the clear directional information. The Z-axis component of the plane normal vector is determined based on the clear band width information, and the X-axis component, Y-axis component and Z-axis component are integrated to generate the plane normal vector.

4. The method of claim 1, wherein After the step of performing pose detection on the screen under test based on the unambiguous plane normal vector to generate screen pose data, the detection method further includes: Adjust the motion platform to perform hand-eye calibration and visual servoing of the microscopic optical system based on screen posture data under different poses.

5. The detection method according to claim 4, characterized in that, The adjustment motion platform, based on screen posture data under different poses, specifically includes the following steps for hand-eye calibration and visual servoing of the microscopic optical system: The position of the end of the motion platform is repeatedly adjusted so that the screen under test is in different poses while within the field of view of the microscope camera, and the motion data of the motion platform is obtained. Calculate the end-effector pose data of the motion platform based on the platform motion data; Images are acquired using a microscope camera, and screen pose data under different poses are obtained. The hand-eye matrix data of the micro-optical system is obtained by solving the screen pose data under different poses, the platform end pose data and the hand-eye model. The adjustment motion of the motion platform when the screen under test reaches the target posture is calculated based on the target posture matrix of the screen under test and the hand-eye matrix data. The motion platform is controlled by adjusting the amount of motion to adjust the pose of the screen under test.

6. The detection method according to claim 5, characterized in that, After the steps of hand-eye calibration and visual servoing of the microscopic optical system based on screen posture data under different poses on the adjustment motion platform, the detection method further includes: The microscope camera of the microscopic optical system is controlled to perform automatic focusing.

7. The detection method according to any one of claims 1 to 6, characterized in that, The step of locating the clear band region in the detected image specifically includes: Edge detection captures the effective screen region in the detected image; Perform initial sharpness analysis within the effective area of ​​the screen to generate an initial sharp area; Obtain a sharpness threshold, and merge the remaining areas with sharpness greater than the sharpness threshold into the initial sharp area to generate a sharp band area.

8. The detection method according to any one of claims 1 to 6, characterized in that, Before the steps of placing the screen under test on the motion platform of the microscopic optical system, acquiring an image of the screen under test through the microscopic camera of the microscopic optical system, and generating a detection image, the detection method further includes: The screen to be tested is placed on the motion platform of the micro-optical system, and a preset dot matrix image is input into the screen to be tested; Acquire a planar image of the screen and extract the coordinates of the dot matrix calibration points in the planar image of the screen; Based on the consistency relationship between the orthogonal components of the screen planar image and the line connecting the image principal point and the product planar feature point in the direction, the external parameters other than the optical axis direction translation component are obtained. The orthogonal component is the orthogonal component of the projection of the line connecting the screen planar feature point and the optical center onto the optical axis. The internal parameters are obtained based on the external parameters; The microscopic optical system is calibrated based on the external parameters, the internal parameters, and the coordinates of the dot matrix calibration points.

9. A device for detecting the microscopic posture of a screen, characterized in that, include: The detection image generation unit is used to place the screen to be tested on the motion platform of the microscopic optical system, and to acquire images of the screen to be tested through the microscopic camera of the microscopic optical system to generate a detection image. A clear band region localization unit is used to locate the clear band region in the detected image; A clear band feature extraction unit is used to extract clear band features in the clear band region, wherein the clear band features include clear band direction information and clear band width information; A plane normal vector calculation unit is used to calculate the plane normal vector of the screen under test based on the clear band features; The intersection line determination unit is used to define the intersection lines of the microscope camera with the screen plane to be measured and the camera focusing plane before and after the displacement as the first intersection line and the second intersection line, respectively, after the microscope camera is displaced along the Z-axis. The ambiguity elimination unit is used to eliminate the ambiguity of the plane normal vector based on the changing trend between the first intersection line and the second intersection line; The screen pose data generation unit is used to perform pose detection on the screen under test based on the unambiguous plane normal vector to generate screen pose data.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains a program that, when executed on a computer, performs the detection method as described in any one of claims 1 to 8.

Images