A point detection method, device, equipment, and medium based on a dual-arm robot
By using a point detection method based on pixel coordinates and module type, the grasping pose and detection points of the dual-arm robot are adaptively adjusted, which solves the problems of low efficiency and accuracy in generating detection points in the existing technology, and realizes automated grasping and efficient defect detection of display modules.
Patent Information
- Application Number
- CN202611610211.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-19
- Publication Date
- 2026-11-17
AI Technical Summary
Existing display module point detection methods based on dual-arm robots suffer from low efficiency and accuracy in generating detection points. In particular, when detecting multiple areas, point-by-point teaching is required, resulting in high replacement and maintenance costs, and a lack of automated closed-loop control for clarity.
The robot determines the grasping pose of the dual-arm robot based on pixel coordinates, uses pixel coordinate transformation to obtain the actual placement deviation of the workpiece, and performs adaptive grasping. It determines multiple target detection points and their poses by combining module categories, performs image acquisition and sharpness adjustment, realizes automatic defect detection, and forms an automatic matching between detection points and algorithms.
It improves the efficiency and accuracy of point generation for display module quality inspection, realizes adaptive and reliable capture, rapid model change and automated defect detection, and improves inspection efficiency and accuracy.
Smart Images

Figure CN123118750A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of robot quality inspection technology, and in particular to a point detection method, device, equipment and medium based on a dual-arm robot. Background Technology
[0002] Display modules, LCD screens, flexible screens, adhesive lines, and the back of modules typically require multi-angle and multi-area quality inspection during production. Existing production lines generally use a combination of fixed inspection fixtures, human-taught robot positioning, camera photography, and visual algorithm inspection to complete quality checks. With the introduction of humanoid robots into industrial settings, utilizing their arms, torso space, and general operational capabilities to replace some dedicated inspection fixtures has become a viable direction for flexible production lines.
[0003] Current robot teaching methods typically require manual point-by-point movement of the robot and saving of each inspection point. For inspection objects with linear or planar distributions, such as screen edges, in-screen areas, and adhesive paths, if dozens or even hundreds of imaging points are needed, point-by-point teaching is required, and a large number of points need to be re-maintained when changing models. Furthermore, while existing Automated Optical Inspection (AOI) systems for display panels can usually select different detection algorithms or accuracy based on regions, the inspection area, robot movement points, screen model, and camera trigger actions are often maintained separately. On-site personnel need to maintain the robot point table, vision algorithm area configuration, and camera trigger parameters separately, which can easily lead to inconsistencies between the points and the quality inspection specifications.
[0004] Therefore, existing display module point detection methods based on dual-arm robots suffer from low efficiency and accuracy in generating detection points. Summary of the Invention
[0005] This application provides a point detection method, device, equipment, and medium based on a dual-arm robot, to solve the problems of low efficiency and accuracy in generating detection points in existing display module point detection methods based on dual-arm robots.
[0006] Firstly, this application provides a point detection method based on a dual-arm robot, the method comprising: Based on pixel coordinates, the grasping pose of the dual-arm robot is determined, and based on the grasping pose, the display module to be detected is grasped. Based on the module category of the display module to be tested, multiple target detection points and their target detection poses are determined. Based on each target detection pose, images are sequentially acquired from the target detection points to obtain the detection images and their clarity values. The target detection points are determined based on a preset number of reference detection points. Based on the preset sharpness threshold and sharpness value, the target detection pose is adjusted to obtain the target detection image; Based on the detection category of the target detection point, defect detection is performed on the target detection image to obtain the target detection result.
[0007] In some embodiments of this application, determining the grasping pose of the dual-arm robot based on pixel coordinates includes: Based on the positioning camera, the pixel coordinates and positioning angle of the display module to be detected are determined, and the pixel coordinates are transformed based on the homography matrix to obtain the two-dimensional grasping coordinates; Based on the two-dimensional grasping coordinates and the preset grasping height, the three-dimensional grasping coordinates are determined, and based on the positioning angle, the grasping posture is determined. Based on the preset workspace of the dual-arm robot, determine whether the three-dimensional grasping coordinates are within the preset workspace; If so, the grasping pose is determined based on the 3D grasping coordinates and grasping posture; If not, based on the positioning camera and the display module to be detected, the pixel coordinates and positioning angle are collected and updated; based on the updated pixel coordinates and positioning angle, the updated 3D grasping coordinates are determined, and if the updated 3D grasping coordinates are within the preset workspace, the grasping pose is determined based on the updated 3D grasping coordinates; if the updated 3D grasping coordinates are not within the preset workspace, the determination of the grasping pose is stopped, and the corresponding prompt information is output. The prompt information is used to inform the user that the grasping has failed.
[0008] In some embodiments of this application, before determining multiple target detection points and their target detection poses based on the module category of the display module to be detected, the method further includes: Based on a dual-arm robot, a preset number of reference detection points are determined on the display module to be tested, and multiple target detection points corresponding to the reference detection points are generated according to the preset number. Based on the baseline detection pose of the baseline detection point, the target detection pose of each target detection point is determined, and the target detection point and its target detection pose are associated with the module category.
[0009] In some embodiments of this application, multiple target detection points corresponding to the baseline detection points are generated according to a preset number, including: Determine the preset quantity; If the preset number is 2, then the direction vector is determined based on the orientation between the reference detection points, and the direction vector is divided equally to obtain multiple target detection points; If the preset quantity is 3, the current point generation mode is determined based on the cross product modulus of the benchmark detection points. When the point generation mode is right angle mode, the diagonal direction is determined based on the first and second reference detection points, and the detection plane normal vector is determined based on the diagonal direction and the third reference detection point. Based on the diagonal direction and the normal vector of the detection plane, the vertical direction perpendicular to the diagonal direction in the detection plane is determined, and based on the diagonal direction and the vertical direction, the local coordinate system is determined; Based on the local coordinate system, the projected position of the third reference detection point is determined, the rectangle orientation is obtained, and the row and column directions are determined based on the diagonal direction, vertical direction and rectangle orientation to obtain the detection area; When the point generation mode is parallelogram mode, the column direction is determined based on the orientation between the first and second reference detection points, and the row direction is determined based on the orientation between the first and third reference detection points; the detection area is determined based on the row and column directions. By performing row and column interpolation on the detection area, multiple target detection points are obtained.
[0010] In some embodiments of this application, the current point generation mode is determined based on the cross product modulus of the reference detection points, including: When the preset quantity is 3, the cross product modulus is calculated based on each benchmark detection point, and it is determined whether the cross product modulus is 0. If so, determine that the benchmark detection points are collinear, and redetermine the preset number of updated benchmark detection points; based on the updated benchmark detection points, determine the point generation mode; If not, then determine that the benchmark detection points are not collinear, and determine the current corresponding point generation mode.
[0011] In some embodiments of this application, the target detection pose of each target detection point is determined based on the reference detection pose of the reference detection point, including: When the preset number is 2, the position coordinates of the target detection points are calculated based on the position coordinates of the reference detection points and the distance between each point; where the point distance is the distance between each target detection point and the reference detection point. When the preset quantity is 3, the ratio of each row number and the ratio of each column number are calculated based on the number of rows and columns corresponding to each target detection point and the number of rows and columns in the detection area. Based on the area row vector, the area column vector, and the ratio of rows and columns, the position coordinates of each target detection point are calculated. Based on the distance between the target detection point and each reference detection point, the pose of each reference detection point is weighted and averaged to obtain the target pose. Then, based on the position coordinates and the target pose, the target detection pose of each target detection point is determined.
[0012] In some embodiments of this application, the target detection pose is adjusted based on a preset sharpness threshold and sharpness value to obtain a target detection image, including: Determine if the clarity value is less than the preset clarity threshold; If the value is less than the target detection pose, the target detection pose will not be adjusted, and the detected image will be determined as the target detection image. If it is not less than, then based on the metadata of the point plane associated with the target detection pose, the detection plane normal vector is determined, and based on the detection plane normal vector, the preset step size and the preset number of attempts, multiple adjustment distances are determined; Based on the target detection pose, the detection plane normal vector, and multiple adjustment distances, multiple adjustment detection poses are determined, and the target joint span is calculated based on each adjustment detection pose and the current joint state. Compare whether the target joint span is less than a preset joint span threshold; When the target joint span is less than the preset joint span threshold, image acquisition is performed based on the adjustment detection pose to obtain the adjustment detection image and its adjustment clarity value; the target detection image is determined based on multiple adjustment clarity values and the preset clarity threshold. When the target joint span is not less than the preset joint span threshold, a prompt message is output, which indicates that the current target joint span cannot be achieved.
[0013] Secondly, this application provides a point detection device based on a dual-arm robot, the device comprising: The grasping module is used to determine the grasping pose of the dual-arm robot based on pixel coordinates, and to grasp the display module to be inspected based on the grasping pose. The acquisition module is used to determine multiple target detection points and their target detection poses based on the module category of the display module to be detected, and to sequentially acquire images of the target detection points based on each target detection pose to obtain the detection image and its clarity value; wherein, the target detection points are points determined according to a preset number of reference detection points; The adjustment module is used to adjust the target detection pose based on a preset sharpness threshold and sharpness value to obtain a target detection image; The detection module is used to perform defect detection on the target detection image based on the detection category of the target detection point and obtain the target detection result.
[0014] Thirdly, this application provides a computer device, including: a processor, and a memory communicatively connected to the processor; The memory stores instructions that the computer executes; The processor executes computer execution instructions stored in memory to implement the method of this application.
[0015] Fourthly, this application provides a computer-readable storage medium storing program code, which, when executed by a processor, is used to implement the method of this application.
[0016] Compared with existing technologies, the method in this application determines the grasping pose of the dual-arm robot based on pixel coordinates and completes the grasping of the display module to be inspected. It achieves a grasping pose adapted to the actual placement deviation of the workpiece by using visual pixel coordinate transformation, eliminating the grasping misalignment problem caused by the positional offset of the workpiece at the loading station, and realizing adaptive and reliable grasping of the display module. Based on the module type of the display module to be inspected, a preset number of reference detection points are determined, and multiple target detection points and their target detection poses are further expanded. Image acquisition is then performed sequentially according to each target detection pose to obtain the detection image and clarity value. This achieves the generation of all target detection points from a small number of reference detection points, reducing the large amount of debugging work required by traditional point-by-point teaching. Simultaneously, it combines module type matching with corresponding... The system uses a set of inspection points to enable rapid switching between different display module models. It adjusts the target detection pose based on preset clarity thresholds and values to obtain the target detection image. For scenes with blurred images, it automatically adjusts the detection pose to complete re-shooting without manual intervention, forming a closed-loop control of clarity. By combining the detection categories corresponding to the target inspection points, it performs defect detection on the target detection image to obtain the target detection result, thus achieving automatic matching of inspection points and detection algorithms. This enables automated multi-region grasping of display modules, rapid point generation, closed-loop control of image clarity, and regional adaptive defect detection, effectively improving the efficiency and accuracy of quality inspection point generation for display modules. This further enhances the efficiency and accuracy of dual-arm robots in detecting defects in display modules based on these quality inspection points. Attached Figure Description
[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0018] Figure 1 A flowchart illustrating a point detection method based on a dual-arm robot provided in an embodiment of this application; Figure 2 A schematic diagram of a point detection device based on a dual-arm robot provided in an embodiment of this application; Figure 3 This is a structural block diagram of a device for performing a point detection method based on a dual-arm robot according to an embodiment of this application. Detailed Implementation
[0019] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0020] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.
[0021] In existing technologies, current robot teaching methods typically require manual movement of the robot point by point and saving the location of each detection point. For detection objects that are distributed in a linear or planar manner, such as screen edges, in-screen areas, and adhesive paths, if dozens or even hundreds of imaging points are required, point-by-point teaching is necessary, and a large number of points need to be re-maintained when changing models.
[0022] Existing Automated Optical Inspection (AOI) systems for display panels typically allow selection of different inspection algorithms or precisions by region. However, the inspection areas, robot movement points, screen models, and camera trigger actions are often maintained in a fragmented manner. On-site personnel need to maintain the robot point table, vision algorithm region configuration, and camera trigger parameters separately, which can easily lead to inconsistencies between the points and the quality inspection specifications.
[0023] Existing autofocus or sharpness assessment methods are usually performed by the camera or vision system alone. In scenarios where dual-arm robots carry or cooperate with cameras and workpieces for inspection, if the clarity of the quality inspection photos does not meet the standards, traditional solutions usually require manual intervention, re-teaching, or re-taking, which makes it difficult to automatically close the loop within the robot's workflow.
[0024] There are also discrepancies between existing offline robot planning and actual machine execution. Planning software can provide the target trajectory, but factors such as the actual joint state of the actual machine, disconnection and reconnection, joint locking, movement span, and execution synchronization can lead to safety risks when the planned trajectory is executed on-site. Therefore, simply piecing together vision, robot teaching, quality inspection, and focusing is insufficient to meet the requirements of display module production lines for changeover efficiency, inspection consistency, and operational safety.
[0025] Therefore, existing methods suffer from several drawbacks: low point-to-point teaching efficiency. Display modules have multiple detection areas, including edges, in-screen areas, FPCs (Flexible Printed Circuits), adhesive paths, and the back side. Teaching each area point-by-point is time-consuming and incurs high maintenance and replacement costs. Furthermore, a limited number of teaching points cannot directly generate executable points. Even if existing methods can generate theoretical points based on geometric relationships, they typically lack unified verification of robot forward and inverse kinematics, end-effector posture interpolation, reachability, and motion span, rendering the generated points unusable for production. Finally, robot points are prone to disconnect from the quality inspection algorithm area. In existing solutions, robot motion points, the vision algorithm area, and camera trigger signals are often maintained by different systems. Changing points on-site may lead to forgetting to synchronize the quality inspection area, resulting in false positives, false negatives, or false negatives. The algorithm's recipe invocation is incorrect; the quality inspection image clarity is abnormal and lacks a workflow loop. That is, if the camera returns a blurry image, traditional solutions often require manual confirmation, retaking the photo, or readjusting the camera distance, and cannot automatically fine-tune and retake the photo based on the plane direction of the inspection point; automatic focusing movement poses safety risks. When the robot fine-tunes the position of the camera or workpiece along the normal direction, directly executing the target pose may cause the robotic arm to swing significantly or collide, especially in scenarios where dual arms, suction cups, and lifting platforms work together, where the risk is even higher; switching between multiple screen models is costly. Different screen models have different sizes, inspection areas, and point sequences, and traditional solutions require manual copying, modification, and verification of a large number of configurations.
[0026] Figure 1 This is a flowchart illustrating a point detection method based on a dual-arm robot, provided as an embodiment of this application. Figure 1 As shown, this point detection method based on a dual-arm robot may include the following steps: S110. Based on pixel coordinates, determine the grasping pose of the dual-arm robot, and grasp the display module to be detected based on the grasping pose.
[0027] In this context, pixel coordinates refer to the two-dimensional position of the feature points of the target object (i.e., the display module to be detected) on the image plane in the image captured by the camera. They are typically represented as (u, v), where u represents the pixel position in the horizontal direction (column direction) and v represents the pixel position in the vertical direction (row direction). Pixel coordinates are discrete coordinates on the camera's imaging plane, with units of pixels, serving as the raw input for visual positioning. Since the camera captures two-dimensional image information, while the robot requires pose information in three-dimensional space for grasping, coordinate transformation is necessary to map the pixel coordinates to the robot's coordinate system. By acquiring pixel coordinates, the actual position and angle of the workpiece can be perceived in real time, thereby dynamically adjusting the grasping pose and improving the grasping success rate and adaptability.
[0028] A dual-arm robot is a robotic system with two robotic arms, each typically possessing multiple degrees of freedom (e.g., 6 or 7 axes). These arms can move independently or collaboratively, and are preferably humanoid robots with a humanoid configuration, including a torso, left arm, right arm, end effector, suction cup, and optionally a lifting platform, head, or movement mechanism. The left and right arms can perform different tasks; for example, the left arm can carry a quality inspection camera for photographic inspection, while the right arm carries a suction cup or gripper for workpiece grasping; or vice versa. Dual-arm robots can also utilize the torso space and lifting platform to adjust their working range, and utilize the head or movement mechanism to expand their field of vision and mobility. This allows them to replace traditional dedicated inspection fixtures and complete multi-area quality inspection tasks for display modules. Multi-area quality inspection of display modules requires multi-angle and multi-area photography. Traditional single-arm robots or fixed inspection fixtures struggle to flexibly adjust the relative positions of the camera and workpiece. Dual-arm robots, like humans, can hold the workpiece in one arm and the camera in the other, taking pictures from different angles to inspect multiple areas such as the screen edge, screen interior, adhesive path, and the back of the module.
[0029] Grasping pose refers to the target position and orientation of a robot end effector (such as a suction cup or gripper) in the robot coordinate system when grasping a display module to be inspected. Position is usually represented as three-dimensional coordinates, and orientation is usually represented as quaternions or Euler angles, thereby driving the robot to move so that the end effector can accurately reach the grasping position and grasp the workpiece.
[0030] Display modules under inspection refer to those requiring multi-area optical quality inspection, including but not limited to the LCD screen, flexible screen, adhesive area, and back of the module. Display modules typically consist of a screen, backlight, driver circuitry, flexible printed circuit board (FPC), and adhesive lines. During production, these components require multi-angle and multi-area quality inspection to ensure product quality.
[0031] Therefore, the placement position and angle of the display module to be inspected on the production line may vary each time. If the robot grasps at a fixed point, it may miss or collide with the workpiece due to positional shift. By acquiring pixel coordinates and converting them into robot coordinates, vision-guided flexible grasping can be achieved. The robot adjusts its grasping posture based on real-time visual information to adapt to changes in workpiece position and angle, improving the grasping success rate and production line flexibility.
[0032] S120. Based on the module category of the display module to be detected, determine multiple target detection points and their target detection poses, and based on each target detection pose, sequentially acquire images of the target detection points to obtain the detection image and its clarity value; wherein, the target detection points are points determined according to a preset number of reference detection points.
[0033] The module category refers to the model or classification of the display module to be tested, used to distinguish display modules of different sizes, testing areas, point sequences, and quality inspection formulas. In practical applications, each screen model corresponds to a set of points and a set of workflow steps. For example, "LCM-7inch" is one module category, and "LCM-10inch" is another; or it can be classified by product type as "LCD module," "flexible screen module," "FPC module," etc. Display modules have multiple models, each with different sizes, testing areas, and point sequences. Traditional solutions require manual copying, modification, and verification of a large number of configurations, resulting in high changeover costs. By introducing module categories, the system can organize points, dot matrices, processes, and parameters by model. When changing models, switching the module category loads the corresponding workflow, improving changeover efficiency and on-site maintainability.
[0034] Target detection points refer to the specific spatial locations where the robot needs to sequentially reach and trigger the camera to take pictures during the quality inspection of the display module. Each target detection point corresponds to the target position and posture of a robot end effector (or camera). Target detection points are determined based on a preset number of benchmark detection points, specifying which spatial locations the camera needs to reach to sample different areas of the display module to be inspected, covering areas requiring quality inspection such as screen edges, screen interiors, FPC, adhesive lines, and the back of the module. In practical applications, each target detection point is bound to a quality inspection area label, which is automatically converted into an area code and written to the Modbus D6 register before taking pictures. The vision algorithm calls the corresponding defect detection algorithm based on the area code, solving the problem of the disconnect between the detection point and the algorithm formula.
[0035] Target detection pose refers to the target position and orientation of the robot end effector (or camera) at the target detection point.
[0036] The sharpness value is a quantitative indicator used to determine the sharpness of a detected image, and it can be returned by the Modbus D7 register. The sharpness value is a numerical value; generally, the larger the value, the blurrier the image, and the smaller the value, the sharper the image. If the sharpness value is less than a set threshold, the image is considered sharp; if the sharpness value is greater than or equal to the set threshold, the image is considered blurry.
[0037] Therefore, the display module has multiple detection areas, including the edges, in-screen area, FPC, adhesive path, and back. If each area is taught point by point, it is time-consuming and has high replacement and maintenance costs. Generating a large number of target detection points by using a small number of reference detection points can avoid point-by-point teaching and improve teaching efficiency and replacement efficiency.
[0038] S130. Based on the preset sharpness threshold and sharpness value, the target detection pose is adjusted to obtain the target detection image.
[0039] The preset clarity threshold refers to a pre-determined benchmark value used to determine whether the detected image is clear. If the clarity value is less than the preset clarity threshold, the image is considered clear; if the clarity value is greater than or equal to the preset clarity threshold, the image is considered blurry. In practical applications, the preset clarity threshold can be configured according to the product model.
[0040] A target detection image refers to the image that, after sharpness assessment and autofocus re-capture, is ultimately used for processing by a visual detection algorithm. A target detection image is an image that meets the sharpness requirement, meaning its corresponding sharpness value is less than a preset sharpness threshold. A target detection image can be a sharp image captured initially, or a sharp image obtained after one or more autofocus re-captures.
[0041] Therefore, the accuracy of the detection algorithm is highly dependent on image quality. If the image is blurry, the detection algorithm may falsely detect or miss detections, leading to product quality issues. In traditional solutions, if the camera returns a blurry image, manual verification, retaking the photo, or readjusting the camera distance are often necessary; automatic fine-tuning and re-taking based on the plane orientation of the detection point are not possible. Through this step, the system can automatically determine whether the image is clear and trigger automatic focusing and re-taking when blurry, ultimately obtaining a target detection image that meets the clarity requirements, ensuring the accuracy and reliability of the detection algorithm.
[0042] S140. Based on the detection category of the target detection point, perform defect detection on the target detection image to obtain the target detection result.
[0043] The detection category refers to the quality inspection area category corresponding to the target detection point. It is used to identify what type of defect should be detected in the image captured at that point, and which visual inspection algorithm, parameters, or template should be used. In practical applications, the detection category is represented by a point tag, which is converted into a quality inspection area code and written into the Modbus D6 register before quality inspection photography is performed. This allows the quality inspection camera or visual algorithm node to know the area to be inspected before receiving the photography trigger. The specific form of the detection category can be a number or a Chinese alias, for example: 0 or "glue + screen" represents the area where glue and screen are combined; 1 or "screen edge" represents the screen edge area; 2 or "inside screen" represents the area inside the screen; 3 or "FPC" represents the flexible circuit board area; 4 or "module back" represents the module back area. This allows for the selection of the corresponding detection algorithm, parameters, or template.
[0044] Defect detection refers to the process of processing the target image based on the detection category of the target detection point, calling the corresponding visual detection algorithm, parameters, or template, identifying whether defects exist in the image, and outputting the detection results. The specific content of defect detection depends on the detection category. For example, screen edge detection: detecting whether there are defects such as chipping, cracks, gaps, and burrs at the screen edges; algorithms may include edge extraction, contour analysis, chipping area calculation, and crack length measurement. In-screen detection: detecting whether there are defects such as bright spots, dark spots, scratches, foreign objects, color spots, and mura (cloudiness) inside the screen; algorithms may include brightness analysis, contrast analysis, texture analysis, and frequency domain analysis. FPC detection: detecting whether the circuits of the flexible circuit board are broken, short-circuited, have poor soldering, bridging, component misalignment, or missing components; algorithms may include circuit connectivity analysis, solder area analysis, and component positioning. The purpose of defect detection is to identify defects in each detection area of the display module, output detection results, and achieve product quality control.
[0045] The target detection result refers to the output of defect detection, used to represent the detection conclusion of the target detection point. In practical applications, the target detection result is returned by the Modbus D2 register: D2=1 indicates OK, meaning no defect was found in the target detection image; D2=2 indicates NG, meaning a defect was found in the target detection image; D2=0 indicates no result or timeout, meaning the visual detection algorithm has not yet returned a result, or the detection process timed out. The target detection result can also include more detailed information, such as defect type, defect location, defect size, defect severity, confidence level, etc. This information can be returned by the visual algorithm node through other communication methods or stored in the quality inspection record. The target detection result is associated with the target detection point, detection category, and target detection image. Each target detection point corresponds to one target detection result; each detection category corresponds to a set of detection algorithms and judgment criteria; each target detection image corresponds to one detection result. Through this association, the system can trace the detection status of each point, statistically analyze the detection results of different detection categories, and analyze the defect distribution patterns.
[0046] Therefore, the display module has multiple detection areas, including edges, in-screen, FPC, adhesive lines, and the back surface. Each area has different defect types, detection algorithms, detection parameters, and judgment criteria. Without distinguishing between detection categories, the vision system cannot know what defects to detect in the current image, which algorithm to use, or what criteria to apply for judgment. Therefore, a clear detection category identifier is needed to associate robot locations with the vision detection algorithm, ensuring that the image at each location is processed by the correct algorithm.
[0047] Based on the feasible implementation of S110 described above, this application further provides a method for determining the grasping pose of a dual-arm robot based on pixel coordinates, including: Based on the positioning camera, the pixel coordinates and positioning angle of the display module to be detected are determined, and the pixel coordinates are transformed based on the homography matrix to obtain the two-dimensional grasping coordinates; Based on the two-dimensional grasping coordinates and the preset grasping height, the three-dimensional grasping coordinates are determined, and based on the positioning angle, the grasping posture is determined. Based on the preset workspace of the dual-arm robot, determine whether the three-dimensional grasping coordinates are within the preset workspace; If so, the grasping pose is determined based on the 3D grasping coordinates and grasping posture; If not, based on the positioning camera and the display module to be detected, the pixel coordinates and positioning angle are collected and updated; based on the updated pixel coordinates and positioning angle, the updated 3D grasping coordinates are determined, and if the updated 3D grasping coordinates are within the preset workspace, the grasping pose is determined based on the updated 3D grasping coordinates; if the updated 3D grasping coordinates are not within the preset workspace, the determination of the grasping pose is stopped, and the corresponding prompt information is output. The prompt information is used to inform the user that the grasping has failed.
[0048] Among them, the positioning camera refers to the vision sensor installed on the end effector, torso, or fixed position in the working area of the dual-arm robot, which is used to capture the display module to be detected and return its pixel coordinates and positioning angle in the image.
[0049] The positioning angle refers to the rotation angle of the display module under test relative to the camera coordinate system or robot coordinate system, usually expressed as θ. The positioning angle is used to determine the yaw angle in the grasping posture, so that the robot end effector (such as a suction cup or gripper) is aligned with the workpiece surface.
[0050] The homography matrix is used to describe the mapping relationship between two planes. Specifically, the homography matrix describes the mapping relationship between the pixel coordinate system (image plane) and the robot coordinate system (work plane).
[0051] The preset gripping height refers to the pre-determined target height of the robot's end effector in the robot coordinate system (z-coordinate) when gripping the display module to be inspected. The preset gripping height is typically determined based on factors such as the height of the workpiece placement plane, the length of the end effector, and the thickness of the workpiece. It is stored in the workflow configuration and can be configured according to the screen model. The preset gripping height is a fixed value and does not change with pixel coordinates. This is because the homography matrix describes the mapping relationship between two-dimensional planes, only providing the x and y coordinates, not the z coordinate. Therefore, the gripping height in the z-direction needs to be configured separately. The preset gripping height can be adjusted according to actual working conditions; for example, it can be modified when the height of the workpiece placement plane changes.
[0052] The preset workspace refers to the three-dimensional spatial range within which a dual-arm robot can safely and reachably perform grasping tasks. In practical applications, the system verifies whether the target's x, y, and z coordinates are within the preset workspace. The preset workspace is typically determined by the robot manufacturer or field engineer based on the robot's structure, joint limitations, arm length, installation location, and safety boundaries, and is stored in the workflow configuration. The preset workspace can be represented as a cuboid, sphere, or other geometric shape, or as a set of joint angle ranges or end effector pose ranges.
[0053] Updating pixel coordinates refers to obtaining a new set of pixel coordinates by re-shooting the display module to be inspected using the positioning camera after the three-dimensional grasping coordinates obtained from the first visual positioning exceed the preset workspace; updating positioning angle refers to the new positioning angle returned by the positioning camera during the reshoot.
[0054] Based on this, when visual positioning and grasping are required first, the system controls the robot to move to the positioning and imaging point, triggering the positioning camera. The positioning camera returns the pixel's horizontal coordinate u, vertical coordinate v, and angle via the Modbus register. The system reads the floating-point data from registers D11-D16, and substitutes the pixel coordinates into the homography matrix: [X,Y,W]^T=H×[u,v,1]^T; H is the 3×3 homography matrix from the pixel plane to the robot's working plane; T is the transpose matrix; [X,Y,W]^T is the transformed homogeneous coordinate, where X is the horizontal component after homogeneous transformation, Y is the vertical component after homogeneous transformation, and W is the scaling factor of the homogeneous coordinate; the robot's horizontal coordinate x = X / W + offset x_offset, vertical coordinate y = Y / W + offset y_offset, and the ordinate z uses the fixed gripping height in the workflow configuration. Then, it verifies whether the target x, y, and z are within the preset workspace range and generates the right arm gripping target pose. The angle can be used to adjust the preset gripping pose of the end effector.
[0055] Furthermore, before executing the grab, a prediction can be made through the motion planning module, that is, only planning without execution, calculating the maximum joint span from the current point to the pre-grab point, the grab point, and the lift point. If the threshold is exceeded, the on-site personnel are required to choose to retake the photo, confirm to continue, skip the grab, or stop.
[0056] Based on the feasible implementation of S120 described above, this application further provides a method that, before determining multiple target detection points and their target detection poses based on the module category of the display module to be detected, includes: Based on a dual-arm robot, a preset number of reference detection points are determined on the display module to be tested, and multiple target detection points corresponding to the reference detection points are generated according to the preset number. Based on the baseline detection pose of the baseline detection point, the target detection pose of each target detection point is determined, and the target detection point and its target detection pose are associated with the module category.
[0057] The preset quantity refers to the number of reference detection points collected by on-site personnel on the display module to be tested when generating target detection points. In practical applications, the preset quantity can be 2 or 3: when the preset quantity is 2, the system generates a line segment matrix for linear detection of screen edges, FPC edges, or adhesive lines; when the preset quantity is 3, the system generates a surface matrix for detection of the screen interior, the back of the module, or other planar areas, and the three points must not be collinear. The system determines the validity of the plane through vector cross product.
[0058] The reference detection pose refers to the target position and orientation of the robot's end effector (or camera) corresponding to the reference detection point. On-site personnel move the robot to a representative position within the screen edge, screen interior, or back detection area and save it as a reference point. The end effector pose corresponding to each reference point is calculated using forward kinematics (FK), yielding the 3D position and orientation of the reference point. This end effector pose is the reference detection pose.
[0059] Based on this, on-site personnel move the robot to representative positions within the screen edge, in-screen area, or back panel detection area and save them as reference points. The system calculates the end effector pose corresponding to each reference point using forward kinematics, obtaining the 3D position and end effector orientation of the reference point. When two reference points are selected, the system generates a line segment matrix for linear detection of screen edges, FPC edges, or adhesive lines. When three reference points are selected, the system generates a surface matrix for detection within the screen, on the back panel of the module, or other planar areas. The three points must not be collinear; the system determines the validity of the plane using vector cross product.
[0060] Furthermore, existing robot teaching methods typically require manual movement of the robot point by point, and saving each detection point. For detection objects distributed linearly or planarly, such as screen edges, in-screen areas, and adhesive lines, if dozens or even hundreds of imaging points are needed, point-by-point teaching is required. During model changes, a large number of points also need to be re-maintained. Display modules have multiple detection areas, including edges, in-screen areas, FPCs, adhesive lines, and the back side. If each area is taught point by point, it is time-consuming and has high maintenance costs during model changes. By limiting the preset number to 2 or 3, on-site personnel only need to teach 2 or 3 reference points, and the system can automatically generate a large number of candidate points. Different detection areas have different geometric characteristics, and the preset number can be flexibly selected according to the area. For example, 2 reference points can be selected for screen edge detection, 3 for in-screen detection, 2 for FPC detection, and 3 for module back side detection. The system automatically selects the corresponding dot matrix generation algorithm based on a preset number of points, requiring no manual intervention. Specifically, two reference points generate a line segment dot matrix, suitable for linear detection of screen edges, FPC edges, or adhesive lines; three reference points generate a surface dot matrix, suitable for detection within the screen, on the back of the module, or other planar areas. This allows the same method to cover both linear and surface detection areas.
[0061] Based on the feasible implementation of S120 described above, this application further provides a method for generating multiple target detection points corresponding to a baseline detection point according to a preset quantity, including: Determine the preset quantity; If the preset number is 2, then the direction vector is determined based on the orientation between the reference detection points, and the direction vector is divided equally to obtain multiple target detection points; If the preset quantity is 3, the current point generation mode is determined based on the cross product modulus of the benchmark detection points. When the point generation mode is right angle mode, the diagonal direction is determined based on the first and second reference detection points, and the detection plane normal vector is determined based on the diagonal direction and the third reference detection point. Based on the diagonal direction and the normal vector of the detection plane, the vertical direction perpendicular to the diagonal direction in the detection plane is determined, and based on the diagonal direction and the vertical direction, the local coordinate system is determined; Based on the local coordinate system, the projected position of the third reference detection point is determined, the rectangle orientation is obtained, and the row and column directions are determined based on the diagonal direction, vertical direction and rectangle orientation to obtain the detection area; When the point generation mode is parallelogram mode, the column direction is determined based on the orientation between the first and second reference detection points, and the row direction is determined based on the orientation between the first and third reference detection points; the detection area is determined based on the row and column directions. By performing row and column interpolation on the detection area, multiple target detection points are obtained.
[0062] Here, orientation refers to the spatial relationship between two reference detection points when the preset quantity is 2. It includes the three-dimensional positional difference between the two reference points in the robot coordinate system and the direction of the connecting line. Specifically, orientation describes the spatial direction from the first reference point P0 to the second reference point P1, that is, the relative positional relationship between P0 and P1 in three-dimensional space.
[0063] The direction vector, when the preset quantity is 2, is a unit vector calculated based on the orientation between two reference detection points, used to describe the extension direction of the line segment matrix; let the first reference point be P0 and the second reference point be P1, calculate the column direction unit vector u: u=(P1-P0) / |P1-P0| The direction is consistent with the orientation from P0 to P1.
[0064] The first, second, and third reference detection points refer to three reference points collected on the display module to be tested when the preset quantity is 3. They are denoted as P0, P1, and P2, respectively. These three reference points are used to determine the plane and range of the detection area and generate a surface dot matrix.
[0065] The cross product magnitude, also known as the vector product or cross product, is the magnitude of the vector obtained by performing the cross product of the direction vectors between two reference detection points and the direction vector related to another reference detection point.
[0066] The diagonal direction, when the preset quantity is 3, refers to the direction determined by the first reference detection point P0 and the second reference detection point P1. It describes a diagonal direction of the rectangular detection area, connecting P0 and P1. In the right-angle dot matrix mode, P0 and P1 are used as the diagonal of the rectangle, and the third point P2 is used to determine the screen plane normal and the rectangle's orientation. The system calculates the diagonal direction, the plane normal, and the direction perpendicular to the diagonal, and then deduces the direction of the right-angled side of the rectangle to obtain the row and column directions.
[0067] The detection plane normal vector is a unit vector perpendicular to the plane containing the detection area, determined by the diagonal direction and the third reference detection point P2. It is the cross product of the diagonal direction vector and the vector from P0 to P2. The result of the cross product is perpendicular to the plane determined by points P0, P1, and P2, which is the normal vector of the plane containing the detection area.
[0068] A local coordinate system is a temporary coordinate system established within the detection plane. It typically uses a reference point as its origin, two directions within the detection plane as the x-axis and y-axis, and the detection plane normal vector as the z-axis. The detection region is a rectangular planar area determined by the diagonal direction and the normal vector, used to generate the surface dot matrix. The detection region is the geometric representation of the area within the screen, on the back of the module, or other planar regions. Its shape is usually rectangular, determined by the row and column directions. In the rectangular dot matrix mode, P0 and P1 are used as the diagonals of the rectangle, and the third point P2 is used to determine the screen plane normal and the rectangle's orientation. The system calculates the diagonal direction, the plane normal, and the direction perpendicular to the diagonal, then deduces the direction of the rectangular right-angled sides to obtain the row and column directions. The extent of the detection region is determined by the row and column lengths.
[0069] Furthermore, the detection area can be defined as a parallelogram pattern, that is, with P0 to P1 as the column direction and P0 to P2 as the row direction, a dot matrix is generated within the parallelogram region. The extent of the detection area is determined by the column direction length and the row direction length.
[0070] Row and column interpolation is a two-dimensional parametric interpolation operation performed within a defined planar detection area, according to the configured total number of rows R and total number of columns C, to generate all candidate target detection points that fill the entire detection area in batches.
[0071] Based on this, candidate point matrices are generated according to the reference points. For two reference points, let the first reference point be P0 and the second reference point be P1. Calculate the column direction unit vector u = (P1 - P0) / |P1 - P0|, and divide P0 to P1 equally according to the number of columns to generate multiple candidate points. For three reference points, two generation modes can be used: a right-angle point matrices mode, where P0 and P1 are used as the diagonals of a rectangle, and the third point P2 is used to determine the screen plane normal and the rectangle orientation. The system calculates the diagonal direction, the plane normal, and the direction perpendicular to the diagonal, and then calculates the direction of the right-angled sides of the rectangle to obtain the row and column directions; and a parallelogram point matrices mode, where P0 to P1 are used as the column direction and P0 to P2 are used as the row direction to generate a point matrices within a parallelogram region.
[0072] Based on the feasible implementation of S120 described above, this application further provides a method for determining the current point generation mode based on the cross product modulus of the reference detection point, including: When the preset quantity is 3, the cross product modulus is calculated based on each benchmark detection point, and it is determined whether the cross product modulus is 0. If so, determine that the benchmark detection points are collinear, and redetermine the preset number of updated benchmark detection points; based on the updated benchmark detection points, determine the point generation mode; If not, then determine that the benchmark detection points are not collinear, and determine the current corresponding point generation mode.
[0073] Therefore, when two reference points are selected, the two points must not coincide, and the distance between the two points ||P1-P0|| must be greater than the preset minimum distance. When three reference points are selected, the system generates a surface point matrix for detection within the screen, on the back of the module, or other planar areas. The three points must not be collinear. The collinearity of the three reference points can be determined by vector cross product, and the detection plane is determined when they are not collinear.
[0074] Based on the feasible implementation of S120 described above, this application further provides a reference detection pose based on the reference detection points, determining the target detection pose of each target detection point, including: When the preset number is 2, the position coordinates of the target detection points are calculated based on the position coordinates of the reference detection points and the distance between each point; where the point distance is the distance between each target detection point and the reference detection point. When the preset quantity is 3, the ratio of each row number and the ratio of each column number are calculated based on the number of rows and columns corresponding to each target detection point and the number of rows and columns in the detection area. Based on the area row vector, the area column vector, and the ratio of rows and columns, the position coordinates of each target detection point are calculated. Based on the distance between the target detection point and each reference detection point, the pose of each reference detection point is weighted and averaged to obtain the target pose. Then, based on the position coordinates and the target pose, the target detection pose of each target detection point is determined.
[0075] Here, the row and column numbers corresponding to the target detection points refer to the row and column indices of each target detection point in the detection area matrix when the preset quantity is 3. Let the number of rows in the detection area be R and the number of columns be C, then the target detection point can be represented by (r, c); the number of rows and columns determines the position of the target detection point in the detection area. The target detection point in the r-th row and c-th column is located at the intersection of the r-th row and c-th column of the detection area.
[0076] The number of rows and columns in a region refers to the total number of rows and columns of the detection region's dot matrix when the preset quantity is 3. Let the number of rows be R and the number of columns be C. Then the detection region is divided into a dot matrix of R rows and C columns, generating a total of R×C target detection points.
[0077] The row ratio and column ratio refer to the ratio between the number of rows corresponding to the target detection point and the number of rows in the region, and the number of columns corresponding to the number of columns in the region, when the preset quantity is 3. They are used to determine the relative position of the target detection point in the detection area.
[0078] Region row vectors and region column vectors refer to the vector representations of the row and column directions of the detection region. A region row vector represents the unit vector and length of the detection region along the row direction, while a region column vector represents the unit vector and length of the detection region along the column direction.
[0079] The target pose refers to the pose of the target detection point obtained by weighting the poses of each reference detection point based on the distance between the target detection point and each reference detection point when the preset number is 2 or 3.
[0080] Based on this, when the preset quantity is 2, the candidate point positions of the line segment lattice are: Pk = P0 + [k / (N-1)](P1-P0), k = 0, ..., N-1; when N = 1, only P0 is taken. N is the total number of candidate points on the line segment dot matrix, which is how many target detection points need to be generated on this line segment; k is the index of the candidate point, indicating which point is currently being generated; Pk is the position of the k-th candidate point.
[0081] When the preset number is 3, the candidate point positions are generated according to the row-column ratio. Let the number of rows be R and the number of columns be C, the column ratio of the r-th row and c-th column be alpha_c, and the row ratio beta_r.
[0082] When C is greater than 1, alpha_c = c / (C-1); otherwise, alpha_c = 0. When R is greater than 1, beta_r = r / (R-1); otherwise, beta_r = 0.
[0083] The candidate point locations are: Q(r,c) = P0 + column direction vector × column direction length × alpha_c + row direction vector × row direction length × beta_r.
[0084] The candidate point attitude is interpolated using the reference point attitude. If a candidate point coincides with a reference point, the reference point attitude is used directly; otherwise, the candidate point is weighted according to its distance from each reference point, with the closer the distance, the greater the weight. The quaternion attitude is then normalized and weighted to make the candidate point's terminal attitude change smoothly within the point matrix.
[0085] Furthermore, the end-effector pose of each candidate point can be input into the inverse kinematics solver to obtain the corresponding robotic arm joint angles. If a solution cannot be found, the candidate point is marked as unreachable and cannot be saved as a production point. For successfully solved candidate points, their joint angles, maximum joint span, and unique lattice key are recorded. Optionally, the maximum joint variation between adjacent selected points is calculated based on the user's selection order. If it exceeds a set threshold, the point or segment is marked as a swing arm risk point, prompting on-site personnel to adjust the saving order or skip it.
[0086] Based on the feasible implementation of S130 described above, this application further provides a method for adjusting the target detection pose based on a preset sharpness threshold and sharpness value to obtain a target detection image, including: Determine if the clarity value is less than the preset clarity threshold; If the value is less than the target detection pose, the target detection pose will not be adjusted, and the detected image will be determined as the target detection image. If it is not less than, then based on the metadata of the point plane associated with the target detection pose, the detection plane normal vector is determined, and based on the detection plane normal vector, the preset step size and the preset number of attempts, multiple adjustment distances are determined; Based on the target detection pose, the detection plane normal vector, and multiple adjustment distances, multiple adjustment detection poses are determined, and the target joint span is calculated based on each adjustment detection pose and the current joint state. Compare whether the target joint span is less than a preset joint span threshold; When the target joint span is less than the preset joint span threshold, image acquisition is performed based on the adjustment detection pose to obtain the adjustment detection image and its adjustment clarity value; the target detection image is determined based on multiple adjustment clarity values and the preset clarity threshold. When the target joint span is not less than the preset joint span threshold, a prompt message is output, which indicates that the current target joint span cannot be achieved.
[0087] The preset step size refers to the distance moved each time the target detection pose is adjusted when the image sharpness value is not less than the preset sharpness threshold. It is usually expressed in millimeters (mm) and is called step_mm.
[0088] Adjusting the detection pose refers to determining a new target detection pose for reshooting based on a preset step size and the target detection pose, provided the image sharpness value is not less than a preset sharpness threshold. In practical applications, adjusting the detection pose involves offsetting the target detection pose by a certain distance along the normal direction of the screen plane containing the point.
[0089] Based on this, if the quality inspection photo is blurry and the automatic sharpness adjustment function is enabled, the metadata of the pixel plane associated with the current point is read to obtain the normal vector n of the screen plane where the point is located, and a set of fine-tuning distances is generated. For example, with step_mm as the step size, offsets such as +step, +2step, -step, and -2step are generated in the order of first moving away from the plane and then moving closer to the plane. The pose Q' of each fine-tuning target is: Q' = Q + n × d; Where Q is the current or corresponding end position, n is the unit normal vector, and d is the current fine-tuning distance.
[0090] Furthermore, when the step is configured for quality inspection photography, the following sub-steps are executed: 1. Read the tag based on the current location name.
[0091] 2. Convert the tag to a quality inspection area code and write it to the ModbusD6 register.
[0092] 3. Read the current value of register D2; if D2 is not 0, clear it first to avoid reading the previous detection result.
[0093] 4. Write register D1=1 to trigger the quality inspection camera to take a picture.
[0094] 5. Waiting for register D1 to be reset to 0 by the camera or vision device indicates that the photo capture trigger has been received or processed.
[0095] 6. Poll D2 within the set timeout period. D2=1 indicates that the detection is OK, D2=2 indicates that the detection is NG, and D2=0 indicates that there is no result or the timeout has occurred.
[0096] 7. Read the sharpness index from register D7. If the sharpness index is less than the set threshold, the image is considered sharp; if the sharpness index is greater than or equal to the set threshold, the image is considered blurry.
[0097] Before each fine-tuning, a safety assessment of the movement is performed: 1. Plan based on the current joint state and target pose.
[0098] 2. Calculate the maximum joint span.
[0099] 3. If the threshold is exceeded, the focusing movement can be automatically skipped, or the on-site personnel can be prompted to confirm whether to continue, skip, or stop.
[0100] If the safety prediction is successful, the robot moves to the fine-tuning pose and writes to register D6, triggers register D1, waits for register D2, and reads register D7 again. If the value of register D7 is less than the threshold, it is determined that the sharpness has been restored, and subsequent fine-tuning stops while the workflow continues. If the image is still blurry after all attempts, the robot chooses to continue, retry, skip, or abort according to the configuration.
[0101] Based on the above steps, it can be seen that the method of this application determines the grasping pose of the dual-arm robot based on pixel coordinates and completes the grasping of the display module to be inspected. It achieves the acquisition of a grasping pose adapted to the actual placement deviation of the workpiece by means of visual pixel coordinate transformation, which can eliminate the grasping misalignment problem caused by the positional offset of the workpiece at the loading station and realize the adaptive and reliable grasping of the display module. According to the module category of the display module to be inspected, a preset number of reference detection points are determined, and multiple target detection points and their target detection poses are further expanded and generated. According to each target detection pose, image acquisition is completed sequentially to obtain the detection image and clarity value. It realizes the generation of all target detection points with a small number of reference detection points, reducing the large amount of debugging work of traditional point-by-point teaching, and at the same time, it combines module category matching. The system uses a set of corresponding detection points to enable rapid switching between different display module models. It adjusts the target detection pose based on preset clarity thresholds and values to obtain the target detection image. This allows for automatic adjustment of the detection pose to complete re-shooting in blurry scenes without manual intervention, forming a closed-loop control of clarity. By combining the detection categories corresponding to the target detection points, it performs defect detection on the target detection image to obtain the target detection result, thus achieving automatic matching of detection points and detection algorithms. This enables automated multi-region grasping of display modules, rapid point generation, closed-loop control of image clarity, and regional adaptive defect detection, effectively improving the efficiency and accuracy of quality inspection point generation for display modules. This further enhances the efficiency and accuracy of defect detection of display modules by the dual-arm robot based on quality inspection points.
[0102] Figure 2 This is a schematic diagram of a point detection device based on a dual-arm robot, provided as an embodiment of this application. Figure 2 As shown, this point detection device based on a dual-arm robot includes: a grasping module, a data acquisition module, an adjustment module, and a detection module; wherein: The grasping module is used to determine the grasping pose of the dual-arm robot based on pixel coordinates, and to grasp the display module to be inspected based on the grasping pose. The acquisition module is used to determine multiple target detection points and their target detection poses based on the module category of the display module to be detected, and to sequentially acquire images of the target detection points based on each target detection pose to obtain the detection image and its clarity value; wherein, the target detection points are points determined according to a preset number of reference detection points; The adjustment module is used to adjust the target detection pose based on a preset sharpness threshold and sharpness value to obtain a target detection image; The detection module is used to perform defect detection on the target detection image based on the detection category of the target detection point and obtain the target detection result.
[0103] In this embodiment of the application, the capture module can also be specifically used for: Based on the positioning camera, the pixel coordinates and positioning angle of the display module to be detected are determined, and the pixel coordinates are transformed based on the homography matrix to obtain the two-dimensional grasping coordinates; Based on the two-dimensional grasping coordinates and the preset grasping height, the three-dimensional grasping coordinates are determined, and based on the positioning angle, the grasping posture is determined. Based on the preset workspace of the dual-arm robot, determine whether the three-dimensional grasping coordinates are within the preset workspace; If so, the grasping pose is determined based on the 3D grasping coordinates and grasping posture; If not, based on the positioning camera and the display module to be detected, the pixel coordinates and positioning angle are collected and updated; based on the updated pixel coordinates and positioning angle, the updated 3D grasping coordinates are determined, and if the updated 3D grasping coordinates are within the preset workspace, the grasping pose is determined based on the updated 3D grasping coordinates; if the updated 3D grasping coordinates are not within the preset workspace, the determination of the grasping pose is stopped, and the corresponding prompt information is output. The prompt information is used to inform the user that the grasping has failed.
[0104] In this embodiment of the application, the acquisition module can also be specifically used for: Based on a dual-arm robot, a preset number of reference detection points are determined on the display module to be tested, and multiple target detection points corresponding to the reference detection points are generated according to the preset number. Based on the baseline detection pose of the baseline detection point, the target detection pose of each target detection point is determined, and the target detection point and its target detection pose are associated with the module category.
[0105] In this embodiment of the application, the acquisition module can also be specifically used for: Determine the preset quantity; If the preset number is 2, then the direction vector is determined based on the orientation between the reference detection points, and the direction vector is divided equally to obtain multiple target detection points; If the preset quantity is 3, the current point generation mode is determined based on the cross product modulus of the benchmark detection points. When the point generation mode is right angle mode, the diagonal direction is determined based on the first and second reference detection points, and the detection plane normal vector is determined based on the diagonal direction and the third reference detection point. Based on the diagonal direction and the normal vector of the detection plane, the vertical direction perpendicular to the diagonal direction in the detection plane is determined, and based on the diagonal direction and the vertical direction, the local coordinate system is determined; Based on the local coordinate system, the projected position of the third reference detection point is determined, the rectangle orientation is obtained, and the row and column directions are determined based on the diagonal direction, vertical direction and rectangle orientation to obtain the detection area; When the point generation mode is parallelogram mode, the column direction is determined based on the orientation between the first and second reference detection points, and the row direction is determined based on the orientation between the first and third reference detection points; the detection area is determined based on the row and column directions. By performing row and column interpolation on the detection area, multiple target detection points are obtained.
[0106] In this embodiment of the application, the acquisition module can also be specifically used for: When the preset quantity is 3, the cross product modulus is calculated based on each benchmark detection point, and it is determined whether the cross product modulus is 0. If so, determine that the benchmark detection points are collinear, and redetermine the preset number of updated benchmark detection points; based on the updated benchmark detection points, determine the point generation mode; If not, then determine that the benchmark detection points are not collinear, and determine the current corresponding point generation mode.
[0107] In this embodiment of the application, the acquisition module can also be specifically used for: When the preset number is 2, the position coordinates of the target detection points are calculated based on the position coordinates of the reference detection points and the distance between each point; where the point distance is the distance between each target detection point and the reference detection point. When the preset quantity is 3, the ratio of each row number and the ratio of each column number are calculated based on the number of rows and columns corresponding to each target detection point and the number of rows and columns in the detection area. Based on the area row vector, the area column vector, and the ratio of rows and columns, the position coordinates of each target detection point are calculated. Based on the distance between the target detection point and each reference detection point, the pose of each reference detection point is weighted and averaged to obtain the target pose. Then, based on the position coordinates and the target pose, the target detection pose of each target detection point is determined.
[0108] In this embodiment of the application, the adjustment module can also be specifically used for: Determine if the clarity value is less than the preset clarity threshold; If the value is less than the target detection pose, the target detection pose will not be adjusted, and the detected image will be determined as the target detection image. If it is not less than, then based on the metadata of the point plane associated with the target detection pose, the detection plane normal vector is determined, and based on the detection plane normal vector, the preset step size and the preset number of attempts, multiple adjustment distances are determined; Based on the target detection pose, the detection plane normal vector, and multiple adjustment distances, multiple adjustment detection poses are determined, and the target joint span is calculated based on each adjustment detection pose and the current joint state. Compare whether the target joint span is less than a preset joint span threshold; When the target joint span is less than the preset joint span threshold, image acquisition is performed based on the adjustment detection pose to obtain the adjustment detection image and its adjustment clarity value; the target detection image is determined based on multiple adjustment clarity values and the preset clarity threshold. When the target joint span is not less than the preset joint span threshold, a prompt message is output, which indicates that the current target joint span cannot be achieved.
[0109] Figure 3 This is a schematic diagram of the structure of a device for performing a point detection method based on a dual-arm robot according to an embodiment of this application. Figure 3 As shown, the device includes: The device may include one or more processors with processing cores, one or more computer-readable storage media such as memory, communication components, etc. The processor, memory, and communication components are connected via a bus.
[0110] In the specific implementation process, at least one processor executes computer execution instructions stored in memory, causing at least one processor to execute the above-described point detection method based on a dual-arm robot.
[0111] The specific implementation process of the processor can be found in the above method embodiments, and its implementation principle and technical effect are similar, so it will not be repeated here.
[0112] Furthermore, the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.
[0113] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.
[0114] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.
[0115] In some embodiments, a computer program product is also provided, including a computer program or instructions that, when executed by a processor, implement the steps in any of the above-described point detection methods based on a dual-arm robot.
[0116] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.
[0117] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.
[0118] Therefore, embodiments of this application provide a computer-readable storage medium storing multiple lines of program code that can be loaded by a processor to execute the steps in any of the point detection methods based on dual-arm robots provided in embodiments of this application.
[0119] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.
[0120] According to one aspect of this application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium.
[0121] Since the instructions stored in the storage medium can execute the steps in any of the point detection methods based on dual-arm robots provided in the embodiments of this application, the beneficial effects that any of the point detection methods based on dual-arm robots provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.
[0122] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope of this application is indicated by the appended claims.
[0123] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.
Claims
1. A point detection method based on a dual-arm robot, characterized in that, The method includes: Based on pixel coordinates, the grasping pose of the dual-arm robot is determined, and based on the grasping pose, the display module to be detected is grasped. Based on the module category of the display module to be detected, multiple target detection points and their target detection poses are determined, and based on each target detection pose, images are sequentially acquired from the target detection points to obtain a detection image and its clarity value; wherein, the target detection points are points determined according to a preset number of reference detection points; Based on the preset clarity threshold and the clarity value, the target detection pose is adjusted to obtain the target detection image; Based on the detection category of the target detection points, defect detection is performed on the target detection image to obtain the target detection result.
2. The method according to claim 1, characterized in that, Determining the grasping pose of the dual-arm robot based on pixel coordinates includes: Based on the positioning camera, the pixel coordinates and positioning angle of the display module to be detected are determined, and the pixel coordinates are transformed based on the homography matrix to obtain two-dimensional grasping coordinates; Based on the two-dimensional grasping coordinates and the preset grasping height, the three-dimensional grasping coordinates are determined, and based on the positioning angle, the grasping posture is determined. Based on the preset workspace of the dual-arm robot, determine whether the three-dimensional grasping coordinates are within the preset workspace; If so, the grasping pose is determined based on the three-dimensional grasping coordinates and the grasping posture; If not, then based on the positioning camera and the display module to be detected, update pixel coordinates and update positioning angle are collected and updated; based on the updated pixel coordinates and the updated positioning angle, update 3D grasping coordinates are determined, and when the updated 3D grasping coordinates are within the preset workspace, the grasping pose is determined based on the updated 3D grasping coordinates; when the updated 3D grasping coordinates are not within the preset workspace, the determination of the grasping pose is stopped, and a corresponding prompt message is output, which is used to prompt the user that the grasping has failed.
3. The method according to claim 1, characterized in that, Before determining multiple target detection points and their target detection poses based on the module category of the display module to be detected, the method further includes: Based on the dual-arm robot, a preset number of reference detection points are determined on the display module to be tested, and multiple target detection points corresponding to the reference detection points are generated according to the preset number. Based on the reference detection pose of the reference detection point, the target detection pose of each target detection point is determined, and the target detection point and its target detection pose are associated with the module category.
4. The method according to claim 3, characterized in that, The step of generating multiple target detection points corresponding to the baseline detection point according to the preset number includes: Determine the preset quantity; If the preset number is 2, then based on the orientation between the reference detection points, a direction vector is determined, and the direction vector is divided equally to obtain multiple target detection points; If the preset quantity is 3, then the current point generation mode is determined based on the cross product modulus of the benchmark detection points. When the point generation mode is right angle mode, the diagonal direction is determined based on the first and second reference detection points, and the detection plane normal vector is determined based on the diagonal direction and the third reference detection point. Based on the diagonal direction and the detection plane normal vector, a vertical direction perpendicular to the diagonal direction within the detection plane is determined, and a local coordinate system is determined based on the diagonal direction and the vertical direction. Based on the local coordinate system, the projected position of the third reference detection point is determined to obtain the rectangle orientation. Based on the diagonal direction, the vertical direction, and the rectangle orientation, the row direction and column direction are determined to obtain the detection area. When the point generation mode is parallelogram mode, the column direction is determined based on the orientation between the first reference detection point and the second reference detection point, and the row direction is determined based on the orientation between the first reference detection point and the third reference detection point; the detection area is determined based on the row direction and the column direction. Row and column interpolation is performed on the detection area to obtain multiple target detection points.
5. The method according to claim 4, characterized in that, The step of determining the current point generation mode based on the cross product modulus of the reference detection points includes: When the preset quantity is 3, the cross product modulus is calculated based on each of the reference detection points, and it is determined whether the cross product modulus is 0. If so, then determine that the benchmark detection points are collinear, and redetermine the preset number of updated benchmark detection points; based on the updated benchmark detection points, determine the point generation mode; If not, then determine that the reference detection points are not collinear, and determine the current point generation mode.
6. The method according to claim 3, characterized in that, The step of determining the target detection pose of each target detection point based on the reference detection pose of the reference detection point includes: When the preset number is 2, the position coordinates of the target detection point are calculated based on the position coordinates of the reference detection point and the distance between each point; wherein, the distance between the points is the distance between each target detection point and the reference detection point; When the preset number is 3, the ratio of each row number and the ratio of each column number are calculated based on the number of rows and columns corresponding to each target detection point and the number of rows and columns in the detection area. The position coordinates of each target detection point are calculated based on the area row vector, the area column vector, and the ratio of rows and columns. Based on the distance between the target detection point and each of the reference detection points, the pose of each of the reference detection points is weighted and averaged to obtain the target pose. Then, based on the position coordinates and the target pose, the target detection pose of each of the target detection points is determined.
7. The method according to claim 1, characterized in that, The step of adjusting the target detection pose based on a preset sharpness threshold and the sharpness value to obtain a target detection image includes: Determine whether the clarity value is less than the preset clarity threshold; If it is less than, then the target detection pose is not adjusted, and the detected image is determined to be the target detection image; If it is not less than, then based on the metadata of the dot plane associated with the target detection pose, the detection plane normal vector is determined, and based on the detection plane normal vector, the preset step size and the preset number of attempts, multiple adjustment distances are determined; Based on the target detection pose, the detection plane normal vector, and the multiple adjustment distances, multiple adjustment detection poses are determined, and based on each of the adjustment detection poses and the current joint state, the target joint span is calculated. Compare whether the target joint span is less than a preset joint span threshold; When the target joint span is less than the preset joint span threshold, image acquisition is performed based on the adjustment detection pose to obtain the adjustment detection image and its adjustment clarity value; the target detection image is determined based on multiple adjustment clarity values and the preset clarity threshold. When the target joint span is not less than the preset joint span threshold, a prompt message is output, which indicates that the current target joint span cannot be achieved.
8. A point detection device based on a dual-arm robot, characterized in that, The device includes: The grasping module is used to determine the grasping pose of the dual-arm robot based on pixel coordinates, and to grasp the display module to be detected based on the grasping pose. The acquisition module is used to determine multiple target detection points and their target detection poses based on the module category of the display module to be detected, and to sequentially acquire images of the target detection points based on each target detection pose to obtain a detection image and its clarity value; wherein, the target detection points are points determined according to a preset number of reference detection points; An adjustment module is used to adjust the target detection pose based on a preset sharpness threshold and the sharpness value to obtain a target detection image; The detection module is used to perform defect detection on the target detection image based on the detection category of the target detection point, and obtain the target detection result.
9. A computer device, characterized in that, include: One or more processors; Memory; One or more programs, wherein the one or more programs are stored in memory and configured to be executed by one or more processors, the one or more programs being configured to perform the method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code that can be called by a processor to perform the method as described in any one of claims 1 to 7.
