Object measurement method, device and equipment based on monocular vision

Abstract

Embodiments of the present application provide a monocular vision-based object measurement method, which can be applied to the fields of artificial intelligence technology and robot technology. The monocular vision-based object measurement method comprises: in response to a shape confidence of target candidate region matching, performing visual geometry refinement processing on a mapping frame image of the target candidate region to determine current pixel size data of the target candidate region in an image coordinate system of the mapping frame image; performing multi-frame target correlation processing on the current pixel size data to generate current frame smooth output data of current camera size data according to historical camera size data corresponding to the current pixel size data; and outputting the current frame smooth output data according to a standard uncertainty corresponding to the current frame smooth output data. Embodiments of the present application also provide a monocular vision-based object measurement device, equipment, storage medium and program product.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to the interdisciplinary field of robot vision technology and image processing technology, and more specifically to an object measurement method, device, equipment, medium and product based on monocular vision. Background Technology

[0002] In desktop CNC machine tools, flexible manufacturing, incoming material inspection, and small-batch processing scenarios, it is often necessary to quickly position and measure the dimensions of rectangular, rounded, or circular planar workpieces placed on a worktable. Traditional vernier calipers, mechanical edge finders, or contact probes require manual edge-by-edge operation, which is inefficient. While binocular vision, structured light, and area array depth cameras can obtain dense depth, they have higher hardware costs, more complex calibration, and require more installation space.

[0003] Monocular cameras offer advantages such as low cost, high resolution, and flexible installation, but a single image cannot determine absolute scale. A single-point Time-of-Flight (ToF) ranging sensor can output the target surface depth along a predetermined measurement optical axis. By fusing this sensor with an intrinsically calibrated monocular camera, the actual dimensions and center 3D coordinates of the target contour can be recovered under the assumption of a planar or approximately planar workpiece. Existing solutions typically employ only fixed-threshold edge detection, which is easily affected by color, texture, reflection, shadows, and background edges; or they use only a general target detection network to obtain an axis-aligned bounding box, making it difficult to meet the contour accuracy and continuous frame stability required for precise measurement. Summary of the Invention

[0004] In view of at least one of the above problems, embodiments of the present invention aim to provide a method, apparatus, device, medium and product for object measurement based on monocular vision. It proposes a method and system based on depth constraint target selection, deep learning coarse positioning, traditional geometric refinement, pinhole model size conversion and multi-frame stable output, so as to realize online measurement of planar workpieces, probe relative positioning and machine tool coarse positioning with less hardware.

[0005] One aspect of the present invention provides a method for object measurement based on monocular vision, comprising: determining a corresponding target candidate region based on at least one frame of monocular vision image of the object to be measured and the axial effective depth of a ranging measurement point, according to a preset intersection-union ratio (IUGR) confidence judgment condition; performing visual geometric refinement processing on the mapped frame image of the target candidate region in response to the shape confidence of the target candidate region matching, and determining the current pixel size data of the target candidate region in the image coordinate system of the mapped frame image; performing multi-frame target association processing on the current pixel size data corresponding to the current camera size data according to historical camera size data, and generating current frame smoothed output data of the current camera size data; and outputting current frame smoothed output data according to the standard uncertainty corresponding to the current frame smoothed output data.

[0006] According to an embodiment of the present invention, before determining the corresponding target candidate region based on at least one frame of monocular visual image of the object to be measured and the axial effective depth of the ranging measurement point according to a preset intersection-union confidence score, the method further includes: extracting multiple reference candidate regions corresponding to the axial effective depth frame by frame from the at least one frame of monocular visual image based on the pre-drawn region; determining the candidate comprehensive score of each reference candidate region by using the center mapping coordinates of the ranging spot of the ranging measurement point corresponding to each reference candidate region in the image coordinate system and the image center coordinates of each reference candidate region; and determining the first candidate box and the second candidate box among the multiple reference candidate regions based on the candidate comprehensive score.

[0007] According to an embodiment of the present invention, before determining the corresponding target candidate region based on at least one frame of monocular visual image of the object to be measured and the effective axial depth of the ranging measurement point, the method further includes: scaling the first candidate box and the second candidate box by a preset scaling ratio to generate a first scaled box and a second scaled box; and outputting a shape confidence and an instance segmentation mask based on the first scaled box and the second scaled box.

[0008] According to an embodiment of the present invention, based on at least one frame of monocular visual image of the object to be measured and the axial effective depth of the ranging measurement point, a corresponding target candidate region is determined according to a preset intersection-union ratio (IUGR) confidence criterion, including:

[0009] Based on shape confidence and instance segmentation mask, the validity of the first and second zoomed boxes is determined; based on the first and second zoomed boxes that have been determined to be valid, an effective cross-union ratio (CUP) that meets the preset CUP confidence determination condition is generated; based on the preset CUP confidence determination condition, when the effective CUP is greater than or equal to a preset confidence threshold, the first or second zoomed box with greater confidence is determined as the target candidate region; the target candidate region is cropped, upsampled, and mapped to the corresponding original frame monocular visual image to generate a mapped frame image.

[0010] According to an embodiment of the present invention, in response to the shape confidence score of the target candidate region matching, visual geometric refinement processing is performed on the mapped frame image of the target candidate region to determine the current pixel size data of the target candidate region in the image coordinate system of the mapped frame image, including: in response to the shape confidence score, performing a candidate region expansion local search on the mapped frame image and performing grayscale conversion to generate a grayscale frame image; performing Gaussian filtering, contrast enhancement and adaptive edge detection processing on the grayscale frame image to generate a refined frame image; and determining the current pixel size data of the candidate region bounding box corresponding to the target candidate region on the refined frame image.

[0011] According to an embodiment of the present invention, in determining the current pixel size data of the candidate region box corresponding to the target candidate region on the refined frame image, the method includes: in response to the shape confidence, when the target candidate region is a rectangle, determining the four pixel side lengths, the average pixel length of the opposite side, and the average pixel width of the opposite side of the candidate region box as the current pixel size data based on the corner pixel coordinates of the intersection of the adjacent sides of the candidate region box after straight line fitting processing.

[0012] According to an embodiment of the present invention, after determining the current pixel size data of the candidate region box corresponding to the target candidate region on the refined frame image, the method further includes: based on the axial effective depth in the camera coordinate system, and according to the joint calibration relationship between the camera coordinate system and the ranging coordinate system, converting the rectangle center pixel coordinates corresponding to the current pixel size data into the rectangle center camera coordinates of the current camera size data; wherein, the rectangle corner camera coordinates corresponding to the rectangle center camera coordinates are used to generate the actual size of the rectangle.

[0013] According to an embodiment of the present invention, the current pixel size data of the candidate region box corresponding to the target candidate region on the refined frame image includes: in response to the shape confidence, when the target candidate region is circular, determining the perimeter and area of ​​the candidate region box; and extracting the center pixel coordinates in the image coordinate system as the current pixel size data for the candidate region box after performing circle fitting processing on the roundness corresponding to the perimeter and area.

[0014] According to an embodiment of the present invention, after determining the current pixel size data of the candidate region box corresponding to the target candidate region on the refined frame image, the method further includes: based on the axial effective depth in the camera coordinate system, and according to the camera intrinsic parameters corresponding to the joint calibration relationship between the camera coordinate system and the ranging coordinate system, generating the center camera coordinates of the current camera size data by back-projecting the center pixel coordinates to the camera coordinate system; and generating the actual diameter of the circle by back-projecting the diameter endpoints in the horizontal and vertical directions.

[0015] According to an embodiment of the present invention, in generating smooth output data of the current frame based on the current camera size data by performing multi-frame target association processing on the current pixel size data corresponding to the current camera size data based on historical camera size data, the following steps are included: based on historical camera size data and current pixel size data, performing association determination on the center distance, scale ratio and shape consistency of pixel length and width parameters and center pixel coordinates, and determining smooth output data of the current frame that meets preset stable output suppression conditions, wherein the preset stable output suppression conditions include dead zone threshold suppression rules, single frame maximum step size suppression rules and exponential smoothing suppression rules.

[0016] According to an embodiment of the present invention, the output of smooth output data of the current frame based on the standard uncertainty corresponding to the smooth output data of the current frame includes: outputting smooth output data of the current frame based on the axial effective depth, combined with the standard uncertainty of the edge positioning error, ranging depth noise and focal length calibration error of the current pixel size data of the smooth output data of the current frame, and then outputting smooth output data of the current frame.

[0017] Another aspect of the present invention provides an object measurement device based on monocular vision for implementing the above-described method, wherein the device includes a region determination module, a data determination module, a data generation module, and a data output module. The region determination module is used to determine a corresponding target candidate region based on at least one frame of monocular vision image of the object to be measured and the axial effective depth of the ranging measurement point, according to a preset intersection-over-union (IoU) confidence level. The data determination module is used to perform visual geometric refinement processing on the mapped frame image of the target candidate region in response to the shape confidence level matched by the target candidate region, determining the current pixel size data of the target candidate region in the image coordinate system of the mapped frame image. The data generation module is used to perform multi-frame target association processing on the current pixel size data corresponding to the current camera size data based on historical camera size data, generating current frame smoothed output data of the current camera size data. The data output module is used to output current frame smoothed output data according to the standard uncertainty corresponding to the current frame smoothed output data.

[0018] Another aspect of the present invention provides an electronic device including one or more processors and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to perform the above-described monocular vision-based object measurement method.

[0019] Another aspect of the present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the above-described monocular vision-based object measurement method.

[0020] Another aspect of the present invention provides a computer program product including a computer program that, when executed by a processor, implements the above-described monocular vision-based object measurement method.

[0021] The object measurement method based on monocular vision provided in this invention can at least partially solve at least one of the problems introduced by the lack of target constraints in monocular vision detection in related technologies, such as poor robustness to complex backgrounds and workpiece textures, and frequent fluctuations in measurement frame, center, and size parameters. Therefore, it can achieve at least one of the following technical effects:

[0022] (1) By fusing monocular high-resolution contour information with single-point ToF absolute depth, the actual size and center three-dimensional coordinates of planar workpieces can be measured with low hardware cost.

[0023] (2) Make full use of the corresponding area of ​​the single-point ranging spot (such as the ToF spot), the image center, the target segmentation area, the confidence level and the candidate geometric features to jointly optimize the target and reduce the background contour misselection.

[0024] (3) Through coarse target localization or instance segmentation in complex backgrounds (which can be achieved using YOLOv8), and then through measurement-level boundary refinement using traditional edge, contour, and line / circle fitting.

[0025] (4) Rectangles, rounded rectangles and circles are represented and drawn in different geometric ways, and their length, width, center or radius are calculated respectively.

[0026] (5) Stable measurement is achieved through corner point arrangement matching, center and scale joint association, continuous multi-frame hysteresis switching, missed detection hold, dead zone, amplitude limiting and EMA.

[0027] (6) When the target segmentation region (such as the ROI region) exists, candidate content outside the region is strictly blocked, thereby effectively eliminating the influence of complex background; when deep learning fails, a multi-strategy traditional visual backoff is adopted to balance accuracy and usability.

[0028] (7) Simultaneously output the target pixel size, actual size, center pixel coordinates, center camera coordinates, coordinates of the rectangle center or circle center in the current probe coordinate system, depth, detection source and confidence status.

[0029] Therefore, the object measurement method based on monocular vision described above in the embodiments of the present invention, under the conditions of low-cost monocular camera and single-point ToF, requires fewer hardware components, has lower cost, and requires less installation space compared to traditional binocular or area array depth solutions; compared to pure traditional vision, it is more robust to handling changes in color, texture, shadow, and complex background; compared to traditional solutions that directly use neural network bounding boxes, it can significantly improve contour and dimensional accuracy by utilizing geometric edge refinement; in addition, multi-frame stabilization and hysteresis mechanisms can significantly reduce the jump of the bounding box, center, and size, while maintaining response speed through the latest frame background inference, so as to simultaneously provide pixel quantity, physical quantity, center camera coordinates, and center probe coordinates, which is convenient for quality inspection, workpiece centering, probe relative positioning, and CNC coarse positioning.

[0030] As can be seen, the object measurement method based on monocular vision described in the embodiments of the present invention can perform fast, accurate and stable contour detection of rectangular, rounded rectangle and circular planar workpieces under the conditions of low-cost monocular camera and single-point ToF, convert pixel size into actual physical size, and output the pixel coordinates and camera coordinates of the target center efficiently and accurately at low cost, with higher robustness.

[0031] It should be understood that the above general description and the following specific embodiments are merely exemplary and illustrative, and do not limit the scope of the invention. Attached Figure Description

[0032] The above-described features, other objects, and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0033] Figure 1 The illustrations schematically depict application scenarios of the monocular vision-based object measurement method and the pixel size transformation method, apparatus, device, medium, and program product based on object measurement according to embodiments of the present invention.

[0034] Figure 2A A flowchart illustrating an object measurement method based on monocular vision according to an embodiment of the present invention is shown schematically.

[0035] Figure 2B A flowchart illustrating a pixel size transformation method based on object measurement according to an embodiment of the present invention is shown schematically.

[0036] Figure 3 The diagram illustrates an overall system flowchart of detection and measurement based on monocular vision-based object measurement method and object measurement-based pixel size transformation method according to embodiments of the present invention.

[0037] Figure 4 The diagram illustrates the principle of the conversion relationship between pixel coordinates, camera coordinates and ToF depth in a monocular vision-based object measurement method according to an embodiment of the present invention.

[0038] Figure 5A The schematic diagram illustrates the principle of the geometric parameters and center definition of a rectangular workpiece in an object measurement method based on monocular vision according to an embodiment of the present invention;

[0039] Figure 5B The diagram illustrates the principle of the geometric parameters and center definition of a circular workpiece in a monocular vision-based object measurement method according to an embodiment of the present invention.

[0040] Figure 6The diagram illustrates the principle of multi-frame target association, anomaly suppression, and stable output in the monocular vision-based object measurement method according to an embodiment of the present invention.

[0041] Figure 7A A schematic diagram illustrating the structure of an object measurement device based on monocular vision according to an embodiment of the present invention is shown.

[0042] Figure 7B A schematic diagram illustrating a pixel size transformation device based on object measurement according to an embodiment of the present invention is shown; and

[0043] Figure 8 A block diagram schematically illustrates an electronic device suitable for implementing a monocular vision-based object measurement method and a pixel size transformation method based on object measurement, according to embodiments of the present invention.

[0044] The accompanying drawings mentioned above are part of the specification of embodiments of the present invention, illustrating exemplary embodiments of the invention. The drawings, together with the description in the specification, serve to illustrate the principles of the embodiments of the present invention. It should be understood that the above general description with reference to the drawings and the following detailed description are merely exemplary and illustrative, and do not limit the scope of the invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the spirit of the contents disclosed in the present invention will be clearly explained below with reference to the accompanying drawings and detailed description. After understanding the embodiments of the present invention, any person skilled in the art can make changes and modifications based on the technology taught in the present invention without departing from the spirit and scope of the present invention.

[0046] The illustrative embodiments and descriptions of the present invention are used to explain the invention, but are not intended to limit the invention. Furthermore, elements / components using the same or similar reference numerals in the drawings and embodiments are used to represent the same or similar parts.

[0047] The terms "first," "second," etc., used in this invention do not specifically refer to any order or sequence, nor are they intended to limit the invention; they are merely used to distinguish elements or operations described using the same technical terms.

[0048] The directional terms used in this invention, such as up, down, left, right, front, or back, are merely for reference to the accompanying drawings. Therefore, the directional terms used are for illustrative purposes and not for limiting the scope of this invention.

[0049] The terms “comprising,” “including,” “having,” “containing,” etc., used in this invention are all open-ended terms, meaning that they include but are not limited to.

[0050] The term "and / or" as used in this invention includes any or all combinations of the things mentioned.

[0051] In this invention, "multiple" includes "two" and "more than two"; in this invention, "multiple groups" includes "two groups" and "more than two groups".

[0052] The terms "approximately," "about," etc., used in this invention are intended to modify any quantity or error that may vary slightly, but these slight variations or errors do not change the essence of the quantity or error. Generally, the range of slight variations or errors modified by such terms may be 20% in some embodiments, 10% in some embodiments, 5% in some embodiments, or other values. Those skilled in the art should understand that the aforementioned values ​​can be adjusted according to actual needs and are not limited thereto.

[0053] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0054] When expressions such as "at least one of A, B, and C" are used, they should generally be interpreted in accordance with the meaning commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, systems having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.). When expressions such as "at least one of A, B, or C" are used, they should generally be interpreted in accordance with the meaning commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, or C" should include, but is not limited to, systems having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.). A person skilled in the art should also understand that any conjunction and / or phrase that substantially arbitrarily indicates two or more optional items, whether in the specification, claims, or drawings, should be understood to indicate the possibility of including one of these items, either of these items, or both items. For example, the phrase “A or B” should be understood as including the possibility of “A” or “B”, or “A and B”.

[0055] Among existing visual measurement solutions for workpiece recognition, there are five main traditional approaches: First, a two-dimensional scale measurement scheme based on a monocular camera and a known scale, requiring the scale and workpiece to be on the same plane, and the scale varying with depth. Second, triangulation based on binocular vision, which can recover depth but relies on left-right image matching, and is prone to parallax holes on weakly textured surfaces and reflective metals. Third, point cloud measurement based on area array ToF or structured light, which has high hardware costs and data processing volume. Fourth, contour detection based on traditional Canny, threshold segmentation, and Hough transform, whose parameters are sensitive to environmental changes. Fifth, localization based on the universal YOLO bounding box, where the output box usually cannot accurately coincide with the workpiece's outer contour, and class or candidate box switching may occur in consecutive frames.

[0056] However, the above five traditional schemes still have the following technical defects. First, the traditional low-cost monocular scheme lacks absolute depth consideration, resulting in poor output accuracy and more complex calculations. Second, when there is no spatial constraint between the single-point depth and the image target, the background depth may be used for target conversion. Furthermore, the contour boundary detected by pure deep learning is limited by the network input resolution. Moreover, pure traditional vision is not robust enough to complex backgrounds and workpiece textures. In addition, when there is no target association, anomaly suppression, and hysteresis switching in consecutive frames, the measurement box, center, and size will frequently jump.

[0057] In view of the above-mentioned defects of traditional object measurement technology, the present invention provides an object measurement method and apparatus based on monocular vision as in Embodiment 1, and a pixel size transformation method and apparatus based on object measurement as in Embodiment 2, in order to solve at least one of the above-mentioned technical problems and provide an object measurement process with lower cost, higher efficiency, higher accuracy and better robustness.

[0058] This provides a corresponding object measurement system for the methods mentioned in Embodiments 1 and 2 of the present invention. This system includes at least a monocular camera fixedly mounted on the machine tool frame, a single-point ranging sensor (such as a ToF sensor), a probe that moves with the spindle, a CNC axis position reading interface, a processor, an image acquisition module, a camera calibration module, a target detection module, a geometric refinement module, a dimension calculation module, a timing stabilization module, a camera-probe calibration module, a probe offset compensation module, and a display / output module. The monocular camera and the single-point ranging sensor are in fixed relative positions. Preferably, the optical axis for single-point ranging (such as the ToF optical axis) is parallel to the camera's optical axis, or mapped to a predetermined spot position in the image after extrinsic parameter calibration.

[0059] Furthermore, in response to the aforementioned object measurement system, the following detailed descriptions of the product composition and software modules can be provided: a monocular camera can be used to acquire high-resolution object images; a single-point ranging sensor can be used to acquire the object surface depth, such as a ToF (Time of Flight) sensor; in this embodiment of the invention, the single-point ranging sensor may include laser ranging, ultrasonic ranging, multi-point ToF, structured light, or binocular depth; the planar assumption is maintained while still using single-point depth. The processor and memory can implement specific algorithms such as image preprocessing, YOLOv8 inference, geometric refinement, size calculation, temporal stabilization, and coordinate transformation; YOLOv8 may be a deep learning object detection / instance segmentation network. The calibration components may include a checkerboard grid, camera intrinsic parameter files, camera-probe corresponding point acquisition and rigid transformation calibration tools (such as rotation matrix R and translation vector t), probe mounting offset, and sphere center compensation parameters. The rigid transformation calibration parameters can be solved using methods such as SVD / Kabsch, where SVD stands for Singular Value Decomposition and Kabsch is a least-squares algorithm for finding the optimal rigid transformation between two sets of points. The interaction and output components, in conjunction with the processor and memory, can output data or information such as ROI drawing, parameter configuration, real-time contour overlay, pixel / mm dimensions, center camera coordinates, center probe coordinates, detection source, and status. Furthermore, the machine tool and probe interface can be used to read the probe's touch coordinates and the coordinate system of the currently active workpiece; when necessary, it can read the motion axis coordinates corresponding to the image time for data synchronization and load the calibrated camera-probe rigid transformation (R, t).

[0060] As will be discussed later, in conjunction with, Figure 3 The overall flow of the object measurement method based on monocular vision in Embodiment 1 and the pixel size transformation method based on object measurement in Embodiment 2 of the present invention includes operations S311 & S312-S309.

[0061] For Example 1, image acquisition is performed in operation S311; single-point ranging (e.g., ToF depth) is obtained in operation S312; then camera distortion correction, depth conversion and filtering are performed, and the region of interest (ROI) is determined; in operation S302, candidate targets are obtained through instance segmentation or detection (specifically, YOLOv8 can be used); in operation S303, targets are filtered using ranging spot, image center, candidate region and confidence level; in operation S304, edge and line / circle refinement is performed on candidate contours; in operation S305, after the rectangle or circle judgment is completed, the actual size and camera coordinates of the rectangle center or circle center are calculated based on depth and camera intrinsic parameters; in operation S306, stable results are output through multi-frame association, hysteresis switching, dead zone and exponential smoothing. Regarding Example 2, in operation S307, during the calibration phase, the corresponding point pairs of the camera detection point and the probe touch point are acquired. In operation S308, the rigid transformation (R, t) from the camera coordinate system to the workpiece coordinate system is solved using least squares (specifically, SVD / Kabsch, etc. can be used). In operation S309, the calibrated rigid transformation (R, t) is applied to convert the center camera coordinates to the center coordinates in the workpiece coordinate system and output. Therefore, in combination with the above... Figure 3 The following will provide a detailed technical description of Embodiments 1 and 2, as shown in the figures.

[0062] Figure 1 The illustrations schematically depict application scenarios of the object measurement method based on monocular vision and the pixel size transformation method, apparatus, device, medium, and program products based on object measurement according to embodiments of the present invention.

[0063] like Figure 1 As shown, application scenario 100 according to this embodiment may include terminal devices 101, 102, and 103, network 104, and server 105. Network 104 is used as a medium to provide a communication link between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0064] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0065] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0066] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0067] It should be noted that the method provided in the embodiments of the present invention can generally be executed by server 105. Correspondingly, the apparatus provided in the embodiments of the present invention can generally be located in server 105. The method provided in the embodiments of the present invention can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the apparatus provided in the embodiments of the present invention can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105.

[0068] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0069] Example 1:

[0070] The following will be based on Figure 1 The described scene, through Figures 2A to 7A The object measurement method based on monocular vision of Embodiment 1 is described in detail.

[0071] like Figure 2A As shown, one aspect of an embodiment of the present invention provides an object measurement method based on monocular vision, which includes operations S211 to S214.

[0072] In operation S211, based on at least one frame of monocular visual image of the object to be measured and the effective axial depth of the ranging measurement point, the corresponding target candidate region is determined according to the preset intersection-union confidence judgment condition.

[0073] In operation S212, in response to the shape confidence of the target candidate region matching, visual geometric refinement processing is performed on the mapped frame image of the target candidate region to determine the current pixel size data of the target candidate region in the image coordinate system of the mapped frame image.

[0074] In operation S213, multi-frame target association processing is performed on the current pixel size data corresponding to the current camera size data based on historical camera size data to generate smooth output data of the current frame for the current camera size data; and

[0075] In operation S214, the smoothed output data of the current frame is output according to the standard uncertainty corresponding to the smoothed output data of the current frame.

[0076] The object to be measured can be a target object with a three-dimensional size to be measured, such as a workpiece. A monocular vision image can be generated by preprocessing the visual detection images detected by a monocular camera (such as image distortion correction). Specifically, the monocular camera can generate continuous video of the object to be measured; these videos can be compared with one frame at each time point, and after image preprocessing, a corresponding set of monocular vision image frames is formed.

[0077] The measurement point can be the point of contact of a single-point ranging sensor on the surface of the object to be measured. As mentioned above, the single-point ranging sensor can be a sensor device capable of distance measurement, such as a Time-of-Flight (ToF) ranging sensor, or a ranging sensor based on principles such as laser ranging, ultrasonic ranging, multi-point ToF, structured light, or binocular depth. Therefore, the axial effective depth can be the distance parameter mapped from the actual physical distance of the measurement point obtained by the ranging sensor to the camera coordinate system.

[0078] The preset crossover ratio (CR) confidence threshold can be a preset condition used to determine the CR of candidate regions (such as Regions of Interest) in the same frame of monocular vision image. Specifically, it can select the optimal target candidate region from the candidate regions of the frame image based on the preset CR confidence threshold, as described below. The target candidate region can be an image region, such as a Region of Interest, selected from the image set of a frame of monocular vision image that is closest to the visual features such as shape and material of the object to be measured.

[0079] Shape confidence can be a threshold for the proximity of the target candidate region to the target shape (such as a rectangle or circle) of the object to be measured, used to determine the matching shape of the target candidate region. The mapped frame image can be understood as a mapped image formed by cropping the target candidate region into a candidate box, upsampling it, and mapping it to its original monocular visual image. Visual geometric refinement processing can include image processing procedures such as expanding the local search band for the target candidate region, grayscale conversion and Gaussian filtering, contrast enhancement, and adaptive edge detection. Current pixel size data can include the shape and size (such as length, width, or perimeter) and center coordinates (such as center or center point) of the mapped frame image of the target candidate region in an image coordinate system (such as a pixel coordinate system) at the current time.

[0080] Current camera size data can be the representation of the shape and size of the object to be measured, mapped from the current pixel size data to camera coordinates. Historical camera size data can be the shape, size, and center coordinates of the object in the camera coordinate system for the same object at a historical time relative to the current moment, based on a determined historical frame of monocular visual image. Multi-frame target association processing involves associating the current camera size data with historical camera size data using multi-frame center distance d_c, scale ratio r_s, and shape consistency I_s, followed by exponential smoothing. The current frame smoothed output data can be the exponentially smoothed data output after multi-frame association processing of the current camera size data.

[0081] The standard uncertainty can be the camera coordinate measurement accuracy fusion error value determined by the edge positioning error σ_l, ToF depth noise σ_Z, and focal length calibration error σ_f corresponding to the smooth output data of the current frame. It is used to feed back the output error of the smooth output data of the current frame to ensure that its measurement range is effective.

[0082] Among them, such as Figure 3 As shown, for the image of the object to be measured acquired in operation S311, a preprocessing process such as monocular camera calibration and distortion correction is first performed to generate a monocular vision image.

[0083] In this embodiment of the invention, a pinhole camera model can be used to achieve image acquisition. Therefore, for any spatial point P_c=(X,Y,Z)^T, its coordinates on the ideal normalized image plane can be represented as x=X / Z, y=Y / Z, where the pixel coordinates p=(u,v)^T. Specifically, P_c represents the three-dimensional coordinates (homogeneous coordinates of the spatial point) of the spatial point in the camera coordinate system, denoted as (X,Y,Z)^T, where X, Y, and Z are the components of the point along the three coordinate axes of the camera coordinate system, and the superscript T indicates vector transpose; x and y are the two-dimensional coordinates of the point on the ideal normalized image plane; p=(u,v)^T is its pixel coordinate, where u and v are the column and row coordinates of the pixel, respectively.

[0084] Therefore, the scale factor s of the projection equation for the pinhole camera model can satisfy:

[0085] (1)

[0086] Where s is the scale factor of the pinhole projection equation; [u,v,1]^T are the homogeneous pixel coordinates; K is a 3×3 camera intrinsic parameter matrix, which can specifically satisfy:

[0087] (2)

[0088] in, These are the equivalent focal lengths in pixels along the horizontal and vertical directions of the image, respectively. The pixel coordinates are the main point (the intersection of the optical axis and the image plane), and 0 in the matrix indicates that the focal lengths in the two directions are decoupled.

[0089] in The equivalent focal length is expressed in pixels. The main point is the camera coordinate system with the camera's optical center as the origin and the camera's Z-axis as the optical axis, where the scale factor s = Z. The camera intrinsic parameter matrix K and distortion vector can be obtained using the checkerboard and Zhang Zhengyou calibration methods. The radial-tangential distortion model for image distortion correction can be expressed as:

[0090] (3)

[0091] Where r is the radial distance on the normalized image plane; The radial distortion coefficient is... , denoted as the tangential distortion coefficient.

[0092] Therefore, x_d and y_d, as the normalized image plane coordinates after distortion, can be expressed as:

[0093] (4)

[0094] (5)

[0095] The acquired frame image can first be subjected to anti-distortion mapping based on the camera intrinsic parameter matrix K and distortion vector D, so that the edge, center and size calculation of the subsequent monocular vision image are in a unified ideal pixel coordinate system, thereby completing the distortion correction and other processes of image preprocessing and generating a monocular vision image.

[0096] like Figure 3 As shown, for single-point distance measurement of the object to be measured in operation S312, depth conversion and filtering are first performed to realize depth acquisition, filtering and validity determination, and generate axial effective depth.

[0097] Specifically, for the original single-point ranging distance d_k of the k-th frame for the ranging measurement points on the surface of the object to be measured, samples that are out of range, have insufficient signal strength, or have jump variables exceeding the threshold are first removed. Then, a median filtering process is performed on a sliding window of length 2m+1 to generate the median filtered distance. :

[0098] (6)

[0099] Where k is the frame number; d_k is the original ToF distance reading of the kth frame; m is the length of one side of the sliding window, so the window length is 2m+1; This is the distance after median filtering. Furthermore, by applying exponential filtering to the median-filtered distance, an exponentially filtered distance can be generated. as follows:

[0100] (7)

[0101] in, This represents the output of the exponential filter; β is the exponential filter smoothing coefficient, with a value range of 0 < β ≤ 1. The exponential filter can suppress random noise.

[0102] In single-point ranging, the output is typically the distance along its transmitting-receiving measurement ray, and not necessarily equal to the Z-depth of the camera coordinate system. Let the unit measurement ray in the single-point ranging sensor coordinate system (such as the ToF coordinate system) be r_t. Then, the extrinsic parameters of the ranging sensor (such as ToF) to the camera coordinate system can be expressed as (R_ct, t_ct). Therefore, the ranging point in the camera coordinate system can be represented as:

[0103] P_c = R_ct P_t + t_ct, (8)

[0104] (9)

[0105] Where r_t is the unit measurement ray direction vector in the ranging sensor coordinate system; R_ct is the 3×3 rotation matrix from the ranging sensor coordinate system to the camera coordinate system, and t_ct is the corresponding 3×1 translation vector; P_t is the three-dimensional coordinate of the ranging point (such as the ToF measured point) in the ranging sensor coordinate system, satisfying P_t=d̂_k r_t; P_c is the three-dimensional coordinate of the ranging point in the camera coordinate system; Z_k is the depth of the ranging point along the Z-axis (i.e., the camera optical axis) in the camera coordinate system, corresponding to the axial effective depth; e3^T represents the transpose of the unit vector along the Z-axis; Δz is the axial offset when the ToF is parallel to the camera's optical axis; Z_0 is the default initial depth value.

[0106] in Furthermore, the axial effective depth can only be simplified to: when the single-point ranging ray is parallel to the camera's optical axis and its lateral offset does not affect the plane depth.

[0107] Z_k= +Δz (10)

[0108] For the default scenario, Z_0 can be set to 200 mm and updated after the effective ToF arrives. This description avoids treating the slant distance directly as the axial depth.

[0109] It should be noted that for fixed planar workpieces, the XY coordinates can be directly calculated using the homography matrix from pixels to the worktable coordinates, and then the height can be compensated by Time-of-Flight (ToF). The homography matrix describes the linear mapping relationship between pixel coordinates between two planes. For non-planar workpieces, point clouds can be generated using area depth analysis, and measured using RANSAC, PCA, and 3D directed bounding boxes. RANSAC is the Random Sample Consensus algorithm; PCA is the Principal Component Analysis algorithm.

[0110] Therefore, by utilizing the aforementioned camera distortion correction, depth conversion, and filtering processes, a corresponding monocular vision image and axial effective depth can be generated. This ensures the accuracy of subsequent image processing and guarantees that all related algorithms are implemented based on a normalized plane of the same depth. Consequently, the computational difficulty is greatly reduced, processing costs are lowered, and processing efficiency is improved.

[0111] like Figures 2A-6 As shown, according to an embodiment of the present invention, before operation S211 determines the corresponding target candidate region based on at least one frame of monocular visual image of the object to be measured and the axial effective depth of the ranging measurement point according to a preset intersection-union ratio confidence judgment condition, the method further includes:

[0112] Based on the pre-drawn region, extract multiple reference candidate regions corresponding to the effective axial depth frame by frame from at least one frame of monocular vision image;

[0113] The candidate comprehensive score of each reference candidate region is determined by the center mapping coordinates of the ranging spot of the corresponding ranging point in each reference candidate region in the image coordinate system and the image center coordinates of each reference candidate region.

[0114] Based on the comprehensive candidate score, the first and second candidate boxes are determined from multiple reference candidate regions.

[0115] like Figure 3 As shown, in operation S302, instance segmentation or detection is performed to obtain candidate targets, which are mainly used to realize the joint constraints of subsequent candidate regions, ranging spots and image centers (such as joint constraints of ROI, ToF spots and image centers).

[0116] For candidate region extraction, the reference candidate region can be extracted based on the user's pre-drawn region as the outline of the pre-extraction region, i.e., the extraction of the image candidate box region (referred to as the candidate box). For example, when the user draws the ROI, detection can be run only inside the ROI, and candidates outside the ROI are directly invalidated. When the ROI is not drawn, the preferred points are the ToF spot mapping point q_t=(u_t,v_t) (i.e., the center mapping coordinates of the ranging spot in the image coordinate system) and / or the image center q_0=(W / 2,H / 2) (i.e., the image center coordinates). For the current frame of monocular vision image, the j-th candidate target reference region can be extracted through the pre-drawn region. Here, ROI is the Region of Interest; q_t=(u_t,v_t) is the mapping point coordinate of the ToF spot in the image; q_0=(W / 2,H / 2) is the image center point, W and H are the image width and height (pixels) respectively; j is the candidate target number. Therefore, for a single frame of monocular vision image, multiple reference candidate regions can be generated.

[0117] Furthermore, such as Figure 3 As shown, in operation S303, the target is selected using the ranging spot, image center, candidate region, and confidence level.

[0118] For the j-th reference candidate region, its candidate comprehensive score S_j can be defined as:

[0119] S_j = w_c C_j + w_q max(0,1-d_j / d_max) + w_h H_j + w_a A_j - w_b B_j (11)

[0120] Where S_j is the comprehensive score of the j-th candidate target; C_j is the network confidence of the candidate; d_j is the distance from the candidate center to the preferred point (ToF spot mapping point and / or image center), and d_max is the upper limit of the distance used for normalization; H_j is an indicator of whether the preferred point is located within the candidate contour, used to indicate whether the preferred point is located within the candidate contour; A_j is the area reasonableness score, used to indicate the area reasonableness; B_j is the penalty term for touching the ROI boundary, area abnormality or depth inconsistency; w_c, w_q, w_h, w_a, and w_b are the weights of each item.

[0121] Then, the two candidate boxes with the largest overall candidate score S_j (i.e., the first candidate box and the second candidate box) can be selected from multiple reference candidate regions as valid candidates for the current frame target, thereby reducing the probability of background objects being mistakenly selected.

[0122] like Figures 2A-6As shown, according to an embodiment of the present invention, before operation S211 determines the corresponding target candidate region based on at least one frame of monocular visual image of the object to be measured and the axial effective depth of the ranging measurement point according to a preset intersection-union ratio confidence judgment condition, the method further includes:

[0123] The first candidate box and the second candidate box are scaled by a preset scaling ratio to generate the first scaled box and the second scaled box.

[0124] Output shape confidence and instance segmentation mask based on the first and second scaling boxes.

[0125] Furthermore, post-processing of the first and second candidate boxes can be achieved based on deep learning coarse localization (such as using the YOLOv8 deep learning network).

[0126] Specifically, the preset scaling ratio can be a preset scaling ratio value that scales the candidate region color images of the first and second scaling boxes to a pre-defined ratio value of the segmentation network input size N×N in a letterbox manner while maintaining the aspect ratio. This ratio value can satisfy r=min(N / W,N / H), and the padding amount is Δx=(N-rW) / 2, Δy=(N-rH) / 2. This allows for the further generation of the corresponding first and second scaling boxes. Here, N is the network input side length (N×N); W and H are the image width and height (or length and width) of the corresponding candidate box (such as the ROI region).

[0127] The N×N input segmentation network can output the target shape confidence (shape) of the corresponding scaled bounding box and an optional instance segmentation mask (which can be understood as a coarse extraction of the shape of the object to be measured). The instance segmentation mask can be used to characterize the target contour within the corresponding candidate box and to determine the validity of the candidate box. YOLOv8 can be replaced with other instance segmentation networks, semantic segmentation networks, or Transformer-based target segmentation networks, where Transformer is a neural network architecture based on a self-attention mechanism.

[0128] like Figures 2A-6 As shown, according to an embodiment of the present invention, in operation S211, based on at least one frame of monocular visual image of the object to be measured and the axial effective depth of the ranging measurement point, the corresponding target candidate region is determined according to a preset intersection-union ratio (IUGR) confidence judgment condition, including:

[0129] The validity of the first and second zoomed boxes is determined based on shape confidence and instance segmentation mask.

[0130] Based on the first and second zoomed boxes that have been validated for validity, generate a valid cross-union ratio that meets the preset confidence level criteria.

[0131] Based on the preset cross-union ratio confidence determination condition, when the effective cross-union ratio is greater than or equal to the preset confidence threshold, the first or second zoomed box with greater confidence is determined as the target candidate region.

[0132] The target candidate region is cropped, upsampled, and mapped to the corresponding original frame monocular vision image to generate the mapped frame image.

[0133] like Figure 3 The confidence screening target of operation S303, for the first and second zoomed boxes generated by the network scaling, can first be used to determine the validity of the zoomed boxes based on the corresponding shape confidence and instance segmentation mask. Specifically, the pixel-level evidence of the instance segmentation mask can be used as the basis for judging whether the zoomed box is valid. The pixel-level evidence can include the coverage, probability response, shape consistency and correlation with the effective ranging depth Z of the instance segmentation mask in the candidate zoomed box. Only when the indicators of these evidences meet the corresponding preset conditions is the candidate zoomed box determined to be valid.

[0134] Considering that the effective intersection-union ratio (IUGR) can be used to measure the degree of overlap between the first and second bounding boxes, for the aforementioned valid first and second bounding boxes, a preset IUGR confidence criterion can be used to compare the effective IUGR with a preset confidence threshold to implement non-maximum suppression (NMS) on duplicate candidate bounding boxes with low confidence levels, thereby retaining candidate bounding boxes with higher confidence levels as target candidate regions. Therefore, the preset IUGR confidence criterion can be a preset judgment condition for performing non-maximum suppression on candidate bounding boxes based on the numerical comparison relationship between the effective IUGR and the preset confidence threshold, and it satisfies the following condition: when the effective IUGR of the bounding box to be judged is greater than or equal to the preset confidence threshold, non-maximum suppression is performed on candidate bounding boxes with lower confidence levels, and candidate bounding boxes with higher confidence thresholds are retained.

[0135] The effective intersection-union ratio (IoU) (A,B) of the first zoomed box A and the second zoomed box B can satisfy:

[0136] IoU(A,B)=|A∩B| / |A∪B| (12)

[0137] Where |A∩B| and |A∪B| are the intersection area and union area between the first scaling box A and the second scaling box B, respectively.

[0138] Based on preset intersection-over-union (IoU) confidence criteria and a preset confidence threshold τ_nms, when the effective IoU of the first scaled bounding box A and the second scaled bounding box B is greater than the preset confidence threshold τ_nms, duplicate candidate boxes with high overlap can be suppressed, while those with higher confidence are retained, thus determining the better scaled bounding box as the target candidate region. Therefore, the preset confidence threshold reflects the degree of overlap in shape between the two candidate scaled bounding boxes, and performs non-maximum suppression (NMS) when the effective IoU exceeds the preset confidence threshold, suppressing duplicate candidate boxes with lower confidence and retaining candidate boxes with higher confidence as the target candidate region for suppressing the target result.

[0139] When the candidate coordinates of the target candidate region (such as a rectangle) in the scaled region are mapped back to the original frame monocular vision image, their pixel coordinates mapped back to the original image can be expressed as:

[0140] u=(u_n-Δx) / r (13)

[0141] v=(v_n-Δy) / r (14)

[0142] Where r is the letterbox scaling ratio; Δx and Δy are the horizontal and vertical fill amounts; u_n and v_n are the network output coordinates, and u and v are the coordinates mapped back to the original frame image.

[0143] For the standard YOLOv8 detection head, the output is typically bounding box parameters and scores for each class, without a separate objectness channel. In this case, the candidate confidence is C = max_c P(c|x). For compatible detection heads that include independent object presence probabilities, the formula C = P(object)·max_c P(c|object,x) is used. Here, C is the candidate confidence; P(c|x) is the conditional probability of class c given input x; max_c represents the maximum value among all classes; P(object) is the object presence probability; and τ_nms is a pre-set confidence threshold for the effective intersection-union ratio (IoU) of non-maximum suppression, used to remove duplicate candidates using non-maximum suppression.

[0144] For the YOLOv8 segmentation model, the instance segmentation mask M can be represented by a combination of the prototype feature P_i and the instance mask coefficients a_i as follows:

[0145] M=σ(Σ_i a_iP_i) (15)

[0146] Where P_i is the segmentation prototype feature, a_i is the instance mask coefficient, σ is the Sigmoid function, and M is the synthetic mask.

[0147] Therefore, the previously selected target candidate regions are further cropped, upsampled, and mapped onto the original frame monocular vision image to generate a mapped frame image, completing the image remapping based on confidence-based target selection. Instance segmentation networks (such as YOLOv8) can handle coarse target localization under complex backgrounds, workpiece color variations, and texture interference, instead of directly using low-resolution network boxes or unrefined mask boundaries as the final measurement edges. This greatly improves the accuracy and efficiency of subsequent image processing and significantly enhances the robustness of the processing.

[0148] like Figures 2A-6 As shown, according to an embodiment of the present invention, in operation S212, in response to the shape confidence of the target candidate region matching, visual geometric refinement processing is performed on the mapped frame image of the target candidate region to determine the current pixel size data of the target candidate region in the image coordinate system of the mapped frame image, including:

[0149] In response to the shape confidence, a candidate region expansion local search is performed on the mapped frame image and grayscale conversion is performed to generate a grayscale frame image;

[0150] Gaussian filtering, contrast enhancement, and adaptive edge detection are applied to grayscale frame images to generate refined frame images;

[0151] Determine the current pixel size data of the candidate region bounding box corresponding to the target candidate region on the refined frame image.

[0152] like Figure 3 As shown, in operation S304, edge and line / circle refinement of the candidate contour (i.e., candidate region of the mapped frame image) is performed, which may include visual geometric refinement and backtracking processes. Specifically, a surrounding extended local search band can be performed on the candidate region of the mapped frame image, and then it can be converted into grayscale to generate a grayscale frame image. Gaussian filtering, contrast enhancement (such as CLAHE), and adaptive edge detection (such as adaptive Canny) can be further performed. Therefore, a closing operation can be performed on the edges to connect broken edges, and then the contour can be extracted to generate the refined frame image.

[0153] Here, CLAHE stands for Contrast Limited Adaptive Histogram Equalization; the Canny threshold can be determined by the Otsu threshold T_o, for example, T_l=max(T_min,0.3T_o), T_h=max(T_hmin,0.9T_o). T_o is the Otsu automatic threshold; T_l and T_h are the low and high thresholds of Canny, respectively; T_min and T_hmin are the minimum lower limits of the low and high thresholds; 0.3 and 0.9 are relative coefficients.

[0154] like Figures 2A-6 As shown, according to an embodiment of the present invention, the current pixel size data for determining the candidate region bounding box corresponding to the target candidate region on the refined frame image includes:

[0155] In response to shape confidence, when the target candidate region is rectangular, the four pixel side lengths of the candidate region box, the average pixel length of the opposite side, and the average pixel width of the opposite side are determined as the current pixel size data based on the corner pixel coordinates of the intersection of the adjacent sides of the candidate region box after straight line fitting.

[0156] like Figure 3 , Figure 4 and Figure 5A As shown, in operation S305, the camera coordinates of the center of the rectangle or the center of the circle are calculated based on the depth and camera intrinsic parameters. To do this, the shape of the target candidate region must first be determined.

[0157] As mentioned above, shape confidence can be a threshold for the proximity of the target shape (such as a rectangle or a circle) of the object to be measured to the target candidate region, and is used to determine the matching shape of the target candidate region, such as a rectangle or a circle.

[0158] For a rectangular target candidate region, a line fitting process can be performed on the corresponding candidate box on the refined frame image to generate a candidate region box. The line fitting process can include polygon approximation, minimum area rotating rectangle, and robust line fitting.

[0159] For a candidate region bounding box that has been fitted with straight lines, each side can be represented as a_ix + b_iy + c_i = 0, and the intersection of two adjacent sides satisfies the following in the image coordinate system:

[0160] (16)

[0161] Where a_i and b_i are the normal vector components of the i-th edge of the corresponding candidate region box, and c_i is a constant term; i is the number of the corresponding edge, and the rectangle can satisfy i=1,2,3,4; [u_i,v_i]^T is the two-dimensional pixel coordinate of the intersection point of the i-th edge and the (i+1)-th edge in the image coordinate system; a_{i+1}, b_{i+1}, and c_{i+1} are the corresponding parameters of the adjacent edges.

[0162] It should be noted that rounded rectangles do not necessarily require the contour to have exactly four sharp corners. The overall length, width, and orientation can be obtained from the minimum area of ​​the rotated rectangle that circumscribed the complete outer contour. The rectangularity is defined as R_e = A_c / A_rr, where A_c is the contour area and A_rr is the area of ​​the minimum area circumscribed by the rotated rectangle. Candidates with R_e below a threshold are rejected. Therefore, if the network output (such as YOLO) does not produce valid results, edge detection (Canny) contour, Otsu foreground segmentation, GrabCut foreground / background separation, connected component and contour merging can be used as a fallback.

[0163] Furthermore, the pixel dimensions of the rectangular object to be measured can be determined for the refined candidate region bounding box, where the four ordered corner points after refinement are p_i=(u_i,v_i), i=1,2,3,4 (i.e., the pixel coordinates of the i-th ordered corner point, i.e., the pixel coordinates of the corner point where adjacent sides intersect). At this time, the pixel side length of the candidate region bounding box is:

[0164] (17)

[0165] (18)

[0166] (19)

[0167] Let L_px be the length of the four pixel sides of the candidate rectangular region; ||·||_2 is the Euclidean norm; L_px and W_px are the average pixel length and width of the opposite sides, i.e., the average pixel length L_px and the average pixel width W_px of the opposite sides.

[0168] Therefore, under the condition that the upper surface of the workpiece is approximately located at the same effective depth Z along the same axis, the ordered corner pixel p_i, when back-projected onto the camera coordinates, satisfies:

[0169] (20)

[0170] Where P_i is the three-dimensional coordinate of the i-th corner point projected onto the camera coordinate system, which reflects the joint calibration relationship between the camera coordinate system and the ranging coordinate system; where Z is the depth of the upper surface of the object to be measured in the camera coordinate system; the ranging coordinate system can be the ranging sensor coordinate system.

[0171] like Figures 2A-6 As shown, according to an embodiment of the present invention, after determining the current pixel size data of the candidate region bounding box corresponding to the target candidate region on the refined frame image, the method further includes:

[0172] Based on the effective axial depth in the camera coordinate system, and according to the joint calibration relationship between the camera coordinate system and the ranging coordinate system, the center pixel coordinates of the rectangle corresponding to the current pixel size data are converted into the center camera coordinates of the rectangle of the current camera size data; among them, the corner camera coordinates of the rectangle corresponding to the center camera coordinates are used to generate the actual size of the rectangle.

[0173] like Figure 3 , Figure 4 and Figure 5A As shown, in operation S305, for the case where the target candidate region is determined by the minimum area rotating rectangle and four fitted lines, the center of the rectangle is preferably the geometric center of the rotating rectangle. Specifically, based on the aforementioned corner pixel coordinates p_i=(u_i,v_i) of the intersection of adjacent sides of the rectangular candidate box, i=1,2,3,4, the center pixel coordinates c_r of the corresponding rectangular candidate box corresponding to the current pixel size data are determined as follows:

[0174] c_r=center(minAreaRect({p_i})) (21)

[0175] Here, minAreaRect(·) represents finding the minimum area of ​​the rotated rectangle, and center(·) represents taking the geometric center.

[0176] Furthermore, by back-projecting the i-th corner point to the three-dimensional coordinate P_i of the camera coordinate system based on the joint calibration relationship between the aforementioned camera coordinate system and the ranging coordinate system, the center pixel coordinates c_r of the rectangle can be mapped to the camera coordinate system to generate the center camera coordinates (X_c, Y_c, Z_c) of the rectangle corresponding to the current camera size data of the target candidate region, as described below.

[0177] If the rectangular candidate box satisfies parallelogram constraints (such as opposite sides being equal, opposite angles being equal, adjacent angles being supplementary, diagonals bisecting each other, and central symmetry), the coordinates c_r of the corresponding rectangle's center pixel can be determined using the four-corner point averaging method as follows:

[0178] (twenty two)

[0179] c_r=(u_c,v_c) (23)

[0180] Furthermore, based on the joint calibration relationship between the aforementioned camera coordinate system and the ranging coordinate system, the three-dimensional coordinates P_i of the i-th corner point back-projected into the camera coordinate system can be used to map the center pixel coordinates c_r of the rectangle to the camera coordinate system, generating the center camera coordinates (X_c, Y_c, Z_c) of the rectangle corresponding to the current camera size data of the target candidate region as follows:

[0181] (twenty four)

[0182] Where u_c and v_c are the components of the center pixel coordinates of their rectangles; f x fᵧ are the equivalent focal lengths in pixels along the horizontal and vertical directions of the image, defined by camera intrinsic parameters, respectively; c x cᵧ represents the pixel coordinates of the principal point (intersection of the optical axis and the image plane) defined by the camera intrinsic parameters; Z represents the effective axial depth; X_c, Y_c, and Z_c represent the three-dimensional coordinate components of the rectangle center in the camera coordinate system.

[0183] Based on the pixel coordinates of the i-th rectangular corner point defined in the aforementioned formula (20), the three-dimensional rectangular corner point camera coordinates P_i (corresponding to the camera coordinates of the rectangular center) are back-projected onto the camera coordinate system. This allows us to generate the actual rectangular size of the object to be measured, which includes the actual length of the object. Actual width Actual diagonal length The actual area is A = L·W. The two sets of side lengths can be further denoted as L = max(L,W) and W = min(L,W).

[0184] like Figures 2A-6 As shown, according to an embodiment of the present invention, the current pixel size data for determining the candidate region bounding box corresponding to the target candidate region on the refined frame image includes:

[0185] In response to shape confidence, when the target candidate region is circular, determine the perimeter and area of ​​the candidate region box;

[0186] After performing circle fitting on the candidate region bounding boxes corresponding to the perimeter and area, the center pixel coordinates in the image coordinate system are extracted as the current pixel size data.

[0187] like Figure 3 , Figure 4 and Figure 5B As shown, in operation S305, for a circular target candidate region, the perimeter P and area A_c of the circular candidate box (i.e., the candidate region box) outline can first be determined, and its corresponding roundness can be determined accordingly. Subsequently, when the roundness exceeds a preset roundness threshold and the radius is within a reasonable range, the following circle fitting process can be used: minimum bounding circle, Hough circle, and / or robust least squares fitted circle.

[0188] (25)

[0189] Where P is the perimeter of the contour, A_c is the area of ​​the contour, C_r is the roundness, r is the radius of the fitted circle, ρ is the robust loss function such as Huber, and u_e and v_e are the pixel coordinates of the e-th sampling point on the current circular contour (circular candidate box) to be tested.

[0190] Therefore, by using the above-mentioned circle fitting process, the center pixel coordinates u_c and v_c in the image coordinate system can be extracted for the candidate region box, so as to determine the center pixel coordinates (u_c, v_c) in the process of circular workpiece detection and measurement.

[0191] like Figures 2A-6 As shown, according to an embodiment of the present invention, after determining the current pixel size data of the candidate region bounding box corresponding to the target candidate region on the refined frame image, the method further includes:

[0192] Based on the effective axial depth in the camera coordinate system, and according to the camera intrinsic parameters corresponding to the joint calibration relationship between the camera coordinate system and the ranging coordinate system, the center camera coordinates of the current camera size data are generated by back-projecting the center pixel coordinates to the camera coordinate system; the actual diameter of the circle is generated by back-projecting the diameter endpoints in the horizontal and vertical directions.

[0193] like Figure 3 , Figure 4 and Figure 5B As shown, in operation S305, to avoid Unlike fᵧ, or the systematic error caused by using a single pixel radius when a circle is represented as an ellipse in the pixel domain, the three-dimensional coordinates P_i of the i-th corner point defined by the joint calibration relationship between the aforementioned camera coordinate system and ranging coordinate system are back-projected to the camera coordinate system. In this embodiment of the invention, it is preferable to obtain two pairs of boundary points q_x^-, q_x^+, q_y^-, q_y^+ (i.e., the endpoint of the circle diameter / boundary point q) in mutually orthogonal directions from the candidate region box contour of the refined frame image after refinement, and back-project them to the normalized target plane (i.e., the target depth plane) determined by the axial effective depth Z respectively:

[0194] (26)

[0195] Where Q(q,Z) is the three-dimensional coordinate of pixel q back-projected onto the target depth plane with the corresponding axial effective depth Z in the camera coordinate system; These are the two boundary points in the horizontal direction. The two boundary points are perpendicular to each other; f is the focal length ( (A uniform value can be taken at any time). The pixel coordinates of the principal point (the intersection of the optical axis and the image plane) defined for the camera intrinsic parameters; u and v are the pixel coordinates of the center point u_c and v_c.

[0196] Furthermore, based on the pixel q back-projected to the target depth plane corresponding to the effective axial depth Z in the camera coordinate system according to the aforementioned formula (26), the three-dimensional coordinates Q(q,Z) can be further generated by using the two boundary points q_x^- and q_x^+ in the horizontal direction as the diameter endpoints in the horizontal direction, and the two boundary points q_y^- and q_y^+ in the vertical direction as the diameter endpoints in the vertical direction. Based on this, the actual diameter Φ and actual radius of the circle can be further determined, as follows:

[0197] (27)

[0198] (28)

[0199] Φ=(D_x+D_y) / 2, R=Φ / 2 (29)

[0200] It should be noted that, for circular candidate regions, only when the corrected image, the circular surface, and the imaging plane are approximately parallel, and Only when q is taken as the center pixel coordinates (u_c, v_c), can the approximation R≈r_px·Z / f be used. Wherein, when q is taken as the center pixel coordinates (u_c, v_c), according to the aforementioned Q(q,Z), the center camera coordinates can be expressed as:

[0201] (30)

[0202] Here, the approximation R≈r_px·Z / f holds only when the image is corrected and the focal length is isotropic; R is the radius, r_px is the pixel radius, Z is the effective axial depth, and f is the focal length; P_c is the three-dimensional coordinate of the center of the circle in the camera coordinate system, i.e., the camera coordinate of the center of the circle. The equivalent focal length and principal point are defined for the camera's intrinsic parameters. These intrinsic parameters can be pre-determined based on the joint calibration relationship between the camera coordinate system and the ranging coordinate system.

[0203] like Figures 2A-6 As shown, according to an embodiment of the present invention, in operation S213, multi-frame target association processing is performed on the current pixel size data corresponding to the current camera size data based on historical camera size data to generate current frame smooth output data of the current camera size data, the following is included:

[0204] Based on historical camera size data and current camera size data, the correlation between pixel length and width parameters and center distance, scale ratio and shape consistency of center pixel coordinates is determined to identify smooth output data of the current frame that meets the preset stable output suppression conditions. The preset stable output suppression conditions include dead zone threshold suppression rules, single frame maximum step size suppression rules and exponential smoothing suppression rules.

[0205] like Figure 3 and Figure 6 As shown, in operation S306, a stable result is output through multi-frame association, hysteresis switching, dead zone and exponential smoothing to realize the multi-frame target association processing.

[0206] Compared to the current camera size data obtained from the current frame monocular visual image at the current moment, the historical camera size data can be the stable output data of the previous frame monocular visual image at a previous moment. Specifically, it can include the historical center camera coordinates of the corresponding historical frame image (such as the historical rectangular center camera coordinates or the historical circle center camera coordinates), which can be represented as T_{k-1}, such as the center camera coordinates of the previous frame image. In contrast, the current center camera coordinates of the current camera size data (such as the current rectangular center camera coordinates or the current circle center camera coordinates) can be represented as T_k, such as the center camera coordinates of the current frame image.

[0207] The center distance can be defined as follows based on the center pixel coordinates (such as the center pixel coordinates of the rectangle or the center pixel coordinates of the circle mentioned above):

[0208] (31)

[0209] Where d_c is the center distance between the center of the shape contour of the current frame and the center of the previous frame; u_k, v_k and u_{k-1}, v_{k-1} are the center pixel coordinates of the current frame and the previous frame, respectively.

[0210] Furthermore, the scaling ratio r_s is defined based on the pixel width and height parameters as follows:

[0211] (32)

[0212] Where r_s is the scale ratio; L_{px,k} and W_{px,k} are the pixel length and width of the k-th frame; L_{px,k-1} and W_{px,k-1} are the pixel length and width of the (k-1)-th frame. L_{px,k}, W_{px,k}, L_{px,k-1}, and W_{px,k-1} are the pixel length and width parameters.

[0213] Furthermore, shape consistency I_s can be provided to compare the shape consistency of the candidate region boxes in the previous frame and the candidate region boxes in the current frame to determine whether their shape types are the same or similar.

[0214] The center distance d_c, scale ratio r_s and shape consistency I_s can be used to determine whether the candidate region box of the previous frame and the candidate region box of the current frame are close or identical. When the shape types are the same (the shape consistency I_s is the same), the center distance d_c satisfies d_c<max(d_0,ηs_{k-1}), and the scale ratio r_s satisfies r_min<r_s<r_max, the candidate region box of the previous frame and the candidate region box of the current frame are determined to be the same target. Wherein, s_{k-1} is the size characteristic length of the candidate region box of the previous frame; η is the size correlation coefficient, and r_min and r_max are the upper and lower limits of the scale ratio. For example, η=0.10, r_min=0.80, and r_max=1.25 can be adopted.

[0215] For the rectangular candidate region box of the refined current frame image, cyclic shifting and forward / reverse sequence traversal can be performed on the four corner points based on the three aspects of the center distance, scale ratio and shape consistency described above, and the permutation with the minimum total distance from the previous frame is selected to effectively eliminate corner point number flipping.

[0216] Further, the data of the current camera coordinates corresponding to the current frame can be used as the current observed value x_k, and the dead zone threshold suppression rule, the single-frame maximum step suppression rule and the exponential smoothing suppression rule can be respectively defined through a preset stable output suppression condition, so as to realize three-layer suppression for the preset stabilizer.

[0217] Wherein, the dead zone threshold suppression rule can be defined as follows:

[0218] (33)

[0219] Wherein, δ is the dead zone threshold (pixel); x_k is the current observed value, and x̂_{k-1} is the smoothed value of the previous frame. The dead zone threshold suppression rule is mainly used to determine the pixel change degree between the current observed value and the smoothed value of the previous frame, and when the change is less than the dead zone threshold δ (pixel), the system enters the dead zone.

[0220] In addition, the single-frame maximum step suppression rule can be defined as follows:

[0221] (34)

[0222] Wherein, x'_k is the amplitude-limited intermediate value, and Δ_max is the maximum allowable single-frame step. The single-frame maximum step suppression rule is mainly used to determine the change degree of the maximum allowable single-frame step between the current observed value and the smoothed value of the previous frame, and when the change is greater than the maximum allowable single-frame step Δ_max, it will be amplitude-limited.

[0223] In addition, according to , the exponential smoothing suppression rule can be defined as follows:

[0224] (35)

[0225] Where x_k is the current observation value, This is the smoothing value from the previous frame; This is the smoothed output for the current frame; α is the exponential smoothing coefficient. This exponential smoothing suppression rule is mainly used to exponentially smooth the changes between the current frame observation and the smoothed value of the previous frame, excluding dead zone changes and maximum allowable step size magnitude changes.

[0226] Therefore, based on the aforementioned three aspects of suppression, a hysteresis handover can be implemented for new targets. Specifically, a stable target is replaced only if the candidate center and size are consistent for N consecutive frames, where N is the number of consecutive consistent frames required for hysteresis handover; in this embodiment, N=3. The previous result is maintained for M frames within a short period of missed detection, where M is the number of frames maintained for short-term missed detection; in this embodiment, M=4. This mechanism simultaneously applies to corner points, center, pixel size, actual size, and depth.

[0227] In this embodiment of the invention, exponential smoothing can be implemented using methods such as exponential moving average (EMA). Specifically, EMA can be replaced by Kalman filtering, One-Euro filtering, moving median and / or particle filtering, etc.

[0228] like Figures 2A-6 As shown, according to an embodiment of the present invention, in operation S214, outputting smoothed output data of the current frame based on the standard uncertainty corresponding to the smoothed output data of the current frame, the following is included:

[0229] Based on the effective axial depth, the standard uncertainty is fused from the edge positioning error, ranging depth noise, and focal length calibration error of the current pixel size data of the current frame smooth output data, and outputs the current frame smooth output data.

[0230] like Figure 3 and Figure 6 As shown, in operation S306, error propagation and determination of the effective measurement range can be further performed.

[0231] Wherein, under the conditions of approximately frontal view and the same depth plane (corresponding to the effective axial depth Z), the directional dimension of the candidate region box in the current frame image in this plane direction can be approximately expressed as L = l_px Z / f, where L is the actual dimension in this plane direction, l_px is the corresponding pixel size, Z is the depth, and f is the focal length. When variable correlation is ignored, the standard uncertainty corresponding to this directional dimension satisfies:

[0232] (36)

[0233] Where σ_L is the standard uncertainty of size; σ_l is the standard uncertainty of pixel edge positioning; σ_Z is the standard uncertainty of single-point ranging depth (such as ToF depth); and σ_f is the standard uncertainty of focal length calibration.

[0234] Therefore, the edge positioning error σ_l, ToF depth noise σ_Z, and focal length calibration error σ_f can be used to jointly determine the camera coordinate measurement accuracy in the camera coordinate system. Among these, it is preferable to reduce the error by increasing the target pixel ratio, reducing the ROI, filtering the single-point ranging (ToF) measurement, using rigid checkerboard calibration, and controlling the angle between the camera and the workpiece plane. Therefore, the final output of the smoothed current frame data has a smaller current center camera coordinate error, higher accuracy, and better robustness.

[0235] It should be noted that the probe coordinate output is also affected by calibration rigidity transformation error, probe repeated triggering error, and single-point ranging stability. The calibration transformation should be verified by independent verification points distributed in the workspace, as described in Example 2 below. Single-point ranging cannot recover different depths at various points on a non-planar workpiece; therefore, this embodiment of the invention can be applied to planar or near-planar workpieces.

[0236] Based on the above-described object measurement method based on monocular vision, this invention also provides an object measurement device based on monocular vision. The following will be combined with... Figure 7A The device is described in detail.

[0237] Figure 7A A schematic block diagram of a monocular vision-based object measurement device 710 according to an embodiment of the present invention is shown.

[0238] like Figure 7A As shown, the object measurement device 710 based on monocular vision in this embodiment includes a region determination module 711, a data determination module 712, a data generation module 713, and a data output module 714.

[0239] The region determination module 711 is used to determine the corresponding target candidate region based on at least one frame of monocular visual image of the object to be measured and the axial effective depth of the ranging measurement point, according to a preset intersection-union confidence score. In one embodiment, it can be used to perform the operation S211 described above, which will not be repeated here.

[0240] The data determination module 712 is used to perform visual geometric refinement processing on the mapped frame image of the target candidate region in response to the shape confidence of the target candidate region matching, and to determine the current pixel size data of the target candidate region in the image coordinate system of the mapped frame image. In one embodiment, it can be used to perform the operation S212 described above, which will not be repeated here.

[0241] The data generation module 713 is used to perform multi-frame target association processing on the current pixel size data corresponding to the current camera size data based on historical camera size data, and generate smooth output data of the current frame for the current camera size data. In one embodiment, it can be used to perform the operation S213 described above, which will not be repeated here.

[0242] The data output module 714 is used to output the smoothed output data of the current frame based on the standard uncertainty corresponding to the smoothed output data of the current frame. In one embodiment, it can be used to perform the operation S214 described above, which will not be repeated here.

[0243] According to embodiments of the present invention, any plurality of modules among the region determination module 711, data determination module 712, data generation module 713, and data output module 714 may be combined into one module, or any one of these modules may be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of the present invention, at least one of the region determination module 711, data determination module 712, data generation module 713, and data output module 714 may be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any one of the three implementation methods or a suitable combination of any of them. Alternatively, at least one of the region determination module 711, data determination module 712, data generation module 713, and data output module 714 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0244] Example 2:

[0245] The following will be based on Figure 1 The described scenario, combined with the object measurement method and apparatus based on monocular vision in Embodiment 1 above, is achieved through... Figures 2B to 7B The pixel size transformation method based on object measurement of the disclosed embodiment 2 will be described in detail.

[0246] like Figure 2B As shown, one aspect of an embodiment of the present invention provides a pixel size transformation method based on object measurement, which includes operations S221 to S223.

[0247] In operation S221, based on at least one frame of monocular vision image of the object to be measured, and based on the axial effective depth of the corresponding ranging measurement point, the camera size information of the object is output.

[0248] In operation S222, for the rigid transformation calibration parameters determined by the joint calibration process between the camera coordinate system and the probe coordinate system, a validity determination process for the calibration parameters is performed to generate the target rigid transformation parameters; and

[0249] In operation S223, the camera size information is transformed based on the target rigidity transformation parameters to generate the target size information in the probe coordinate system.

[0250] Based on the monocular vision-based object measurement method and apparatus of Embodiment 1, a transformation from center pixel coordinates to center camera coordinates can be achieved. Combined with multi-frame correlation and uncertainty error propagation processes, a precise and efficient smooth output of the center camera coordinates of the current frame image can be realized, i.e., smooth output data for the current frame. This smooth output data includes camera size information for object measurement, specifically including the center camera coordinates of the corresponding object. Furthermore, based on this, a method based on... Figure 3 The operations S307-S309 shown are used to achieve the dimensional transformation of the corresponding coordinate parameters to the probe coordinate system.

[0251] like Figures 2B-6 As shown, according to an embodiment of the present invention, before performing the validity determination process of the calibration parameters determined by the joint calibration process between the camera coordinate system and the probe coordinate system in operation S222, and generating the target rigid transformation parameters, the method further includes:

[0252] Determine the camera reference coordinates in the camera coordinate system and the probe reference coordinates in the probe coordinate system for the reference entity point of the preset rigid calibration plate, respectively;

[0253] By minimizing the sum of squared residuals of the reference camera coordinates and the reference probe coordinates, the rigid transformation calibration parameters are determined, and the joint calibration process is completed.

[0254] like Figures 2B-6 As shown, according to an embodiment of the present invention, determining the camera reference coordinates of the reference entity point of the preset rigid calibration plate in the camera coordinate system and the probe reference coordinates in the probe coordinate system includes:

[0255] Based on the target center pixel coordinates of the pre-calibrated rigid calibration plate at the pre-detection position, the pre-calibrated camera intrinsic parameters, and the pre-single-point ranging axial depth, determine the camera reference coordinates;

[0256] Offset compensation is performed on the probe center coordinates generated by the probe moving to the target center at the preset detection position and performing touch detection, and the probe reference coordinates are generated.

[0257] like Figure 3 As shown, in operation S307, corresponding point pairs between camera detection points and probe contact points are acquired. Specifically, it is necessary to implement the transformation process from the calibrated camera coordinate system to the probe coordinate system.

[0258] In this embodiment of the invention, the monocular camera can be fixedly mounted on the machine tool frame, and the pose of the camera coordinate system C relative to the machine tool reference coordinate system M remains unchanged. To obtain the coordinates of the visual inspection result in the probe coordinate system P, this embodiment of the invention preferably adopts a calibration method to directly establish a rigid transformation from the camera coordinate system C to the probe coordinate system P, without having to separately solve for the poses of the camera-machine tool externally involved in the probe-machine tool motion and then perform a composite. Here, the probe coordinate system P is taken as the coordinate system to which the coordinates of the probe ball center (or theoretical tool tip) reported by the CNC controller when the probe touches the contact point belong, i.e., the currently active workpiece coordinate system (e.g., G54); this workpiece coordinate system remains fixed during calibration and measurement. If the controller reports the coordinates of the spindle end face, tool holder reference point, or theoretical tool tip, the probe length, installation offset, and ball center position compensation should be completed uniformly before establishing the corresponding points, and repeated compensation should not be performed in subsequent transformations.

[0259] Therefore, for calibration point acquisition and corresponding point establishment, a pre-set rigid calibration plate can be used for the joint calibration process. This pre-set rigid calibration plate has high-contrast circular dots and / or crosshair target centers, whose centers can be stably detected by the camera and safely touched by the probe. Throughout the calibration and measurement process, the camera, single-point ToF sensor, and probe mounting base remain mechanically fixed, and the activated workpiece coordinate system remains unchanged. The target center and / or crosshair target center are calibration feature points that can be simultaneously detected by the camera and touched by the probe.

[0260] Arrange no fewer than three, preferably nine to sixteen, non-collinear calibration points within the effective working area of ​​the machine tool. For example, a minimum layout consisting of the four corners, the midpoints of the four sides, and the center. The calibration points cover different X and Y positions, and preferably include at least two Z-heights. The larger the covered area and the more uniform the distribution, the more stable the resulting transformation. Corresponding points must be non-collinear. Where N is the number of corresponding point groups; X, Y, and Z are the machine tool coordinate axis directions.

[0261] The preset detection position can be a fixed position of a preset rigid calibration plate, where camera detection and probe touch detection are performed respectively. Therefore, the preset rigid calibration plate can first be set at the preset detection position (such as the i-th position). The target center pixel (u_i, v_i) detected by the camera detection vision system is used as the target center pixel coordinates. Combined with the calibrated camera intrinsic parameters (f_x, f_y, c_x, c_y) and the pre-single-point ranging axial depth Z_i measured by a single-point ranging sensor (such as a ToF sensor), the three-dimensional coordinates of the entity point in the camera coordinate system C are obtained as follows:

[0262] (37)

[0263] in, Let be the three-dimensional coordinates of the i-th calibration point in the camera coordinate system C, i.e., the camera reference coordinates; u_i and v_i are the pixel coordinates of the target center; Z_i is the single-point ranging depth of this point; f_x, f_y, c_x, and c_y are the camera intrinsic parameters (equivalent focal length and principal point).

[0264] Furthermore, the preset probe corresponding to the probe coordinate system is moved to the target center of the preset rigid calibration plate at the same preset detection position for safety contact testing. The probe sphere center coordinates displayed on the controller of the preset probe are read and recorded after compensation for probe length and XY probe offset. That is, the coordinates of the same entity point in the probe coordinate system P. Therefore, p_i^P is the coordinate of the probe sphere center after compensation for probe length and XY probe offset, i.e., the probe reference coordinates.

[0265] Therefore, for each calibration point, a set of corresponding points between camera coordinates and probe coordinates can be established based on the above method. The corresponding relationships are recorded when the target remains stationary. Here, the probe reference coordinates p_i^P represent the three-dimensional coordinates of the same calibration point in the probe coordinate system P. When different preset detection positions are moved, the target moves, and the above process is repeated until all calibration points are covered. This establishes different combinations of camera reference coordinates and probe reference coordinates. N represents the number of point groups. The recording order does not affect the calibration results. To ensure the calibration is solvable, the number of corresponding point groups N ≥ 3 and they should not be collinear in space; preferably N ≥ 9 to suppress noise. All camera reference coordinates and probe reference coordinates should use the same unit of length (millimeters). For example, during calibration, all translations, ToF depths, and probe ball radii should use the same unit of length (millimeters), and machine coordinates (e.g., G53) and workpiece coordinates (e.g., G54) should not be mixed in the same calibration. Here, G53 is the machine coordinate system (corresponding to the probe coordinate system), and G54 is the workpiece coordinate system.

[0266] like Figures 2B-6As shown, according to an embodiment of the present invention, the process of determining the rigid transformation calibration parameters by minimizing the sum of squared residuals of the reference camera coordinates and the reference probe coordinates, and completing the joint calibration process, includes:

[0267] Based on the preset rigid transformation relationship and preset residual sum of squares between the probe reference coordinates and the corresponding camera reference coordinates, the centroid of the probe coordinates corresponding to the probe reference coordinates and the centroid of the camera coordinates corresponding to the camera reference coordinates are determined respectively. Then, the centroids of the probe coordinates and the centroids of the camera coordinates are decentered to generate probe coordinate points and camera coordinate points respectively. A covariance matrix is ​​generated based on the probe coordinate points and camera coordinate points. Singular value decomposition is performed on the covariance matrix. By using a preset corrected diagonal matrix to exclude mirror reflections, an orthogonal transformation matrix and a corresponding translation vector are generated as rigid transformation calibration parameters.

[0268] like Figure 3 As shown, in operation S308, the least squares solution is performed to solve the rigid transformation (R, t) from the camera coordinate system to the workpiece coordinate system, thereby achieving the calibration solution of the rigid transformation.

[0269] First, regarding the aforementioned N sets of corresponding points Based on probe reference coordinates and corresponding camera reference coordinates Pre-defined rigid transformation relationship between them The preset sum of squared residuals between the predefined probe reference coordinates and the corresponding camera reference coordinates is:

[0270] (38)

[0271] Where R is a 3×3 rotation matrix; t is a 3×1 translation vector. st RᵀR = I; det(R) = 1, R^T represents the transpose of R, det(R) represents the determinant of R, and I is the identity matrix. Furthermore, p^P is the probe reference coordinate, and p^c is the camera reference coordinate.

[0272] Based on the aforementioned preset sum of squared residuals, the centroids of the two sets of points are first determined as follows:

[0273] (39)

[0274] (40)

[0275] Where μ^c is the centroid of the camera coordinate point group, that is, the centroid of the camera coordinates corresponding to the camera reference coordinates; μ^P is the centroid of the probe coordinate point group, that is, the centroid of the probe coordinates corresponding to the probe reference coordinates.

[0276] The following decentralization process is then applied to each point:

[0277] (41)

[0278] (42)

[0279] Where q_i^c is the decentralized camera coordinates, i.e., the camera coordinate point, and q_i^P is the decentralized probe coordinates, i.e., the probe coordinate point.

[0280] Furthermore, a 3×3 covariance matrix can be constructed:

[0281] (43)

[0282] Where H is a 3×3 covariance matrix, and (q_i^c)^T is the transpose of the decentralized camera coordinates.

[0283] Next, singular value decomposition is performed on the 3×3 covariance matrix H:

[0284] (44)

[0285] Where U and V are orthogonal matrices, Σ is a diagonal singular value matrix, and V^T is the transpose of V.

[0286] Finally, the pre-set correction diagonal matrix is ​​used. And when det(U Vᵀ) = −1, M = diag(1, 1, −1) to exclude mirror reflection and ensure that R is an orthogonal rotation matrix (det(R) = 1). Here, M is a corrected diagonal matrix, diag(·) represents a diagonal matrix with the elements inside the parentheses as the diagonal; det(UV^T) is used to determine whether mirror reflection occurs; μ^c and μ^P are as described above.

[0287] At this point, calibration involves finding the orthogonal transformation matrix R that minimizes the sum of squared residuals and the corresponding translation vector t.

[0288] The 3×3 rotation matrix (i.e., the orthogonal transformation matrix) can be represented as:

[0289] (45)

[0290] Furthermore, the translation vector can be determined from the centroid as the corresponding 3×1 translation vector:

[0291] (46)

[0292] Therefore, the aforementioned generated orthogonal transformation matrix and corresponding translation vector serve as the rigid transformation calibration parameters in this embodiment of the invention.

[0293] It should be noted that when the corresponding points are approximately collinear or their distribution is too concentrated, the covariance matrix H approaches singularity, and the resulting transformation is unreliable for the uncovered directions. In this case, calibration should be performed again after covering calibration points of different X, Y, and Z directions. Furthermore, the calibration solution for the rigid transformation in this embodiment of the invention can be implemented using a closed-form solution scheme such as SVD / Kabsch least squares, which will not be elaborated upon here.

[0294] like Figures 2B-6 As shown, according to an embodiment of the present invention, in operation S222, the rigid transformation calibration parameters determined by the joint calibration process between the camera coordinate system and the probe coordinate system are used to perform a validity determination process for the calibration parameters, generating target rigid transformation parameters, including:

[0295] Based on the predicted probe coordinates corresponding to the rigid transformation calibration parameters, determine the probe coordinate error between them and the measured probe coordinates;

[0296] The root mean square error and maximum calibration error are determined using the probe coordinate error.

[0297] The validity determination process is performed based on the root mean square error and the maximum calibration error, and the target rigid transformation parameters are generated.

[0298] like Figure 3 As shown, for the rigid transformation process of operation S308, the calibration accuracy of the rigid transformation calibration parameters after calibration can be further evaluated to realize the effectiveness judgment process.

[0299] First, after calibration, based on the preset rigid transformation relationship between the probe reference coordinates pᴾ and the corresponding camera reference coordinates pᶜ, the probe coordinates of each calibration point are predicted using the calibrated rigid transformation calibration parameters (R, t):

[0300] (47)

[0301] Where p̂_i^P is the probe coordinate predicted by the rigid transformation calibration parameter (R,t) for the i-th calibration point, i.e., the predicted probe coordinate.

[0302] Furthermore, the error between the predicted probe coordinates and the measured probe coordinates p_iᴾ can be expressed as follows:

[0303] (48)

[0304] Where p_i^P is the actual measured probe coordinate; e_i is the prediction error of that point, i.e., the probe coordinate error;

[0305] Based on the aforementioned probe coordinate error, the root mean square error and maximum calibration error are determined as follows:

[0306] (49)

[0307] e_max = max_i e_i (50)

[0308] Where RMS is the root mean square error and e_max is the maximum calibration error. The root mean square error and maximum calibration error can be used to evaluate the quality of the aforementioned calibration process. Preferably, RMS ≤ 0.5 mm and e_max ≤ 1 mm; if the error exceeds the threshold, points with questionable errors should be removed and re-acquired, or the camera, ToF sensor, and probe should be checked for displacement. Before being used for automatic probe movement, the transformation must be verified using several independent verification points not involved in the calibration to confirm that their RMS and maximum error meet the requirements, and the calibration results (R, t, RMS, e_max) should be recorded for traceability. If the calibration RMS or verification point error exceeds the set threshold, the system should prohibit the output of automatic positioning results and prompt for recalibration.

[0309] After the validity determination process, the obtained target rigid transformation parameters (R, t) can be calibrated to be constant values ​​under the premise that the camera, single-point ToF, and probe mounting base remain fixed, and do not change with the random bed axis position q. Compared with the composite transformation method that requires real-time reading of axis position, the runtime calculation is simpler and more robust. In particular, since (R, t) is constant, the probe coordinates of any detection center can be obtained by only one matrix-vector multiplication and addition during runtime, which is suitable for embedded or real-time processing.

[0310] It should be noted that, in this embodiment of the invention, for the above calibration process and validity determination process, it is preferable to persistently save the calibration results and load them when the software starts; recalibration is required after changing the workpiece coordinate system or changing the probe / stylus. To suppress single-point noise, the camera coordinates of the corresponding point should preferably be the median of the ToF depth over multiple frames and the stable detection center; probe touch detection should preferably be performed in the same approach direction. If checkerboard corner points are used as targets during calibration, (u_i, v_i) can be directly obtained from camera corner point detection; if a dedicated target ball is used, the center pixel can be obtained from circle / cross detection. For a significantly tilted workpiece plane, the depth Z_i at the target center should be converted to the depth at the target center using the plane equation before participating in the calibration. Calculations are performed to ensure that corresponding points are coplanar. The above calibration process, together with the above rigid transformation solution, constitutes a complete mapping from the camera coordinate system to the probe coordinate system. Subsequently, the target rigid transformation parameters are applied to the dimensional transformation process between the center of the rectangle and the center of the circle, as detailed below. Figure 3 The operation shown is S309.

[0311] like Figures 2B-6As shown, according to an embodiment of the present invention, in operation S223, which transforms the camera size information based on the target rigidity transformation parameters to generate target size information in the probe coordinate system, the following steps are included:

[0312] In response to the shape confidence of the object to be measured, when the target candidate region of the monocular vision image of the object to be measured is rectangular, the camera coordinates of the rectangular center of the camera size information are transformed by the target rigidity transformation parameters to generate the rectangular center probe coordinates of the target size information.

[0313] like Figure 3 As shown, in operation S309, the calibrated rigid transformation (R, t) is applied to convert the center camera coordinates into the center coordinates in the workpiece coordinate system (probe coordinate system) and output them.

[0314] As described in Embodiment 1 above, when the candidate region bounding box of the object to be measured is determined to be rectangular based on the shape confidence level of the object to be measured, the rectangular center camera coordinates output after distortion correction and geometric refinement as mentioned in Embodiment 1 can be converted into rectangular center probe coordinates in the probe coordinate system using the aforementioned target rigidity transformation parameters, thus completing the size transformation process of this embodiment of the invention. The rectangular center probe coordinates can be understood as the position of the center of the rectangle in the probe coordinate system.

[0315] Specifically, as described in Example 1 above, the four ordered corner points of the rectangular candidate region bounding box after distortion correction and geometric refinement can be represented as p_i=(u_i, v_i), i=1,2,3,4. Therefore, the geometric center of the smallest area rotated rectangle is preferably taken as the center pixel coordinate of the rectangle as follows:

[0316] (51)

[0317] Where p_i is the pixel coordinate of the i-th ordered corner point of the rectangle, i=1,2,3,4; (u_r,v_r) is the pixel coordinate of the center of the rectangle; minAreaRect(·) means to find the minimum area of ​​the rotated rectangle; center(·) means to take the geometric center.

[0318] When the four corner points satisfy the parallelogram constraint, the following can also be used:

[0319] (52)

[0320] (53)

[0321] Furthermore, let Z_r be the depth of the rectangular center camera obtained after extrinsic correction by a single-point ranging ray (such as a ToF ray), which is the aforementioned effective axial depth. Specifically, when the upper surface of the workpiece is approximately parallel to the camera image plane, the Z-depth of the ranging measurement point (ToF measurement point) within the same plane can be used as Z_r; when the workpiece plane has a significant inclination, the depth of the ToF measurement point should first be converted to the rectangular center depth based on the known or fitted plane equation.

[0322] Therefore, for the aforementioned center pixel coordinates of the rectangle, the corresponding camera coordinates of the rectangle's center are:

[0323] (54)

[0324] Where p_r^c is the camera coordinates of the center of the rectangle, corresponding to the camera coordinates (X_c, Y_c, Z_c) of the center of the rectangle as determined by formula (24) in Example 1; Z_r is the depth at the center of the rectangle, i.e. the aforementioned effective axial depth; f_x, f_y, c_x, and c_y are the camera intrinsic parameters corresponding to the distortion correction image (the aforementioned monocular vision image), which will not be elaborated in detail.

[0325] Based on the joint calibration relationship between the camera coordinate system and the probe coordinate system, as described above... Figure 3 The target rigidity transformation parameters (R, t) calibrated by operations S307 and S308, and the position of the rectangular center camera coordinate point in the probe coordinate system, can be expressed as:

[0326] (55)

[0327] Where p_r^P represents the three-dimensional coordinates of the rectangular center camera in the probe coordinate system P, i.e., the rectangular center probe coordinates. The rectangular center probe coordinates can be represented by [X_r^P, Y_r^P, Z_r^P]^T, which describes its three components along the X, Y, and Z axes of the probe coordinate system, and is denoted as... In other words, these three components represent the coordinates of the rectangular center along the X, Y, and Z axes in the probe coordinate system P (i.e., the current workpiece coordinate system). Therefore, the aforementioned rectangular center probe coordinates can be directly used as the target coordinates that the probe (tool tip) should move to, for workpiece centering, probe relative positioning, and CNC coarse positioning. Since the target rigidity transformation parameter (R, t) is a constant, the rectangular center probe coordinates p_rᴾ are independent of the machine tool axis position q, depending only on the detection center and effective axial depth of the current frame, thus eliminating the need to read real-time axis coordinates during runtime.

[0328] like Figures 2B-6As shown, according to an embodiment of the present invention, in operation S223, which transforms the camera size information based on the target rigidity transformation parameters to generate target size information in the probe coordinate system, the following steps are included:

[0329] In response to the shape confidence of the object to be measured, when the target candidate region of the monocular vision image of the object to be measured is circular, the center camera coordinates of the camera size information are transformed by the target rigidity transformation parameters to generate the center probe coordinates of the target size information.

[0330] like Figure 3 As shown, the size transformation process in operation S309 can also be used for size transformation of camera coordinates when the candidate region box is circular, so as to determine the position of the center of the circle in the probe coordinate system.

[0331] As described in Embodiment 1 above, when the candidate region bounding box of the object to be measured is determined to be circular based on the shape confidence, the center camera coordinates output after distortion correction and geometric refinement as mentioned in Embodiment 1 above can be converted into the center probe coordinates in the probe coordinate system through the above-mentioned target rigidity transformation parameters, thereby completing the size transformation process of this embodiment of the invention.

[0332] Specifically, as mentioned in Example 1 above, the center pixel coordinates of the circular candidate region bounding box after distortion correction and circular contour refinement are p_o=(u_o, v_o). When the upper surface of the circular workpiece is approximately parallel to the camera image plane, the axial effective depth of the single-point ranging measurement point in the same plane can be used as the center depth Z_o; when the workpiece plane has a significant tilt, the center depth Z_o at the center should be calculated first according to the plane equation.

[0333] Therefore, for the aforementioned center pixel coordinates, the corresponding center camera coordinates are:

[0334] (56)

[0335] Where p_o^c is the coordinate of the center camera, corresponding to the center camera coordinates determined by formula (30) in Example 1. Where the center pixel coordinates q=(u_c,v_c); Z_o is the depth at the center, i.e., the effective axial depth. f_x, f_y, c_x, and c_y are the camera intrinsic parameters corresponding to the distortion-corrected image (the aforementioned monocular vision image), which will not be elaborated further.

[0336] Therefore, based on the joint calibration relationship between the camera coordinate system and the probe coordinate system, and according to the aforementioned... Figure 3 The target rigid transformation parameters (R, t) calibrated for the same purpose in operations S307 and S308, and the position of the center camera coordinates in the probe coordinate system, can be expressed as:

[0337] (57)

[0338] Where p_o^P represents the three-dimensional coordinates of the center of the circle in the probe coordinate system P, i.e., the probe coordinates of the center. The probe coordinates of the center can be represented by [X_o^P, Y_o^P, Z_o^P]^T, which describes its three components along the X, Y, and Z axes of the P system, and is denoted as... In other words, this vector represents the three-dimensional coordinates of the center of the circle in the probe coordinate system P (the current workpiece coordinate system).

[0339] It should be noted that the center of the rectangle and the center of the circle use the same target rigid transformation parameters (R, t) for calibration transformation. The only difference is the way the center pixel is obtained. The output coordinates of both are located in the same probe / workpiece coordinate system and can be directly used for positioning and centering.

[0340] In summary, after completing the dimensional transformation from the center camera coordinates to the probe coordinate system, the corresponding probe coordinates can be synchronized and calibrated to ensure successful measurement in each process. The calibrated target rigidity transformation parameters (R, t) are constant; therefore, the rectangular center probe coordinates are output each time. Or the coordinates of the center probe At this time, only the effective depth of single-point ranging corresponding to the image exposure time and the calibrated target rigidity transformation parameters (R, t) need to be obtained, without needing to read the real-time machine tool axis position q. Preferably, timestamps are added to the image and ToF data, and when the time difference between the two exceeds a set threshold, no positioning result will be output.

[0341] In this embodiment of the invention, for calibration maintenance, recalibration is typically performed under any of the following circumstances: moving the camera, changing its focal length / magnification, moving a single-point ToF sensor, changing the probe / stylus geometry without updating its offset, changing the origin of the currently active workpiece coordinate system, or a collision occurs. Fixed top-view cameras typically only require periodic verification; if a camera that moves with the spindle is used, a more complete hand-eye calibration process is required when the spindle attitude changes. Hand-eye calibration refers to the calibration process that solves for the pose relationship between the camera and the moving end effector.

[0342] Furthermore, in this embodiment of the invention, all translation vectors, ToF depths, probe lengths, and sphere radii should use the same unit of length and should not be mixed with the direction of the inverse matrix of the transformation. Calibration quality should still be confirmed using independent verification points distributed throughout the workspace; if the root mean square error (RMS), probe repetitive triggering error, ToF stability, or time synchronization error exceeds a set threshold, recalibration should be performed and automatic probe approach to the workpiece should be prohibited.

[0343] Based on the above-described pixel size transformation method based on object measurement, this invention also provides a pixel size transformation device based on object measurement. The following will be combined with... Figure 7B The device is described in detail.

[0344] Figure 7B A schematic block diagram of a pixel size transformation device 720 based on object measurement according to an embodiment of the present invention is shown.

[0345] like Figure 7B As shown, the pixel size transformation device 720 based on object measurement in this embodiment includes an information output module 721, a parameter generation module 722, and a size transformation module 723.

[0346] The information output module 721 is used to output camera size information of the object based on at least one frame of monocular vision image of the object to be measured, and based on the axial effective depth of the corresponding ranging measurement point. In one embodiment, it can be used to perform the operation S221 described above, which will not be repeated here.

[0347] The parameter generation module 722 is used to perform a validity determination process for the rigid transformation calibration parameters determined by the joint calibration process between the camera coordinate system and the probe coordinate system, and to generate target rigid transformation parameters. In one embodiment, it can be used to perform the operation S222 described above, which will not be repeated here.

[0348] The size transformation module 723 is used to transform the camera size information based on the target rigidity transformation parameters to generate target size information in the probe coordinate system. In one embodiment, it can be used to perform the operation S223 described above, which will not be repeated here.

[0349] According to embodiments of the present invention, any plurality of modules among the information output module 721, parameter generation module 722, and size transformation module 723 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules can be combined with at least a portion of the functionality of other modules and implemented in one module. According to embodiments of the present invention, at least one of the information output module 721, parameter generation module 722, and size transformation module 723 can be at least partially implemented as a hardware circuit, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented by any other reasonable means of integrating or packaging the circuit, or implemented in any one of the three methods of software, hardware, and firmware, or in a suitable combination of any of these. Alternatively, at least one of the information output module 721, parameter generation module 722, and size transformation module 723 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0350] like Figures 2A-7B As shown, in conjunction with the aforementioned Embodiment 1 and Embodiment 2, the following specific implementation examples for monocular measurement of workpieces are further provided for detailed explanation:

[0351] A monocular camera with an image resolution of approximately 1920×1080 (about 2 megapixels), a single-point ToF sensor with an effective range covering the workpiece distance, a wireless probe that moves with the spindle, and a YOLOv8 segmentation model were used. The camera was fixedly mounted on the machine tool frame, and at least 15 images of different postures were acquired using a 9×6 checkerboard grid to complete the intrinsic parameter calibration. Subsequently, nine to sixteen non-collinear calibration points (including the four corners, the midpoints of the four sides, and the center, covering different X, Y, and at least two Z heights) were arranged in the workspace. For each calibration point, the target center pixel was detected by the vision system, and the camera coordinates were obtained by combining the intrinsic parameters with the ToF depth. Then, the probe safely touched the same target center, and the probe coordinates were obtained by compensating for the probe length and probe offset, forming corresponding point pairs. The rigid transformation (R, t) from the camera to the workpiece coordinate system was solved using SVD / Kabsch least squares, requiring a root mean square error (RMS) ≤ 0.5 mm and a maximum calibration error (e_max) ≤ 1 mm, and was verified by independent verification points. The workpiece is measured at a depth of approximately 200 mm. The camera and ToF sensor are fixedly mounted so that the ToF spot falls in the neighborhood of the image center.

[0352] Each analysis cycle retrieves the latest frame from the acquisition buffer; older frames are not queued. YOLOv8 first outputs candidate masks on the colored ROI; if the confidence level is not lower than the threshold, grayscale enhancement, Canny, and contour fitting are performed near the mask edges; if the network has no valid candidates, it proceeds to traditional visual backoff. Rectangular results are displayed as rotated quadrilaterals, and circular results are displayed as circles. ToF depth is used for size conversion after median and exponential filtering. A continuous frame stabilizer performs target association on the candidates, switching targets only after three frames of confirmation; if a target is missed within four frames, the previous result is retained.

[0353] It should be noted that the measurement results are displayed above the image. The main output items involved in the measurement results include: target pixel size, actual target size, center pixel coordinates, center camera coordinates, center probe coordinates, depth, detector, and confidence.

[0354] For a rectangle, the center pixel (u_r, v_r) is obtained by refining the four corner points, and the camera coordinates of the rectangle's center are obtained by back projection. Then, the coordinates of the rectangular center probe are obtained by applying the calibrated rigid transformation. For a circle, the center pixel coordinates (u_o, v_o) are obtained by refining the center, and the camera coordinates of the center are obtained by back projection. Then, the coordinates of the center probe are obtained by applying the same calibration transformation. Since the target rigidity transformation (R, t) is constant, this transformation does not require reading the real-time machine tool axis position, but only the ToF depth corresponding to the image time.

[0355] Therefore, the above specific implementation examples can achieve at least the following technical effects:

[0356] 1. By fusing monocular high-resolution contour information with single-point ToF absolute depth, the actual size and center three-dimensional coordinates of planar workpieces can be measured with low hardware cost.

[0357] 2. Joint target optimization is performed using the corresponding region of the ToF spot, image center, ROI, confidence level, and candidate geometric features to reduce background contour misselection.

[0358] 3. Use YOLOv8 to perform coarse target localization or instance segmentation in complex backgrounds, and then use traditional edge, contour, and line / circle fitting for measurement-level boundary refinement.

[0359] 4. Rectangles, rounded rectangles, and circles are represented and drawn using different geometric methods, and their length, width, center or radius are calculated separately.

[0360] 5. Stable measurement is achieved through corner point arrangement matching, joint correlation between center and scale, continuous multi-frame hysteresis switching, missed detection hold, dead zone, amplitude limiting and EMA.

[0361] 6. Strictly block candidates outside the ROI when it exists; adopt a multi-strategy traditional visual backoff when deep learning fails, balancing accuracy and usability.

[0362] 7. Simultaneously output the target pixel size, actual size, center pixel coordinates, center camera coordinates, coordinates of the rectangle center or circle center in the current probe coordinate system, depth, detection source, and confidence status.

[0363] 8. For fixed cameras and probes that move with the spindle, a calibration method is used to directly establish a rigid transformation (R, t) from the camera coordinate system to the workpiece coordinate system (the coordinate system to which the probe touch coordinates belong), so that the position of the visual inspection center can be directly represented as the three-dimensional coordinates in the workpiece coordinate system, which can be used as the target positioning coordinates of the probe (tool tip).

[0364] In summary, Embodiments 1 and 2 of the present invention can achieve at least one of the following technical effects:

[0365] 1. Compared to binocular or area array depth solutions, it requires fewer hardware components, has lower costs, and requires less installation space.

[0366] 2. Compared to purely traditional vision, it is more robust to changes in color, texture, shadow, and complex backgrounds.

[0367] 3. Compared to directly using neural network bounding boxes, geometric edge refinement improves contour and dimensional accuracy.

[0368] 4. Multi-frame stabilization and hysteresis mechanisms reduce fluctuations in borders, center, and size, while maintaining responsiveness through background inference of the latest frame.

[0369] 5. Simultaneously provides pixel quantity, physical quantity, center camera coordinates and center probe coordinates, which facilitates quality inspection, workpiece centering, probe relative positioning and CNC coarse positioning.

[0370] Figure 8 A block diagram schematically illustrates an electronic device suitable for implementing a monocular vision-based object measurement method and a pixel size transformation method based on object measurement, according to embodiments of the present invention.

[0371] The electronic device provided in the embodiments of the present invention includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to perform the above-described object measurement method based on monocular vision and the pixel size transformation method based on object measurement.

[0372] like Figure 8 As shown, an electronic device 800 according to an embodiment of the present invention includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage portion 808 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0373] RAM 803 stores various programs and data required for the operation of electronic device 800. Processor 801, ROM 802, and RAM 803 are interconnected via bus 804. Processor 801 executes various operations of the method flow according to embodiments of the present invention by executing programs in ROM 802 and / or RAM 803. It should be noted that the programs may also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 may also execute various operations of the method flow according to embodiments of the present invention by executing programs stored in said one or more memories.

[0374] According to an embodiment of the present invention, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to a bus 804. The electronic device 800 may also include one or more of the following components connected to the I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 810 as needed so that computer programs read from it can be installed into the storage section 808 as needed.

[0375] The present invention also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the above-described object measurement method based on monocular vision and the pixel size transformation method based on object measurement.

[0376] The computer-readable storage medium may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0377] According to embodiments of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of the present invention, a computer-readable storage medium may include ROM 802 and / or RAM 803 and / or one or more memories other than ROM 802 and RAM 803 described above.

[0378] Embodiments of the present invention also include a computer program product comprising a computer program that, when executed by a processor, implements the above-described monocular vision-based object measurement method and the object measurement-based pixel size transformation method.

[0379] The computer program includes program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the methods provided in the embodiments of the present invention.

[0380] When the computer program is executed by the processor 801, it performs the functions defined in the system / apparatus of this invention. According to embodiments of the invention, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0381] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 809, and / or installed from a removable medium 811. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0382] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by processor 801, it performs the functions defined in the system of this embodiment of the invention. According to embodiments of the invention, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0383] According to embodiments of the present invention, program code for executing the computer programs provided in the embodiments of the present invention can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0384] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0385] Furthermore, all actions involving the acquisition of information, signals, or data in this invention are carried out in compliance with the relevant data protection laws, regulations, and policies of the country where the invention is located, and with the authorization granted by the owner of the corresponding device.

[0386] Those skilled in the art will understand that the features described in the various embodiments and / or claims of the present invention can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments and / or claims of the present invention can be combined or combined in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or combinations fall within the scope of the present invention.

[0387] The embodiments of the present invention have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of the invention. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of the invention is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the invention, and all such substitutions and modifications should fall within the scope of the invention.

Claims

1. A method for measuring objects based on monocular vision, characterized in that, include: Based on at least one frame of monocular visual image of the object to be measured and the effective axial depth of the ranging measurement point, the corresponding target candidate region is determined according to the preset cross-union ratio confidence judgment condition. In response to the shape confidence score of the target candidate region matching, a visual geometric refinement process is performed on the mapped frame image of the target candidate region to determine the current pixel size data of the target candidate region in the image coordinate system of the mapped frame image. This includes: in response to the shape confidence score, performing a candidate region expansion local search and grayscale conversion on the mapped frame image to generate a grayscale frame image; performing Gaussian filtering, contrast enhancement, and adaptive edge detection processing on the grayscale frame image to generate a refined frame image; and determining the current pixel size data of the candidate region bounding box corresponding to the target candidate region on the refined frame image. Based on historical camera size data, multi-frame target association processing is performed on the current pixel size data corresponding to the current camera size data to generate smooth output data of the current frame for the current camera size data; and Based on the standard uncertainty corresponding to the smoothed output data of the current frame, output the smoothed output data of the current frame; Wherein, the current pixel size data of the candidate region bounding box corresponding to the target candidate region on the refined frame image includes: In response to the shape confidence, when the target candidate region is a rectangle, the four pixel side lengths, the average pixel length of the opposite side, and the average pixel width of the opposite side of the candidate region box are determined as the current pixel size data based on the corner pixel coordinates of the intersection of the adjacent sides of the candidate region box after line fitting. In response to the shape confidence score, when the target candidate region is circular, the perimeter and area of ​​the candidate region box are determined; for the candidate region box after performing circle fitting processing on the roundness corresponding to the perimeter and area, the center pixel coordinates in the image coordinate system are extracted as the current pixel size data.

2. The method according to claim 1, characterized in that, Before determining the corresponding target candidate region based on at least one frame of monocular visual image of the object to be measured and the axial effective depth of the ranging measurement point according to a preset intersection-union confidence score, the method further includes: Based on the pre-drawn region, multiple reference candidate regions corresponding to the effective axial depth are extracted frame by frame from the at least one frame of monocular vision image; The candidate comprehensive score of each reference candidate region is determined by the center-mapped coordinates of the ranging spot at the corresponding ranging point of each reference candidate region in the image coordinate system and the image center coordinates of each reference candidate region. Based on the comprehensive candidate score, a first candidate box and a second candidate box are determined in the plurality of reference candidate regions.

3. The method according to claim 2, characterized in that, Before determining the corresponding target candidate region based on at least one frame of monocular visual image of the object to be measured and the axial effective depth of the ranging measurement point according to a preset intersection-union confidence score, the method further includes: The first candidate box and the second candidate box are scaled by a preset scaling ratio to generate a first scaled box and a second scaled box. The shape confidence and instance segmentation mask are output based on the first and second zoom boxes.

4. The method according to claim 3, characterized in that, Based on at least one frame of monocular visual image of the object to be measured and the effective axial depth of the ranging measurement point, the corresponding target candidate region is determined according to a preset intersection-union ratio (IUU) confidence criterion, including: Based on the shape confidence and instance segmentation mask, the validity of the first and second zoomed boxes is determined. Based on the first and second zoomed boxes that have been determined to be valid, a valid cross-union ratio that meets the preset cross-union ratio confidence determination condition is generated. Based on the preset cross-union ratio confidence determination condition, when the effective cross-union ratio is greater than or equal to the preset confidence threshold, the first zoomed box or the second zoomed box with greater confidence is determined as the target candidate region. The target candidate region is cropped, upsampled, and mapped to the corresponding original frame monocular visual image to generate the mapped frame image.

5. The method according to claim 1, characterized in that, After determining the current pixel size data of the candidate region bounding box corresponding to the target candidate region on the refined frame image, the method further includes: Based on the effective axial depth in the camera coordinate system, and according to the joint calibration relationship between the camera coordinate system and the ranging coordinate system, the rectangular center pixel coordinates corresponding to the current pixel size data are converted into the rectangular center camera coordinates of the current camera size data. Among them, the camera coordinates of the rectangle's center point, corresponding to the camera coordinates of the rectangle's corner points, are used to generate the actual size of the rectangle.

6. The method according to claim 1, characterized in that, After determining the current pixel size data of the candidate region bounding box corresponding to the target candidate region on the refined frame image, the method further includes: Based on the effective axial depth in the camera coordinate system, and according to the camera intrinsic parameters corresponding to the joint calibration relationship between the camera coordinate system and the ranging coordinate system, the center camera coordinates of the current camera size data are generated by back-projecting the center pixel coordinates to the camera coordinate system; the actual diameter of the circle is generated by back-projecting the diameter endpoints in the horizontal and vertical directions.

7. The method according to claim 1, characterized in that, The step of performing multi-frame target association processing on the current pixel size data corresponding to the current camera size data based on historical camera size data to generate smooth output data of the current frame for the current camera size data includes: Based on the historical camera size data and the current camera size data, the correlation determination of the center distance of the center pixel coordinates, the scale ratio of the pixel length and width parameters, and the shape consistency is performed to determine the smooth output data of the current frame that meets the preset stable output suppression conditions. The preset stable output suppression conditions include dead zone threshold suppression rules, single frame maximum step size suppression rules, and exponential smoothing suppression rules.

8. The method according to claim 1, characterized in that, The step of outputting the smoothed output data of the current frame based on the standard uncertainty corresponding to the smoothed output data of the current frame includes: Based on the effective axial depth, and combined with the standard uncertainty fused from the edge positioning error, ranging depth noise, and focal length calibration error of the current pixel size data of the current frame smooth output data, the current frame smooth output data is output.

9. An object measurement device based on monocular vision, used to implement the method according to any one of claims 1-8, characterized in that, include: The region determination module is used to determine the corresponding target candidate region based on at least one frame of monocular visual image of the object to be measured and the axial effective depth of the ranging measurement point, according to a preset intersection-union confidence judgment condition. The data determination module is used to perform visual geometric refinement processing on the mapped frame image of the target candidate region in response to the shape confidence of the target candidate region matching, and determine the current pixel size data of the target candidate region in the image coordinate system of the mapped frame image; The data generation module is used to perform multi-frame target association processing on the current pixel size data corresponding to the current camera size data based on historical camera size data, and generate smooth output data of the current frame of the current camera size data. as well as The data output module is used to output the smoothed output data of the current frame based on the standard uncertainty corresponding to the smoothed output data of the current frame.

10. An electronic device, comprising: One or more processors; Memory, used to store one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method of any one of claims 1 to 8.

11. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 8.

12. A computer program product comprising a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8.

Images