An online imaging correction method and related apparatus

Abstract

The application discloses an online imaging correction method and related equipment, and the method comprises the following steps: collecting the first pixel coordinates of the mark points in the reference correction image and the first profile point set of the visual haptic sensor; solving the non-ideal camera pose based on the first pixel coordinates and the three-dimensional reference coordinates; based on the three-dimensional sampling points on the visual haptic sensor, constructing the pixel corresponding sample by using the non-ideal camera pose and the ideal camera pose, fitting to obtain the geometric mapping function from the non-ideal imaging to the ideal imaging to generate the remapping data structure; using the remapping data structure to perform online correction on the real-time original image to obtain the real-time correction image; and counting the correction error, and if the correction error does not meet the preset condition, updating the remapping data structure. The application can assist in realizing the online fast correction of the real-time image, thereby significantly reducing the calculation overhead and the delay, meeting the real-time requirement of the visual haptic, and can be widely applied to the technical field of data processing.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to an online imaging correction method and related equipment. Background Technology

[0002] Visor-tactile sensors typically image elastomers, reflective films, markers, or surface textures using a camera, and calculate contact morphology and force information based on image difference, photometric stereo, or learned models. These algorithms are highly sensitive to pixel position consistency: even when the same sensor is used in different assembly batches or on different sensors, any deviation in the camera's pose relative to the colloidal / reflective structure will cause marker position drift, contour scale changes, and inconsistent geometric deformation of the contact area. This leads to increased tactile calculation errors and decreased model generalization ability.

[0003] Existing imaging correction methods mostly rely on general camera calibration and distortion correction, or employ image-level methods such as feature-point registration and optical flow / non-rigid registration. The former typically focuses on lens distortion and intrinsic / extrinsic parameter estimation, making it difficult to cover nonlinear differences caused by the internal structure of visual-tactile sensors; while the latter can compensate for geometric deviations to some extent, it often requires online iterative optimization or dense matching, resulting in high computational overhead and latency, making it difficult to meet the real-time requirements of visual-tactile sensing. Furthermore, existing correction methods typically employ manual or periodic updates of correction parameters, resulting in poor generalization ability. Summary of the Invention

[0004] The main objective of this invention is to provide an online imaging correction method, apparatus, electronic device, storage medium, and program product, aiming to solve at least one problem in the prior art.

[0005] To achieve the above objectives, one aspect of the present invention proposes an online imaging correction method, the method comprising: Acquire target data; wherein, the target data includes an ideal imaging reference and a reference correction image of the visual-touch sensor. The ideal imaging reference includes the ideal camera pose and the three-dimensional reference coordinates of several marked points on the visual-touch sensor. Acquire the first pixel coordinates of the marked points in the reference correction image and the first set of contour points of the visual-tactile sensor; The non-ideal camera pose is obtained by solving based on the first pixel coordinates and the three-dimensional reference coordinates; Based on the 3D sampling points on the visual-touch sensor, pixel-corresponding samples are constructed using non-ideal camera poses and ideal camera poses; Based on pixel-corresponding samples, a geometric mapping function from non-ideal imaging to ideal imaging is fitted, and a remapping data structure is generated according to the geometric mapping function. The real-time original image is corrected online using a remapping data structure to obtain a real-time corrected image; The second pixel coordinates of the marked points in the real-time corrected image and the second contour point set of the visual-tactile sensor are collected. The correction error is obtained by combining the first pixel coordinates and the first contour point set. If the correction error does not meet the preset conditions, the remapping data structure is updated.

[0006] In some embodiments, acquiring the first pixel coordinates of marker points in a reference correction image and the first contour point set of a visual-tactile sensor includes the following steps: Threshold segmentation is performed on the reference correction image based on a preset intensity threshold to extract candidate regions for marker points; The reference correction image is binarized based on the candidate regions of the marker points to obtain a binary mask; Connectivity analysis is performed on the binary mask to obtain the connected components of the candidate marker points; wherein, the connected components include the pixel set of the candidate marker points; The geometric center of the connected region is obtained based on pixel set quantization, and the first pixel coordinates of the marker point are obtained; wherein, the marker point represents a set of center points selected from the set of geometric centers, and the geometric consistency of each region is pre-configured in the visual tactile sensor, and the probability of selecting the geometric center as the marker point is positively correlated with the level of geometric consistency of the region where the geometric center is located; Edge detection based on connected component analysis is performed on the reference correction image to obtain the first set of contour points outside the effective working area of ​​the visual-tactile sensor.

[0007] In some embodiments, the method further includes the following steps: Before acquiring the first pixel coordinates of the marked points in the reference correction image, the reference correction image is subjected to distortion correction processing based on pre-calibrated distortion parameters; Alternatively, distortion correction can be performed on the coordinates of the first pixel based on the distortion parameters.

[0008] In some embodiments, the non-ideal camera pose is obtained by quantization based on the three-dimensional reference coordinates and the first pixel coordinates, including the following steps: Use the first marked point in the reference correction image as the target point; Based on the three-dimensional reference coordinates of the target point, the target point is projected through a preset projection function to obtain the two-dimensional pixel coordinates of the target point. For the target point, the difference between the vector of the two-dimensional pixel coordinates and the vector of the first pixel coordinates is squared using the L2 norm to obtain the quantized value of the target point. Take the next marker point in the reference correction image as the target point, return to perform the step of perspective projection on the target point based on the three-dimensional reference coordinates of the target point and through the preset projection function, until all marker points in the reference correction image have been traversed; The non-ideal camera pose is obtained by minimizing the sum of all quantized values.

[0009] In some embodiments, constructing pixel-corresponding samples using non-ideal camera poses and ideal camera poses includes the following steps: By using the non-ideal camera pose to project the 3D sampling points onto the image plane, the coordinates of the fourth pixel are obtained; The coordinates of the fifth pixel are obtained by projecting the 3D sampling points onto the image plane using the ideal camera pose. The fourth and fifth pixel coordinates corresponding to the same 3D sampling point are associated as a set of pixel-corresponding samples.

[0010] In some embodiments, the pixel-corresponding samples include the fourth pixel coordinates corresponding to the non-ideal camera pose and the fifth pixel coordinates corresponding to the ideal camera pose. Based on the pixel-corresponding samples, a geometric mapping function from non-ideal imaging to ideal imaging is fitted. A remapping data structure is generated according to the geometric mapping function, including the following steps: Construct a two-dimensional polynomial based on the fourth and fifth pixel coordinates; A least-squares optimization problem is constructed based on samples corresponding to multiple pixels, and then the fitting coefficients of the two-dimensional polynomial are obtained by solving the problem, thus obtaining the geometric mapping function. By performing a forward mapping and inverse calculation on the geometric mapping function, a remapping data structure is constructed. The method also includes the following steps: Based on the cache structure and instruction set of the target hardware, the remapped data structure adopts mixed precision encoding and pre-interpolated weight storage; wherein, the sampling coordinates in the remapped data structure are stored using fixed-point or half-precision floating-point compression, and the remapped data structure is associated with bilinear interpolation weight coefficients.

[0011] In some embodiments, the correction error includes root mean square error, maximum profile error, and average profile error. The correction error is statistically obtained. If the correction error does not meet the preset conditions, the remapping data structure is updated, including the following steps: The root mean square error is constructed based on the second pixel coordinates and the first pixel coordinates through root mean square operation. The nearest distance from each reference contour point in the first contour point set to the second contour point set is obtained through nearest neighbor distance statistical processing. The largest nearest distance is taken as the maximum profile error; The average contour error is obtained by averaging the nearest distances corresponding to all reference contour points in the first contour point set. When the root mean square error is less than or equal to the first threshold, the average profile error is less than or equal to the second threshold, and the maximum profile error is less than or equal to the third threshold, the correction error is determined to meet the preset conditions. If the correction error does not meet the preset conditions, return to the step of obtaining the target data to reconstruct the remapping data structure; Alternatively, if the correction error does not meet the preset conditions, the remapped data structure will be rolled back to a previously valid version.

[0012] To achieve the above objectives, another aspect of the present invention provides an online imaging correction device, the device comprising: The first module is used to acquire target data; the target data includes an ideal imaging reference and a reference correction image of the visual-tactile sensor. The ideal imaging reference includes the ideal camera pose and the three-dimensional reference coordinates of several marked points on the visual-tactile sensor. The second module is used to acquire the first pixel coordinates of the marked points in the reference correction image and the first contour point set of the visual-touch sensor. The third module is used to solve for the non-ideal camera pose based on the first pixel coordinates and the three-dimensional reference coordinates. The fourth module is used to construct pixel-to-pixel samples based on the three-dimensional sampling points on the visual-tactile sensor, using non-ideal camera poses and ideal camera poses; The fifth module is used to fit a geometric mapping function from non-ideal imaging to ideal imaging based on pixel-corresponding samples, and to generate a remapping data structure based on the geometric mapping function. The sixth module is used to perform online correction on the real-time original image using the remapping data structure to obtain the real-time corrected image; The seventh module is used to collect the second pixel coordinates of the marked points in the real-time correction image and the second contour point set of the visual-tactile sensor. It combines the first pixel coordinates and the first contour point set to statistically obtain the correction error. If the correction error does not meet the preset conditions, the remapping data structure is updated.

[0013] To achieve the above objectives, another aspect of the present invention provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned method.

[0014] To achieve the above objectives, another aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.

[0015] To achieve the above objectives, another aspect of the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method.

[0016] The embodiments of the present invention include at least the following beneficial effects: The present invention provides an online imaging correction method, device, electronic device, storage medium, and program product. This solution acquires target data; wherein, the target data includes an ideal imaging reference and a reference correction image of a visual-tactile sensor, the ideal imaging reference including an ideal camera pose and three-dimensional reference coordinates of several marked points on the visual-tactile sensor; the first pixel coordinates of the marked points in the reference correction image and the first contour point set of the visual-tactile sensor are acquired; a non-ideal camera pose is obtained based on the first pixel coordinates and the three-dimensional reference coordinates; pixel-corresponding samples are constructed using the non-ideal camera pose and the ideal camera pose based on the three-dimensional sampling points on the visual-tactile sensor; a geometric mapping function from non-ideal imaging to ideal imaging is fitted based on the pixel-corresponding samples, and a remapping data structure is generated according to the geometric mapping function; the real-time original image is corrected online using the remapping data structure to obtain a real-time corrected image; the second pixel coordinates of the marked points in the real-time corrected image and the second contour point set of the visual-tactile sensor are acquired, and the correction error is statistically obtained by combining the first pixel coordinates and the first contour point set; if the correction error does not meet the preset conditions, the remapping data structure is updated. This invention, by pre-constructing a geometric mapping function from non-ideal imaging to ideal imaging and generating a remapping data structure, can assist in the online and rapid correction of real-time images, thereby significantly reducing computational overhead and latency and meeting the real-time requirements of visual and tactile perception. Furthermore, by using the statistical correction error of marker points and contour point sets to determine the updated mapping structure, this invention can adaptively compensate for structural deviations between different assembly batches or individual sensors, avoiding manual periodic calibration and improving the model's generalization ability and tactile solution accuracy. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of an implementation environment for the online imaging correction method provided in this embodiment of the invention; Figure 2 This is a schematic flowchart of the online imaging correction method provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the overall process framework of the online imaging correction method provided in the embodiments of the present invention; Figure 4 This is a structural block diagram of the online correction system for visual-tactile sensor imaging provided in an embodiment of the present invention; Figure 5 This is a schematic diagram illustrating a comparison of correction effects provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the scatter points and clustering circles of the intact hole centers in the template diagram provided in the embodiment of the present invention; Figure 7 This is a schematic diagram illustrating a comparison example of pallet contour consistency provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of the structure of the online imaging correction device provided in an embodiment of the present invention; Figure 9 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of this invention; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this invention as detailed in the appended claims.

[0019] It is understood that the terms "first," "second," etc., used in this invention may be used to describe various concepts, but unless specifically stated otherwise, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of embodiments of this invention, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words "if" or "when" as used herein may be interpreted as "when," "in response to determination," or "in the event of a determination."

[0020] The terms “at least one,” “multiple,” “each,” “any,” etc., used in this invention, “at least one” includes one, two, or more than two; “multiple” includes two or more than two; “each” refers to each of the corresponding multiple; and “any” refers to any one of the multiple.

[0021] Unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this invention is for descriptive purposes only and is not intended to limit the invention.

[0022] In related technologies, there is an urgent need for an imaging correction method for visual and tactile sensors that can perform pixel-by-pixel geometric correction with extremely low online computation while taking into account lens distortion and assembly pose deviation, and has the ability to automatically verify and prevent errors in the correction configuration.

[0023] In view of this, this invention provides an online imaging correction method and related equipment. This method involves acquiring target data, including an ideal imaging reference and a reference correction image of a visual-tactile sensor. The ideal imaging reference includes an ideal camera pose and three-dimensional reference coordinates of several marker points on the visual-tactile sensor. The method acquires the first pixel coordinates of the marker points in the reference correction image and the first contour point set of the visual-tactile sensor. Based on the first pixel coordinates and the three-dimensional reference coordinates, a non-ideal camera pose is obtained. Based on the three-dimensional sampling points on the visual-tactile sensor, pixel-corresponding samples are constructed using the non-ideal and ideal camera poses. Based on the pixel-corresponding samples, a geometric mapping function from non-ideal imaging to ideal imaging is fitted, and a remapping data structure is generated according to the geometric mapping function. The remapping data structure is used to perform online correction on the real-time original image to obtain a real-time corrected image. The method acquires the second pixel coordinates of the marker points in the real-time corrected image and the second contour point set of the visual-tactile sensor, and statistically obtains the correction error by combining the first pixel coordinates and the first contour point set. If the correction error does not meet preset conditions, the remapping data structure is updated. This invention, by pre-constructing a geometric mapping function from non-ideal imaging to ideal imaging and generating a remapping data structure, can assist in the online and rapid correction of real-time images, thereby significantly reducing computational overhead and latency and meeting the real-time requirements of visual and tactile perception. Furthermore, by using the statistical correction error of marker points and contour point sets to determine the updated mapping structure, this invention can adaptively compensate for structural deviations between different assembly batches or individual sensors, avoiding manual periodic calibration and improving the model's generalization ability and tactile solution accuracy.

[0024] It is understood that the online imaging correction method provided by this invention can be applied to any computer device with data processing and computing capabilities, and this computer device can be various terminals or servers. When the computer device in the embodiment is a server, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the terminal can be a smartphone, tablet, laptop, or desktop computer, but it is not limited to these.

[0025] like Figure 1 The diagram shown is a schematic representation of an implementation environment provided by an embodiment of the present invention. (Refer to...) Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected via a network, either wirelessly or via a wired connection, to complete data transmission and exchange.

[0026] Server 101 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0027] Additionally, server 101 can also be a node server in a blockchain network. Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms.

[0028] Terminal 102 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. Terminal 102 and server 101 can be directly or indirectly connected via wired or wireless communication, and this embodiment of the invention does not impose any limitations.

[0029] For example, based on Figure 1 The implementation environment shown in this embodiment of the invention provides an online imaging correction method. The following description uses the application of this online imaging correction method in server 101 as an example. It can be understood that this online imaging correction method can also be applied in terminal 102.

[0030] Reference Figure 2 , Figure 2 This is an optional flowchart of an online imaging correction method provided in an embodiment of the present invention. The executing entity of the online imaging correction method can be any of the aforementioned computer devices (including servers or terminals). Figure 2 The method may include, but is not limited to, steps S100 to S700.

[0031] Step S100: Obtain target data; The target data includes an ideal imaging reference and a reference correction image of the visual-touch sensor. The ideal imaging reference includes the ideal camera pose and the three-dimensional reference coordinates of several marked points on the visual-touch sensor. For example, in some specific implementations, the 3D structural model established based on sensor design drawings can be exported as an STP file. After importing the STP file, the key geometric parameters (including position, orientation, and size information) of components such as the sensor housing, light source, and camera are extracted and analyzed. Based on this, the algorithm constructs a sensor geometric layout model to determine the ideal camera pose. ,in: The subscript "ideal" indicates the meaning of "ideal" and is used to indicate quantities related to an ideal imaging reference. This represents the rotation matrix of the camera coordinate system relative to the sensor coordinate system under ideal camera pose. This represents the translation vector of the camera coordinate system relative to the sensor coordinate system under the ideal camera pose. Based on the above geometric layout model, combined with the actual data acquired by the image measuring instrument and the sensor surface model, the three-dimensional positions of several preset marker points are determined. Specifically, the two-dimensional coordinates of the markers on the sensor surface can be measured using the image measuring instrument. And the corresponding height coordinates are obtained by interpolation using the surface model. This allows us to obtain the set of three-dimensional coordinates of the marker. , where: subscript Used to identify the first One marker, ; Indicates the first The three-dimensional coordinate vector of a marker in the sensor coordinate system, denoted as . Through the above steps, the ideal imaging benchmark is established, including the sensor geometric layout model and the ideal camera pose. and a set of 3D reference coordinates for markers used for pose estimation. .

[0032] Step S200: Acquire the first pixel coordinates of the marked points in the reference correction image and the first contour point set of the visual-tactile sensor; It should be noted that, in some embodiments, step S200 may include the following steps: thresholding the reference correction image based on a preset intensity threshold to extract candidate regions for marker points; binarizing the reference correction image based on the candidate regions for marker points to obtain a binary mask; performing connected component analysis on the binary mask to obtain the connected components of the candidate marker points; wherein, the connected components include a set of pixels of the candidate marker points; obtaining the geometric center of the connected components based on the pixel set quantization to obtain the first pixel coordinates of the marker points; wherein, the marker points represent a set of center points selected from the set of geometric centers, and the visual tactile sensor is pre-configured with the geometric consistency of each region, and the probability of selecting the geometric center as a marker point is positively correlated with the level of the geometric consistency of the region where the geometric center is located; performing edge detection based on connected component analysis on the reference correction image to obtain a first set of contour points outside the effective working area of ​​the visual tactile sensor.

[0033] For example, in some specific embodiments, in this step, at least one frame of the calibration image from the visual-tactile sensor is first acquired, denoted as the input image. ,in: , These represent the horizontal and vertical pixel coordinates on the image plane, respectively; Indicates the pixel position The pixel intensity value at that location. Subsequently, two types of features are detected from this corrected image: 1. J marker pixel coordinates used for pose estimation , where: subscript Indicates the first One marker; , They represent the first The horizontal and vertical pixel coordinates of each marker in the image. One implementation is to extract candidate marker regions based on thresholding. Specifically, a binary mask function is defined:

[0034] in: Indicates the pixel position Binary mask value at the location; Indicates the intensity threshold used to segment the marker; symbol This indicates an indicator function that takes the value 1 when the condition within the parentheses is met, and 0 otherwise.

[0035] In binary mask Connectivity analysis is performed to obtain the pixel set of each candidate marker, denoted as . For each connected component Calculate its geometric center as the pixel coordinates of the marker:

[0036] in: Represents connected components The number of pixels contained therein; This represents the pixel coordinates (horizontal coordinates are...) of the marker corresponding to the connected component. The vertical coordinate is All candidate markers are screened based on shape and size (e.g., roundness, fill rate, ellipticity, etc.) to obtain those that meet the quality requirements. Each marker pixel coordinate, i.e., a set .

[0037] 2. Set of contour features of the effective area of ​​the sensor : By using edge detection and connected component analysis, the set of outer contour points of the effective working area of ​​the visual-touch sensor is extracted from the calibrated image, denoted as: Where: subscript Indicates the first One contour point, ; , They represent the first The horizontal and vertical pixel coordinates of each contour point are determined. During contour extraction, morphological opening and closing operations can be used to remove isolated noise points, ensuring the continuity and uniformity of the contour. Simultaneously, constraints can be placed on the number and quality of markers to ensure the stability and reliability of the features used for pose estimation.

[0038] It should be noted that, in some embodiments, the method may further include the following steps: before acquiring the first pixel coordinates of the marked points in the reference correction image, performing distortion correction processing on the reference correction image based on pre-calibrated distortion parameters; or, performing distortion correction processing on the first pixel coordinates based on distortion parameters.

[0039] For example, in some specific embodiments, the distortion parameters can be determined based on pre-calibrated distortion parameters. Distortion is removed from the entire image, including: These represent the first-order, second-order, and third-order radial distortion coefficients, respectively. These represent the first-order tangential distortion coefficients.

[0040] The aforementioned marker pixel coordinates Mapped to the distortion-free pixel coordinates The superscript "~" indicates "distortion-free". They represent the first The horizontal and vertical pixel coordinates of each marker in the distortion-free image are determined. To improve the accuracy of subsequent marker detection and localization, in this embodiment, it is preferable to first perform the above distortion-free processing on the original image, and then perform marker detection and sub-pixel coordinate extraction on the distortion-free image, thereby reducing the impact of lens distortion on the marker detection results.

[0041] In some preferred embodiments, the marker points are not simply selected arbitrarily from the centers of all detected connected domains, but rather probabilistically sampled based on the geometric consistency level of each region on the sensor surface to further improve the robustness and tolerance of pose estimation. Specifically, during the design phase of the visual-tactile sensor, the deformation stability of different regions on the sensor surface under typical assembly errors can be pre-analyzed using CAD models or finite element simulations. The sensor imaging area is then divided into multiple sub-regions, and each sub-region is assigned a geometric consistency score. A higher score indicates that the region can maintain a relatively stable spatial positional relationship under conditions such as assembly deviations and colloidal deformation. During the offline calibration configuration generation process, after all candidate marker points (i.e., the geometric centers of all connected domains) are detected from the calibration image, the system reads the pre-configured geometric consistency score of the sub-region where each candidate marker point is located and calculates the probability that it will be selected as a marker point for the final PnP pose estimation. This probability is positively correlated with the geometric consistency score; that is, the higher the consistency of a region, the greater the probability that its marker point will be selected. One specific implementation involves assigning a weight to each candidate marker (determined by the product of the geometric consistency score of the region where the marker is located and a preset adjustment factor); then, random sampling with or without replacement is performed according to the normalized weights, ultimately selecting a preset number (e.g., 10) of markers to form the target marker set for PnP solving. This probabilistic selection strategy effectively suppresses the interference of randomly occurring low-quality markers or markers located in low-consistency regions on pose estimation. Even if errors occur in the detection of individual markers, their low probability of being selected in the sampling means they will not have a decisive impact on the final pose, thus significantly improving the robustness and repeatability of pose estimation. In other implementations, the top K markers with the highest consistency scores can be directly selected without introducing randomness.

[0042] Step S300: Solve for the non-ideal camera pose based on the first pixel coordinates and the three-dimensional reference coordinates; It should be noted that in some embodiments, step S300 may include the following steps: taking the first marker point in the reference correction image as the target point; performing perspective projection on the target point using a preset projection function based on the three-dimensional reference coordinates of the target point to obtain the two-dimensional pixel coordinates of the target point; performing a L2 norm square operation on the difference between the vector of the two-dimensional pixel coordinates and the vector of the first pixel coordinates for the target point to obtain the quantization value of the target point; taking the next marker point in the reference correction image as the target point, returning to perform the step of performing perspective projection on the target point using a preset projection function based on the three-dimensional reference coordinates of the target point, until all marker points in the reference correction image have been traversed; performing a minimization optimization based on the sum of all quantization values ​​to obtain the non-ideal camera pose.

[0043] For example, in some specific implementations, in this step, based on the distortion-corrected marker pixel coordinates and the corresponding 3D reference coordinates, the non-ideal camera pose in the current actual assembly state can be estimated using the PnP (Perspective-n-Point) algorithm. Let the camera intrinsic parameter matrix be... ,in: for The camera intrinsic parameter matrix contains parameters such as focal length and principal point coordinates. Let the camera extrinsic parameters to be estimated be... ,in: Represents the rotation matrix; Let represent the translation vector. Let the projection function be... This is used to project three-dimensional points onto a two-dimensional pixel plane. For the first... 3D coordinates of each marker Its projected pixel coordinates can be written as ,in: Indicates in internal reference External reference Under the condition of three-dimensional points The two-dimensional pixel coordinate vector obtained after perspective projection is denoted as... To estimate the non-ideal camera pose corresponding to the actual assembly state, it is denoted as... The subscript "non" indicates "non-ideal". Represents the rotation matrix under non-ideal camera pose; This represents the translation vector under non-ideal camera pose.

[0044] By minimizing the reprojection error between the marker pixel coordinates after distortion correction and the 3D point projection pixel coordinates, we can obtain: Pose that minimizes reprojection error :

[0045] in: The second norm of a vector; Represents the square of the L2 norm; Indicates the first The distortion-free pixel coordinate vectors of each marker. By solving the above optimization problem, the optimal [structure / parameter] is obtained. and This is recorded as a non-ideal camera pose. .

[0046] Step S400: Based on the three-dimensional sampling points on the visual-tactile sensor, construct pixel-corresponding samples using non-ideal camera pose and ideal camera pose; It should be noted that in some embodiments, constructing pixel-corresponding samples using non-ideal camera poses and ideal camera poses may include the following steps: projecting the three-dimensional sampling points onto the image plane using the non-ideal camera pose to obtain the fourth pixel coordinates; projecting the three-dimensional sampling points onto the image plane using the ideal camera pose to obtain the fifth pixel coordinates; and associating the fourth pixel coordinates and fifth pixel coordinates corresponding to the same three-dimensional sampling point as a set of pixel-corresponding samples.

[0047] For example, in some specific implementations, the same batch of 3D sampling points are projected under both the "ideal pose" and the "current estimated pose" to obtain pixel correspondences. This applies to densely sampled points on the sensor surface / curved model. , where: subscript Indicates the first Three-dimensional sampling points; Indicates the first The three-dimensional coordinate vector of each sampling point in the sensor coordinate system. For each sampling point In non-ideal camera pose Below, through the projection function Obtain its pixel coordinates on the image plane:

[0048] Wherein: the superscript "non" indicates the corresponding "non-ideal" pose; These represent the first and second positions under non-ideal poses. The horizontal and vertical pixel coordinates of each sampling point.

[0049] At the same time, in the ideal camera pose Below, for the same sampling point Projecting, we get:

[0050] Wherein: the superscript "ideal" indicates the corresponding "ideal" pose; They represent the first, second, and third positions under the ideal pose. The horizontal and vertical pixel coordinates of each sampling point.

[0051] Through the above operations, we can obtain Group of pixels corresponding to samples:

[0052] In some implementations, the above sampling can adopt a hierarchical / multi-resolution strategy: for example, first perform sparse sampling on the whole to estimate large-scale deformation, and then perform local densification sampling in contact sensitive areas or marker-dense areas, thereby improving the mapping accuracy of key areas while keeping the overall computational load controllable.

[0053] Step S500: Based on the corresponding pixel samples, fit the geometric mapping function from non-ideal imaging to ideal imaging, and generate the remapping data structure according to the geometric mapping function; It should be noted that the pixel-corresponding samples include the fourth pixel coordinates corresponding to the non-ideal camera pose and the fifth pixel coordinates corresponding to the ideal camera pose. In some embodiments, step S500 may include the following steps: constructing a two-dimensional polynomial based on the fourth pixel coordinates and the fifth pixel coordinates; constructing a least-squares optimization problem based on multiple pixel-corresponding samples, and then solving for the fitting coefficients of the two-dimensional polynomial to obtain the geometric mapping function; performing a forward mapping inverse on the geometric mapping function to construct a remapping data structure; wherein, in some preferred embodiments, the method further includes the following steps: based on the cache structure and instruction set of the target hardware, using mixed-precision encoding and pre-interpolated weight storage for the remapping data structure; wherein, the sampling coordinates in the remapping data structure are stored using fixed-point or half-precision floating-point compression, and the remapping data structure is associated with bilinear interpolation weight coefficients.

[0054] For example, in some specific embodiments, in this step, a two-dimensional geometric mapping function from non-ideal imaging to ideal imaging is fitted using the aforementioned pixel-corresponding samples to correct the non-ideal imaging. Let the mapping function be:

[0055] in: Represents the coordinates of any pixel under non-ideal imaging conditions; This represents the pixel coordinates under the corresponding ideal imaging.

[0056] One implementation is to use a two-dimensional polynomial (e.g., the highest order is 1). The order is used to fit this mapping. Specifically, we can let:

[0057] Where: subscript A non-negative integer represents a pair of corresponding terms in a polynomial. , powers of; This represents the maximum total order of the polynomial, i.e. ; Indicates the use of fitting In the two-dimensional polynomial, of The power of sum of The coefficient of the power term; Indicates the use of fitting In a two-dimensional polynomial, the coefficients of terms corresponding to the same power are given.

[0058] For all Given training samples, construct a least squares optimization problem:

[0059] in: Indicates the sample points The predicted coordinates are obtained by substituting the above polynomial mapping function. Represents the corresponding true ideal coordinates; Let L2 be the square of the difference between the two-dimensional coordinates. By solving the above least squares problem, all coefficients can be obtained. and Thus, the complete two-dimensional geometric mapping function is obtained. In this embodiment, the above-mentioned two-dimensional polynomial mapping is a global fitting model, whose domain covers the entire effective field of view of the sensor imaging plane, and different polynomial models are not established for local regions. When it is necessary to further improve the accuracy of local regions, the mapping can also be extended to a block local polynomial or other local adaptive model without departing from the spirit of the present invention.

[0060] The above mapping It is established after radial and tangential distortion correction of the original image. It is used to describe the residual geometric error relationship between the "geometrically non-ideal sampling coordinate system" and the "target ideal geometric coordinate system". Therefore, it can be regarded as a unified compensation for the residual geometric distortion caused by lens distortion, installation deviation and elastic medium and other comprehensive factors.

[0061] In other alternative implementations, the effective imaging region of the sensor can be divided into blocks according to polar angles or grids, and an independent two-dimensional mapping function can be fitted to each sub-region. Alternatively, a piecewise polynomial / radial basis function (RBF) approach can be used to approximate strong local nonlinear deformations. Such block or piecewise strategies automatically select the order and regularization weights through cross-validation in the offline stage to suppress overfitting while also considering the storage overhead of the LUT (Look-Up Table, also referred to as geometric remapping in this invention for constructing the online remapping module) in the online stage.

[0062] Furthermore, in order to perform image resampling, online remapping uses inverse mapping; therefore, the inverse of the forward mapping can be obtained in the offline stage.

[0063] The geometric mapping is discretized into a data structure that can be quickly resampled online and then embedded into the calibration configuration file.

[0064] For example, let the pixel coordinates in the ideal output image be... Its sampling coordinates in a non-ideal image are For each pixel on the ideal imaging coordinate grid By performing a reverse solution (e.g., through numerical inversion or finding the nearest neighbor), the sampled coordinates in a non-ideal image coordinate system can be obtained. , where: subscript This indicates the meaning of "sample"; These represent the horizontal and vertical coordinates used for sampling in a non-ideal image, respectively. These sampling coordinates are stored as:

[0065] in: Indicates the ideal pixel position Below is a lookup table of horizontal coordinates for the corresponding sampling positions in a non-ideal image; This represents the corresponding vertical coordinate lookup table.

[0066] In a preferred embodiment, the LUT can be processed using mixed-precision encoding and pre-interpolated weight storage, taking into account the target hardware's cache structure and instruction set. For example, the sampled coordinates can be compressed and stored using fixed-point or half-precision floating-point encoding, while the bilinear interpolation weight coefficients are pre-calculated and written into the LUT. This way, remapping can be completed in the online phase with only integer addition and a few multiplication operations, which is beneficial for further reducing latency in embedded platforms or high frame rate scenarios.

[0067] During the online phase, remapping is performed on any frame of the original image:

[0068] Where: the subscript "raw" indicates the original (uncorrected) image; the subscript "corr" indicates the corrected image; Indicates the pixel position of the corrected image. Pixel value at; Indicates the pixel position of the original image. The pixel value at that location. In this embodiment, the mapping... Typically, it is a nonlinear two-dimensional polynomial, whose analytic inverse function is difficult to give explicitly. Therefore, the inverse of the forward mapping is achieved by numerical discretization: in the offline stage, for each output sampling point on the ideal imaging plane... The corresponding sampling position in the non-ideal image is found through numerical iteration or table lookup. And store the results in a lookup table. This is used for fast queries during online remapping. For potential non-bijective scenarios (such as partial occlusion, folding, or many-to-one or one-to-many relationships caused by non-monotonic mapping), this embodiment preferably uses... The grid is sampled sufficiently densely, and the nearest neighbor or weighted average criterion is used during the solution process to select from multiple candidate... The system selects the set of points closest to the ideal position as the final sampling points, thus ensuring the numerical continuity and stability of the mapping lookup table. In actual measurements, due to the small geometric deviation of the imaging system, the mapping remains approximately injective within the effective field of view, and the above strategy is sufficient to avoid significant artifacts caused by folding.

[0069] In some alternative implementations, for embedded platforms or visual-haptic perception tasks requiring ultra-high frame rates (e.g., ≥200fps), the storage and usage of the remapped data structure (LUT) are deeply optimized to further reduce computational latency and memory bandwidth consumption during the online calibration phase. Specific steps can be implemented as follows: (1) Mixed precision encoding: The sampling coordinate table originally stored in single precision floating-point (float32) format is converted into a mixed precision encoding format. Convert to fixed-point or half-precision floating-point (float16) format. For example, for an ideal output image, each sampled coordinate value originally occupies 4 bytes. When using fixed-point encoding, the bit width allocation of the integer and fractional parts can be dynamically determined according to the image width (e.g., using an 8-bit integer part + 8-bit fractional part 16-bit fixed-point number), compressing the storage space by half. Using half-precision floating-point (1 sign bit + 5 exponent bits + 10 mantissa bits) can retain approximately 10... -3 While maintaining relative precision, it also achieves half the storage requirement, and the vector instruction sets of modern embedded platforms typically have native support for half-precision operations.

[0070] (2) Pre-interpolation weight storage: Traditional bilinear interpolation requires real-time calculation of the weight coefficients of four neighboring pixels based on the fractional part of the sampled coordinates during the online stage, involving multiple floating-point multiplications. To eliminate this overhead, this invention performs pre-interpolation weight storage for each ideal output pixel during the offline stage. Pre-calculate its sampling coordinates in non-ideal images And further extract the integer part and decimal part Then, the four interpolation weights are calculated directly. , , and These weights, along with the integer coordinates, are stored in the LUT. To balance storage efficiency and computation speed, the integer coordinates can be stored using 16-bit unsigned integers, and the weight coefficients can be quantized as 8-bit unsigned integers (e.g., multiplying the floating-point weights by 255 and then rounding down). Finally, the LUT's data structure is organized as a contiguous array of structures, each containing: (16-bit) (16 bits), four interpolation weights (4×8 bits), totaling 64 bits (8 bytes). Compared to traditional floating-point LUTs (each pixel requires storing two float32s, totaling 8 bytes), this embodiment of the invention can simplify the pixel interpolation operation in the online stage to integer address calculation and four integer multiplication / shift operations without changing the memory usage, completely avoiding floating-point operations.

[0071] (3) Online remapping execution: In the online phase, the following fast interpolation is performed on each ideal output pixel:

[0072] The 128 before addition is used for rounding, and shifting right by 8 bits is equivalent to dividing by 255. The entire operation uses only integer addition and multiplication, with an extremely low instruction cycle, making it very suitable for pipelined parallelism or SIMD acceleration.

[0073] The above-mentioned hybrid precision LUT and pre-interpolated weight storage scheme can significantly improve the computational efficiency and energy efficiency ratio in the online stage without sacrificing the correction accuracy, enabling the method of the present invention to be deployed in robot fingertip embedded systems or low-power edge computing devices with limited computing power.

[0074] Step S600: The real-time original image is corrected online using the remapping data structure to obtain the real-time corrected image; For example, in some specific implementations, the configuration file is loaded at runtime, and geometric remapping (LUT) is performed on the real-time frame to obtain a real-time corrected image. In addition, during the execution of geometric remapping (LUT), the color of the corresponding pixel can also be synchronously mapped based on the pixel correspondence mapping relationship.

[0075] Step S700: Collect the second pixel coordinates of the marked points in the real-time correction image and the second contour point set of the visual-touch sensor. Combine the first pixel coordinates and the first contour point set to obtain the correction error. If the correction error does not meet the preset conditions, update the remapping data structure. It should be noted that the specific principle logic of acquiring the second pixel coordinates of the marked points in the real-time correction image and the second contour point set of the visual-tactile sensor is the same as that of step S200, and will not be elaborated here.

[0076] It should be noted that the correction error includes root mean square error, maximum contour error, and average contour error. In some embodiments, the correction error is statistically obtained. If the correction error does not meet the preset conditions, the remapping data structure is updated, which may include the following steps: constructing the root mean square error based on the second pixel coordinates and the first pixel coordinates through root mean square operation; obtaining the nearest distance from each reference contour point in the first contour point set to the second contour point set through nearest neighbor distance statistical processing; taking the largest nearest distance as the maximum contour error; averaging the nearest distances corresponding to all reference contour points in the first contour point set to obtain the average contour error; determining that the correction error meets the preset conditions when the root mean square error is less than or equal to the first threshold, the average contour error is less than or equal to the second threshold, and the maximum contour error is less than or equal to the third threshold simultaneously; if the correction error does not meet the preset conditions, returning to the step of obtaining target data to reconstruct the remapping data structure; or, if the correction error does not meet the preset conditions, reverting the remapping data structure to a historical valid version.

[0077] For example, in some specific implementations, marker and contour features are detected again on the corrected image, and a deviation index from the baseline features in the configuration file is calculated, such as: Root mean square error (RMSE) of marker coordinates:

[0078] in: The root mean square error of the marker pixel coordinates; Indicates the current number The pixel coordinate vector of each marker; Indicates the benchmark number The pixel coordinate vector of each marker; This represents the squared L2 norm of the difference between two-dimensional coordinates.

[0079] Contour distance error (nearest neighbor distance statistics): Let the reference profile point set be The current set of contour points is For each reference profile point, calculate its nearest distance to the current profile:

[0080] in: Indicates the first One reference contour point; Indicates the first The nearest neighbor distance from each reference profile point to the current profile is calculated; and the average profile error and maximum profile error are calculated.

[0081] in: This represents the average distance from all reference contour points to the current contour. This represents the maximum value among these distances.

[0082] Set marker error threshold Average contour error threshold and maximum contour error threshold ,in: Indicates the maximum allowable RMSE; Indicates the upper limit of the allowable average profile error; This represents the upper limit of the allowable maximum profile error. When both conditions are met... and The configuration is deemed valid only if it is valid; otherwise, it is deemed invalid and a prompt to regenerate is displayed. The above thresholds... The settings can be configured based on the target application's requirements for imaging geometric accuracy and the results of calibration experiments. In this embodiment, it is preferable to first obtain the RMSE under different configurations through multiple calibration experiments. and The distribution range is determined by combining tactile reconstruction error, 3D reconstruction error, and other business indicators with regression analysis and comparative experiments to identify the threshold range within which the overall system performance meets requirements. Specific values ​​within this range can be adjusted by engineers based on experience. For different sensor models or different resolution configurations, the corresponding thresholds can also be recalibrated through the aforementioned statistical analysis process. In some implementations, the system can also maintain timestamps and usage counts for configuration files and monitor the long-term drift trend of verification indicators: when a monotonically increasing trend in RMSE or contour error is observed during multiple startups or running cycles, even if the threshold has not yet been exceeded, an early warning can be issued or the trust weight of tactile computation can be automatically reduced to improve the robustness and stability of the overall system.

[0083] To explain in detail the principle of the technical solution of the present invention, the overall process of the present invention will be described below with reference to some specific embodiments. It is easy to understand that the following is an explanation of the technical principle of the present invention and should not be regarded as a limitation of the present invention.

[0084] First, it should be noted that, from an imaging geometry perspective, the projection of a camera onto a three-dimensional point on the sensor surface can typically be described using a pinhole model: Where (X,Y,Z) are three-dimensional points in the sensor coordinate system or world coordinate system, (u,v) are pixel coordinates, s is the scale factor, K is the camera intrinsic parameter matrix, and R,t are the rotation matrix and translation vector of the camera relative to the sensor, respectively. For visual-tactile sensors, assembly errors will cause the offset of R,t; at the same time, lens distortion will introduce additional nonlinear mapping deviations, causing a systematic shift in the pixel coordinates of the same physical location in the image. In general camera imaging, lens distortion is often described using radial distortion and tangential distortion models. Let the normalized image plane coordinates be (x,y), let... Then the distorted coordinates ( This can be represented as: in, The radial distortion coefficient is... This represents the tangential distortion coefficient. It is evident that even under ideal pinhole projection conditions, pixel coordinates may still exhibit significant deviations without distortion compensation. Furthermore, in visual-tactile sensors, in addition to lens distortion, there are often assembly pose deviations and nonlinear geometric differences caused by internal optical structures, making it difficult to guarantee imaging consistency between different replicas by relying solely on general calibration / distortion correction.

[0085] In view of the shortcomings of the prior art, the purpose of this invention is to provide a low-latency imaging geometry correction method and system based on markers for visual-tactile sensors, so as to solve the problem of imaging geometry inconsistency caused by pose deviations of different machine assembly and lens distortion in the prior art.

[0086] To achieve the above objectives, the present invention provides a marker-based low-delay imaging geometric correction method for visual-tactile sensors, comprising: (1) Establish an ideal imaging reference: predefine the ideal camera pose or ideal imaging coordinate system, and obtain the set of three-dimensional reference coordinates of the marker points for pose estimation; (2) Acquire calibration frames and extract features: Acquire at least one frame of calibration image, detect the pixel coordinates of the marker points and the contour features of the effective area of ​​the sensor, and perform quantity and quality screening; (3) Distortion compensation and pose estimation: Based on the pre-calibrated lens distortion parameters, the feature points (or images) are subjected to distortion correction, and then the current non-ideal camera pose is solved by PnP using the three-dimensional-two-dimensional correspondence of the marked points. (4) Construct pixel correspondence and fit geometric mapping: Select dense sampling points on the sensor surface, project them under the ideal pose and the current pose respectively, construct the "non-ideal pixel - ideal pixel" corresponding samples, and fit them through two-dimensional polynomial or other function forms to obtain the geometric mapping relationship from non-ideal imaging to ideal imaging; in some implementations, the sampling points can also be partitioned and weighted or local rigid constraints can be introduced to reduce the local fitting deviation caused by high curvature areas or edge areas.

[0087] (5) Offline generation of online remapping data structure: Discretize the geometric mapping into a pixel-by-pixel reverse sampling coordinate table (map_x / map_y) or a regular grid sampling table, and solidify it into the calibration configuration file along with the benchmark marker / contour features and threshold parameters; (6) Online low-latency correction: Load the configuration file at runtime, perform remapping based on lookup table and interpolation on the real-time image (and optionally perform distortion correction and color mapping), and output the corrected image for tactile processing; (7) Verification and error prevention: Detect markers and contours again on the corrected image, calculate the deviation index of the baseline features in the configuration file (such as marker coordinate RMSE, contour average / maximum nearest neighbor distance), and prevent the continued use of the configuration when the deviation exceeds the threshold and prompt to regenerate or roll back to the previous valid configuration.

[0088] like Figure 3 As shown, the online imaging correction method for the visual-tactile sensor of the present invention includes an offline processing stage and an online processing stage. The offline processing stage includes steps S100-S700: establishing an ideal imaging reference, acquiring correction frames and detecting markers / contours, lens distortion correction, estimating non-ideal camera poses based on PnP, constructing training samples, fitting two-dimensional geometric mappings, and generating LUT remapping data structures. The online processing stage includes steps S800-S900: online correction execution, verification, and error prevention.

[0089] The specific methods and steps are detailed below: S1. Establish an ideal imaging benchmark: Specifically, the 3D structural model based on sensor design drawings can be exported as an STP file. After importing the STP file, key geometric parameters (including position, orientation, and dimensional information) of components such as the sensor housing, light source, and camera can be extracted and analyzed. Based on this, the algorithm constructs a sensor geometric layout model, thereby determining the ideal camera pose. ,in: The subscript "ideal" indicates the meaning of "ideal" and is used to indicate quantities related to an ideal imaging reference. This represents the rotation matrix of the camera coordinate system relative to the sensor coordinate system under ideal camera pose. This represents the translation vector of the camera coordinate system relative to the sensor coordinate system under the ideal camera pose. Based on the above geometric layout model, combined with the actual data acquired by the image measuring instrument and the sensor surface model, the three-dimensional positions of several preset marker points are determined. Specifically, the two-dimensional coordinates of the markers on the sensor surface can be measured using the image measuring instrument. And the corresponding height coordinates are obtained by interpolation using the surface model. This allows us to obtain the set of three-dimensional coordinates of the marker. , where: subscript Used to identify the first One marker, ; Indicates the first The three-dimensional coordinate vector of a marker in the sensor coordinate system, denoted as . Through the above steps, the ideal imaging benchmark is established, including the sensor geometric layout model and the ideal camera pose. and a set of 3D reference coordinates for markers used for pose estimation. .

[0090] S2. Acquire calibration frames and detect markers / contours: In this step, at least one frame of the calibration image from the visual-touch sensor is first acquired, and this is recorded as the input image. ,in: , These represent the horizontal and vertical pixel coordinates on the image plane, respectively; Indicates the pixel position The pixel intensity value at that location. Subsequently, two types of features are detected from this corrected image: 1. J marker pixel coordinates used for pose estimation , where: subscript Indicates the first One marker; , They represent the first The horizontal and vertical pixel coordinates of each marker in the image. One implementation is to extract candidate marker regions based on thresholding. Specifically, a binary mask function is defined:

[0091] in: Indicates the pixel position Binary mask value at the location; Indicates the intensity threshold used to segment the marker; symbol This indicates an indicator function that takes the value 1 when the condition within the parentheses is met, and 0 otherwise.

[0092] In binary mask Connectivity analysis is performed to obtain the pixel set of each candidate marker, denoted as . For each connected component Calculate its geometric center as the pixel coordinates of the marker:

[0093] in: Represents connected components The number of pixels contained therein; This represents the pixel coordinates (horizontal coordinates are...) of the marker corresponding to the connected component. The vertical coordinate is All candidate markers are screened based on shape and size (e.g., roundness, fill rate, ellipticity, etc.) to obtain those that meet the quality requirements. Each marker pixel coordinate, i.e., a set .

[0094] 2. Set of contour features of the effective area of ​​the sensor

[0095] By using edge detection and connected component analysis, the set of outer contour points of the effective working area of ​​the visual-touch sensor is extracted from the calibrated image, denoted as:

[0096] Where: subscript Indicates the first One contour point, ; , They represent the first The horizontal and vertical pixel coordinates of each contour point are determined. During contour extraction, morphological opening and closing operations can be used to remove isolated noise points, ensuring the continuity and uniformity of the contour. Simultaneously, constraints can be placed on the number and quality of markers to ensure the stability and reliability of the features used for pose estimation.

[0097] S3, Lens Distortion Correction: Based on the pre-calibrated distortion parameters Distortion is removed from the entire image, including: These represent the first-order, second-order, and third-order radial distortion coefficients, respectively. These represent the first-order tangential distortion coefficients.

[0098] The aforementioned marker pixel coordinates Mapped to the distortion-free pixel coordinates The superscript "~" indicates "distortion-free". They represent the first The horizontal and vertical pixel coordinates of each marker in the distortion-free image are determined. To improve the accuracy of subsequent marker detection and localization, in this embodiment, it is preferable to first perform the above distortion-free processing on the original image, and then perform marker detection and sub-pixel coordinate extraction on the distortion-free image, thereby reducing the impact of lens distortion on the marker detection results.

[0099] S4. Estimating non-ideal camera pose based on PnP: In this step, based on the distortion-corrected marker pixel coordinates and the corresponding 3D reference coordinates, the non-ideal camera pose in the current actual assembly state is estimated using the PnP (Perspective-n-Point) algorithm. Let the camera intrinsic parameter matrix be... ,in: for The camera intrinsic parameter matrix contains parameters such as focal length and principal point coordinates. Let the camera extrinsic parameters to be estimated be... ,in: Represents the rotation matrix; Let represent the translation vector. Let the projection function be... This is used to project three-dimensional points onto a two-dimensional pixel plane. For the first... 3D coordinates of each marker Its projected pixel coordinates can be written as ,in: Indicates in internal reference External reference Under the condition of three-dimensional points The two-dimensional pixel coordinate vector obtained after perspective projection is denoted as... To estimate the non-ideal camera pose corresponding to the actual assembly state, it is denoted as... The subscript "non" indicates "non-ideal". Represents the rotation matrix under non-ideal camera pose; This represents the translation vector under non-ideal camera pose.

[0100] By minimizing the reprojection error between the marker pixel coordinates after distortion correction and the 3D point projection pixel coordinates, we can obtain: Pose that minimizes reprojection error :

[0101] in: The second norm of a vector; Represents the square of the L2 norm; Indicates the first The distortion-free pixel coordinate vectors of each marker. By solving the above optimization problem, the optimal [structure / parameter] is obtained. and This is recorded as a non-ideal camera pose. .

[0102] S5. Construct training samples: The same batch of 3D sampling points are projected under both the "ideal pose" and the "current estimated pose" to obtain pixel correspondences. This applies to densely sampled points on the sensor surface / curved model. , where: subscript Indicates the first Three-dimensional sampling points; Indicates the first The three-dimensional coordinate vector of each sampling point in the sensor coordinate system. For each sampling point In non-ideal camera pose Below, through the projection function Obtain its pixel coordinates on the image plane:

[0103] Wherein: the superscript "non" indicates a "non-ideal" pose; These represent the first and second positions under non-ideal poses. The horizontal and vertical pixel coordinates of each sampling point.

[0104] At the same time, in the ideal camera pose Below, for the same sampling point Projecting, we get:

[0105] Wherein: the superscript "ideal" indicates the corresponding "ideal" pose; They represent the first, second, and third positions under the ideal pose. The horizontal and vertical pixel coordinates of each sampling point.

[0106] Through the above operations, we can obtain Group of pixels corresponding to samples:

[0107] In some implementations, the above sampling can adopt a hierarchical / multi-resolution strategy: for example, first perform sparse sampling on the whole to estimate large-scale deformation, and then perform local densification sampling in contact sensitive areas or marker-dense areas, thereby improving the mapping accuracy of key areas while keeping the overall computational load controllable.

[0108] S6. Fitting a two-dimensional geometric mapping (non-ideal → ideal): In this step, a two-dimensional geometric mapping function from non-ideal imaging to ideal imaging is fitted using the aforementioned pixel-corresponding samples to correct the non-ideal imaging. Let the mapping function be:

[0109] in: Represents the coordinates of any pixel under non-ideal imaging conditions; This represents the pixel coordinates under the corresponding ideal imaging.

[0110] One implementation is to use a two-dimensional polynomial (e.g., the highest order is 1). The order is used to fit this mapping. Specifically, we can let:

[0111] Where: subscript A non-negative integer represents a pair of corresponding terms in a polynomial. , powers of; This represents the maximum total order of the polynomial, i.e. ; Indicates the use of fitting In the two-dimensional polynomial, of The power of sum of The coefficient of the power term; Indicates the use of fitting In a two-dimensional polynomial, the coefficients of terms corresponding to the same power are given.

[0112] For all Given training samples, construct a least squares optimization problem:

[0113] in: Indicates the sample points The predicted coordinates are obtained by substituting the above polynomial mapping function. Represents the corresponding true ideal coordinates; Let L2 be the square of the difference between the two-dimensional coordinates. By solving the above least squares problem, all coefficients can be obtained. and Thus, the complete two-dimensional geometric mapping function is obtained. In this embodiment, the above-mentioned two-dimensional polynomial mapping is a global fitting model, whose domain covers the entire effective field of view of the sensor imaging plane, and different polynomial models are not established for local regions. When it is necessary to further improve the accuracy of local regions, the mapping can also be extended to a block local polynomial or other local adaptive model without departing from the spirit of the present invention.

[0114] The above mapping It is established after radial and tangential distortion correction of the original image. It is used to describe the residual geometric error relationship between the "geometrically non-ideal sampling coordinate system" and the "target ideal geometric coordinate system". Therefore, it can be regarded as a unified compensation for the residual geometric distortion caused by lens distortion, installation deviation and elastic medium and other comprehensive factors.

[0115] In other alternative implementations, the effective imaging region of the sensor can be divided into blocks according to polar angles or grids, and an independent two-dimensional mapping function can be fitted to each sub-region. Alternatively, a piecewise polynomial / radial basis function (RBF) approach can be used to approximate strong local nonlinear deformations. Such block or piecewise strategies automatically select the order and regularization weights through cross-validation in the offline stage to suppress overfitting while also considering the LUT storage overhead in the online stage.

[0116] S7. Generate the data structure for low-latency online correction: To perform image resampling, online remapping uses inverse mapping; therefore, the inverse of the forward mapping can be obtained in the offline stage.

[0117] The geometric mapping is discretized into a data structure that can be quickly resampled online and then embedded into the calibration configuration file.

[0118] For example, let the pixel coordinates in the ideal output image be... Its sampling coordinates in a non-ideal image are For each pixel on the ideal imaging coordinate grid By performing a reverse solution (e.g., through numerical inversion or finding the nearest neighbor), the sampled coordinates in a non-ideal image coordinate system can be obtained. , where: subscript This indicates the meaning of "sample"; These represent the horizontal and vertical coordinates used for sampling in a non-ideal image, respectively. These sampling coordinates are stored as:

[0119] in: Indicates the ideal pixel position Below is a lookup table of horizontal coordinates for the corresponding sampling positions in a non-ideal image; This represents the corresponding vertical coordinate lookup table.

[0120] In a preferred embodiment, the LUT can be processed using mixed-precision encoding and pre-interpolated weight storage, taking into account the target hardware's cache structure and instruction set. For example, the sampled coordinates can be compressed and stored using fixed-point or half-precision floating-point encoding, while the bilinear interpolation weight coefficients are pre-calculated and written into the LUT. This way, remapping can be completed in the online phase with only integer addition and a few multiplication operations, which is beneficial for further reducing latency in embedded platforms or high frame rate scenarios.

[0121] During the online phase, remapping is performed on any frame of the original image:

[0122] Where: the subscript "raw" indicates the original (uncorrected) image; the subscript "corr" indicates the corrected image; Indicates the pixel position of the corrected image. Pixel value at; Indicates the pixel position of the original image. The pixel value at that location. In this embodiment, the mapping... Typically, it is a nonlinear two-dimensional polynomial, whose analytic inverse function is difficult to give explicitly. Therefore, the inverse of the forward mapping is achieved by numerical discretization: in the offline stage, for each output sampling point on the ideal imaging plane... The corresponding sampling position in the non-ideal image is found through numerical iteration or table lookup. And store the results in a lookup table. This is used for fast queries during online remapping. For potential non-bijective scenarios (such as partial occlusion, folding, or many-to-one or one-to-many relationships caused by non-monotonic mapping), this embodiment preferably uses... The grid is sampled sufficiently densely, and the nearest neighbor or weighted average criterion is used during the solution process to select from multiple candidate... The system selects the set of points closest to the ideal position as the final sampling points, thus ensuring the numerical continuity and stability of the mapping lookup table. In actual measurements, due to the small geometric deviation of the imaging system, the mapping remains approximately injective within the effective field of view, and the above strategy is sufficient to avoid significant artifacts caused by folding.

[0123] S8. Online calibration execution: At runtime, the configuration file is loaded, and geometric remapping (LUT) and color mapping are performed on the real-time frame.

[0124] S9. Correction Configuration Verification and Mistake Prevention: After correction, marker and contour features are detected again in the image, and the deviation index from the baseline features in the configuration file is calculated, such as: Root mean square error (RMSE) of marker coordinates:

[0125] in: The root mean square error of the marker pixel coordinates; Indicates the current number The pixel coordinate vector of each marker; Indicates the benchmark number The pixel coordinate vector of each marker; This represents the squared L2 norm of the difference between two-dimensional coordinates.

[0126] Contour distance error (nearest neighbor distance statistics): Let the reference profile point set be The current set of contour points is For each reference profile point, calculate its nearest distance to the current profile:

[0127] in: Indicates the first One reference contour point; Indicates the first The nearest neighbor distance from each reference profile point to the current profile is calculated; and the average profile error and maximum profile error are calculated.

[0128] in: This represents the average distance from all reference contour points to the current contour. This represents the maximum value among these distances.

[0129] Set marker error threshold Average contour error threshold and maximum contour error threshold ,in: Indicates the maximum allowable RMSE; Indicates the upper limit of the allowable average profile error; This represents the upper limit of the allowable maximum profile error. When both conditions are met... and The configuration is deemed valid only if it is valid; otherwise, it is deemed invalid and a prompt to regenerate is displayed. The above thresholds... The settings can be configured based on the target application's requirements for imaging geometric accuracy and the results of calibration experiments. In this embodiment, it is preferable to first obtain the RMSE under different configurations through multiple calibration experiments. and The distribution range is determined by combining tactile reconstruction error, 3D reconstruction error, and other business indicators with regression analysis and comparative experiments to identify the threshold range within which the overall system performance meets requirements. Specific values ​​within this range can be adjusted by engineers based on experience. For different sensor models or different resolution configurations, the corresponding thresholds can also be recalibrated through the aforementioned statistical analysis process. In some implementations, the system can also maintain timestamps and usage counts for configuration files and monitor the long-term drift trend of verification indicators: when a monotonically increasing trend in RMSE or contour error is observed during multiple startups or running cycles, even if the threshold has not yet been exceeded, an early warning can be issued or the trust weight of tactile computation can be automatically reduced to improve the robustness and stability of the overall system.

[0130] In some alternative implementations, configuration file management can also be performed: Save: LUT parameters, baseline marker / contour data, and corrected image.

[0131] In addition, in some specific embodiments, the present invention also provides a system corresponding to the above method, which includes: a camera module, a feature detection module, a distortion correction module, a pose estimation module, a geometric mapping fitting module, an online remapping module, a verification and error prevention module, and a configuration file management module. The modules work together to achieve low-latency imaging geometric consistency and online reliable verification.

[0132] like Figure 4 As shown, the online correction system for visual-tactile sensor imaging of the present invention includes: a visual-tactile sensor (colloid / reflective film / marker), a camera module (including lens distortion parameters), an image acquisition module, a marker detection module, a distortion correction module, a pose estimation module (PnP), a geometric mapping fitting module, an online remapping module (LUT), a verification and error prevention module, a configuration file management module, and a tactile calculation module; wherein, the marker detection module, the distortion correction module, the pose estimation module, and the geometric mapping fitting module constitute an offline processing stage for generating a correction configuration file; the configuration file management module, the online remapping module, and the verification and error prevention module constitute an online processing stage for performing rapid geometric correction on the real-time acquired images.

[0133] In some specific application scenarios, taking the imaging correction of a visual-tactile sensor based on 10 markers as an example, a second-order polynomial mapping and LUT online correction scheme is used (hereinafter referred to as the imaging example as object surface 3). The specific implementation steps of the technical solution of this invention are as follows: (1) Establishing an ideal imaging reference: Define the ideal camera pose, and obtain the three-dimensional reference coordinates of 10 markers through an image measuring instrument. The main focus is on selecting marker points with good consistency in the lower half of the image. Their 3D coordinates (X, Y) are obtained through the image measuring instrument, and the Z coordinate is obtained through interpolation using a surface model. In this embodiment, the consistency judgment may include the following steps: First, for each frame of the corrected image, calculate its position error using a preset reference marker template, and compare it with the reference position obtained from offline calibration.

[0134] (2) Camera parameter settings: sensor size 3.674×2.760 mm, resolution 3264×2448, focal length 1.790 mm, distortion parameters were obtained through pre-calibration.

[0135] (3) Offline calibration configuration generation: After acquiring the calibration frame, dense sampling (N≈45000 sampling points) is performed on the sensor surface. After distortion correction and PnP pose estimation, a second-order polynomial is used to fit the geometric mapping relationship to generate a pixel-by-pixel sampling coordinate table map_x / map_y (float32 format), and the baseline marker / contour data and threshold parameters are saved. In this embodiment, spatial statistical analysis can also be optionally performed on the fitting residuals: when it is detected that the residuals in some local areas are significantly higher than the global mean, the dense sampling and local refitting of the area are automatically triggered, thereby further compressing the worst-case RMSE without significantly increasing the overall sample size.

[0136] (4) Online correction execution: The configuration file is loaded at runtime, and geometric remapping (LUT lookup) is performed on each frame, which takes about 1.2ms, meeting the millisecond-level real-time requirements.

[0137] like Figure 5 The image shown is a comparison of the correction effect of object surface 3 in the simulation. The left side is the ideal image, the middle side is the uncorrected non-ideal image (RMSE=342.362 pixels), and the right side is the image after correction by the method of this invention (RMSE=1.786 pixels).

[0138] In specific application scenarios, the effectiveness and reliability of the method of this invention in real-world scenarios with distortion are verified through the complete configuration generation and online usage process of the foolproof verification mechanism. The specific implementation steps are as follows: (1) Initial configuration generation: Acquire 1 frame of correction image, detect 100 general markers and contours, and select 10 specified reference markers as PnP inputs; after distortion removal, PnP pose estimation and polynomial fitting, generate correction configuration file.

[0139] (2) Benchmark Feature Extraction: Secondary detection is performed on the generated corrected image to extract marker coordinates and contour features, which are then saved as a benchmark file for subsequent verification. For example... Figure 6 As shown, this is a schematic diagram of the scatter points and clustering circles of the intact hole centers in the template image, demonstrating the spatial distribution characteristics of the baseline marker points.

[0140] (3) Online operation and verification: Upon each startup, the configuration is loaded, a calibration is performed, and the marker / contour is detected and compared with the baseline. Contour consistency verification is as follows: Figure 7 As shown, the correction effect is evaluated by calculating the difference between the pallet profile participating in the averaging and the average reference profile. In some implementations, online verification can also introduce an adaptive threshold based on historical distribution: the system continuously records the RMSE and profile error distribution obtained from multiple verifications during normal operation, and dynamically adjusts the judgment threshold based on quantiles (e.g., the 95th percentile), thereby maintaining high sensitivity and low false alarm rate under slowly changing conditions such as ambient light and minor assembly loosening.

[0141] (4) Mistake prevention mechanism: If the verification passes, the configuration file is used for real-time correction; if the verification fails, the system prompts for regeneration of the correction configuration file.

[0142] In summary, the purpose of this invention is to provide a low-latency imaging geometry correction method and system based on markers for visual-tactile sensors, in order to solve the problem of inconsistency in imaging geometry caused by pose deviations in different machine assembly and lens distortion in the prior art.

[0143] Specifically, the present invention aims to: 1) In the presence of camera-sensor assembly errors and individual structural differences, non-ideal imaging is uniformly mapped to a preset ideal imaging benchmark to achieve pixel geometric consistency between different emulators; 2) Integrate lens distortion compensation and geometric errors caused by pose deviation into the same correction framework to improve correction accuracy and applicability; 3) By generating pixel-by-pixel remapping data structures offline (such as inverse mapping tables map_x / map_y), correction can be completed online only by looking up the table and interpolating, thus meeting the low-latency real-time processing requirements of visual and tactile perception. 4) Provides an online consistency verification and error prevention mechanism for calibration configuration files. When a marker or contour deviation is detected to exceed the threshold, it will automatically prompt that the configuration has failed and trigger a regeneration or rollback strategy to avoid systemic errors caused by long-term erroneous calibration.

[0144] Compared with the prior art, the present invention has at least the following beneficial effects: 1. Significantly improved prototype consistency: The average diameter of marker clustering circles was reduced from approximately 45.120 pixels to approximately 4.56 pixels. The contour difference of prototype images was reduced from menDis=23 to 4.

[0145] 2. Significantly reduce simulation consistency error: In both simulation and actual measurement, the RMSE decreases significantly before and after correction (for example, the simulation error drops from hundreds of pixels to about ten pixels; the error increases significantly when distortion is ignored, but recovers to a low RMSE after considering distortion).

[0146] 3. Low latency: By fixing the geometric mapping offline and performing only table lookup and color interpolation online, the online correction time can be reduced to the millisecond level (1.2ms).

[0147] 4. Reliable error prevention: By verifying the consistency between the calibrated marker and the outline, it automatically identifies configuration errors, lens replacements, assembly abnormalities, and other situations, avoiding long-term operation of erroneous calibration.

[0148] like Figure 8 As shown, this embodiment of the invention also provides an online imaging correction device 900, which can implement the above-described method. This device may include: The first module 901 is used to acquire target data; wherein, the target data includes an ideal imaging reference and a reference correction image of the visual-tactile sensor, and the ideal imaging reference includes the ideal camera pose and the three-dimensional reference coordinates of several marked points on the visual-tactile sensor; The second module 902 is used to acquire the first pixel coordinates of the marked points in the reference correction image and the first contour point set of the visual-touch sensor. The third module 903 is used to solve for the non-ideal camera pose based on the first pixel coordinates and the three-dimensional reference coordinates. The fourth module 904 is used to construct pixel-to-pixel samples based on the three-dimensional sampling points on the visual-tactile sensor, using non-ideal camera poses and ideal camera poses; The fifth module 905 is used to fit a geometric mapping function from non-ideal imaging to ideal imaging based on pixel-corresponding samples, and to generate a remapping data structure based on the geometric mapping function. The sixth module 906 is used to perform online correction on the real-time original image using the remapping data structure to obtain the real-time corrected image; The seventh module 907 is used to collect the second pixel coordinates of the marked points in the real-time correction image and the second contour point set of the visual-tactile sensor. The correction error is obtained by combining the first pixel coordinates and the first contour point set. If the correction error does not meet the preset conditions, the remapping data structure is updated.

[0149] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0150] This invention also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0151] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0152] like Figure 9 As shown, Figure 9 The hardware structure of an electronic device 1000 according to another embodiment is illustrated. The electronic device 1000 includes: The processor 1001 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (aSIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention. The memory 1002 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RaM). The memory 1002 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called and executed by the processor 1001. Input / output interface 1003 is used to implement information input and output; The communication interface 1004 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1005 transmits information between various components of the device (e.g., processor 1001, memory 1002, input / output interface 1003, and communication interface 1004); The processor 1001, memory 1002, input / output interface 1003 and communication interface 1004 are connected to each other within the device via bus 1005.

[0153] The electronic device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0154] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0155] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0156] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0157] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0158] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0159] This invention provides an online imaging correction method, apparatus, electronic device, storage medium, and program product. It acquires target data, including an ideal imaging reference and a reference correction image of a visual-tactile sensor. The ideal imaging reference includes an ideal camera pose and three-dimensional reference coordinates of several marker points on the visual-tactile sensor. The method involves: acquiring the first pixel coordinates of the marker points in the reference correction image and the first contour point set of the visual-tactile sensor; solving for a non-ideal camera pose based on the first pixel coordinates and the three-dimensional reference coordinates; constructing pixel-corresponding samples using the non-ideal and ideal camera poses based on the three-dimensional sampling points on the visual-tactile sensor; fitting a geometric mapping function from non-ideal to ideal imaging based on the pixel-corresponding samples; generating a remapping data structure based on the geometric mapping function; performing online correction on the real-time original image using the remapping data structure to obtain a real-time corrected image; acquiring the second pixel coordinates of the marker points in the real-time corrected image and the second contour point set of the visual-tactile sensor, and statistically calculating the correction error by combining the first pixel coordinates and the first contour point set; updating the remapping data structure if the correction error does not meet preset conditions. This invention, by pre-constructing a geometric mapping function from non-ideal imaging to ideal imaging and generating a remapping data structure, can assist in the online and rapid correction of real-time images, thereby significantly reducing computational overhead and latency and meeting the real-time requirements of visual and tactile perception. Furthermore, by using the statistical correction error of marker points and contour point sets to determine the updated mapping structure, this invention can adaptively compensate for structural deviations between different assembly batches or individual sensors, avoiding manual periodic calibration and improving the model's generalization ability and tactile solution accuracy.

[0160] The embodiments described in this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0161] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present invention, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0162] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0163] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0164] The preferred embodiments of the present invention have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of the present invention should be within the scope of the claims of the present invention.

Claims

1. An online imaging correction method, characterized in that, The method includes the following steps: Acquire target data; wherein, the target data includes an ideal imaging reference and a reference correction image of the visual-tactile sensor, the ideal imaging reference including an ideal camera pose and three-dimensional reference coordinates of several marked points on the visual-tactile sensor; Acquire the first pixel coordinates of the marked points in the reference correction image and the first contour point set of the visual-tactile sensor; The non-ideal camera pose is obtained based on the first pixel coordinates and the three-dimensional reference coordinates, including the following steps: The first marked point in the reference correction image is taken as the target point; Based on the three-dimensional reference coordinates of the target point, the target point is subjected to perspective projection using a preset projection function to obtain the two-dimensional pixel coordinates of the target point; For the target point, the difference between the vector of the two-dimensional pixel coordinates and the vector of the first pixel coordinates is squared using the L2 norm to obtain the quantized value of the target point; Take the next marker point in the reference correction image as the target point, return to execute the step of performing perspective projection on the target point based on the three-dimensional reference coordinates of the target point through a preset projection function, until all the marker points in the reference correction image have been traversed; The non-ideal camera pose is obtained by minimizing the sum of all the quantized values. Based on the three-dimensional sampling points on the visual-tactile sensor, pixel-corresponding samples are constructed using the non-ideal camera pose and the ideal camera pose; Based on the pixel-corresponding samples, a geometric mapping function from non-ideal imaging to ideal imaging is fitted, and a remapping data structure is generated according to the geometric mapping function. The real-time original image is corrected online using the remapping data structure to obtain a real-time corrected image; The second pixel coordinates of the marked points in the real-time corrected image and the second contour point set of the visual-tactile sensor are collected. The correction error is obtained by combining the first pixel coordinates and the first contour point set. If the correction error does not meet the preset conditions, the remapping data structure is updated.

2. The method according to claim 1, characterized in that, The acquisition of the first pixel coordinates of the marked points in the reference correction image and the first contour point set of the visual-tactile sensor includes the following steps: The reference correction image is segmented based on a preset intensity threshold to extract candidate regions for marker points; The reference correction image is binarized based on the candidate regions of the marker points to obtain a binary mask; Connectivity analysis is performed on the binary mask to obtain the connected components of the candidate marker points; wherein, the connected components include the pixel set of the candidate marker points; The geometric center of the connected region is obtained based on the quantization of the pixel set, and the first pixel coordinates of the marker point are obtained; wherein, the marker point represents a set of center points selected from the set of geometric centers, the visual tactile sensor is pre-configured with the geometric consistency of each region, and the probability of the geometric center being selected as the marker point is positively correlated with the level of geometric consistency of the region where the geometric center is located; Edge detection based on connected component analysis is performed on the reference correction image to obtain the first set of contour points outside the effective working area of ​​the visual-tactile sensor.

3. The method according to claim 1, characterized in that, The method further includes the following steps: Before the step of acquiring the first pixel coordinates of the marker point in the reference correction image, the reference correction image is subjected to distortion removal processing based on pre-calibrated distortion parameters; Alternatively, the first pixel coordinates can be distorted based on the distortion parameters.

4. The method according to claim 1, characterized in that, The process of constructing pixel-corresponding samples using the non-ideal camera pose and the ideal camera pose includes the following steps: The three-dimensional sampling points are projected onto the image plane using the non-ideal camera pose to obtain the coordinates of the fourth pixel. The three-dimensional sampling points are projected onto the image plane using the ideal camera pose to obtain the coordinates of the fifth pixel. The coordinates of the fourth pixel and the fifth pixel corresponding to the same three-dimensional sampling point are associated as a set of corresponding pixel samples.

5. The method according to claim 1, characterized in that, The pixel-corresponding samples include the fourth pixel coordinates corresponding to the non-ideal camera pose and the fifth pixel coordinates corresponding to the ideal camera pose. The step of fitting a geometric mapping function from non-ideal imaging to ideal imaging based on the pixel-corresponding samples, and generating a remapping data structure according to the geometric mapping function, includes the following steps: A two-dimensional polynomial is constructed based on the coordinates of the fourth pixel and the fifth pixel. A least-squares optimization problem is constructed based on multiple pixel-corresponding samples, and then the fitting coefficients of the two-dimensional polynomial are obtained by solving the problem to obtain the geometric mapping function. The geometric mapping function is forward-mapped and inverted to construct the remapped data structure. The method further includes the following steps: Based on the cache structure and instruction set of the target hardware, the remapping data structure adopts mixed precision encoding and pre-interpolated weight storage; wherein, the sampling coordinates in the remapping data structure are stored using fixed-point or half-precision floating-point compression, and the remapping data structure is associated with bilinear interpolation weight coefficients.

6. The method according to claim 1, characterized in that, The correction error includes root mean square error, maximum contour error, and average contour error. The correction error is obtained by statistics. If the correction error does not meet the preset conditions, the remapping data structure is updated, including the following steps: The root mean square error is constructed based on the second pixel coordinates and the first pixel coordinates through root mean square operation. The nearest distance from each reference contour point in the first contour point set to the second contour point set is obtained through nearest neighbor distance statistical processing. The largest of the nearest distances is taken as the maximum contour error; The average contour error is obtained by averaging the nearest distances corresponding to all the reference contour points in the first contour point set. When the root mean square error is less than or equal to the first threshold, the average profile error is less than or equal to the second threshold, and the maximum profile error is less than or equal to the third threshold, the correction error is determined to meet the preset condition. If the correction error does not meet the preset condition, return to the step of obtaining the target data to reconstruct the remapping data structure; Alternatively, if the correction error does not meet the preset conditions, the remapped data structure will be rolled back to a previously valid version.

7. An online imaging correction device, characterized in that, The device includes: The first module is used to acquire target data; wherein, the target data includes an ideal imaging reference and a reference correction image of the visual-tactile sensor, and the ideal imaging reference includes an ideal camera pose and three-dimensional reference coordinates of several marked points on the visual-tactile sensor; The second module is used to acquire the first pixel coordinates of the marker points in the reference correction image and the first contour point set of the visual-tactile sensor. The third module is used to solve for the non-ideal camera pose based on the first pixel coordinates and the three-dimensional reference coordinates, and specifically includes the following operations: The first marked point in the reference correction image is taken as the target point; Based on the three-dimensional reference coordinates of the target point, the target point is subjected to perspective projection using a preset projection function to obtain the two-dimensional pixel coordinates of the target point; For the target point, the difference between the vector of the two-dimensional pixel coordinates and the vector of the first pixel coordinates is squared using the L2 norm to obtain the quantized value of the target point; Take the next marker point in the reference correction image as the target point, return to execute the step of performing perspective projection on the target point based on the three-dimensional reference coordinates of the target point through a preset projection function, until all the marker points in the reference correction image have been traversed; The non-ideal camera pose is obtained by minimizing the sum of all the quantized values. The fourth module is used to construct pixel-corresponding samples based on the three-dimensional sampling points on the visual-tactile sensor, using the non-ideal camera pose and the ideal camera pose; The fifth module is used to fit a geometric mapping function from non-ideal imaging to ideal imaging based on the pixel corresponding samples, and to generate a remapping data structure according to the geometric mapping function. The sixth module is used to perform online correction on the real-time original image using the remapping data structure to obtain a real-time corrected image; The seventh module is used to collect the second pixel coordinates of the marked points in the real-time corrected image and the second contour point set of the visual-tactile sensor, and to statistically obtain the correction error by combining the first pixel coordinates and the first contour point set. If the correction error does not meet the preset conditions, the remapping data structure is updated.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 6.

9. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images