Multi-sensor data fusion positioning method for portable printing and copying machine

Abstract

The invention provides a multi-sensor data fusion positioning method for a portable printing and copying machine, which comprises the following steps of: 1, arranging a contact type image sensor at the bottom of the portable printing and copying machine, and acquiring a surface image of a scanning or printing medium; 2, at least two laser mouse sensors are arranged at the bottom of the portable printing and copying machine, or at least one inertial measurement unit IMU is arranged on the portable printing and copying machine; 3, at least one printing nozzle is arranged at the bottom of the portable printing and copying machine; 4, acquiring parameters between the contact image sensor arranged in the step 1 and the laser mouse sensor or IMU arranged in the step 2; step 5, acquiring space parameters between the contact type image sensor arranged in the step 1 and the printing nozzle arranged in the step 3; and 6, in an image scanning or printing task, positioning the portable printing and copying machine by utilizing the parameters among the sensors obtained in the step 4 and fusing the measurement data of the sensors arranged in the steps 1 and 2.

Description

Technical field

[0001] The present invention proposes a multi-sensor data fusion positioning method for a portable printer / copier, and in particular relates to a method for information fusion and positioning using a visual sensor, a laser mouse sensor, and an IMU, and belongs to the fields of printer / copier, artificial intelligence, or information technology. [Background Technology]

[0002] Portable printers and copiers have enormous market potential due to their flexibility, versatility, and environmental friendliness. While portable printers based on vision positioning can scan and print images of any size, their positioning accuracy on the surface of the scanned and printed media lags behind that of traditional printers, limiting their further adoption.

[0003] Patent application CN117549678A proposes a method for improving the image stitching and positioning accuracy of portable printers using linear image sensors. This method overcomes image stitching and positioning accuracy issues in vision-based printers caused by local motion measurement errors and sparse image features. However, this method still exhibits certain positioning errors when the printer body is not constrained by linear guides or wheels, or when the printer body exhibits significant vibration or rotation.

[0004] Therefore, this patent application proposes a method for improving the positioning accuracy of a portable printer and copier by integrating visual sensors, laser mouse sensors, and IMU data without using linear guides or wheel structures to constrain the movement direction of the printer body, thereby overcoming the problem of increased errors caused by the shaking and rotation of the printer body during image scanning and printing. [Summary of the invention]

[0005] The present invention proposes a multi-sensor data fusion positioning method for a portable printer and copier, comprising the following steps:

[0006] Step 1: A contact image sensor is arranged at the bottom of the portable printer / copier to obtain an image of the surface of a scanned or printed medium.

[0007] Step 2: at least two laser mouse sensors (or photoelectric mouse sensors, blue light mouse sensors) are arranged at the bottom of the portable printer / copier, or at least one IMU (Inertial Measurement Unit) is arranged on the portable printer / copier.

[0008] Step 3: at least one print head is arranged at the bottom of the portable printer / copier for printing an image on a surface of a print medium during a printing task.

[0009] Step 4: Obtain parameters between the contact image sensor set in step 1 and the laser mouse sensor or IMU set in step 2.

[0010] Step 5: Obtain the spatial parameters between the contact image sensor set in step 1 and the print head set in step 3.

[0011] Step 6: In the image scanning or printing task, the inter-sensor parameters obtained in step 4 are used and the measurement data of the sensors set in steps 1 and 2 are integrated to locate the portable printer / copier.

[0012] Step 7: In the image scanning task, based on the positioning results obtained in step 6, the scanned image is dedistorted and spliced ​​to obtain a scanned image with higher accuracy.

[0013] Step 8: During the printing task, the position of the print head is calculated in real time based on the online positioning result obtained in step 6 and the spatial parameters between the contact image sensor and the print head obtained in step 5, thereby obtaining the data to be printed.

[0014] Assume that the sensors installed in the portable printer and copier system in steps 1 and 2 include 1 contact image sensor, 2 laser mouse sensors, and 1 IMU. Establish the sensor and medium coordinate system as follows: Figure 1 As shown in the figure, the contact image sensor coordinate system has both 2D and 3D forms. Its y-axis represents the pixel distribution direction of the contact image sensor, while the x-axis is perpendicular to the y-axis and, according to the right-hand rule, the z-axis points perpendicular to the media plane. The 2D form only has the x and y axes. The media coordinate system is established based on the printed or scanned media plane and also has both 2D and 3D forms depending on computational requirements. The coordinate systems of IMU and laser mouse sensors are established based on the measurement characteristics of the sensors themselves.

[0015] In order to improve the accuracy of data fusion positioning of portable printers and copiers, it is necessary to accurately obtain the parameters between sensors. The parameters between sensors include the spatial parameters of the laser mouse sensor o (o = 1, 2) and the contact image sensor. (2D rigid body transformation), spatial parameters of IMU and contact image sensor (3D rigid body transformation).

[0016] The inter-sensor parameters are obtained by calibrating the structure parameters of the portable printer or the inter-sensor parameters. Inter-sensor parameter calibration is a data collection and processing process for the purpose of accurately estimating the inter-sensor parameters. In step 4, a method for calibrating the inter-sensor parameters further includes:

[0017] Step 4.1: Preset an image or pattern with known size and content on the scanning medium plane.

[0018] Step 4.2: Place the portable printer on the medium plane and perform a translation and rotation compound motion within the plane. An example of a motion trajectory of the contact image sensor in this process is as follows: Figure 2 shown.

[0019] Step 4.3: Synchronously collect the measurement data and corresponding timestamps of all sensors within a period of time [t1, t2] in step 4.2.

[0020] Step 4.4: Construct and solve the following optimization problem to obtain the 2D motion trajectory of the contact image sensor and the estimation of the parameters between different sensors:

[0021]

[0022] Among them, w is the medium coordinate system, which has two forms: 2D and 3D. Traj is the 2D motion trajectory of the contact image sensor in the 2D medium coordinate system. are the spatial parameters of IMU and contact image sensor, are the spatial parameters of the laser mouse sensor and the contact image sensor, st.‖g w ‖=9.8m / s 2 is the value of gravitational acceleration in the 3D medium coordinate system, is the acceleration bias of the IMU, is the angular velocity bias of the IMU, (·) * represents the solution of the optimization problem, M cis is the measurement set of the contact image sensor, f cis is the cost function constructed based on the contact image sensor measurement model, Σ cis is the covariance matrix of the contact image sensor measurement error, is the set of measurements of the laser mouse o, f opt is the cost function constructed based on the laser mouse sensor measurement model, Σ opt is the covariance matrix of the laser mouse sensor measurement error, M acc is the set of acceleration measurements from the IMU, f acc is the cost function constructed based on the IMU acceleration measurement model, Σ acc is the covariance matrix of IMU acceleration measurement error, M gyr is the set of angular velocity measurements from the IMU, f gyr is the cost function constructed based on the IMU angular velocity measurement model, Σ gyr is the covariance matrix of the IMU angular velocity measurement error.

[0023] During printing, according to the positioning status of the contact image sensor The spatial parameters of the contact image sensor and the print head are used to calculate the position of the print head in the medium coordinate system w, thereby obtaining the printed content in real time. The spatial parameters of the contact image sensor and the print head refer to the 2D coordinates of the print head endpoints e (each nozzle has at least two endpoints) in the contact image sensor coordinate system.

[0024] During the printing process, the 2D position of the print head in the medium coordinate system w at time t is calculated based on the real-time positioning results of the printer, the contact image sensor and the spatial parameters of the print head. The method is to use a contact image sensor to locate the state Coordinate transformation of the spatial parameters of the contact image sensor and the print head

[0025] In the method for calculating the print head position based on the printer positioning result, the spatial parameters of the contact image sensor and the print head are replaced with the spatial parameters of the laser mouse sensor or IMU and the print head, and then equivalent calculation is achieved based on the spatial parameters of the laser mouse sensor or IMU and the touch image sensor.

[0026] The spatial parameters of the contact image sensor and the print head are obtained by calibrating the assembly structural parameters of the portable printer or the spatial parameters of the contact image sensor and the print head. In step 5, the method for calibrating the spatial parameters of the contact image sensor and the print head further includes:

[0027] Step 5.1: Preset a pattern or marking line on the surface of the printing medium for contact image sensor positioning.

[0028] Step 5.2: Place the portable printer on the print medium, move the printer to collect data from the contact image sensor and motion tracking sensor (laser mouse or IMU), trigger the print head to print ink lines at intervals, and record the printing time t for L ink lines. l (l=1,2,...,L).

[0029] Step 5.3: For a certain endpoint e of the print head, measure the position p of the corresponding endpoint of the printed ink line in the medium coordinate system w l (l=1,2,...,L).

[0030] Step 5.4: Use the pattern or mark preset in step 5.1, the sensor data collected in step 5.2, and the known inter-sensor parameters to locate the contact image sensor and obtain its 2D positioning state at the time of ink line printing.

[0031] Step 5.5: Calculate the 2D coordinates of the print head endpoint e in the contact image sensor coordinate system

[0032]

[0033] 2D motion tracking of printers is performed by fusing multiple sensor data during image scanning or printing tasks. 2D motion tracking is the estimation of the relative motion of a contact image sensor over a period of time [t1, t2]. Relative motion refers to the transformation between the sensor coordinate system of the contact image sensor at time t1 and the sensor coordinate system at time t2. like Figure 3 shown.

[0034] like Figure 4 As shown, in step 6, at least two laser mouse sensors are used to perform 2D motion tracking of a portable printer / copier with three degrees of freedom in a plane. Taking two laser mouse sensors as an example, the process further includes:

[0035] Step 6.1.1: Get the spatial parameters of the laser mouse sensor o and the contact image sensor. The values ​​are:

[0036]

[0037] Among them, (px o ,py o ) is the translation component of the spatial parameters of the laser mouse sensor o and the contact image sensor, θ o is the rotation component.

[0038] Step 6.1.2: Get the measurement value (Δx) of the laser mouse sensor o during the time period [t1, t2] for motion tracking o ,Δy o ), o = 1, 2, are the displacements of the laser mouse in the directions of its two coordinate axes during this period.

[0039] Step 6.1.3: Assume that the motion of the contact image sensor to be solved in the time period [t1, t2] is Among them, (tx,ty) is the translation component of the motion, is the rotational component of the motion; according to the measurement model of the laser mouse sensor, the spatial parameter relationship between the laser mouse sensor and the contact image sensor is:

[0040]

[0041] Arranged:

[0042]

[0043] Step 6.1.4: Combining the above equations based on the measurements of each laser mouse sensor, we get:

[0044]

[0045] Solve the above equation or find the least squares solution of the above equation to obtain tx, ty, The estimated value of , thus obtaining the 2D motion of the contact image sensor in the time period [t1, t2]

[0046] like Figure 5 As shown in the figure, an IMU is used to track the 2D motion of a portable printer. Although the motion and measurement of the contact image sensor are only performed on the 2D plane of the printed scanning medium, the motion measurement of the IMU is in 3D space. In order to facilitate the solution of the relationship between the two sensor coordinate systems, the contact image sensor coordinate system is expanded into a 3D coordinate system according to the right-hand rule. Let the spatial parameters of the IMU and the contact image sensor be The method for tracking the 2D motion of the portable printer on the medium surface using IMU data in step 6 further includes:

[0047] Step 6.2.1: Static initialization of IMU: estimate gravity acceleration, IMU acceleration bias, IMU angular velocity bias, IMU initial position, and IMU initial velocity. Place the printer on the printing medium surface, collect IMU acceleration and angular velocity measurements for a short period of time (e.g., 3 seconds), and calculate the average value to obtain At this time, the initial position of the IMU is the unit matrix, and the velocity value of the IMU is (0,0,0) T , the angular velocity bias of IMU is The acceleration due to gravity is The acceleration bias of the IMU is

[0048] Step 6.2.2: When the gravity acceleration, IMU acceleration bias, IMU angular velocity bias, IMU pose and velocity values ​​at time t1 are known, integrate the IMU acceleration and angular velocity measurements over a period of time [t1, t2] to obtain the IMU pose and velocity values ​​at time t2, and then obtain the relative motion of the IMU during this period of time. It is the transformation relationship of the 3D rigid body coordinate system.

[0049] Step 6.2.3: Using the parameters between the IMU and the contact image sensor, obtain the relative motion of the contact image sensor in the time period [t1, t2] neglect The translation component along the z axis and the rotation components along the x and y axes are used to obtain the 2D relative motion of the contact image sensor on the medium plane within the time period [t1, t2].

[0050] like Figure 6 As shown in the figure, an IMU and a laser mouse sensor are used to track the 2D motion of a portable printer on the surface of a medium. Let the rotation component of the spatial parameters of the IMU and the contact image sensor be is the rotation component of the 3D rigid body coordinate system transformation relationship; the spatial parameters of the laser mouse sensor and the contact image sensor are is the transformation relationship of the 2D rigid body coordinate system, satisfying:

[0051]

[0052] Among them, (px,py) is the translation component of the spatial parameters of the laser mouse sensor and the contact image sensor, and θ is the rotation component.

[0053] In step 6, the IMU measurements and the laser mouse sensor measurements (Δx, Δy) over a period of time [t1, t2] are used to calculate the relative motion of the contact image sensor during this period. (where (tx, ty) is relative motion The translation component of For relative motion The method further comprises:

[0054] Step 6.3.1: Estimate the IMU angular velocity bias using the static method.

[0055] Step 6.3.2: Using the IMU angular velocity bias, integrate the IMU angular velocity measurements during the time period [t1, t2] to obtain the IMU's rotational motion during this period. It is the rotation component of the 3D rigid body coordinate system transformation relationship.

[0056] Step 6.3.3: Use the rotational components of the IMU and contact image sensor spatial parameters to obtain the contact image sensor's rotational motion during this period. Only the rotation component along the z-axis is retained to obtain the rotation component of the contact image sensor movement

[0057] Step 6.3.4: Using the measurements of the laser mouse sensor and the spatial parameters of the laser mouse sensor and the contact image sensor, we can obtain:

[0058]

[0059] Thus, the relative motion of the touch image sensor in the time period [t1, t2] is obtained

[0060] When using a portable printer to print multiple lines of content, a Kalman filter, an extended Kalman filter, or an error-state Kalman filter is constructed based on sensor data to online estimate the 2D positioning state of the portable printer on the printing medium surface.

[0061] The method for estimating the contact image sensor state and performing online positioning of the portable printer / copier using data from at least two laser mouse sensors and one contact image sensor using an extended Kalman filter method in step 6 further includes:

[0062] Step 6.4.1: Define and initialize the contact image sensor state to be estimated in, is the translation component, is the rotation component, and its covariance matrix

[0063] Step 6.4.2: Prediction step. Using [t i-1 ,t i ] The measured values ​​of at least two laser mouse sensors and the inter-sensor parameters are calculated for the relative motion of the contact image sensor in the time period (where (Δtx i ,Δty i ) is the translation component of the motion measurement value, is the rotational component of the motion measurement), and the covariance matrix of the motion equation noise is assumed to be a known constant Σ pre , for contact image sensors at t i The state at the moment is predicted

[0064] is the state prediction value;

[0065] is the Jacobian matrix of the equation of motion for the state;

[0066] is the covariance matrix of the predicted state;

[0067] Step 6.4.3: Update step. Using the contact image sensor at t i Constructing observation equations based on time data Where h is the observation equation function, the specific form depends on the implementation, is the independent variable of the observation equation, μ is the noise of the observation equation and its covariance matrix Σ is known μ , perform measurement update:

[0068] is the Jacobian matrix of the observation equation to the state;

[0069] is the Kalman gain;

[0070] Update the value for the state,

[0071] in, is the translation component, is the rotational component;

[0072] is the covariance matrix of the updated state;

[0073] Among them, I is the unit matrix;

[0074] Step 6.4.4: Using [t i ,t] time period laser mouse sensor extension interpolation measurement value calculation time period contact image sensor relative motion Among them, (Δtx, Δty) is the relative motion translation component, is the relative motion rotation component, and then the positioning state of the contact image sensor at any time t is calculated have:

[0075]

[0076] Among them, (tx t ,ty t ) is the translation component of the positioning state, The rotation component of the positioning state.

[0077] The method of using an error state Kalman filter method in step 6 to estimate the IMU state using an IMU and contact image sensor data, and then calculating the contact image sensor positioning state, further includes:

[0078] Step 6.5.1: Use the static initialization method to estimate the gravity acceleration, IMU acceleration bias, IMU angular velocity bias, and IMU initial velocity value, and calculate the contact image sensor positioning state on the printing medium plane at that moment. The spatial parameters of the IMU and the contact image sensor are used to calculate the initial IMU pose, and the estimated gravitational acceleration is converted to the medium coordinate system w.

[0079] Step 6.5.2: Define and initialize the IMU nominal state in, represents the position of the IMU in the medium coordinate system w, set to the estimated value obtained according to step 6.5.1, represents the velocity of the IMU in the medium coordinate system w, is the unit quaternion representing the rotation of the IMU in the medium coordinate system w, set to the estimated value obtained according to step 6.5.1, represents the IMU acceleration bias, set to the estimated value obtained in step 6.5.1, represents the IMU angular velocity bias, set to the estimated value obtained in step 6.5.1, represents the gravitational acceleration in the medium coordinate system and is set to the estimated value obtained in step 6.5.1;

[0080] Defining Error States in, represents the IMU position error, represents the IMU velocity error, is the IMU rotation error expressed as a rotation vector, Indicates the IMU acceleration bias error, Indicates the IMU angular velocity bias error, Represents the gravity acceleration error; sets the error state covariance matrix

[0081] Step 6.5.3: Use t i The IMU measurement is performed at the time of prediction. Let the acquisition time interval of IMU measurement be Δt, t i The measured value at time is in, is the acceleration measurement value, is the angular velocity measurement value. Calculate t i Nominal state at the moment and the error state covariance matrix

[0082]

[0083] in, Indicates t i The position of the IMU in the medium coordinate system w at the moment, Indicates t i The velocity of the IMU in the medium coordinate system w at the moment, Indicates t i The rotation of the IMU in the medium coordinate system w at time t, Indicates t iIMU acceleration zero bias at this moment, Indicates t i IMU angular velocity zero bias at this moment, Indicates t i The gravitational acceleration in the medium coordinate system at the moment, yes The corresponding rotation matrix, q{} represents the conversion of the rotation vector into a unit quaternion. Represents quaternion multiplication.

[0084]

[0085] in,

[0086] is the Jacobian matrix of the error state motion equation with respect to the error state;

[0087] is the Jacobian matrix of the error state motion equation with respect to noise;

[0088] is the covariance matrix of the noise;

[0089] Among them, R{} represents the conversion of the rotation vector into a rotation matrix, [] × Indicates converting a 3D vector into an antisymmetric matrix. is the acceleration measurement noise parameter of the IMU, is the angular velocity measurement noise parameter of the IMU, is the acceleration random walk noise parameter of IMU, It is the angular velocity random walk noise parameter of the IMU. All noise parameters of the IMU are obtained from the IMU technical manual or through calibration methods.

[0090] Step 6.5.4: Using t i The contact image sensor measures the error state at every moment. Assume that at t i The measurement is built based on the contact image sensor data at all times (If there is no contact image sensor measurement data at this moment, skip this step and go to the next moment), where: are the spatial parameters of the IMU and contact image sensor and are known, h is the observation equation function, the specific form depends on the implementation, η is the covariance matrix Σ η Gaussian measurement noise.

[0091] Calculate the nominal state error based on the measurement and its covariance matrix

[0092] is the Kalman gain;

[0093]

[0094] in, Indicates t i IMU position error at time t, Indicates t i IMU velocity error at time, is represented by the rotation vector t i IMU rotation error at time t, Indicates t i IMU acceleration bias error at time t, Indicates t i IMU angular velocity zero bias error at this moment, Indicates t i Gravity acceleration error at this moment.

[0095] is the Jacobian matrix of the observation equation with respect to the error state;

[0096]

[0097]

[0098] is the Jacobian matrix of the rotation component of the nominal state to the rotation error;

[0099] in, It needs to be calculated according to the specific measurement equation, [q w ,q x ,q y ,q z ] T for

[0100] Then update the nominal state and the error state covariance matrix

[0101]

[0102] The Jacobian matrix of the error state reset equation for the error state;

[0103] Step 6.5.5: Using t i The nominal state of IMU at the moment [t i ,t] time period IMU measurement is integrated to obtain the IMU pose at time t (p t ,q t ), and then use the spatial parameters of the IMU and the contact image sensor to obtain the positioning state of the contact image sensor in the medium coordinate system w at time t

[0104] In single-pass image scanning on a portable printer / copier, the 2D motion tracking results of the contact image sensor are obtained using sensor measurement data and inter-sensor parameter calculations, and the scanned image is then dedistorted to obtain a single-pass scanned image with higher accuracy.

[0105] In step 7, assume that a total of N frames of images are collected during a single image scan, each frame corresponds to a pixel value vector of length h, and the acquisition time of the kth (k = 1, 2, ..., N) frame image is t k 、Data is v k , the calculated 2D motion tracking result of the contact image sensor is The method for performing single-pass scan image dedistortion further comprises:

[0106] Step 7.1: Initialize the stitched image I and set the image width to x max -x min +1, image height y max -y min +1, among which, (·) d , (d=1,2) means taking the d-th dimension coordinate value of the vector, Indicates 2D coordinate transformation using a coordinate transformation matrix.

[0107] Step 7.2: Map the k-th frame of the collected image to the spliced ​​image I and assign pixel values ​​I(x kr ,y kr )=v k (r), where k = 1, 2, ..., N, r = 1, 2, ..., h, I(x,y) represents the pixel of the image at coordinate (x,y), v k (r) represents the r-th pixel value of the k-th frame image.

[0108] Step 7.3: Assign values ​​to the unassigned pixels of the stitched image I using an interpolation method to obtain a dedistorted scanned image.

[0109] A portable printer / copier performs multi-pass image scanning. The collected multi-pass sensor data and the spatial parameters between sensors are used to jointly optimize and estimate the multi-pass 2D motion trajectory of the contact image sensor. The multi-pass trajectory is then used to generate a stitched scanned image. The specific stitching method is the same as for single-pass scanning, resulting in a larger image. Assume that the sensors on the portable printer / copier include one contact image sensor, two laser mouse sensors, and one IMU. In a multi-pass scanning task, a total of s passes are scanned. One method for multi-pass motion trajectory estimation is to construct and solve the following optimization problem:

[0110]

[0111] Among them, Traj j (j=1,2,...,s) is the s-pass 2D motion trajectory of the contact image sensor, st.‖g w ‖=9.8m / s 2 is the acceleration due to gravity, is the IMU acceleration bias, is the IMU angular velocity bias, (·) * represents the solution of the optimization problem, are the spatial parameters of the IMU and contact image sensor and are known, are the spatial parameters of the laser mouse sensor and the contact image sensor and are known. M cis It is a set of contact image sensor measurements obtained by stitching the scanned images of adjacent passes. Each measurement includes matching features and matching track numbers (j1, j2). cis is the cost function constructed based on the contact image sensor measurement model, Σ cis is the covariance matrix of the contact image sensor measurement error, is the set of measurements of the laser mouse o at the jth pass, s is the number of trajectories, f opt is the cost function constructed based on the laser mouse sensor measurement model, Σ opt is the covariance matrix of the laser mouse sensor measurement error, is the acceleration measurement set of IMU in the jth pass, f acc is the cost function constructed based on the IMU acceleration measurement model, Σ acc is the covariance matrix of the IMU acceleration measurement error, is the set of angular velocity measurements of the IMU at the jth pass, f gyr is the cost function constructed based on the IMU angular velocity measurement model, Σ gyris the covariance matrix of the IMU angular velocity measurement error.

[0112] Without compromising positioning accuracy, replace the laser mouse sensor with a wheel encoder or stepper motor in steps 1-8 to measure and locate the portable printer. Assume that the wheel displacement measured by the wheel encoder or stepper motor during the time period (t1, t2) is Δx. This is converted into a 2D displacement measurement (Δx, 0), which can be used to replace the laser mouse sensor measurement for motion tracking and positioning.

[0113] In the printing scenario, without affecting the positioning accuracy, remove some or all of the settings and use of the laser mouse sensor or IMU in steps 1-8, and use the contact image sensor to position the image of the printing medium surface.

[0114] Without affecting the positioning accuracy, in steps 1-8, the contact image sensor is replaced with a linear image sensor or an area array image sensor imaging system (including a lens, an area array image sensor chip, an image acquisition circuit, and a light source) to position the portable printer copier.

[0115] In a portable printer / copier, if an area array image sensor imaging system is used, a dedistorted, original-size (or 1:1) image of the scanned or printed medium surface is obtained from the captured single-frame image based on its internal parameters (including focal length, principal point coordinates, distortion parameters) and its spatial relationship with the scanned or printed medium plane (including height, roll angle, and pitch angle), thereby replacing the contact image sensor image for image scanning or print positioning.

[0116] In a portable printer / copier, if an area array image sensor imaging system is used, then in steps 1-8, the setting and use of some or all of the laser mouse sensor or IMU are removed, and the imaging of the scanned or printed medium surface image is positioned by the area array image sensor imaging system.

[0117] In portable printers and copiers, if an area array image sensor imaging system is used, measurement information is extracted from the resulting dedistorted, full-size scans or printed media surface images for positioning. This extracted measurement information is categorized into three types: tracking of visual features between consecutive frames during a single-pass image scan or print; matching of visual features between overlapping frames during different scan passes; and matching of visual features between a single-frame image and the printed media surface image during printing. These visual features include point features, line features, and grayscale features calculated from 2D images, as well as features extracted using neural network methods.

[0118] In portable printers and copiers, if the area array imaging system used uses global shutter exposure, the acquisition time of a single frame image is unique, and the measurement time extracted for the same frame image is also consistent; if rolling shutter exposure is used, the acquisition time of each row of pixels in a single frame image is different. For the same frame image, the measurement time needs to be determined based on the exposure time of the corresponding row, thereby improving measurement accuracy and positioning result precision.

[0119] The multi-sensor data fusion positioning method for portable printers and copiers proposed in this invention has the beneficial effect of improving the positioning accuracy of portable printers and copiers without using linear guides or wheel structures to constrain the movement direction of the printer body. This overcomes the problem of increased positioning errors caused by printer body vibration and rotation during image scanning and printing. Experimental results show that the implementation of this method reduces the average stitching error between adjacent lines in multi-line stitching printing tasks to less than 40 μm. This improves the practicality and flexibility of portable printers and copiers based on vision positioning principles and provides a technical foundation for the large-scale commercial use of this type of equipment.

Brief Description of the Drawings

[0120] Figure 1 It is a schematic diagram of the sensor coordinate system of a portable printer copier.

[0121] Figure 2 This is an example of the motion trajectory of a contact image sensor during parameter calibration.

[0122] Figure 3 It is a schematic diagram of the relative motion of the contact image sensor.

[0123] Figure 4 is a schematic diagram of motion tracking using at least two laser mouse sensors.

[0124] Figure 5 This is a schematic diagram of using an IMU for motion tracking.

[0125] Figure 6 This is a diagram of motion tracking using an IMU and a laser mouse sensor.

[0126] Figure 7 It is a portable printing copier and print media setting diagram. [Specific implementation method]

[0127] In a portable printer and copier hardware implemented based on the method of the present invention, two laser mouse sensors and one contact image sensor are set at the bottom of the portable printer and copier for positioning, and a print head is set for printing; the laser mouse sensor, contact image sensor, and print head are fixedly connected together through the printer body. Preset marking lines are used for positioning on the surface of the printing medium. The marking lines are set as two groups of parallel and equally spaced straight lines within the groups and vertically distributed between the groups. The spacing between the straight lines is 1 inch (600 pixels at 600dpi). The settings of the portable printer and the printing medium are as follows Figure 7 shown.

[0128] To achieve high-precision stitching printing, the spatial parameters between the laser mouse sensor and the contact image sensor, as well as between the print head and the contact image sensor, must be calibrated. During the printing process, positioning is achieved by measuring the laser mouse sensor and imaging the marking line with the contact image sensor. The print head's position in the print medium's coordinate system is then calculated, allowing for real-time adjustments to the print content based on the positioning results.

[0129] Accurate sensor parameter calibration is a prerequisite for high-precision positioning using sensor data fusion. In a portable printer / copier system consisting of a laser mouse sensor, an IMU, and a contact image sensor, a specific sensor parameter calibration method is implemented as follows: the portable printer / copier is placed on a marked print media surface. While moving and rotating the printer / copier, data from all sensors is collected. The laser mouse sensor's measurement frequency is 4000 Hz, the IMU's measurement frequency is 100 Hz, and the contact image sensor's measurement frequency is 1000 Hz. Using the collected data, a joint optimization problem is constructed and solved to obtain the contact image sensor's 2D motion trajectory and inter-sensor parameters. The contact image sensor's 2D motion trajectory is modeled using a 4th-order B-spline curve Traj with three dimensions: two dimensions represent in-plane translation (expressed in pixel coordinates at 600 dpi) and one dimension represents in-plane rotation (expressed in radians). The control node interval of the B-spline curve Traj is 50 ms, and the time range covers the sensor measurement data. In the optimization problem modeling and solution, the B-spline curve is used to calculate the 3D motion state Traj(t) at any time t, and then the position and posture of the contact image sensor at time t are obtained:

[0130] in,(·) d , (d = 1, 2, 3) represents the d-th dimension of the vector; the first-order derivative of the 3D motion state at time t is calculated using the B-spline curve Then the angular velocity of the contact image sensor at time t is obtained Calculate the second-order derivative of the 3D motion state at time t using B-spline curve Then we can get the acceleration of the contact image sensor at time t: angular acceleration The optimization problem constructed for parameter calibration is:

[0131]

[0132] Among them, M cis It is a set of measurements obtained by scanning the marker line using a contact image sensor. Specifically, a measurement m is the image coordinate p at time t. m =[0,y m ] T The image point of the equation is a m x+b m y+c m =0(st.a m 2 +b m 2 =1) On the marking line, the coordinates of the image point in the medium coordinate system are calculated based on the positioning state of the contact image sensor. At this time f cis (m,Traj)=a m x' m +b m y' m +c m Indicates the distance from the image point to the marker line. Σ cis According to the measurement characteristics of the contact image sensor, Σ cis =[4] 1×1 (at 600dpi).

[0133] in, is the set of measurements of the laser mouse sensor 1. Specifically, a measurement is the measurement value (Δx, Δy) of the laser mouse sensor from time t1 to t2. Let is the spatial parameter of the laser mouse sensor 1 and the contact image sensor to be solved, and the relative motion of the contact image sensor in this time period is obtained according to the trajectory calculation. but Σ opt According to the measurement characteristics of the laser mouse sensor, it is selected as (at 600 dpi). The measurement set for Laser Mouse Sensor 2 is similar to that for Laser Mouse Sensor 1.

[0134] Among them, M acc is the set of acceleration measurements of the IMU. The acceleration measurement of the IMU at time t is set up are the spatial parameters of the IMU and contact image sensor to be solved, then where [τ] × Indicates converting the scalar τ into a 2D antisymmetric matrix (·) 1,2 Indicates taking the first 2 dimensions of a 3-dimensional vector, since It is not observable, so it is set to a value obtained from the assembly dimensions and set as a constant in solving the optimization problem. acc According to the IMU acceleration measurement characteristics, it is selected as (Distances are measured in pixel coordinates at 600 dpi).

[0135] Among them, M gyr is the set of angular velocity measurements of the IMU. The angular velocity measurement of the IMU at time t is Σ gyr According to the IMU angular velocity measurement characteristics, it is selected

[0136] During a single-pass image scanning process, at least two laser mouse sensors are used to track the 2D motion of a portable printer and copier, obtaining the 2D motion process of the contact image sensor, thereby removing the distortion of the scanned image caused by the sensor movement, thereby obtaining a scanned image with higher accuracy. In one example, a contact image sensor with a resolution of 600 dpi and 2592 pixels was fixedly mounted on the bottom of a portable printer / copier, with its pixels arranged roughly perpendicular to the machine's direction of motion. It was used to capture a 1:1 scale image of the scanned media surface at a measurement frame rate of 3 kHz. Two laser mouse sensors with a resolution of 5 μm were fixedly mounted on the bottom of the portable copier to capture the motion of the portable copier relative to the scanned media at a measurement frequency of 4 kHz, with the two laser mouse sensors synchronized in time. The spatial parameters of laser mouse sensor 1 relative to the contact image sensor were a translational shift of (-998, 1699) pixels (at 600 dpi) and a rotational shift of -0.015 radians. The spatial parameters of laser mouse sensor 2 relative to the contact image sensor were a translational shift of (-967, -481) pixels (at 600 dpi) and a rotational shift of 0.017 radians. During a single-line image scanning task, 2.5 seconds of data were collected. First, the 2D motion trajectory of the contact image sensor is calculated using the synchronously collected data from two laser mouse sensors and the spatial parameters of the laser mouse sensor and the contact image sensor. The 2D motion trajectory is then interpolated to obtain the contact image sensor pose at any moment, thereby dedistorting the captured contact image sensor image to obtain a scanned image.

[0137] In an example of multi-pass image scanning and stitching using a portable printer, the system includes a contact image sensor and two laser mouse sensors. The method for optimizing the 2D motion trajectory of the s passes using the sensor data is to construct and solve the following optimization problem:

[0138]

[0139] Among them, M cis is a set of contact image sensor measurements, obtained by stitching together adjacent scans with overlapping areas. A contact image sensor measurement creation method is to first obtain a dedistorted scanned image using single-pass acquisition data; then perform visual feature point extraction and matching on the dedistorted scanned images of adjacent passes, resulting in a measurement m with j1 passes and image coordinates at time t1 of the trajectory p1 = [0, y1] T The visual feature points and the image coordinates of j2 trip trajectory at time t2 are p2 = [0, y2] T The visual feature points of are matched. in, Represents a function that calculates the conversion matrix from the contact image sensor coordinate system to the medium coordinate system based on the 2D motion trajectory and time. cis Determined by the feature matching accuracy and the accuracy of the contact image sensor, the pixel coordinates are set at 600dpi

[0140] In multi-line splicing printing, the position of the print head is determined by using the positioning results of the contact image sensor in the medium coordinate system and the spatial parameters of the contact image sensor and the print head. An example of an online positioning method uses data from two laser mouse sensors and one contact image sensor, and estimates the contact image sensor posture online through an extended Kalman filter. In the state initialization stage, the positioning result of the last frame of the scanned small segment image is determined to obtain the posture of the contact image sensor at that moment. During the printing process, the contact image sensor is used to collect images of marker lines preset on the surface of the printing medium with known coordinates to correct the positioning results in real time. The measurement m created based on the image collected by the contact image sensor at time t is the image coordinate p m =[0,y m ] T The image point of the equation is a m x+b m y+c m =0(st.a m 2 +b m 2 =1) mark line, the observation equation is 0=h(X t )=a m x'm +b m y' m +c m , where p' m =[x' m ,y' m ] T =T(X t )p m , is the contact image sensor pose to be estimated, In the extended Kalman filter, the method for calculating the Jacobian matrix of the observation equation to the estimated state is:

[0141]

[0142] In multi-line stitching printing, an online positioning method uses an IMU and a contact image sensor data. The IMU pose is estimated online using an error state Kalman filter, and the contact image sensor positioning state is calculated based on the IMU and contact image sensor spatial parameters. During the printing process, the contact image sensor captures an image of a marker line with known coordinates pre-set on the print medium surface to correct the positioning state in real time. The intersection of the marker lines is extracted based on the image captured by the contact image sensor at time t. The coordinates of the intersection point in the contact image sensor coordinate system are set to be The equation of the marking line in the medium coordinate system is a m x+b m y+c m =0(st.a m 2 +b m 2 =1), the spatial parameters of IMU and contact image sensor are And it is known that the pose of the IMU to be estimated is (p t ,q t ), q t =[w t v t ] T (w t is the real part of the unit quaternion, v t is the imaginary part of the unit quaternion). Then transform the image point to the IMU coordinate system According to the image point on the marker line, the observation equation is: in, The method to calculate the Jacobian matrix is in,

[0143]

[0144] ([] × represents the calculation of the antisymmetric matrix based on the 3D vector).

[0145] A method for using an area array image sensor instead of a contact image sensor for image scanning and printer positioning employs an area array image sensor imaging system and a laser mouse sensor. The area array image sensor imaging system is fixedly mounted on a portable printer / copier and includes an area array image sensor, a camera, an image acquisition circuit, and a light source to capture images of the surface of the scanned or printed media. The laser mouse sensor is fixedly mounted on the bottom of the portable printer / copier to capture local motion of the portable printer / copier. During image scanning or printing positioning, the image captured by the area array image sensor imaging system is dedistorted and homographically transformed based on the imaging system's internal parameters (including focal length, principal point coordinates, distortion parameters), as well as its distance from the planar media, pitch angle, and roll angle. This produces an image (referred to as the dedistorted area array image) with a 1:1 ratio to the planar media surface image. This image coordinate system is used as the area array image sensor coordinate system. During image scanning or printing positioning, the area array image sensor is positioned using the dedistorted area array image, laser mouse data, and the spatial parameters of the area array image sensor and laser mouse sensor, obtaining the relationship between the area array image sensor coordinate system and the media coordinate system. In the printing task, the position of the print head in the printing medium is calculated according to the spatial parameters of the area array image sensor and the print head to achieve splicing printing.

[0146] A method for single-pass image scanning motion estimation using an area array image sensor imaging system and a laser mouse sensor is to formulate and solve the following optimization problem:

[0147]

[0148] Among them, Traj is the 2D motion trajectory of the area array image sensor coordinate system, which is used to calculate the transformation relationship between the area array image sensor coordinate system and the medium coordinate system at time t M image is a set of visual measurements created based on the dedistorted area array image. A visual measurement m is a pair of visual feature matching points extracted and created from two adjacent frames at time t1 and t2, where the coordinates of the feature point in the frame at time t1 in the dedistorted area array image are p1 = [x1 y1] T , the coordinates of the feature point in the frame at time t2 in the dedistorted array image are p2 = [x2 y2] T ,but (600dpi image coordinate system). opt The spatial parameters of the laser mouse sensor and the area array image sensor that have been calibrated in advance are used in the calculation

Claims

1. A multi-sensor data fusion positioning method for a portable printer and copier, characterized in that: The steps include: Step 1: A contact image sensor is provided at the bottom of the portable printer / copier to obtain an image of the surface of a scanned or printed medium; Step 2: at least two laser mouse sensors are provided at the bottom of the portable printer / copier, or at least one inertial measurement unit (IMU) is provided on the portable printer / copier; Step 3: at least one print head is provided at the bottom of the portable printer / copier for printing an image on a surface of a print medium during a printing task; Step 4: Obtain parameters between the contact image sensor set in step 1 and the laser mouse sensor or IMU set in step 2; Step 5: Obtain the spatial parameters between the contact image sensor set in step 1 and the print head set in step 3; Step 6: In the image scanning or printing task, the inter-sensor parameters obtained in step 4 are used and the measurement data of the sensors set in steps 1 and 2 are integrated to locate the portable printer / copier.

2. The multi-sensor data fusion positioning method for a portable printer and copier according to claim 1, characterized in that: The method further includes step 7: in the image scanning task, according to the positioning result obtained in step 6, dedistorting and splicing the scanned image to obtain a scanned image with higher accuracy.

3. The multi-sensor data fusion positioning method for a portable printer and copier according to claim 1, characterized in that: The method further includes step 8: in the printing task, according to the online positioning result obtained in step 6 and the spatial parameters between the contact image sensor and the print head obtained in step 5, the position of the print head is calculated in real time to obtain the data to be printed.

4. A portable printer / copier multi-sensor data fusion positioning method according to claim 1, 2 or 3, characterized in that: Assume that the sensors installed in the portable printer copier system in steps 1 and 2 include 1 contact image sensor, 2 laser mouse sensors, and 1 IMU; establish sensor and medium coordinate systems, where the contact image sensor coordinate system has two forms: 2D and 3D; its y-axis is the pixel distribution direction of the contact image sensor, the x-axis is perpendicular to the y-axis, and the z-axis obtained according to the right-hand rule points perpendicular to the medium plane; the 2D form only has the x and y axes; the medium coordinate system is a coordinate system established based on the printing or scanning medium plane, and has two forms: 2D and 3D according to calculation requirements; the IMU and laser mouse sensor coordinate systems are established based on the measurement characteristics of the sensor itself.

5. The multi-sensor data fusion positioning method for a portable printer and copier according to claim 1, characterized in that: The inter-sensor parameters include the spatial parameters of the laser mouse sensor o (o=1, 2) and the contact image sensor Spatial parameters of IMU and contact image sensor 6. The multi-sensor data fusion positioning method for a portable printer and copier according to claim 1 or 5, characterized in that: In step 4, the inter-sensor parameter calibration method includes: Step 4.1: Preset an image or pattern with known size and content on the scanning medium; Step 4.2: Place the portable printer / copier flat on the medium plane and perform a composite translation and rotation motion within the plane; Step 4.3: Synchronously collect the measurement data and corresponding timestamps of all sensors within the time [t1, t2] in step 4.2; Step 4.4: Construct and solve the optimization problem to obtain the 2D motion trajectory of the contact image sensor and the estimation of the parameters between different sensors.

7. The multi-sensor data fusion positioning method for a portable printer and copier according to claim 1 or 3, characterized in that: During printing, according to the positioning status of the contact image sensor The spatial parameters of the contact image sensor and the print head are used to calculate the position of the print head in the medium coordinate system w, thereby obtaining the printed content in real time; the spatial parameters of the contact image sensor and the print head refer to the 2D coordinates of the print head endpoint e in the contact image sensor coordinate system Each print head has at least two endpoints.

8. The multi-sensor data fusion positioning method for a portable printer and copier according to claim 1, 3 or 7, characterized in that: During the printing process, the 2D position of the print head in the medium coordinate system w at time t is calculated based on the real-time positioning results of the printer, the contact image sensor and the spatial parameters of the print head. The method is to use a contact image sensor to locate the state Coordinate transformation of the spatial parameters of the contact image sensor and the print head 9. The multi-sensor data fusion positioning method for a portable printer and copier according to claim 8, characterized in that: In the method for calculating the print head position based on the printer positioning result, the spatial parameters of the contact image sensor and the print head are replaced with the spatial parameters of the laser mouse sensor or IMU and the print head, and then equivalent calculation is achieved based on the spatial parameters of the laser mouse sensor or IMU and the touch image sensor.

10. The multi-sensor data fusion positioning method for a portable printer and copier according to claim 1, characterized in that: In step 5, further comprising: Step 5.1: Preset a pattern or marking line on the surface of the printing medium for contact image sensor positioning; Step 5.2: Place the portable printer on the print medium. Move the printer to collect data from the contact image sensor and motion tracking sensor. Trigger the print head to print ink lines at intervals and record the printing time t for each L ink line. l (l=1, 2, ..., L); Step 5.3: For a certain endpoint e of the print head, measure the position p of the corresponding endpoint of the printed ink line in the medium coordinate system w l (l=1, 2, ..., L); Step 5.4: Use the pattern or mark preset in step 5.1, the sensor data collected in step 5.2, and the known inter-sensor parameters to locate the contact image sensor and obtain its 2D positioning state at the time of ink line printing. Step 5.5: Calculate the 2D coordinates of the print head endpoint e in the contact image sensor coordinate system 11. The multi-sensor data fusion positioning method for a portable printer and copier according to claim 1, characterized in that: In step 6, the method further includes: using at least two laser mouse sensors in step 6 to perform 2D motion tracking of the portable printer and copier with three degrees of freedom in a plane, specifically: Step 6.1.1: Get the spatial parameters of the laser mouse sensor o and the contact image sensor. The values ​​are: Among them, (px o ,py o ) is the translation component of the spatial parameters of the laser mouse sensor o and the contact image sensor, θ o is the rotation component; Step 6.1.2: Obtain the measurement value (Δx) of the laser mouse sensor o during the time period [t1, t2] to be tracked. o , Δy o ), o=1, 2, are the displacements of the laser mouse in the directions of its two coordinate axes during this period; Step 6.1.3: Assume that the motion of the contact image sensor to be solved in the time period [t1, t2] is Among them, (tx, ty) is the translation component of the motion, is the rotational component of the motion; according to the measurement model of the laser mouse sensor, the spatial parameter relationship between the laser mouse sensor and the contact image sensor is: Arranged: Step 6.1.4: Combining the above equations based on the measurements of each laser mouse sensor, we get: Solve the above equation or find the least squares solution of the above equation to obtain tx, ty The estimated value of , thus obtaining the 2D motion of the contact image sensor in the time period [t1, t2] 12. The multi-sensor data fusion positioning method for a portable printer and copier according to claim 1, characterized in that: In step 6, the method further includes: using the IMU data in step 6 to track the 2D motion of the portable printer on the medium surface, specifically: Step 6.2.1: Static initialization of IMU: estimate gravity acceleration, IMU acceleration bias, IMU angular velocity bias, IMU initial position, and IMU initial velocity; place the printer on the printing medium surface, collect the IMU acceleration and angular velocity and calculate the average value to obtain At this time, the initial position of the IMU is the unit matrix, and the velocity value of the IMU is (0, 0, 0) T , the angular velocity bias of IMU is The acceleration due to gravity is The acceleration bias of the IMU is Step 6.2.2: When the gravity acceleration, IMU acceleration bias, IMU angular velocity bias, IMU pose and velocity values ​​at time t1 are known, integrate the IMU acceleration and angular velocity measurements within the time [t1, t2] to obtain the IMU pose and velocity values ​​at time t2, and obtain the relative motion of the IMU during this time. is the transformation relationship of the 3D rigid body coordinate system; Step 6.2.3: Using the parameters between the IMU and the contact image sensor, obtain the relative motion of the contact image sensor in the time period [t1, t2] neglect The translation component along the z axis and the rotation components along the x axis and y axis are used to obtain the 2D relative motion of the contact image sensor on the medium plane within the time period [t1, t2].

13. The multi-sensor data fusion positioning method for a portable printer and copier according to claim 1, characterized in that: In step 6, the method further includes: using an IMU and a laser mouse sensor to track the 2D motion of the portable printer on the medium surface in step 6; assuming that the rotation component of the spatial parameters of the IMU and the contact image sensor is is the rotation component of the 3D rigid body coordinate system transformation relationship; the spatial parameters of the laser mouse sensor and the contact image sensor are is the transformation relationship of the 2D rigid body coordinate system, satisfying: Among them, (px, py) is the translation component of the spatial parameters of the laser mouse sensor and the contact image sensor, and θ is the rotation component; Use the IMU measurements and the laser mouse sensor measurements (Δx, Δy) during time [t1, t2] to calculate the relative motion of the contact image sensor during this period. Among them, (tx, ty) is the relative motion The translation component of For relative motion The rotation component of ; specifically: Step 6.3.1: Estimate the IMU angular velocity bias using the static method; Step 6.3.2: Using the IMU angular velocity bias, integrate the angular velocity measurements of the IMU in the time period [t1, t2] to obtain the rotational motion of the IMU at that time is the rotation component of the 3D rigid body coordinate system transformation relationship; Step 6.3.3: Use the rotational components of the IMU and contact image sensor spatial parameters to obtain the contact image sensor's rotational motion during this period. Only the rotation component along the z-axis is retained to obtain the rotation component of the contact image sensor movement Step 6.3.4: Using the measurements of the laser mouse sensor and the spatial parameters of the laser mouse sensor and the contact image sensor, we can obtain: Thus, the relative motion of the touch image sensor in the time period [t1, t2] is obtained 14. The multi-sensor data fusion positioning method for a portable printer and copier according to claim 1, characterized in that: In step 6, the method further includes: using data from at least two laser mouse sensors and one contact image sensor in step 6 to estimate the contact image sensor state and perform online positioning of the portable printer / copier, specifically: Step 6.4.1: Define and initialize the contact image sensor state to be estimated in, is the translation component, is the rotation component, and its covariance matrix Step 6.4.2: Prediction step; using [t i-1 , t i ] The measured values ​​of at least two laser mouse sensors and the inter-sensor parameters are calculated for the relative motion of the contact image sensor in the time period Where (Δtx i , Δty i ) is the translation component of the motion measurement value, is the rotational component of the motion measurement, and the covariance matrix of the motion equation noise is a known constant ∑ pre , for contact image sensors at t i The state at the moment is predicted Step 6.4.3: Update step; use the contact image sensor to i Constructing observation equations based on time data Among them, h is the observation equation function, is the independent variable of the observation equation, μ is the noise of the observation equation and its covariance matrix ∑ is known μ , and update the measurement to get Step 6.4.4: Using [t i ,t] time period laser mouse sensor extension interpolation measurement value calculation time period contact image sensor relative motion Among them, (Δtx, Δty) is the relative motion translation component, is the relative motion rotation component, and then the positioning state of the contact image sensor at any time t is calculated have: Among them, (tx t ,ty t ) is the translation component of the positioning state, The rotation component of the positioning state.

15. The multi-sensor data fusion positioning method for a portable printer and copier according to claim 1, characterized in that: In step 6, the method further includes: using an IMU and contact image sensor data in step 6 to estimate the IMU state, and then calculating the contact image sensor positioning state, specifically: Step 6.5.1: Use the static initialization method to estimate the gravity acceleration, IMU acceleration bias, IMU angular velocity bias, and IMU initial velocity value, and calculate the contact image sensor positioning state on the printing medium plane at that moment. The spatial parameters of the IMU and the contact image sensor are used to calculate the initial pose of the IMU, and the estimated gravitational acceleration is converted to the medium coordinate system w; Step 6.5.2: Define and initialize the IMU nominal state in, represents the position of the IMU in the medium coordinate system w, set to the estimated value obtained according to step 6.5.1, represents the velocity of the IMU in the medium coordinate system w, is the unit quaternion representing the rotation of the IMU in the medium coordinate system w, set to the estimated value obtained according to step 6.5.1, represents the IMU acceleration bias, set to the estimated value obtained in step 6.5.1, represents the IMU angular velocity bias, set to the estimated value obtained in step 6.5.1, represents the gravitational acceleration in the medium coordinate system and is set to the estimated value obtained in step 6.5.1; Defining Error States And set the error state covariance matrix Step 6.5.3: Use t i The IMU measurement is performed at the time of prediction; let the acquisition time interval of IMU measurement be Δt, t i The measured value at time is in, is the acceleration measurement value, is the angular velocity measurement value; calculate t i Nominal state at the moment and the error state covariance matrix in, Indicates t i The position of the IMU in the medium coordinate system w at the moment, Indicates t i The velocity of the IMU in the medium coordinate system w at the moment, Indicates t i The rotation of the IMU in the medium coordinate system w at time t, Indicates t i IMU acceleration zero bias at this moment, Indicates t i IMU angular velocity zero bias at this moment, Indicates t i The gravitational acceleration in the medium coordinate system at the moment, yes The corresponding rotation matrix, q{} represents the conversion of the rotation vector into a unit quaternion. Represents quaternion multiplication; Step 6.5.4: Using t i The contact image sensor measures the error state at all times; let t i The measurement is built based on the contact image sensor data at all times in, are the spatial parameters of IMU and contact image sensor and are known, h is the observation equation function, η is the covariance matrix ∑ η Gaussian measurement noise; Calculate the nominal state error based on the measurement and its covariance matrix is the Kalman gain; in, Indicates t i IMU position error at time t, Indicates t i IMU velocity error at time, is represented by the rotation vector t i IMU rotation error at time t, Indicates t i IMU acceleration bias error at time t, Indicates t i IMU angular velocity zero bias error at this moment, Indicates t i Gravity acceleration error at all times; Then update the nominal state and the error state covariance matrix Step 6.5.5: Using t i The nominal state of IMU at the moment [t i , t] time period IMU measurement is integrated to obtain the IMU pose at time t (p t ,q t ), and then use the spatial parameters of the IMU and the contact image sensor to obtain the positioning state of the contact image sensor in the medium coordinate system w at time t 16. The multi-sensor data fusion positioning method for a portable printer and copier according to claim 1 or 2, characterized in that: In single-pass image scanning on a portable printer / copier, the 2D motion tracking results of the contact image sensor are obtained using sensor measurement data and inter-sensor parameter calculations, and the scanned image is then dedistorted to obtain a single-pass scanned image with higher accuracy.

17. The multi-sensor data fusion positioning method for a portable printer and copier according to claim 1, 2 or 16, characterized in that: In step 7, it is further included that: in step 7, a total of N frames of images are collected during a single image scanning process, each frame corresponds to a pixel value vector with a length of h, and the collection time of the kth frame image is t k 、Data is v k , the calculated 2D motion tracking result of the contact image sensor is Where k = 1, 2, ..., N, and the method for performing single-pass image dedistortion is as follows: Step 7.1: Initialize the stitched image I and set the image width to x max -x min +1, image height y max -y min +1, among which, (·) d , (d=1,2) means taking the d-th dimension coordinate value of the vector, Indicates 2D coordinate transformation using coordinate transformation matrix; Step 7.2: Map the k-th frame of the collected image to the spliced ​​image I and assign pixel values ​​I(x kr ,y kr )=v k (r), where k = 1, 2, ..., N, r = 1, 2, ..., h, I(x, y) represents the pixel of the image at coordinate (x, y), v k (r) represents the r-th pixel value of the k-th frame image; Step 7.3: Assign values ​​to the unassigned pixels of the stitched image I using an interpolation method to obtain a dedistorted scanned image.

18. The multi-sensor data fusion positioning method for a portable printer and copier according to claim 1 or 2, characterized in that: Multi-pass image scanning is performed using a portable printer and copier. The collected multi-pass sensor data and the spatial parameters between sensors are used to jointly optimize and estimate the multi-pass 2D motion trajectory of the contact image sensor. The multi-pass trajectory is then used to obtain a stitched scanned image. The specific stitching method is the same as that for single-pass scanning.

19. A multi-sensor data fusion positioning method for a portable printer / copier based on claim 1, characterized in that: The laser mouse sensor is replaced with a wheel encoder or stepper motor to measure and locate the motion of a portable printer and copier. The wheel displacement obtained by the wheel encoder or stepper motor in the time period (t1, t2) is Δx, which is converted into a 2D displacement measurement of (Δx, 0), thereby replacing the laser mouse sensor measurement value for motion tracking and positioning.

20. A multi-sensor data fusion positioning method for a portable printer / copier according to claim 1, characterized in that: In the printing scenario, some or all of the laser mouse sensors or IMU settings and usage are removed, and the image on the surface of the printing medium is positioned using a contact image sensor.

21. A multi-sensor data fusion positioning method for a portable printer / copier according to claim 1, characterized in that: Replace contact image sensors with linear image sensors or area array image sensor imaging systems to position portable printers and copiers.

22. The multi-sensor data fusion positioning method for a portable printer and copier according to claim 21, characterized in that: In a portable printer / copier, if an area array image sensor imaging system is used, a dedistorted, original-size image of the scanned or printed medium surface is obtained from the captured single-frame image based on its internal parameters and its spatial relationship with the scanned or printed medium plane, thereby replacing the contact image sensor image for image scanning or printing positioning.

23. The multi-sensor data fusion positioning method for a portable printer and copier according to claim 21, characterized in that: In a portable printer / copier, if an area array image sensor imaging system is used, the setting and use of some or all laser mouse sensors or IMUs are removed, and the imaging of the scanned or printed medium surface image is positioned by the area array image sensor imaging system.

24. The multi-sensor data fusion positioning method for a portable printer and copier according to claim 21 or 22, characterized in that: In portable printers and copiers, if an area array image sensor imaging system is used, measurement information is extracted from the resulting dedistorted, full-size scan or printed media surface image for positioning. The extracted measurement information is categorized into three types: tracking results of visual features between preceding and following frames during a single-pass image scan or print; matching results of visual features between overlapping frames during different image scans; and matching results of visual features between a single-frame image and the printed media surface image during printing. The visual features referred to include point features, line features, grayscale value features calculated based on 2D images, and features extracted based on neural network methods.

25. The multi-sensor data fusion positioning method for a portable printer and copier according to claim 21, 22 or 24, characterized in that: In a portable printer / copier, if the area array imaging system used is a global shutter exposure, the acquisition time of a single frame image is unique, and the measurement time extracted from the same frame image is also consistent; If rolling shutter exposure is used, the acquisition time of each row of pixels in a single frame image is different. For the same frame image, the measurement time needs to be determined based on the exposure time of the corresponding row to improve the measurement accuracy and positioning result precision.

Images