Code-free intelligent pen control system based on multi-sensor fusion

Abstract

The invention discloses a code-free intelligent pen control system based on multi-sensor fusion, which relates to the technical field of intelligent hardware and comprises a pressure sensor module, an acceleration sensor module, a camera module, a microcontroller unit, a wireless communication module and a power management module. According to the method, handwriting content digitization on common paper is realized, data of a pressure sensor, an acceleration sensor and a camera are fused, the accuracy of writing tracks and positioning is improved, SIFT feature extraction, RANSAC model estimation, Kalman filtering, deep learning and other algorithms are adopted, high-precision content recognition and positioning are realized, hardware and software design is optimized, and the accuracy of content recognition and positioning is improved. The system architecture supports modular design, other functional modules can be added according to needs, digitalization of written content can be achieved without special paper, the cost is reduced, and the application range is widened.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent hardware, and in particular to a codeless intelligent pen control system based on multi-sensor fusion. Background Art

[0002] Screenless digitization has many application scenarios in today's digital education. A smart pen that looks similar to a traditional pen but can digitize the content written by students or teachers has a large market space. Currently, the market mainly uses dot matrix code technology to solve the positioning and recognition problems of smart pens. Some also use electromagnetic induction for positioning, but the scope of use is relatively small.

[0003] Smart pen solutions based on dot code technology can accurately locate the page number and position of writing during use, but a lot of work needs to be invested in the early stage to process the paper before printing, which greatly increases the cost of using smart pens based on this technology and also sets more application restrictions. This has greatly slowed down the large-scale promotion and popularization of smart pens. Summary of the invention

[0004] In order to solve the deficiencies mentioned in the above background technology, the purpose of the present invention is to provide a code-free smart pen control system based on multi-sensor fusion, which realizes the digital collection and precise positioning of handwritten content on ordinary paper.

[0005] In the first aspect, the purpose of the present invention can be achieved by the following technical solution: a codeless smart pen control system based on multi-sensor fusion, comprising:

[0006] Pressure sensor module, acceleration sensor module, camera module, microcontroller unit, wireless communication module and power management module;

[0007] The pressure sensor module is used to detect the contact pressure data between the pen tip and the paper surface, and send the contact pressure data between the pen tip and the paper surface to the microcontroller unit;

[0008] The acceleration sensor module is used to collect the three-axis acceleration data of the pen and send the three-axis acceleration data of the pen to the microcontroller unit;

[0009] The camera module is used to collect images of the writing area and send the collected images of the writing area to the microcontroller unit;

[0010] The microcontroller unit determines the writing state of the pen according to the contact pressure data between the pen tip and the paper surface, and sends an acceleration acquisition signal to the acceleration sensor module when the pen enters the writing state; and calculates the writing trajectory after receiving the three-axis acceleration data of the pen, and sends an image acquisition signal to the camera module; identifies and locates the writing content according to the image of the writing area, and controls the operation of the pen in combination with the writing trajectory to obtain control data, and sends the control data to the wireless communication module;

[0011] The wireless communication module is used to transmit control data to the mobile device, so as to operate and control the pen;

[0012] The power management module is used to supply power to the entire system.

[0013] In combination with the first aspect, in some implementations of the first aspect, the system further includes: the pressure sensor module sets a preprocessing process for contact pressure data between the pen tip and the paper surface:

[0014] A first-order IIR filter is used to filter out high-frequency noise. The recursive formula of the filter is:

[0015] P filtered (n) = αP raw (n)+(1-α)P filtered (n-1)

[0016] Among them, α is the preset coefficient, P raw (n) is the original pressure value of the nth sampling.

[0017] In combination with the first aspect, in some implementations of the first aspect, the system further includes: the acceleration sensor module is set to obtain three-axis acceleration data (a x (n),a y (n),a z (n));

[0018] Preprocessing steps:

[0019] Zero bias calibration: When the device is stationary, collect multiple sets of data and calculate the zero bias value (a x0 ,a y0 ,a z0 );

[0020] Deduct zero bias:

[0021]

[0022] Gravity compensation: Since the accelerometer also senses gravity acceleration, it is necessary to compensate for the gravity component according to the pen's posture. Assuming that the pen mainly moves in a plane, gravity compensation is processed;

[0023] Filtering: Use Kalman filter to filter the acceleration data;

[0024] Kalman filter algorithm:

[0025] Status definition:

[0026] Define the state vector as

[0027] State transfer equation:

[0028] x k =x k-1 +w k

[0029] Among them, w k is the process noise, assumed to be zero-mean Gaussian white noise

[0030] Observation equation:

[0031] z k =x k +v k

[0032] Among them, z k is the measured value, v k To measure noise;

[0033] Filtering process:

[0034] Prediction Steps:

[0035]

[0036] in, is the prediction covariance, Q is the process noise covariance matrix;

[0037] Update steps:

[0038]

[0039] Where R is the measurement noise covariance matrix.

[0040] In combination with the first aspect, in certain implementations of the first aspect, the system further includes: during the writing process, the camera module collects a frame of image at fixed intervals or when a special event is detected;

[0041] Preprocessing steps:

[0042] Image scaling: scale the original high-resolution image to 640x480;

[0043] Grayscale: Convert a color image to a grayscale image;

[0044] Image enhancement: Adaptive histogram equalization (CLAHE) is used to improve image contrast;

[0045] Denoising: Use a Gaussian filter to smooth the image and remove high-frequency noise.

[0046] In combination with the first aspect, in some implementations of the first aspect, the system further includes: a process in which the microcontroller unit determines the writing state of the pen:

[0047] Set the pressure threshold P threshold , when P filtered (n)>P threshold When the writing state is entered, the data collection process is started.

[0048] In combination with the first aspect, in some implementations of the first aspect, the system further includes: a calculation process of the writing trajectory of the microcontroller unit is as follows:

[0049] Based on the preprocessed acceleration data, the velocity and displacement are calculated using the numerical integration method;

[0050] Speed ​​calculation:

[0051]

[0052] Among them, v x (n), v y (n) represents the velocity of the nth sampling point in the X-axis and Y-axis directions, v x (n-1), v y (n-1) represents the speed of the n-1th sampling point in the X-axis and Y-axis directions, is the calibrated X-axis and Y-axis acceleration of the nth sampling point, and It is the acceleration data after zero bias calibration and filtering, and Δt is the time interval (sampling period) between two adjacent sampling points.

[0053] Displacement calculation:

[0054]

[0055] Among them, s x (n), s y (n) represents the displacement velocity of the nth sampling point in the X-axis and Y-axis directions, s x (n-1), s y (n-1) represents the displacement of the n-1th sampling point in the X-axis and Y-axis directions, v x (n), v y (n) is the speed of the nth sampling point in the X-axis and Y-axis directions, and Δt is the time interval (sampling period) between two adjacent sampling points.

[0056] In combination with the first aspect, in certain implementations of the first aspect, the system also includes: the process by which the microcontroller unit identifies and locates the writing content based on the image of the writing area includes: feature point extraction, feature point matching, elimination of mismatched feature points, and writing position positioning.

[0057] In combination with the first aspect, in some implementations of the first aspect, the system further includes: the feature point extraction process:

[0058] The SIFT algorithm is used to extract feature points. The steps are as follows:

[0059] Scale space construction:

[0060] Perform Gaussian blur on the image I(x,y) to generate images L(x,y,σ) of different scales:

[0061] L(x,y,σ)=G(x,y,σ)*I(x,y)

[0062] Among them, G(x,y,σ) is the Gaussian kernel function, * represents the convolution operation;

[0063] Difference scale space:

[0064] Calculate the difference between adjacent scale images and obtain the difference image D(x,y,σ):

[0065] D(x,y,σ)=L(x,y,kσ)-L(x,y,σ)

[0066] Key point detection:

[0067] In the difference scale space, find the extreme points;

[0068] Accurate positioning of key points:

[0069] The initial key points are processed precisely to remove low-contrast and edge response points;

[0070] Direction assignment:

[0071] Compute the gradient magnitude and direction in the neighborhood of the keypoint:

[0072]

[0073] Determine the main direction of the key point based on the gradient direction histogram;

[0074] Feature descriptor generation:

[0075] Centering on the key point, a neighborhood window is established, and the domain window is divided into multiple sub-areas;

[0076] In each sub-region, the gradient histogram in each direction is calculated to generate a feature vector;

[0077] The feature point matching adopts the KNN algorithm, and the steps are as follows:

[0078] Compute the distance between eigenvectors:

[0079] The Euclidean distance is used to measure the similarity between two feature vectors;

[0080] Nearest neighbor search:

[0081] For each feature point to be matched, find the two feature points closest to each other in the template feature set;

[0082] Ratio test:

[0083] Calculate the nearest neighbor distance d1 and the next nearest neighbor distance d2 if The match is considered valid;

[0084] The RANSAC algorithm is used to eliminate mismatched feature points;

[0085] Writing position calculation method:

[0086] Use the obtained affine transformation matrix A to transform the points in the camera coordinate system to the paper coordinate system;

[0087] For the image point (x', y') captured by the camera, its corresponding paper coordinates (x, y) are:

[0088]

[0089] In combination with the first aspect, in some implementations of the first aspect, the system further includes: the microcontroller unit uses an extended Kalman filter EKF to fuse the displacement estimation of the acceleration sensor with the positioning result of the camera, and the process is as follows:

[0090] State vector definition:

[0091]

[0092] State transition model:

[0093] x k =Fx k-1 +Bu k-1 +w k-1

[0094] in,

[0095] F is the state transfer matrix:

[0096]

[0097] B is the control input matrix:

[0098]

[0099] u k-1 For acceleration input:

[0100]

[0101] w k-1 is the process noise;

[0102] Observation model:

[0103] z k =Hx k +v k

[0104] in,

[0105] H is the observation matrix:

[0106]

[0107] v k is the observation noise;

[0108] The prediction step of filtering:

[0109] Status prediction:

[0110]

[0111] Covariance prediction:

[0112]

[0113] Where Q is the process noise covariance matrix;

[0114] Update steps:

[0115] Calculate the Kalman gain:

[0116]

[0117] Where R is the observation noise covariance matrix;

[0118] Status Update:

[0119]

[0120] Covariance update:

[0121]

[0122] In another aspect of the present invention, in order to achieve the above-mentioned purpose, a terminal device is disclosed, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the memory stores a computer program capable of running on the processor, and when the processor loads and executes the computer program, a non-coding smart pen control system based on multi-sensor fusion as described above is adopted.

[0123] Beneficial effects of the present invention:

[0124] The present invention: does not require special paper or pre-printed dot codes, realizes the digitization of handwritten content on ordinary paper, integrates data from pressure sensors, acceleration sensors and cameras, improves the accuracy of writing trajectory and positioning, adopts SIFT feature extraction, RANSAC model estimation, Kalman filtering, deep learning and other algorithms, realizes high-precision content recognition and positioning, optimizes hardware and software design, reduces system power consumption, and extends the battery life of the device. The system architecture supports modular design, and other functional modules can be added as needed. The digitization of written content can be realized without the need for special paper, which reduces costs and increases the scope of application. BRIEF DESCRIPTION OF THE DRAWINGS

[0125] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0126] Figure 1 It is a schematic diagram of the content processing flow of the smart pen of the present invention;

[0127] Figure 2 It is a hardware principle block diagram of the smart pen system of the present invention;

[0128] Figure 3 is a schematic diagram of the structure of the smart pen of the present invention;

[0129] Figure 4 It is the flow chart of the App side of the present invention. DETAILED DESCRIPTION

[0130] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0131] Embodiment 1:

[0132] The following is an introduction to the relevant terms involved in the embodiments of the present application:

[0133] Sensor: A sensor (English name: transducer / sensor) is a detection device that can sense the information being measured and convert the sensed information into electrical signals or other required forms of information output according to certain rules to meet the requirements of information transmission, processing, storage, display, recording and control.

[0134] The existence and development of sensors have given objects senses such as touch, taste and smell, making objects come alive. Sensors are an extension of human's five senses.

[0135] Sensors have the characteristics of miniaturization, digitization, intelligence, multi-function, systematization and networking. They are the primary link in realizing automatic detection and automatic control.

[0136] like Figure 1 As shown, a codeless smart pen control system based on multi-sensor fusion includes:

[0137] Pressure sensor module, acceleration sensor module, camera module, microcontroller unit, wireless communication module and power management module;

[0138] The pressure sensor module is used to detect the contact pressure data between the pen tip and the paper surface, and send the contact pressure data between the pen tip and the paper surface to the microcontroller unit;

[0139] Specifically, the pressure sensor module functions to detect the contact pressure between the pen tip and the paper surface to determine whether it is in a writing state.

[0140] Implementation details:

[0141] Sensor selection: Use a highly sensitive piezoelectric pressure sensor, such as the FSR400 series from Interlink Electronics.

[0142] Installation location: The sensor is installed at the tip of the pen and can directly sense the pressure applied by the tip of the pen.

[0143] Signal conditioning: The analog signal output by the sensor passes through the resistor divider and filter circuit and is connected to the ADC (analog-to-digital converter) channel of the MCU.

[0144] Pressure sensor data acquisition and preprocessing

[0145] Collection method:

[0146] Sampling frequency: Set the ADC sampling frequency to 1kHz to obtain the output voltage signal of the pressure sensor in real time.

[0147] Preprocessing steps:

[0148] Filtering: Use a first-order IIR filter to filter out high-frequency noise. The recursive formula of the filter is:

[0149] P filtered (n) = αP raw (n)+(1-α)P filtered (n-1)

[0150] Where α=0.1,P raw (n) is the original pressure value of the nth sampling.

[0151] The acceleration sensor module is used to collect the three-axis acceleration data of the pen and send the three-axis acceleration data of the pen to the microcontroller unit;

[0152] Specifically, the acceleration sensor module:

[0153] Function: Collect the three-axis acceleration data of the pen to calculate the writing trajectory.

[0154] Implementation details:

[0155] Sensor selection: Use a three-axis MEMS accelerometer such as the BMA280 from Bosch Sensortec or the LIS3DH from STMicroelectronics.

[0156] Communication interface: through I 2 C or SPI interface to connect to MCU and support high-speed data transmission.

[0157] Data output: Output three-axis acceleration data with a resolution of up to 12 bits and a sampling rate of up to 1kHz.

[0158] Acceleration sensor data acquisition and preprocessing

[0159] Collection method:

[0160] Sampling frequency: Set the sampling frequency to 200Hz to obtain the three-axis acceleration data (a x (n),a y (n),a z (n)).

[0161] Technical principle: Calculate velocity and displacement by integrating the acceleration signal.

[0162] v(t)=∫a(t)dt+v0

[0163] s(t)=∫v(t)dt+s0

[0164] Preprocessing steps:

[0165] Zero bias calibration: When the device is stationary, collect multiple sets of data and calculate the zero bias value (a x0 ,a y0 ,a z0 ).

[0166] Deduct zero bias:

[0167]

[0168] Gravity compensation: Since the accelerometer also senses the acceleration of gravity, the gravity component needs to be compensated according to the pen's posture. Assuming that the pen moves mainly in a plane, gravity compensation can be handled approximately.

[0169] Filtering: Use Kalman filter to filter acceleration data to reduce noise interference.

[0170] Kalman filter algorithm:

[0171] Status definition:

[0172] Define the state vector as

[0173] State transfer equation:

[0174] x k =x k-1 +w k

[0175] Among them, w k is the process noise, which is assumed to be zero-mean Gaussian white noise.

[0176] Observation equation:

[0177] z k =x k +v k

[0178] Among them, z k is the measured value, v k To measure noise.

[0179] Filtering process:

[0180] Prediction Steps:

[0181]

[0182] in, is the prediction covariance, and Q is the process noise covariance matrix.

[0183] Update steps:

[0184]

[0185] Where R is the measurement noise covariance matrix.

[0186] The camera module is used to collect images of the writing area and send the collected images of the writing area to the microcontroller unit;

[0187] Specifically, the camera module

[0188] Function: Capture images of writing areas for content recognition and positioning.

[0189] Implementation details:

[0190] Camera selection: Choose a low-power, small-size camera module, such as OmniVision's OV7670 or OV5640.

[0191] Optical design: The camera is facing the pen tip, the field of view covers the writing area, and the focal length and aperture are optimized.

[0192] Image transmission: connected to MCU via DVP or SPI interface, supporting QVGA or VGA resolution.

[0193] Camera image acquisition and preprocessing

[0194] Collection method:

[0195] During the writing process, a frame of image is collected every fixed time (such as 500ms) or when a special event (such as page turning) is detected.

[0196] Preprocessing steps:

[0197] Image Scaling: Scale the original high-resolution image to 640x480 to reduce computational effort.

[0198] Grayscale: Convert a color image to a grayscale image.

[0199] Image enhancement: Adaptive histogram equalization (CLAHE) is used to improve image contrast.

[0200] Denoising: Use a Gaussian filter to smooth the image and remove high-frequency noise.

[0201] The microcontroller unit determines the writing state of the pen according to the contact pressure data between the pen tip and the paper surface, and sends an acceleration acquisition signal to the acceleration sensor module when the pen enters the writing state; and calculates the writing trajectory after receiving the three-axis acceleration data of the pen, and sends an image acquisition signal to the camera module; identifies and locates the writing content according to the image of the writing area, and controls the operation of the pen in combination with the writing trajectory to obtain control data, and sends the control data to the wireless communication module;

[0202] Specifically, the microcontroller unit (MCU)

[0203] Function: Responsible for sensor data acquisition, algorithm processing and system control.

[0204] Implementation details:

[0205] Chip selection: Use a high-performance, low-power MCU, such as STMicroelectronics' STM32F4 series (based on the Cortex-M4 core) or STM32H7 series (based on the Cortex-M7 core).

[0206] Resource configuration: At least 256KB of RAM and 1MB of Flash to meet the running requirements of complex algorithms.

[0207] Writing state determination: Setting pressure threshold P threshold , when P filtered (n)>P threshold When the writing state is entered, the data collection process is started.

[0208] Writing trajectory calculation

[0209] Based on the preprocessed acceleration data, the velocity and displacement are calculated using the numerical integration method.

[0210] Speed ​​calculation:

[0211]

[0212] Displacement calculation:

[0213]

[0214] Here, Δt is a sampling period (eg, Δt=1 / 200 seconds).

[0215] Error handling:

[0216] Integration drift problem: Due to the zero bias error of the accelerometer, the integration calculation will lead to the accumulation of displacement error.

[0217] Solution: Use Kalman filter and camera positioning results to correct the displacement (see the multi-sensor fusion section for details).

[0218] Camera content recognition and positioning algorithm details

[0219] Feature point extraction and description (SIFT algorithm)

[0220] Algorithm steps:

[0221] Scale space construction:

[0222] Perform Gaussian blur on the image I(x,y) to generate images L(x,y,σ) of different scales:

[0223] L(x,y,σ)=G(x,y,σ)*I(x,y)

[0224] Among them, G(x,y,σ) is the Gaussian kernel function, and * represents the convolution operation.

[0225] Difference scale space:

[0226] Calculate the difference between adjacent scale images and obtain the difference image D(x,y,σ):

[0227] D(x,y,σ)=L(x,y,kσ)-L(x,y,σ)

[0228] Key point detection:

[0229] In the differential scale space, find the extreme point, that is, the pixel value of the point is the maximum or minimum value in the 3x3x3 neighborhood.

[0230] Accurate positioning of key points:

[0231] The initial key points are refined to remove low-contrast and edge response points.

[0232] Direction assignment:

[0233] Compute the gradient magnitude and direction in the neighborhood of the keypoint:

[0234]

[0235] According to the gradient direction histogram, the main direction of the key point is determined.

[0236] Feature descriptor generation:

[0237] Centered on the key point, a 16x16 neighborhood window is established and divided into 4x4 sub-regions.

[0238] In each sub-region, the gradient histograms in 8 directions are calculated to generate a 128-dimensional feature vector.

[0239] Feature point matching (KNN algorithm)

[0240] Algorithm steps:

[0241] Compute the distance between eigenvectors:

[0242] The Euclidean distance is used to measure the similarity between two feature vectors.

[0243] Nearest neighbor search:

[0244] For each feature point to be matched, find the two feature points with the closest distance in the template feature set.

[0245] Ratio test:

[0246] Calculate the nearest neighbor distance d1 and the next nearest neighbor distance d2 if The match is considered valid.

[0247] False match elimination (RANSAC algorithm)

[0248] Algorithm steps:

[0249] Random Sampling:

[0250] A minimum number of points (e.g., 3 pairs) are randomly selected from the matching feature point pairs to estimate the affine transformation matrix A.

[0251] Model estimation:

[0252] Calculate the affine transformation matrix based on the selected point pairs.

[0253] Interior point determination:

[0254] Calculate the projection error of all matching points under the model. If the error is less than the threshold, it is considered to be an internal point.

[0255] Iterative Optimization:

[0256] Repeat the above steps until the model with the largest number of inliers is found.

[0257] Model optimization:

[0258] The affine transformation matrix is ​​re-estimated using all inliers to obtain a more accurate model.

[0259] Writing position positioning

[0260] Calculation method:

[0261] Using the obtained affine transformation matrix A, the points in the camera coordinate system are transformed into the paper coordinate system.

[0262] For the image point (x', y') captured by the camera, its corresponding paper coordinates (x, y) are:

[0263]

[0264] Multi-sensor data fusion

[0265] The extended Kalman filter (EKF) is used to fuse the displacement estimation of the acceleration sensor with the positioning result of the camera to improve the accuracy of the writing trajectory.

[0266] State vector definition:

[0267]

[0268] State transition model:

[0269] x k =Fx k-1 +Bu k-1 +w k-1

[0270] in,

[0271] F is the state transfer matrix:

[0272]

[0273] B is the control input matrix:

[0274]

[0275] u k-1 For acceleration input:

[0276]

[0277] w k-1 is the process noise.

[0278] Observation model:

[0279] z k =Hx k +v k

[0280] in,

[0281] H is the observation matrix:

[0282]

[0283] v k is the observation noise.

[0284] Filtering steps:

[0285] Prediction Steps:

[0286] Status prediction:

[0287]

[0288] Covariance prediction:

[0289]

[0290] Where Q is the process noise covariance matrix.

[0291] Update steps:

[0292] Calculate the Kalman gain:

[0293]

[0294] Where R is the observation noise covariance matrix.

[0295] Status Update:

[0296]

[0297] Covariance update:

[0298]

[0299] Deep learning model training and fine-tuning

[0300] In order to improve the accuracy and robustness of content recognition, a deep learning model is used to recognize handwritten content.

[0301] Dataset preparation

[0302] Data collection: Collect handwritten text images in multiple fonts and different writing habits.

[0303] Data labeling: Use tools such as LabelImg to label handwritten text word by word and generate corresponding label files.

[0304] Data enhancement: Rotate, scale, transform perspective, add noise, etc. to the original image to increase data diversity.

[0305] Model selection

[0306] Network structure: CRNN (Convolutional Recurrent Neural Network), an architecture that combines convolutional neural network (CNN) and recurrent neural network (RNN).

[0307] Model composition:

[0308] Convolutional layer: used to extract spatial features of images.

[0309] Recurrent layer: Use bidirectional LSTM to capture sequence information.

[0310] Output layer: CTC (Connectionist Temporal Classification) layer is used to realize the recognition of indefinite length sequences.

[0311] Model Training

[0312] Loss function: CTC loss function is used to solve the problem of mismatch between label and prediction sequence length.

[0313] Optimizer: Use the Adam optimizer and set the initial learning rate to 0.001.

[0314] Training process:

[0315] Batch training: Set the batch size to 64 and input training data batch by batch.

[0316] Iterations: Train until the validation set accuracy no longer improves to prevent overfitting.

[0317] Model saving: The model that achieves the best performance on the validation set is saved.

[0318] Model fine-tuning

[0319] Purpose: To further improve the recognition performance of the model for specific users or scenarios.

[0320] method:

[0321] Use user-specific handwriting data for transfer learning or fine-tuning.

[0322] Adjust the parameters of the last few layers of the model, retaining the feature extraction part of the previous training.

[0323] Model deployment

[0324] Model compression: Use model pruning, quantization and other technologies to reduce model size and facilitate deployment on embedded devices.

[0325] Reasoning optimization: Use tools such as TensorRT to optimize the reasoning speed of the model.

[0326] The wireless communication module is used to transmit control data to the mobile device, so as to operate and control the pen;

[0327] Specifically, the wireless communication module

[0328] Function: Realize data communication with mobile devices.

[0329] Implementation details:

[0330] Module selection: Use a wireless module that supports BLE 5.0, such as Nordic Semiconductor's nRF52840.

[0331] Protocol support: Support BLE's GATT protocol to achieve stable data transmission and low power consumption design.

[0332] Data transmission and communication

[0333] Data Packaging and Encryption:

[0334] The writing trajectory, location information, timestamp and other data are packaged into a data frame.

[0335] AES-128 algorithm is used for data encryption, and the key is stored in a secure area.

[0336] BLE communication protocol design:

[0337] Connection establishment: The device starts broadcasting, and the mobile device scans and connects.

[0338] Data transmission: GATT protocol is used, and custom characteristics are used to transmit data.

[0339] Data reception and decryption: The mobile device receives the data frame and decrypts it using the same key.

[0340] The power management module is used to supply power to the entire system.

[0341] Specifically, the power management module

[0342] Function: Manage battery power, provide stable power output, and extend device life.

[0343] Implementation details:

[0344] Power chip: Use a power management IC that integrates charging management, over-discharge protection and voltage regulation, such as TI's BQ24075.

[0345] Power output: Provides 3.3V and 1.8V regulated output to meet the voltage requirements of different modules.

[0346] Specifically, the present invention is further described below by way of embodiments:

[0347] Mobile device software design

[0348] Main functional modules:

[0349] Data receiving module: receives data from the smart pen through the BLE interface.

[0350] Data decryption and analysis module: decrypts the received data and analyzes the writing trajectory and location information.

[0351] Trajectory reconstruction module: reconstructs the user's writing trajectory based on the received position information.

[0352] Content recognition module: calls the deep learning model to recognize handwriting traces and output text content.

[0353] User interface module: provides a friendly interface, displays written content, and supports functions such as editing, saving, and sharing.

[0354] See the mobile end flow chart Figure 4 .

[0355] Based on the same inventive concept, the present invention also provides a computer device, which includes: one or more processors, and a memory for storing one or more computer programs; the program includes program instructions, and the processor is used to execute the program instructions stored in the memory. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is used to implement one or more instructions, specifically for loading and executing one or more instructions in a computer storage medium to implement the above method.

[0356] It needs to be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium, on which a computer program is stored, and the computer program is executed by a processor to execute the above method. The storage medium can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electrical, magnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0357] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0358] The above shows and describes the basic principles, main features and advantages of the present disclosure. Those skilled in the art should understand that the present disclosure is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present disclosure. Without departing from the spirit and scope of the present disclosure, the present disclosure may have various changes and improvements, and these changes and improvements fall within the scope of the present disclosure to be protected.

Claims

1. A codeless smart pen control system based on multi-sensor fusion, characterized in that: include: Pressure sensor module, acceleration sensor module, camera module, microcontroller unit, wireless communication module and power management module; The pressure sensor module is used to detect the contact pressure data between the pen tip and the paper surface, and send the contact pressure data between the pen tip and the paper surface to the microcontroller unit; The acceleration sensor module is used to collect the three-axis acceleration data of the pen and send the three-axis acceleration data of the pen to the microcontroller unit; The camera module is used to collect images of the writing area and send the collected images of the writing area to the microcontroller unit; The microcontroller unit determines the writing state of the pen according to the contact pressure data between the pen tip and the paper surface, and sends an acceleration acquisition signal to the acceleration sensor module when the pen enters the writing state; After receiving the three-axis acceleration data of the pen, the writing trajectory is calculated and an image acquisition signal is sent to the camera module; Identify and locate the writing content according to the image of the writing area, and control the operation of the pen in combination with the writing trajectory to obtain control data, and send the control data to the wireless communication module; The wireless communication module is used to transmit control data to the mobile device, so as to operate and control the pen; The power management module is used to supply power to the entire system.

2. The codeless smart pen control system based on multi-sensor fusion according to claim 1, characterized in that: The pressure sensor module preprocesses the contact pressure data between the pen tip and the paper surface: A first-order IIR filter is used to filter out high-frequency noise. The recursive formula of the filter is: P filtered (n)=αP raw (n)+(1-α)P filtered (n-1) Among them, α is the preset coefficient, P raw (n) is the original pressure value of the nth sampling.

3. The codeless smart pen control system based on multi-sensor fusion according to claim 1, characterized in that: The acceleration sensor module acquires three-axis acceleration data (a x (n), a y (n), a z (n)); Preprocessing steps: Zero bias calibration: When the device is stationary, collect multiple sets of data and calculate the zero bias value (a x0 , a y0 , a z0 ); Deduct zero bias: Gravity compensation: Since the accelerometer also senses gravity acceleration, it is necessary to compensate for the gravity component according to the pen's posture. Assuming that the pen mainly moves in a plane, gravity compensation is processed; Filtering: Use Kalman filter to filter the acceleration data; Kalman filter algorithm: Status definition: Define the state vector as State transfer equation: x k =x k-1 +w k Among them, w k is the process noise, assumed to be zero-mean Gaussian white noise Observation equation: z k =x k +V k Among them, z k is the measured value, v k To measure noise; Filtering process: Prediction Steps: in, is the prediction covariance, Q is the process noise covariance matrix; Update steps: Where R is the measurement noise covariance matrix.

4. The codeless smart pen control system based on multi-sensor fusion according to claim 1, characterized in that: During the writing process, the camera module collects a frame of image at fixed intervals or when a special event is detected; Preprocessing steps: Image scaling: scale the original high-resolution image to 640x480; Grayscale: Convert a color image to a grayscale image; Image enhancement: Adaptive histogram equalization (CLAHE) is used to improve image contrast; Denoising: Use a Gaussian filter to smooth the image and remove high-frequency noise.

5. The codeless smart pen control system based on multi-sensor fusion according to claim 1, characterized in that: The process of the microcontroller unit judging the writing state of the pen: Set the pressure threshold P threshold , when P filtered (n)>P threshold When the writing state is entered, the data collection process is started.

6. The codeless smart pen control system based on multi-sensor fusion according to claim 5, characterized in that: The calculation process of the writing trajectory of the microcontroller unit is as follows: Based on the preprocessed acceleration data, the velocity and displacement are calculated using the numerical integration method; Speed ​​calculation: Among them, v x (n), v y (n) represents the velocity of the nth sampling point in the X-axis and Y-axis directions, v x (n-1), v y (n-1) represents the speed of the n-1th sampling point in the X-axis and Y-axis directions, is the calibrated X-axis and Y-axis acceleration of the nth sampling point, and is the acceleration data after zero bias calibration and filtering, Δt is the time interval between two adjacent sampling points; Displacement calculation: Among them, s x (n), s y (n) represents the displacement velocity of the nth sampling point in the X-axis and Y-axis directions, s x (n-1), s y (n-1) represents the displacement of the n-1th sampling point in the X-axis and Y-axis directions, v x (n), v y (n) is the velocity of the nth sampling point in the X-axis and Y-axis directions, and Δt is the time interval between two adjacent sampling points.

7. The codeless smart pen control system based on multi-sensor fusion according to claim 6, characterized in that: The process of the microcontroller unit identifying and locating the writing content according to the image of the writing area includes: feature point extraction, feature point matching, mismatched feature point elimination and writing position positioning.

8. The codeless smart pen control system based on multi-sensor fusion according to claim 7, characterized in that: The process of feature point extraction: The SIFT algorithm is used to extract feature points. The steps are as follows: Scale space construction: Perform Gaussian blur on the image I(x, y) to generate images L(x, y, v) of different scales: L(x,y,σ)=G(x,y,σ)*I(x,y) Among them, G(x, y, σ) is the Gaussian kernel function, and * represents the convolution operation; Difference scale space: Calculate the difference between adjacent scale images and obtain the difference image D(x, y, σ): D(x,y,σ)=L(x,y,kσ)-L(x,y,σ) Key point detection: In the difference scale space, find the extreme points; Accurate positioning of key points: The initial key points are processed precisely to remove low-contrast and edge response points; Direction assignment: Compute the gradient magnitude and direction in the neighborhood of the keypoint: Determine the main direction of the key point based on the gradient direction histogram; Feature descriptor generation: Centering on the key point, a neighborhood window is established, and the domain window is divided into multiple sub-areas; In each sub-region, the gradient histogram in each direction is calculated to generate a feature vector; The feature point matching adopts the KNN algorithm, and the steps are as follows: Compute the distance between eigenvectors: The Euclidean distance is used to measure the similarity between two feature vectors; Nearest neighbor search: For each feature point to be matched, find the two feature points closest to each other in the template feature set; Ratio test: Calculate the nearest neighbor distance d1 and the next nearest neighbor distance d2 if The match is considered valid; The RANSAC algorithm is used to eliminate mismatched feature points; Writing position calculation method: Use the obtained affine transformation matrix A to transform the points in the camera coordinate system to the paper coordinate system; For the image point (x′, y′) captured by the camera, its corresponding paper coordinates (x, y) are:

9. The codeless smart pen control system based on multi-sensor fusion according to claim 8, characterized in that: The microcontroller unit uses an extended Kalman filter (EKF) to fuse the displacement estimation of the acceleration sensor with the positioning result of the camera. The process is as follows: State vector definition: State transition model: x k =Fx k-1 +This k-1 +w k-1 in, F is the state transfer matrix: B is the control input matrix: u k-1 Enter the acceleration: w k-1 is the process noise; Observation model: z k =Hx k +V k in, H is the observation matrix: vk is the observation noise; The prediction step of filtering: Status prediction: Covariance prediction: Where Q is the process noise covariance matrix; Update steps: Calculate the Kalman gain: Where R is the observation noise covariance matrix; Status Update: Covariance update:

10. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: The memory stores a computer program that can be run on the processor. When the processor loads and executes the computer program, a codeless smart pen control system based on multi-sensor fusion according to any one of claims 1 to 9 is adopted.

Images