Method for collecting electronic writing data, and related device and storage medium
Patent Information
- Application Number
- CN202611092236.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-22
- Publication Date
- 2026-10-16
AI Technical Summary
[0009] A fourth aspect of the present invention provides a computer program product, including instructions that, when executed by a processor, implement the steps of the method for acquiring electronic writing data provided in the first aspect of the above embodiments.
Smart Images
Figure CN122929578A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer peripheral input device technology, and in particular to a method for acquiring electronic writing data, as well as related equipment and storage media. Background Technology
[0002] With the development of electronic technology, electronic pens are widely used in scenarios such as electronic writing, drawing, and signing. Electronic pens typically have a miniature camera at the tip, which captures images of the paper in real time along the area the tip passes through, and transmits the captured image data to an external display device, thereby realizing the digital presentation and real-time display of handwriting.
[0003] Currently, most electronic pens use full-frame image transmission, which means that every complete frame of the image captured by the camera is sent to the external display device. However, during writing, the electronic pen's camera usually captures images continuously at a high sampling frequency. The full-frame transmission method leads to a sharp increase in data volume, consuming a large amount of communication bandwidth and causing significant transmission delays, making it difficult to meet the low-latency requirements for a smooth writing experience. Summary of the Invention
[0004] To address the technical problem of insufficient writing smoothness caused by the full-frame image transmission method in existing electronic pens, this invention provides a method for acquiring electronic writing data, as well as related equipment and storage media.
[0005] The first aspect of this invention provides a method for collecting electronic writing data, applied to an electronic pen, comprising: Detect pen tip pressure; When the pen tip pressure exceeds a preset threshold, images of the local area corresponding to the pen tip are continuously acquired to obtain a sequence of frame images arranged in the acquisition order. Perform difference processing on the current frame image and the previous frame image to extract the newly added writing data area of the current frame image relative to the previous frame image; The newly added writing data area is packaged into an incremental data packet corresponding to the current frame image; The incremental data packet is sent to an external display device so that the external display device can overlay the incremental data packet onto the canvas for display.
[0006] A second aspect of this invention provides a device for collecting electronic writing data, including a pen body comprising a pen tip and a pen shaft, and further comprising: The pressure detection unit is located inside the pen body; A motion detection unit, located inside the pen body, is used to detect the motion parameters of the pen body; An image acquisition unit is located on the pen body and is used to acquire images of a local area corresponding to the pen tip; A control unit, located within the pen body, is electrically connected to the pressure detection unit, the motion detection unit, and the image acquisition unit. It controls the pressure detection unit to detect the pen tip pressure. When the pen tip pressure exceeds a preset threshold, it controls the image acquisition unit to continuously acquire images of the local area corresponding to the pen tip, obtaining a sequence of frame images arranged in the acquisition order. It performs differential processing on the current frame image and the previous frame image to extract the newly added writing data area relative to the previous frame image. It packages the newly added writing data area into an incremental data packet corresponding to the current frame image. The incremental data packet is sent to an external display device, allowing the external display device to overlay the incremental data packet onto a canvas for display.
[0007] A third aspect of the present invention provides an electronic writing system, including a device for collecting electronic writing data and a display device, wherein the device for collecting electronic writing data and the display device are communicatively connected, and the device for collecting electronic writing data is the same as the device for collecting electronic writing data provided in the second aspect of the above embodiments.
[0008] A fourth aspect of the present invention provides a computer-readable storage medium having instructions stored thereon, which, when executed by a processor, implement the steps of the method for acquiring electronic writing data provided in the first aspect of the above embodiments.
[0009] A fourth aspect of the present invention provides a computer program product, including instructions that, when executed by a processor, implement the steps of the method for acquiring electronic writing data provided in the first aspect of the above embodiments.
[0010] Compared to existing technologies, the method for acquiring electronic writing data provided in this invention extracts newly added writing data areas through differential processing and only packages and sends these newly added writing data areas, rather than transmitting the entire image of each frame. Since the newly added handwriting area in each frame typically occupies only a small proportion of the image area during the writing process, the data transmission volume of the method for acquiring electronic writing data in this invention is significantly reduced compared to the full-frame image transmission method. This effectively reduces communication bandwidth usage and transmission latency, thereby improving the smoothness of writing. Attached Figure Description
[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. The drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein: Figure 1 This is a schematic diagram of the hardware architecture of an electronic writing system provided in an embodiment of the present invention; Figure 2 A flowchart illustrating a method for acquiring electronic writing data according to an embodiment of the present invention; Figure 3 for Figure 2 The diagram shows a detailed process flow diagram of one embodiment of the method for collecting electronic writing data. Detailed Implementation
[0012] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0013] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.
[0014] Please see Figure 1 This is a schematic diagram of the hardware architecture of an electronic writing system provided in an embodiment of the present invention. The electronic writing system 100 includes a device 10 for collecting electronic writing data and a display device 20 connected in communication. The display device 20 is a smart device with a display screen, such as a tablet computer, smartphone, electronic whiteboard, computer monitor, etc.
[0015] The device 10 for collecting electronic writing data includes a pen body 11, which includes a pen tip 111 and a pen body 112. The pen body 112 is equipped with a pressure detection unit 12, a motion detection unit 13, an image acquisition unit 14, and a control unit 15.
[0016] The pressure detection unit 12 is located inside the pen body 112 and is electrically connected to the control unit 15. It is used to detect the pressure on the pen tip.
[0017] In one specific embodiment, the pressure detection unit 12 is installed at the rear end of the contact point of the pen tip 111. When the pen body 11 writes on the writing surface, the pressure on the pen tip 111 can be transmitted to the pressure detection unit 12, thereby detecting the pen tip pressure. The pressure detection unit 12 can specifically adopt a piezoresistive or capacitive thin-film pressure sensor with a range of 0~5N, an accuracy of ±0.01N, and a signal sampling rate of ≥200Hz.
[0018] The motion detection unit 13 is located inside the pen body 112 and is electrically connected to the control unit 15. It is used to detect the motion parameters of the pen body 11, including triaxial acceleration and angular velocity data.
[0019] In one specific embodiment, the motion detection unit 13 includes an inertial measurement module (IMU), which includes an accelerometer and a gyroscope. It consists of a 6-axis IMU (3-axis accelerometer + 3-axis gyroscope), and the model can be a low-power device such as BMI160 or ICM-20948. The sampling rate is 200~400Hz, and it supports hardware FIFO buffering. It outputs the pen's three-axis acceleration and angular velocity data to the control unit 15 in real time.
[0020] The image acquisition unit 14 is located on the pen body 112 and is electrically connected to the control unit 15. It is used to acquire images of the local area corresponding to the pen tip 111.
[0021] In one specific embodiment, the image acquisition unit 14 includes a camera, which is located on the side of the pen body 11 near the pen tip 111 and exposed outside the pen body 11. The camera is installed at the front end of the pen body 11, 5-15mm behind the pen tip 111. The optical axis of the camera forms an angle of 15°-45° with the axis of the pen body 11 and faces the writing surface. The field of view of the camera covers the local area where the pen tip 111 contacts the writing surface.
[0022] When a user holds the pen 11 and writes on the writing surface, the pen tip 111 moves across the surface, and the camera moves synchronously with the pen 11. The local area covered by the camera's field of view always follows the pen tip 111, allowing the camera to continuously capture the writing currently being produced near the pen tip 111. Because the camera's field of view is limited, and the writing surface is usually much larger than the field of view, the camera captures an image of a local area of the writing surface, rather than a complete image of the entire writing surface.
[0023] Preferably, the camera's field of view covers a local area centered on the point of contact between the pen tip 111 and the writing surface.
[0024] The control unit 15 is located inside the pen body 112 and is used to control the device 10 for collecting electronic writing data to perform writing data acquisition operations.
[0025] In one embodiment, the control unit 15 includes a processor and a memory. The processor performs arithmetic and logical operations and instruction control, while the memory is electrically connected to the processor and stores the computer program executed by the processor and the data generated during processing. Preferably, the processor is an ARM Cortex-M4 microcontroller (MCU) or an equivalent embedded processor chip, and the memory includes flash memory and static random access memory (SRAM). The flash memory is used to store firmware programs, and the SRAM is used as a runtime data cache.
[0026] The device 10 for collecting electronic writing data also includes a communication unit 16 and a power supply unit 17.
[0027] The communication unit 16 is located inside the pen body 112 and is electrically connected to the control unit 15 for communication with the display device 20.
[0028] In a further embodiment, the image acquisition unit 14 also includes a supplementary lighting unit 18, which is located at the front end of the pen body 11 and arranged around the lens of the image acquisition unit 14. The supplementary lighting unit 18 includes 2 to 4 infrared LEDs (wavelength 850nm) or white LEDs with adjustable brightness (e.g., by controlling the current through PWM to ensure that the illuminance on the writing surface is not less than 100 Lux), to ensure clear image texture under different paper materials.
[0029] In one specific embodiment, the communication unit 16 uses a Bluetooth Low Energy (BLE) or Wi-Fi module to communicate with the display device 20 via a wireless protocol.
[0030] The power supply unit 17 is located inside the pen body 112 and is electrically connected to the power supply terminals of the pressure detection unit 12, motion detection unit 13, image acquisition unit 14, control unit 15 and communication unit 16, respectively, to provide working power to each unit.
[0031] In one embodiment, the power supply unit 17 includes a rechargeable battery and a power management module (PMU). The power management module includes a load switch and a low-dropout regulator (LDO).
[0032] The hardware architecture of the electronic writing system 100 has been described above. The following describes a method for collecting electronic writing data provided by an embodiment of the present invention. This method can be applied to the above-mentioned device 10 for collecting electronic writing data.
[0033] Please see Figure 2 This is a flowchart illustrating a method for acquiring electronic writing data according to an embodiment of the present invention. The method includes: 101. Test the pen tip pressure.
[0034] In this embodiment, the pressure detection unit 12 detects the contact pressure between the pen tip 111 and the writing surface in real time. Specifically, when the pen tip 111 is not in contact with the writing surface, the pressure value output by the pressure detection unit 12 is zero or close to zero; when the user holds the pen body 11 and presses the pen tip 111 against the writing surface, the pressure value detected by the pressure detection unit 12 increases with the increase of writing pressure.
[0035] 102. When the pen tip pressure exceeds the preset threshold, continuously acquire images of the local area corresponding to the pen tip to obtain a sequence of frame images arranged in the acquisition order.
[0036] In this embodiment, the device 10 for acquiring electronic writing data can determine whether the pen body 11 is in a writing state by judging whether the pen tip pressure is greater than a preset threshold. When the pen body 11 is in a writing state, the image acquisition unit 14 is controlled to continuously take pictures to obtain a sequence of frame images that can display the process of writing handwriting changes. The acquired images are arranged in chronological order to form a frame image sequence F1, F2, F3, ..., Ft, ...
[0037] When the pen tip pressure exceeds a preset threshold, images of the local area corresponding to the pen tip are continuously acquired to obtain a sequence of frame images arranged in the acquisition order, which may specifically include: When the pen tip pressure exceeds a preset threshold, a trigger signal is output; In response to the trigger signal, images of the local area corresponding to the pen tip are continuously acquired at a preset frame rate to obtain a sequence of frame images.
[0038] The device 10 for collecting electronic writing data can achieve pressure threshold detection and trigger signal output through pure hardware circuitry, software, or a combination of hardware and software. Hardware implementation: The pressure detection unit 12 of the electronic writing acquisition device 10 is connected to the control unit 15 through an analog front-end and comparator circuit. When the detected pressure value exceeds a preset threshold (e.g., 0.1N), the comparator outputs a rising edge interrupt signal as a trigger signal to the control unit 15. This method has the advantages of fast response speed and low power consumption.
[0039] Software implementation: The pressure detection unit 12 outputs the real-time detected pressure value to the control unit 15 after analog-to-digital conversion. The control unit 15 determines whether the pressure value exceeds the preset threshold through the software program running on it, and generates a trigger signal when it does.
[0040] The hardware and software combination method is as follows: the pressure signal is acquired and initially amplified by the hardware circuit, and the threshold judgment and trigger signal generation are performed by the control unit 15 through software.
[0041] All three implementation methods described above fall within the protection scope of this invention.
[0042] It should be noted that during the process of acquiring frame image sequences in response to the trigger signal, the control unit 15 needs to ensure that the camera can completely capture the changes in the writing, and there must be a sufficient overlap area in the field of view between two adjacent frames in the direction of pen tip movement. If the overlap area is insufficient, some newly generated writing between adjacent frames may fall outside the camera's field of view, resulting in the loss of writing data. Therefore, the control unit 15 controls the camera of the image acquisition unit 14 to continuously acquire images of the local area corresponding to the pen tip 111 at a preset frame rate (e.g., 60fps). The preset frame rate must match the movement speed of the pen tip on the writing surface to ensure that any new writing generated between any two adjacent frames falls within the camera's field of view.
[0043] Those skilled in the art will understand that parameters such as the field of view, installation position, optical axis angle, and frame rate of the camera can be adjusted according to actual application requirements. For example, the field of view can be increased or the frame rate can be improved to accommodate faster writing speeds, or the field of view can be reduced to improve image resolution. These adjustments do not exceed the protection scope of the embodiments of the present invention.
[0044] During image acquisition, parameters such as the brightness, color temperature, and activation strategy of the supplementary lighting module can be adjusted according to the actual application scenario. For example, in scenarios with sufficient ambient light, the supplementary lighting brightness can be reduced or the supplementary lighting module can be turned off to reduce power consumption; in scenarios with low light or writing on low-reflectivity paper (such as kraft paper or rough paper), the supplementary lighting brightness can be increased to ensure that the image acquisition unit 14 can capture clearly discernible paper texture and handwriting outlines; in scenarios with colored handwriting, white LED supplementary lighting can be used to preserve the true color information of the handwriting; in scenarios where it is necessary to reduce ambient light interference, infrared LED supplementary lighting can be used in conjunction with a narrowband filter to filter out stray light interference in the ambient light and improve the image signal-to-noise ratio. The above adjustments to the supplementary lighting parameters do not exceed the protection scope of the embodiments of the present invention.
[0045] It should also be noted that the image acquisition unit 14 only operates during writing. Disconnecting its power supply when not writing reduces energy consumption and extends battery life. The control unit 15 can respond to the trigger signal from the pressure detection unit 12 to control the power supply to the image acquisition unit 14 and the supplementary lighting module. Specifically, when the pen tip pressure exceeds a preset threshold, the power supply unit 17 connects the power supply to the image acquisition unit 14; when the pen tip pressure is below the preset threshold, the power supply unit 17 disconnects the power supply to the image acquisition unit 14, and the image acquisition unit 14 stops acquiring images. That is, when not writing, this part of the circuit is completely powered off, with standby power consumption <10µA, thereby extending battery life. In other words, the acquisition control unit 15, based on the trigger signal, first controls the image acquisition unit 14 to power on, and then controls it to continuously take pictures at a preset frame rate.
[0046] Preferably, the control unit 15 also activates the motion detection unit 13 at the same time to start recording the acceleration and angular velocity data of the pen body 11.
[0047] In a preferred embodiment, because the posture of the pen 11 changes continuously during writing, the angle of the camera optical axis of the image acquisition unit 14 relative to the writing surface also changes accordingly, which may result in perspective distortion in the acquired image. The method for acquiring electronic writing data may further include: Obtain the current motion parameters of the electronic pen at the moment of acquisition of the current frame image; Based on the current motion parameters, determine the three-dimensional spatial posture of the electronic pen at the moment of acquisition of the current frame image; Based on the three-dimensional spatial pose, perform perspective correction preprocessing on the current frame image.
[0048] Specifically, while acquiring each frame of image, the control unit 15 synchronously acquires the gyroscope data (angular velocity) and accelerometer data (acceleration) at the current moment through the motion detection unit 13. Based on the gyroscope data and accelerometer data, the control unit 15 calculates the attitude angles of the pen body 11 in three-dimensional space at the current moment, including pitch, yaw, and roll.
[0049] Based on the aforementioned attitude angle, and combined with the installation geometry parameters of the camera of the image acquisition unit 14 on the pen body 11 (i.e., the fixed angle and distance of the camera optical axis relative to the pen shaft axis), the control unit 15 calculates the spatial tilt angle of the camera optical axis relative to the normal of the writing surface at the current moment.
[0050] Then, the control unit 15 constructs a perspective transformation matrix (i.e., a homography matrix) based on the spatial tilt angle. This matrix describes the pixel mapping relationship between the tilted shooting viewpoint and the normal shooting viewpoint. The method for constructing the perspective transformation matrix is a conventional technique in the field of image processing: the homography matrix between the normal and tilted viewing views is calculated based on the intrinsic parameter matrix of the camera and the extrinsic parameter rotation matrix determined by the spatial tilt angle. This embodiment of the invention will not elaborate further on this method.
[0051] Finally, the control unit 15 uses the perspective transformation matrix to perform inverse perspective transformation on the current frame image, correcting the current frame image acquired from the tilted viewpoint to the current frame image from the normal viewpoint (i.e., the view viewed from a direction perpendicular to the paper).
[0052] 103. Perform differential processing on the current frame image and the previous frame image to extract the newly added writing data area in the current frame image relative to the previous frame image.
[0053] In this embodiment, after the control unit 15 obtains the frame image sequence, it performs differential processing on the current frame image and the previous frame image to extract the newly added writing data area of the current frame image relative to the previous frame image.
[0054] It is understandable that the frame image sequence is a collection of frame images obtained by the image acquisition unit 14 at a preset frame rate. In order to ensure real-time feedback of the handwriting on the display device 20, the control unit 15 can perform differential processing on each new frame image acquired.
[0055] Based on the above, the control unit 15 performs frame-by-frame iterative processing on the frame image sequence. For each new frame acquired, the difference between the current frame Ft and the previous frame Ft-1 is calculated. This difference processing extracts the newly added writing data region in the current frame relative to the previous frame, which may specifically include: Calculate the pixel difference between the current frame image and the previous frame image to obtain the difference image.
[0056] Threshold segmentation is performed on the difference image, and pixel regions whose difference exceeds the noise threshold and whose continuous area is larger than the preset area are identified as newly added writing data regions.
[0057] Specifically, the control unit 15 can first calculate the difference image using the following formula: D(x,y) = |Ft(x,y) – Ft-1(x,y)| Where (x,y) represents the coordinates of the pixel in the image coordinate system.
[0058] Subsequently, the control unit 15 can process the difference image D i Binarization is performed, marking pixels with a difference greater than a preset noise threshold Tnoise as 1, and pixels with a difference less than or equal to Tnoise as 0, to obtain a binary difference mask image M. i : M i (x, y) = 1, if D i (x,y)>Tnoise; M i (x,y) = 0, if D i (x,y) ≤ Tnoise.
[0059] The preset noise threshold Tnoise can be set and adjusted according to the noise level of the image sensor of the image acquisition unit 14, for example, it can be set to 20 / 255 of the grayscale difference.
[0060] Then, the control unit 15 performs binary difference masking on the image M. iPerform connected component analysis to extract all connected pixel regions. For each connected component, calculate its area (i.e., the number of pixels it contains), and discard connected components with an area smaller than a preset area threshold, classifying them as noise. The preset area is the area of the minimum valid writing point, for example, 3×3 pixels.
[0061] After area filtering, the remaining connected components are the valid new writing data regions (i.e., the newly generated handwriting in the current frame).
[0062] 104. The newly added writing data area is the incremental data packet corresponding to the current frame image.
[0063] In this embodiment, the control unit 15 packages the newly added writing data area into an incremental data packet corresponding to the current frame image according to a predetermined format. It is understood that the incremental data packet is a data interaction format agreed upon between the control unit 15 and the display device 20, and its specific structure can be defined and adjusted according to the communication protocol and actual application requirements.
[0064] The newly added writing data area will be the incremental data packet corresponding to the current frame image, which may specifically include: 1041. Perform image matching between the current frame image and the previous frame image to determine the inter-frame displacement.
[0065] In this embodiment, the control unit 15 performs frame-by-frame iterative processing on the frame image sequence. Each time a new image is acquired, image matching is performed between the current frame Ft and the previous frame Ft-1. The image matching between the current frame and the previous frame to determine the inter-frame displacement specifically includes the following steps: Divide the previous frame image into several pixel blocks; For each pixel block, template matching is performed within a preset search range of the current frame image to obtain the inter-frame motion vector of that pixel block; Calculate the inter-frame motion vectors of all pixel blocks, and take the median or average value as the inter-frame displacement.
[0066] Specifically, the control unit 15 first divides the previous frame image into several non-overlapping pixel blocks (each pixel block is N×N pixels in size, where N can be 8 or 16, but is not limited to these values); then, for each pixel block in the previous frame image, template matching is performed in the current frame image with the position of the pixel block as the center and within a preset search range (e.g., ±15 pixels), and the normalized cross-correlation coefficient is calculated. The position with the largest correlation coefficient is taken as the best matching position of the pixel block in the current frame image, thereby obtaining the inter-frame motion vector of the pixel block; finally, the motion vectors of all pixel blocks in the previous frame image are statistically analyzed, outliers that deviate from the overall trend are removed (e.g., median filtering or standard deviation filtering is used), and the average or median value of the remaining motion vectors is taken to obtain the overall inter-frame displacement.
[0067] It should be noted that there is no sequential dependency between step 1041 (image matching) and step 103 (differential processing). Both steps take the current frame image and the previous frame image as input, and their respective output results (inter-frame displacement, newly added written data area) are independent of each other's input conditions. Therefore, they can be executed sequentially or in parallel, and the specific execution order can be flexibly determined according to the computing resources and processing strategy of the control unit 15. This embodiment uses the example of executing step 103 first and then step 1041 as an example, but this order does not constitute a limitation on the present invention.
[0068] 1042. Determine the local coordinates of the newly added writing data area in the current frame image.
[0069] In this embodiment, a field of view origin is defined in the frame image, and a coordinate system of the frame image is established with this point as the coordinate origin. After the control unit 15 extracts the newly added writing data area, it determines the local coordinates of the newly added writing data area in the coordinate system.
[0070] It is understandable that the pixel size of each frame image acquired by the image acquisition unit 14 is the same, and the origin of the field of view can be any point in the frame image. The selection rule for the origin of the field of view is consistent across all frames, ensuring that the coordinate system definition rule is the same for each frame image. Specifically, the origin of the field of view can be selected as a position with clear geometric meaning, such as the upper left corner, upper right corner, lower left corner, lower right corner, or geometric center of the frame image, as long as the selection rule for the origin of the field of view is consistent across all frames.
[0071] In this embodiment, the top-left pixel of the frame image is used as the origin of the field of view. A coordinate system for the frame image is established with this point as the origin and the row and column directions of the pixels as the coordinate axes. Under this coordinate system, the local coordinates of any pixel within the frame image are the row and column indices of that pixel in the image.
[0072] Determining the local coordinates of the newly written data region within the current frame image can specifically include: Determine the minimum bounding rectangle of the newly added writing data area in the frame image coordinate system, and obtain the local coordinates of the reference point of the minimum bounding rectangle in the frame image coordinate system, as well as the width and height of the minimum bounding rectangle.
[0073] The reference point is the positioning reference point of the smallest bounding rectangle. The control unit 15 uses the upper left corner vertex of the smallest bounding rectangle as the reference point of the region. In other embodiments, other reference point definition methods can also be used, such as using the geometric center or centroid of the newly added writing data area as the reference point.
[0074] In other embodiments, the control unit 15 may also determine the coordinates of each pixel in the newly written data area in the frame image coordinate system to determine the position of the newly written data area in the current frame image.
[0075] 1043. Determine the global coordinates of the newly added writing data area in the canvas based on the inter-frame displacement and local coordinates.
[0076] In this embodiment, the control unit 15 determines the global coordinates of the newly added writing data area on the canvas based on the inter-frame displacement and local coordinates, which may specifically include: Obtain the global coordinates of the view origin of the previous frame image in the canvas; The global coordinates of the view origin of the current frame image in the canvas are determined based on the inter-frame displacement and the global coordinates of the view origin of the previous frame image in the canvas. The global coordinates of the newly added writing data area on the canvas are determined based on the global coordinates of the current frame image's view origin on the canvas and the local coordinates corresponding to the newly added writing data area.
[0077] Specifically, the control unit 15 stores the global coordinates of the view origin of the previous frame image in the canvas. For the first frame image, the global coordinates of its view origin in the canvas can be predefined, that is, the initial display position of the handwriting in the canvas can be customized. For example, it can be set to the preset starting position of the canvas, such as the coordinate origin (0, 0), indicating that the view origin of the first frame image is positioned at the starting drawing position of the canvas; it can also be set to other preset values according to actual display needs, such as starting drawing in the central area of the canvas. After determining the initial global coordinates of the view origin of the first frame, for the view origin of each subsequent frame, through this step 106, the global coordinates of the view origin in the canvas are obtained by superimposing the global coordinates of the view origin of the previous frame with the inter-frame displacement. Through the above frame-by-frame recursion, the view origin of each frame image is accurately mapped to the global coordinates of the canvas.
[0078] It should be noted that the "canvas" involved in this embodiment of the invention is a canvas in a logical coordinate system, and its resolution is consistent with the resolution of the camera of the image acquisition unit 14. That is, the local coordinates in the frame image coordinate system in step 105, the inter-frame displacement in step 103, and the global coordinates in step 106 are all based on the camera acquisition resolution. The canvas maintained locally by the display device 20 also adopts a logical coordinate system consistent with the camera resolution. Therefore, the coordinate mapping in step 106 and the drawing position calculation in step 109 are both completed in the same logical coordinate system.
[0079] At this point, the global coordinates of the view origin of the current frame image in the canvas can be determined using the following formula: O Ft,x= O Ft-1,x + u O Ft,y = O Ft-1,y + v Among them, (O) Ft-1,x O Ft-1,y (O) represents the global coordinates of the view origin of the previous frame in the canvas. Ft,x O Ft,y (u,v) represents the global coordinates of the view origin of the current frame image in the canvas, and (u,v) represents the inter-frame displacement.
[0080] The global coordinates of the newly added writing data area on the canvas can be determined using the following formula: Xt= O Ft,x +x t Yt= O Ft,y +y t Where (Xt, Yt) are the global coordinates of the reference point of the minimum bounding rectangle of the newly added writing data area in the canvas, (xt, Yt) t y t () represents the local coordinates of the reference point of the smallest bounding rectangle of the newly added writing data area.
[0081] When the physical display resolution of the display device 20 differs from the aforementioned logical resolution, the display driver layer or graphics rendering engine of the display device 20 completes the viewport mapping from the logical canvas to the physical screen. This mapping is independent of the incremental data packet stitching algorithm of this embodiment. For example, the logical canvas can be stretched to full-screen display by setting a scaling factor, or displayed in a specific area of the screen according to the actual pixel ratio. Regardless of the viewport mapping method used, its essence is to transform the display of the already stitched canvas as a whole, without affecting the relative positional relationship between the pixels of the existing handwriting in the canvas, nor affecting the stitching position of the newly added pixel data in the canvas.
[0082] 1044. Based on local coordinates, extract the pixel data corresponding to the newly added writing data area from the current frame image; In this embodiment, the control unit 15 extracts the pixel data corresponding to the newly added writing data area from the current frame image based on the local coordinates of the newly added writing data area determined in step 1042.
[0083] Specifically, the control unit 15 can backtrack to the current frame image based on the local coordinates of the reference point of the smallest bounding rectangle of the newly added writing data area in the frame image coordinate system, as well as the width and height of the rectangle, and crop out the corresponding rectangular pixel block from the image.
[0084] Further, the control unit 15 uses the binary difference mask image M obtained in step 103 to perform pixel filtering on the cropped rectangular pixel block. For each pixel within the rectangular pixel block, the control unit 15 obtains the value of its corresponding position in the binary difference mask image M: if M(x, y) = 1, it indicates that the pixel belongs to the newly added writing data area, then the original color value of the pixel is retained, and its Alpha channel value is set to 255 (fully opaque); if M(x, y) = 0, it indicates that the pixel does not belong to the newly added writing data area, then the Alpha channel value of the pixel is set to 0 (fully transparent).
[0085] After the above processing, only the newly added pen stroke pixels in the rectangular pixel block are non-transparent, while the original old pen strokes and paper background pixels are transparent. When the display device subsequently overlays pixel data onto the canvas, the transparent pixels will not obscure the existing old pen strokes on the canvas; only the actual newly added pen strokes are drawn onto the canvas, thus achieving accurate overlay of the new pen strokes.
[0086] In other implementations, if the control unit 15 determines the local coordinates of each pixel in the newly added writing data area according to step 1042, then the control unit 15 can directly extract the corresponding pixel data from the current frame image based on the local coordinates of each pixel.
[0087] It should be noted that there is no mandatory sequential dependency between steps 1043 (global coordinate determination) and 1044 (pixel data extraction). Both steps take the local coordinates of the newly added writing data area determined in step 1042 as input, and their respective outputs (global coordinates, pixel data) are independent of each other's input conditions. Therefore, they can be executed sequentially or in parallel, and the specific execution order can be flexibly determined according to the computing resources and processing strategy of the control unit 15. This embodiment uses the example of executing step 1043 first and then step 1044, but this order does not constitute a limitation on the present invention.
[0088] 1045. Pack the global coordinates and pixel data into an incremental data packet corresponding to the current frame image.
[0089] In this embodiment, the control unit 15 packages the global coordinates and pixel data of the newly added writing data area into an incremental data package.
[0090] 105. Send the incremental data packet to the external display device so that the external display device can overlay the incremental data packet onto the canvas for display.
[0091] In this embodiment, the control unit 15 sends incremental data packets to the display device 20 via the communication unit 16. After receiving the incremental data packets, the display device 20 parses the global coordinates, pixel data, width, and height of the newly added writing data area on the canvas from the incremental data packets.
[0092] The display device 20 uses global coordinates as the drawing reference point to draw pixel data to the corresponding positions on the canvas. Specifically, for each pixel in the pixel data, the display device 20 maps its row and column numbers to the smallest bounding rectangle, draws the pixel value of that pixel on the canvas, and iterates through all pixels in the pixel data to complete the drawing. If the pixel data contains an alpha channel (transparency channel), pixels with full transparency (Alpha=0) are not drawn.
[0093] In other embodiments, the control unit 15 determines the local coordinates of each pixel in the newly added writing data area according to step 105, and the display device 20 parses the global coordinates and pixel value of each pixel in the newly added writing data area on the canvas from the incremental data packet, and draws each pixel one by one on the corresponding position of the canvas.
[0094] It should be noted that the above steps describe the complete processing flow for the second frame and subsequent frames. For the first frame image F1 in the frame image sequence, which is a blank reference frame captured at the moment the pen touches the paper, there is no pen mark on it, so no differential processing is required. Only the global coordinates of the viewpoint origin of the first frame image in the canvas need to be initialized to a preset starting value. Afterward, starting from the second frame, the process is repeated frame by frame according to steps 103 to 109.
[0095] In summary, the method for acquiring electronic writing data provided by this invention determines the relative positional relationship between newly added handwriting regions and non-newly added handwriting regions through differential detection. It does not rely on corners, inflection points, or any significant texture features in the image, and can stably extract positional information even on smooth handwriting, demonstrating strong adaptability. Furthermore, the method for acquiring electronic writing data provided by this invention only transmits the pixel data of the newly added writing data region and its relative positional information, significantly reducing the amount of data transmission and facilitating a low-latency, real-time writing experience.
[0096] This invention also provides a computer-readable storage medium storing instructions thereon, which, when executed by a processor, implement the steps of the method for acquiring electronic writing data provided in the above embodiments.
[0097] The present invention also provides a computer program or a computer program product including a computer program, which, when executed, will implement the steps of the method for collecting electronic writing data provided in the above embodiments.
[0098] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for collecting electronic writing data, applied to an electronic pen, characterized in that, include: Detect pen tip pressure; When the pen tip pressure exceeds a preset threshold, images of the local area corresponding to the pen tip are continuously acquired to obtain a sequence of frame images arranged in the acquisition order. Perform differential processing on the current frame image and the previous frame image to extract the newly added writing data area of the current frame image relative to the previous frame image; The newly added writing data area is packaged into an incremental data packet corresponding to the current frame image; The incremental data packet is sent to an external display device so that the external display device can overlay the incremental data packet onto the canvas for display.
2. The method according to claim 1, characterized in that, The step of packaging the newly added written data region into an incremental data packet corresponding to the current frame image includes: Image matching is performed between the current frame image and the previous frame image to determine the inter-frame displacement. Determine the local coordinates of the newly added written data region in the current frame image; The global coordinates of the newly added writing data area in the canvas are determined based on the inter-frame displacement and the local coordinates. Based on the local coordinates, extract the pixel data corresponding to the newly added writing data area from the current frame image; The global coordinates and the pixel data are packaged into an incremental data packet corresponding to the current frame image.
3. The method according to claim 2, characterized in that, The step of performing image matching between the current frame image and the previous frame image to determine the inter-frame displacement includes: The previous frame image is divided into several pixel blocks; For each pixel block, template matching is performed within a preset search range of the current frame image to obtain the inter-frame motion vector of each pixel block; The inter-frame motion vectors of all the pixel blocks are counted, and the median or average value is taken as the inter-frame displacement.
4. The method according to claim 3, characterized in that, Determining the global coordinates of the newly added writing data area in the canvas based on the inter-frame displacement and the local coordinates includes: Obtain the global coordinates of the view origin of the previous frame image in the canvas; The global coordinates of the view origin of the current frame image in the canvas are determined based on the inter-frame displacement and the global coordinates of the view origin of the previous frame image in the canvas. The global coordinates of the newly added writing data area in the canvas are determined based on the global coordinates of the origin of the current frame image in the canvas and the local coordinates.
5. The method according to claim 1, characterized in that, The step of performing difference processing on the current frame image and the previous frame image to extract the newly added written data region in the current frame relative to the previous frame image includes: Calculate the pixel difference between the current frame image and the previous frame image to obtain a difference image; The difference image is segmented by thresholding, and pixel regions whose difference exceeds the noise threshold and whose continuous area is greater than a preset area are extracted as the newly added writing data region.
6. The method according to claim 1, characterized in that, Also includes: Obtain the current motion parameters of the electronic pen at the acquisition time of the current frame image; Based on the current motion parameters, determine the three-dimensional spatial posture of the electronic pen at the acquisition time of the current frame image; Based on the three-dimensional spatial pose, the current frame image is subjected to perspective correction preprocessing.
7. A device for collecting electronic writing data, comprising a pen body, said pen body including a pen tip and a pen shaft, characterized in that, Also includes: The pressure detection unit is located inside the pen body; A motion detection unit, located inside the pen body, is used to detect the motion parameters of the pen body; An image acquisition unit is located on the pen body and is used to acquire images of a local area corresponding to the pen tip; A control unit, located within the pen body, is electrically connected to the pressure detection unit, the motion detection unit, and the image acquisition unit. It controls the pressure detection unit to detect the pen tip pressure. When the pen tip pressure exceeds a preset threshold, it controls the image acquisition unit to continuously acquire images of the local area corresponding to the pen tip, obtaining a sequence of frame images arranged in the acquisition order. It performs differential processing on the current frame image and the previous frame image to extract the newly added writing data area relative to the previous frame image. It packages the newly added writing data area into an incremental data packet corresponding to the current frame image. The incremental data packet is sent to an external display device, allowing the external display device to overlay the incremental data packet onto a canvas for display.
8. The device according to claim 7, characterized in that, The image acquisition unit includes a camera, which is located on the pen body near the pen tip and exposed outside the pen body. The optical axis of the camera forms a preset angle with the axis of the pen body and faces the writing surface. The field of view of the camera covers the local area where the pen tip contacts the writing surface.
9. An electronic writing system, comprising a device for acquiring electronic writing data and a display device, wherein the device for acquiring electronic writing data and the display device are communicatively connected, characterized in that, The device for collecting electronic writing data is the device for collecting electronic writing data as described in claim 8.
10. A computer-readable storage medium having instructions stored thereon, characterized in that, When executed by the processor, this instruction implements the steps of the method for acquiring electronic writing data as described in any one of claims 1 to 6.