Image display method and device and scanning equipment

Abstract

The invention provides an image display method and apparatus, and a scanning device. The method comprises the steps of obtaining a current frame image and a previous frame image of the current frame image; determining transformation parameters for correcting the current frame image based on the relative motion between the current frame image and the previous frame image; correcting the current frame image by using the transformation parameter to obtain a target image; and displaying the target image on an electronic observation window on the display screen. According to the invention, the continuous image frames are obtained and the transformation parameters for correction are dynamically determined based on the relative motion between the frames, so that adaptive distortion correction of the images collected in real time is realized, and the problem of poor correction effect or display jitter caused by the change of scanning angle and speed is effectively avoided; and the display effect and the user experience of the electronic observation window are obviously improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to an image display method, apparatus, and scanning device. Background Technology

[0002] As a convenient tool for scanning and translating text and images, dictionary pens typically use a scanning window at the tip, allowing users to select and recognize text on paper or a screen. To enhance product integration and user experience, dictionary pens are increasingly adopting large screen designs, sometimes even covering the entire front of the pen. This design eliminates the physical viewing window, instead using the screen as an "electronic viewing window" to display the real-time image of the scanned area captured by the camera.

[0003] However, in actual use, when a user holds the dictionary pen, there may be a tilt angle between the pen body and the scanning medium (such as paper). This angle causes the image captured by the camera to have significant trapezoidal distortion, that is, the image area near the pen tip is narrow, while the image area far from the pen tip is wide. This distorted image, which does not conform to the normal observation habits of the human eye, is directly presented on the screen, which seriously affects the user's aiming and reading experience and makes the device inconvenient to operate. Summary of the Invention

[0004] This invention provides an image display method, apparatus, and scanning device to address the deficiencies in the prior art.

[0005] This invention provides an image display method, comprising the following steps: Obtain the current frame image and the previous frame image of the current frame image; Based on the relative motion between the current frame image and the previous frame image, transformation parameters for correcting the current frame image are determined; The target image is obtained by applying the transformation parameters to the current frame image; The target image is displayed in an electronic viewing window on the screen.

[0006] According to an image display method provided by the present invention, the relative motion includes the motion distance of each of at least two preset image regions within the current frame image relative to the previous frame image; The step of determining the transformation parameters for correcting the current frame image based on the relative motion between the current frame image and the previous frame image includes: The target smoothing angle of the current frame image is determined based on the motion distance of each image region. Based on the target smoothing angle, the transformation parameters used to correct the current frame image are determined.

[0007] According to an image display method provided by the present invention, the motion distance of each image region relative to the previous frame image is determined based on the following steps: Feature point detection is performed on each image region to obtain the effective feature points of each image region; If the effective feature point of a specified image region in at least two preset image regions moves a distance greater than a preset distance relative to the previous frame image, the movement distance of each image region relative to the previous frame image is calculated based on the movement distance of the effective feature point of each image region relative to the previous frame image. If the movement distance of the valid feature points in the specified image region relative to the previous frame image is less than or equal to the preset distance, the smoothing angle of the previous frame image is taken as the target smoothing angle of the current frame image, and the next frame image is taken as the current frame image. Then, the process returns to perform feature point detection on each image region in the current frame image.

[0008] According to an image display method provided by the present invention, the feature point detection of each image region includes: Feature point detection is performed on each image region when all image regions are non-blank regions. If any image region is a blank region, the smoothing angle of the previous frame image is taken as the target smoothing angle of the current frame image, and the next frame image is taken as the current frame image, and at least two preset image regions in the current frame image are determined.

[0009] According to an image display method provided by the present invention, the step of detecting feature points in each image region to obtain effective feature points in each image region includes: Feature point detection is performed on each image region to obtain multiple initial feature points in each image region; If the gradient value of any initial feature point is less than the gradient threshold, then that initial feature point is discarded. If the motion distance of any initial feature point relative to the previous frame image is greater than a distance threshold, then that initial feature point is discarded. If the motion direction of any initial feature point relative to the previous frame image is not specified, then the initial feature point is discarded. The remaining initial feature points of each image region are taken as valid feature points.

[0010] According to an image display method provided by the present invention, determining the target smoothing angle of the current frame image based on the motion distance of each image region includes: The ratio of optical flow velocities in each image region is determined based on the motion distance of each region. The target smoothing angle of the current frame image is determined based on the mapping relationship between the ratio of optical flow velocities and the smoothing angle.

[0011] According to an image display method provided by the present invention, the step of determining the target smoothing angle of the current frame image based on the motion distance of each image region further includes: If the difference between the target smoothing angle and the smoothing angle of the previous frame image is greater than a threshold, the smoothing angle of the previous frame image is increased. If the difference between the target smoothing angle and the smoothing angle of the previous frame image is less than the threshold, the smoothing angle of the previous frame image is reduced.

[0012] According to an image display method provided by the present invention, determining the transformation parameters for correcting the current frame image based on the target smoothing angle includes: From a plurality of candidate smoothing angles, determine two candidate smoothing angles that are numerically adjacent to the target smoothing angle; Based on the perspective transformation point coordinates corresponding to the two candidate smoothing angles, the perspective transformation point coordinates corresponding to the target smoothing angle are determined. The transformation parameters are determined based on the coordinates of the perspective transformation point corresponding to the target smoothing angle.

[0013] According to an image display method provided by the present invention, determining the perspective transformation point coordinates corresponding to the target smoothing angle based on the perspective transformation point coordinates corresponding to the two candidate smoothing angles respectively includes: The interpolation ratio is determined based on the relative position of the target smoothing angle within the interval formed by the two candidate smoothing angles. Based on the interpolation ratio, the perspective transformation point coordinates corresponding to the two candidate smoothing angles are weighted and summed to obtain the perspective transformation point coordinates corresponding to the target smoothing angle.

[0014] According to an image display method provided by the present invention, the electronic viewing window is set as the entire display area of ​​the display screen, or the electronic viewing window is set as a partial display area of ​​the display screen, wherein the partial display area is a segmented screen area or a floating window on the display screen.

[0015] According to an image display method provided by the present invention, the partial display area is adjacent to the pen tip of the scanning device.

[0016] The present invention also provides an image display device, comprising the following modules: The acquisition unit is used to acquire the current frame image and the previous frame image of the current frame image; The determining unit is configured to determine transformation parameters for correcting the current frame image based on the relative motion between the current frame image and the previous frame image; The application unit is used to apply the transformation parameters to correct the current frame image to obtain the target image; The display unit is used to display the target image in an electronic viewing window on the display screen.

[0017] The present invention also provides a scanning device, comprising: an image display device as described above, and an image acquisition element disposed on a scanning window, the image acquisition element being used to acquire the content selected by the scanning window.

[0018] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the image display method as described above.

[0019] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image display method as described above.

[0020] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the image display method as described above.

[0021] The image display method, apparatus, and scanning device provided by this invention achieve adaptive distortion correction of real-time acquired images by acquiring continuous image frames and dynamically determining the transformation parameters for correction based on the relative motion between frames. Because this invention can dynamically adjust the correction parameters according to the real-time changes in the pen's posture during scanning, rather than simply using a fixed correction model, it effectively avoids problems such as poor correction effects or display jitter caused by changes in scanning angle and speed, significantly improving the display effect and user experience of the electronic viewing window. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating the image display method provided by the present invention.

[0024] Figure 2 This is a flowchart illustrating the motion distance method provided by the present invention.

[0025] Figure 3 This is a flowchart illustrating the content validity screening process provided by the present invention.

[0026] Figure 4 This is a schematic diagram of the smooth transition process provided by the present invention.

[0027] Figure 5 This is a schematic diagram of a frame image obtained when the dictionary pen is tilted at 30° according to the present invention.

[0028] Figure 6 This is a flowchart illustrating the image correction method provided by the present invention.

[0029] Figure 7 This is a schematic diagram of image correction when the dictionary pen is tilted at 30°, as provided by the present invention.

[0030] Figure 8 This is a schematic diagram of the structure of the image display device provided by the present invention.

[0031] Figure 9 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0033] In actual use, when a user holds the dictionary pen, there may be a tilt angle between the pen body and the scanning medium (such as paper). This angle causes the image captured by the camera to have significant trapezoidal distortion, that is, the image area near the pen tip is narrow, while the image area far from the pen tip is wide. This distorted image, which does not conform to the normal observation habits of the human eye, is directly presented on the screen, which seriously affects the user's aiming and reading experience and makes the device inconvenient to operate.

[0034] To address this, the present invention provides an image display method. This method can be applied to image correction and display of video streams captured in real time during scanning by a scanning device (such as a dictionary pen), as well as to image correction and display of images captured by mobile terminals such as smartphones during document scanning or screen recording. It can also be applied to image correction and display of video streams captured by vehicle-mounted driver assistance systems or drones. The present invention does not specifically limit the application to these methods. For ease of explanation, the following embodiments all use the application of image correction and display of video streams captured in real time during the scanning process of a dictionary pen as an example.

[0035] in, Figure 1 This is a flowchart illustrating the image display method provided by the present invention, as shown below. Figure 1 As shown, the method includes steps 110, 120, 130 and 140.

[0036] Step 110: Obtain the current frame image and the previous frame image.

[0037] In this embodiment, the above method is applied to a scanning device with an image acquisition element (such as a camera), such as a dictionary pen. When the user starts the scanning function, the camera set in the scanning window begins to continuously acquire a video stream, which consists of a series of static images arranged in chronological order, each static image being a "frame" image.

[0038] In this context, the current frame image can be understood as the latest frame image that needs to be processed in the processing flow. The previous frame image refers to the frame image immediately preceding the current frame image in the video stream's time series. For example, if the camera captures images at a rate of 30 frames per second, then when processing the Nth frame image, that Nth frame is the "current frame image," and the (N-1)th frame is the "previous frame image." Obtaining these two frames is the foundation for subsequent motion analysis and image correction.

[0039] Step 120: Based on the relative motion between the current frame image and the previous frame image, determine the transformation parameters used to correct the current frame image.

[0040] Specifically, the relative motion between the current frame and the previous frame refers to the change in position of corresponding content (such as text, background texture, etc.) in the image due to the user's movement of the scanning device. This relative motion reflects not only the scanning speed but also, indirectly, the possible changes in the pen's posture (especially the tilt angle) during the scanning process. This relative motion can be determined using image processing techniques, such as calculating the displacement vector or movement distance of pixels or feature blocks between the two frames using methods like optical flow or feature matching.

[0041] Transformation parameters are a set of mathematical parameters used to perform geometric transformations on an image. Depending on the type of distortion to be corrected, transformation parameters can define an affine transformation, perspective transformation, or other nonlinear transformation. In the application scenario described here, perspective transformation is preferred for correcting trapezoidal distortion, and its transformation parameters are typically represented by a 3×3 transformation matrix.

[0042] Determining transformation parameters based on relative motion is based on the core idea of ​​establishing a relationship between "motion change" and "distortion change." During scanning, even slight hand tremors or adjustments to the scanning angle can alter relative motion, leading to changes in the degree of image distortion. Therefore, by monitoring the relative motion between frames, the appropriate correction intensity for the current frame can be dynamically and in real-time calculated, and the most suitable transformation parameters can be generated accordingly.

[0043] This embodiment uses a method based on relative motion to determine transformation parameters, which has significant adaptive advantages compared to schemes that use a single, fixed preset transformation parameter for correction. Fixed-parameter schemes cannot cope with real-time changes in the user's posture during actual scanning, such as pen grip angle and scanning speed, thus easily leading to under-correction or over-correction under non-standard postures. The method in this embodiment can adjust the correction intensity in real time according to actual motion changes, therefore, regardless of whether the user is swiping quickly or moving slowly, or holding the pen upright or at an angle, it can adaptively provide a continuously stable and visually appropriate corrected image.

[0044] Step 130: Apply the transformation parameters to correct the current frame image to obtain the target image.

[0045] Specifically, after determining the transformation parameters used for correction, this step applies these transformation parameters to the current frame image. Specifically, this process performs a geometric transformation on the coordinates of each pixel in the current frame image. For example, if perspective transformation is used, the pixel coordinates of the current frame image are used in a matrix operation with the 3×3 transformation matrix determined in step 120 to obtain new pixel coordinates. The original pixel values ​​are then filled into the new coordinate positions, thereby generating a new image, i.e., the target image.

[0046] The target image is the corrected image. Ideally, trapezoidal distortion in the target image is eliminated, and the image content (such as a line of text) appears as a rectangle when viewed vertically from directly above, which is more in line with human visual perception.

[0047] Step 140: Display the target image in the electronic viewing window on the screen.

[0048] Specifically, the display screen is the physical screen on the scanning device used for human-computer interaction. As devices such as dictionary pens have increasingly higher screen-to-body ratios, and some even have the entire front surface as a display screen, setting up a physical viewing window has become impractical.

[0049] Therefore, this embodiment sets up an electronic viewing window on the display screen to display the content captured and corrected by the camera in real time, as an alternative to the physical viewing window. The electronic viewing window is a logical display area on the display screen, and its specific form can be flexible and varied. For example, the electronic viewing window can be the entire display area of ​​the display screen; or, in order to display other information at the same time (such as translation results, battery level, etc.), the electronic viewing window can also be a partial display area of ​​the display screen. This partial display area can be a segmented screen area (such as the upper 1 / 3 area of ​​the screen), or a draggable floating window. This embodiment does not specifically limit this. Preferably, in order to simulate the usage habits of a traditional physical viewing window, the electronic viewing window (especially when it is a partial display area) is set on the end of the display screen near the tip of the dictionary pen.

[0050] The image display method provided in this embodiment achieves adaptive distortion correction of real-time acquired images by acquiring continuous image frames and dynamically determining the transformation parameters for correction based on the relative motion between frames. Because this embodiment can dynamically adjust the correction parameters according to the real-time changes in the pen's posture during scanning, rather than simply using a fixed correction model, it effectively avoids problems such as poor correction effects or display jitter caused by changes in scanning angle and speed, significantly improving the display effect and user experience of the electronic viewing window.

[0051] Based on the above embodiments, relative motion includes the motion distance of at least two preset image regions within the current frame image relative to the previous frame image; Step 120, based on the relative motion between the current frame image and the previous frame image, determines the transformation parameters used to correct the current frame image, including: Step 121: Determine the target smoothing angle of the current frame image based on the motion distance of each image region.

[0052] In this embodiment, the preset image region refers to a specific area within the current frame image that is pre-defined for motion information calculation in subsequent steps. There are at least two preset image regions, and their shape, size, and position can be adjusted according to the actual application scenario. For example, the preset image region can be a rectangular, circular, elliptical, or other arbitrarily shaped region, or it can be the edge region, center region, or region where a specific target is located. To facilitate subsequent image correction processing, regions with regular shapes and easy positioning are typically selected as preset image regions.

[0053] In this embodiment, the ROI (Region of Interest) image regions on the left and right sides are used as an example for explanation. The ROI image region refers to a specific area of ​​interest in an image, which usually contains the main information or target in the image.

[0054] In the current frame image, a rectangular region is divided on both the left and right sides of the image, using the image center line as the boundary, as the ROI image region. The width of the rectangular region can be adjusted according to actual needs, such as setting it to 1 / 4 or 1 / 5 of the image width. The height of the rectangular region can be set to the image height to ensure that it covers the entire vertical direction of the image, or it can be set to other heights; this embodiment does not specifically limit this.

[0055] For example, assuming the current frame image size is 640×480 pixels, the left ROI image region can be set to (0, 0, 160, 480), and the right ROI image region can be set to (480, 0, 160, 480). Here, (0, 0) represents the top-left corner coordinate of the left ROI image region, 160 represents the width of the rectangle, and 480 represents the height of the rectangle. Similarly, (480, 0) represents the top-left corner coordinate of the right ROI image region, 160 represents the width of the rectangle, and 480 represents the height of the rectangle.

[0056] It should be noted that by defining a preset image region, this embodiment can focus the subsequent motion estimation algorithm on key information areas, avoiding complex global and indiscriminate calculations on the entire image, thereby improving the overall algorithm's computational efficiency and saving computing resources. Furthermore, using the left and right ROI image regions as preset image regions can maximize the capture of parallax motion differences caused by perspective effects resulting from device tilt, effectively detecting and correcting image distortion caused by tilt or rotation during dictionary pen scanning.

[0057] Furthermore, motion distance refers to the relative displacement of the same feature point between two consecutive image frames (the current frame and the previous frame). Here, a feature point can be understood as a pixel in the image that has local saliency and is stable between frames, such as corner points, edge points, or spots with unique texture structures. Motion distance can be expressed in pixels or other units, such as millimeters or centimeters.

[0058] The motion distance of an image region can be estimated by calculating the motion vector of each pixel in the image, or by extracting and matching feature points in two frames of images. This embodiment does not specifically limit the specific method used.

[0059] Taking the estimation of motion distance of an image region by calculating the motion vector of each pixel in the image as an example, suppose the calculated motion distance of the left ROI image region relative to the previous frame is (5, 0), and the motion distance of the right ROI image region relative to the previous frame is (7, 0). Here, (5, 0) indicates that the left ROI image region has a horizontal motion distance of 5 pixels and a vertical motion distance of 0 pixels. (7, 0) indicates that the right ROI image region has a horizontal motion distance of 7 pixels and a vertical motion distance of 0 pixels.

[0060] The target smoothing angle refers to the angle value used to correct the current frame image. Determining the target smoothing angle requires comprehensive consideration of the motion distance of each image region and the smoothing angles of historical frame images to ensure the stability and smoothness of the image correction. For example, if the target smoothing angle is 30°, it indicates that the current frame image has detected a geometric deflection that needs to be compensated for by the correction model; this deflection can be equivalent to a 30-degree rotation.

[0061] Since the motion distance of each image region can directly reflect the parallax effect caused by perspective distortion, and the difference of this parallax effect between different image regions (e.g., the ratio or difference of motion distance between the left and right sides) has a definite and quantifiable functional relationship with the tilt angle of the current frame image, the target smoothing angle of the current frame image can be determined based on the motion distance of each image region.

[0062] As an optional embodiment, the original target angle can be determined by calculating the ratio of optical flow velocities in each image region and using a preset mapping relationship. Then, the original target angle is smoothed by time-dimension smoothing filtering to finally obtain the target smoothing angle.

[0063] If the current frame is the first frame, it lacks a reference frame (i.e., the "previous frame") for motion information comparison, making it impossible to determine the target smoothing angle using the inter-frame motion analysis method described in this invention. In this case, the smoothing angle state variable needs to be initialized. In this embodiment, the target smoothing angle can be directly set to a preset initial value, such as 45°. This 45° can be considered a default initial correction posture, providing an effective starting reference for the smoothing filtering algorithm of subsequent frames. After completing this initialization step, the task of the first frame is completed, and the next frame (i.e., the second frame) in the video stream is then used as the new current frame, and the process returns to step 110 to begin the normal correction process based on inter-frame motion comparison.

[0064] Step 122: Determine the transformation parameters used to correct the current frame image based on the target smoothing angle.

[0065] Specifically, transformation parameters refer to parameters used to perform geometric transformations on the image. In this embodiment, the geometric transformation can be a perspective transformation, used to correct a trapezoidal image into a rectangular image. The perspective transformation parameters can be represented by a 3×3 matrix, or by a matrix of other sizes; this embodiment of the invention does not impose a specific limitation on this.

[0066] As an optional embodiment, after determining the target smoothing angle, the transformation parameters can be determined based on the following steps: First, to improve computational efficiency and ensure the smoothness of the transformation, this embodiment can pre-establish and store a mapping relationship in the device storage. This mapping relationship includes multiple discrete candidate smoothing angles and the preset perspective transformation point coordinates corresponding to each candidate smoothing angle. The perspective transformation point coordinates typically refer to the coordinates of the four vertices used to define the region of the image to be corrected (e.g., a quadrilateral). This set of coordinates defines how to "crop" a quadrilateral from the original image to transform it into the final rectangular image.

[0067] Next, two candidate smoothing angles that are numerically adjacent to the target smoothing angle are determined from a plurality of candidate smoothing angles. For example, suppose the preset candidate smoothing angles are 0°, 10°, 30°, 40°, 50°, etc., and the target smoothing angle calculated in the current step 130 is 32°. At this time, by retrieving the preset mapping relationship, the two candidate smoothing angles that are numerically adjacent to 32° can be determined to be 30° and 40°.

[0068] Then, based on the relative position of the target smoothing angle within the interval formed by the two candidate smoothing angles, an interpolation ratio is determined. This interpolation ratio reflects which candidate smoothing angle the target smoothing angle is biased towards and the degree of bias. The calculation formula is: Interpolation ratio = (Target smoothing angle - First candidate smoothing angle) / (Second candidate smoothing angle - First candidate smoothing angle). Substituting the above example, we get the interpolation ratio = (32 - 30) / (40 - 30) = 0.2.

[0069] Next, based on the interpolation ratio, the coordinates of the perspective transformation points corresponding to the two candidate smoothing angles are weighted and summed to obtain the coordinates of the perspective transformation points corresponding to the target smoothing angle. This step performs a weighted mixing of the four sets of vertex coordinates corresponding to the two candidate angles, and the calculation formula can be: target coordinates = (1 - interpolation ratio) × coordinates corresponding to the first candidate angle + interpolation ratio × coordinates corresponding to the second candidate angle.

[0070] Finally, based on the coordinates of the perspective transformation points corresponding to the target smoothing angle, the final transformation parameters are determined. Specifically, the coordinates of the four perspective transformation points corresponding to the target smoothing angle calculated in the previous step can be used as the source coordinates, and the four vertices of the target output image (e.g., the four corner points of a standard rectangle) can be used as the target coordinates. A 3×3 perspective transformation matrix is ​​calculated using a standard perspective transformation function (e.g., the getPerspectiveTransform function commonly used in the OpenCV computer vision library). This matrix is ​​the transformation parameter determined in this step that can be directly used for image correction.

[0071] Based on any of the above embodiments, the motion distance of each image region relative to the previous frame image is determined based on the following steps: Step 210: Perform feature point detection on each image region to obtain the effective feature points of each image region.

[0072] This step aims to automatically identify salient feature points that are easy to track across consecutive frames from each predefined image region (e.g., left and right ROI image regions). These feature points are typically locations in the image where brightness or color changes drastically, such as corners, edge intersections, or spots with unique textures. Feature point detection for each image region can be performed using the `cv2.goodFeaturesToTrack()` interface provided by the OpenCV library. pts=cv2.goodFeaturesToTrack(gray,maxCorners=maxCorners,qualityLevel=qualityLevel, minDistance=minDistance, mask=mask) Wherein, gray represents the input single-channel grayscale image; maxCorners represents the maximum number of corner points returned (upper limit of feature points), such as 50, which means that a maximum of 50 strongest corner points will be returned; qualityLevel represents the minimum acceptable quality of corner points, with a value of 0 or 1. The larger the value, the more likely it is to retain stronger corner points, such as 0.01 or 0.2; minDistance represents the minimum Euclidean distance, which ensures that the returned corner points are at least this many pixels apart to prevent dense repetition; mask represents an optional mask image, which only detects corner points in the non-zero areas of the mask.

[0073] Considering that not all detected feature points are suitable for motion estimation, in dictionary pen scanning scenarios, there may be large blank areas or weak, gradient-less textures on the paper. Feature points detected in these areas are unstable or unreliable; if directly used in subsequent calculations, they will introduce noise and affect the accuracy of motion distance estimation. Therefore, "effective feature points" refer to feature points that have been considered high-quality and highly stable after one or more screening processes.

[0074] As an optional embodiment, the image gradient magnitude within a small window (e.g., 5×5 pixels) surrounding each feature point can be calculated. When the gradient magnitude is greater than a preset gradient threshold, the feature point is considered to be located in a content-rich, non-blank area and is thus determined to be a "valid feature point." This effectively filters out feature points that are falsely detected in blank paper areas.

[0075] For example, suppose 100 candidate feature points are detected using the FAST algorithm within the left-hand ROI image region. This step then calculates the gradient magnitude of the neighborhood for each of these 100 points. If the gradient threshold is set to 20, only feature points with a neighborhood gradient magnitude greater than 20 (e.g., 60) will be retained as the "valid feature point" set for that region, used in subsequent steps.

[0076] Step 220: If the effective feature points of a specified image region in at least two preset image regions have a motion distance greater than a preset distance relative to the previous frame image, calculate the motion distance of each image region relative to the previous frame image based on the motion distance of the effective feature points of each image region relative to the previous frame image.

[0077] Step 230: If the effective feature point in the specified image region moves at a distance less than or equal to a preset distance relative to the previous frame image, take the smoothing angle of the previous frame image as the target smoothing angle of the current frame image, and take the next frame image as the current frame image, and return to perform feature point detection on each image region in the current frame image.

[0078] Specifically, the designated image region can be any one of several preset image regions selected as a representative for initial judgment. For example, considering that the dictionary pen usually moves from left to right when scanning, in order to respond to the start of the scanning action as quickly as possible and determine whether the device is in motion with the highest efficiency, the left ROI image region can be selected as the "designated image region".

[0079] The motion distance of valid feature points in a specified image region relative to the previous frame can be understood as a statistical measure characterizing the overall translational magnitude of all valid feature points within that region. For example, the average displacement vector of all valid feature points between the current frame and the previous frame can be calculated first, then the magnitude of this average vector can be calculated, and this magnitude can be used as the motion distance.

[0080] If the movement distance is less than or equal to the preset distance, it indicates that the dictionary pen is currently stationary, meaning the tilt angle between the dictionary pen and the scanning medium has not changed. Therefore, there is no need to recalculate; instead, the smoothing angle of the previous frame is directly assigned to the current frame as its target smoothing angle, ensuring absolute stability of the image in a stationary state and preventing unnecessary correction jitter due to minor noise. Next, the next frame is used as the new current frame image, and the process returns directly to step 121 to begin feature point detection on the new current frame image, forming an efficient loop.

[0081] If the motion distance is greater than the preset distance, it indicates that the dictionary pen is performing effective scanning movement. In this case, the motion distance can continue to be calculated for all effective feature points in all preset image regions (including the left and right ROIs). The motion distance can be calculated using sparse optical flow methods (such as the Lucas-Kanade algorithm), which involves tracing the corresponding positions of the effective feature points in the current frame in the previous frame and calculating their displacement vectors. Finally, the average magnitude of the displacement vectors of all effective feature points in the region is taken as the motion distance of that region.

[0082] In other words, this embodiment sets a "motion state" judgment threshold to distinguish whether the dictionary pen is in an "effective scanning" state or a "stationary / micro-movement" state, thereby realizing intelligent allocation of computing resources.

[0083] In this embodiment, step 210 filters "effective feature points" to eliminate noise interference from blank and weak texture areas, thereby improving the accuracy of motion estimation. The judgment and branching mechanism constructed in steps 122 and 123 enables intelligent differentiation between the dynamic and static states of the dictionary pen, avoiding a large number of unnecessary and potentially error-inducing complex calculations when the dictionary pen is stationary. This significantly saves computing resources, reduces power consumption, and enhances the stability of the output image in a static state.

[0084] Figure 2 This is a flowchart illustrating the motion distance method provided by the present invention, as shown below. Figure 2As shown, the average displacement (mean_dist) of the effective feature points in the left ROI image region in the x direction is calculated. If mean_dist > preset distance, the current frame image is determined to be a moving frame, and step 220 can be executed. If mean_dist ≤ preset distance, the current frame image is determined to be a stationary frame, and step 230 can be executed.

[0085] Based on any of the above embodiments, feature point detection is performed on each image region, including: Feature point detection is performed on each image region when all image regions are non-blank regions. If any image region is a blank region, the smoothing angle of the previous frame image is used as the target smoothing angle of the current frame image, and the next frame image is used as the current frame image. The system then returns the results of determining at least two preset image regions in the current frame image.

[0086] Before performing formal, computationally expensive feature point detection, this embodiment introduces a pre-detection step, namely blank area detection. This is to address special cases where the dictionary pen scans areas such as book margins, large blank spaces between paragraphs, or areas without text on tickets, thereby further improving the robustness and efficiency of the algorithm.

[0087] In this context, blank areas can be understood as image regions lacking sufficient texture information for stable tracking. Blank areas can be identified by calculating the standard deviation of the grayscale values ​​of all pixels within the region. If the standard deviation is below a preset very low threshold (e.g., a threshold of 5 in the grayscale range of 0-255), it indicates that the pixel values ​​within the region are essentially uniform, lacking contrast and texture, and can be classified as blank areas. Alternatively, blank areas can also be identified by calculating the average gradient magnitude within the region. If the average gradient magnitude is below a preset threshold, it can also be classified as a blank area.

[0088] Before performing step 210, this embodiment will first perform the above-mentioned blank area detection on each preset image region (e.g., left and right ROI image regions).

[0089] If the detection results show that all preset image regions (e.g., the left and right ROIs) are determined to be non-blank regions, it indicates that each region contains sufficient information for reliable motion estimation. In this case, the program will proceed normally, that is, continue to detect feature points in each image region and execute subsequent steps 210, 220, 230, etc.

[0090] If the detection results show that at least one preset image region is identified as a blank area (for example, the left half of the dictionary pen is in the blank space of the scanned page, resulting in a blank ROI on the left), motion estimation is unreliable because effective motion information cannot be extracted from the blank area. Forcing calculation may lead to errors in subsequent parameters such as the ratio of optical flow velocity, resulting in a drastic jump and an incorrect correction angle. Therefore, to ensure the continuity and stability of the output image, the target smoothing angle finally adopted by the previous frame image is inherited and assigned to the current frame as its target smoothing angle. The next frame image is then used as the new current frame image, and the process returns to step 110, where the preset image region is re-determined in this new current frame image, and a new round of blank area pre-detection begins.

[0091] This embodiment significantly enhances the robustness of the algorithm by introducing a blank area pre-detection mechanism, enabling it to handle various complex document layouts with ease and avoiding erroneous or unstable correction results due to the scanning of blank areas. Furthermore, the pre-detection mechanism significantly improves operational efficiency. Through a computationally low pre-detection step, all subsequent complex image processing and calculation processes can be skipped when necessary, effectively saving valuable computing resources and reducing power consumption in embedded devices (such as dictionary pens). Ultimately, this ensures that users obtain smoother, more stable, and more reliable corrected images regardless of changes in the scanned content.

[0092] Based on any of the above embodiments, feature point detection is performed on each image region to obtain the effective feature points of each image region, including: Feature point detection is performed on each image region to obtain multiple initial feature points in each image region; If the gradient value of any initial feature point is less than the gradient threshold, then any initial feature point is discarded. If the motion distance of any initial feature point relative to the previous frame image is greater than the distance threshold, then any initial feature point is discarded. If the motion direction of any initial feature point relative to the previous frame image is not specified, then discard any initial feature point. The remaining initial feature points of each image region are taken as valid feature points.

[0093] Specifically, this embodiment is the same as the initial detection stage in the previous embodiments, that is, in each preset image region (e.g., left and right ROI image regions), a batch of candidate feature points are detected using standard feature point detection algorithms such as cv2.goodFeaturesToTrack(). At this time, these candidate feature points only meet the algorithm's own definition of "corner point" and have not yet been optimized for the specific application scenario of this invention, so they are called "initial feature points".

[0094] Next, this embodiment filters the initial feature points from three aspects: content validity, motion rationality, and motion consistency. Unreliable feature points introduced due to poor content, abnormal motion, or inconsistent direction are filtered out, resulting in valid feature points. The specific filtering process is as follows: The purpose of content validity filtering is to filter out feature points located in areas with sparse image content (such as blank areas on paper). For each initial feature point, the gradient magnitude within its neighboring small window (e.g., 5×5 pixels) can be calculated. This gradient magnitude is compared with a preset gradient threshold (e.g., 20). If the gradient magnitude of an initial feature point is less than this threshold, the point is considered to be located in a flat or weakly textured region, and its positioning accuracy and tracking stability cannot be guaranteed; therefore, it should be removed from the initial feature point set.

[0095] Figure 3 This is a flowchart illustrating the content validity filtering process provided by the present invention, such as... Figure 3 As shown, the gradients of the initial feature points in each image region in the x and y directions are calculated, and the gradient magnitude (grad_mag) of each image region is determined based on the gradients in the x and y directions. If grad_mag ≥ gradient threshold, the corresponding initial feature point is retained; if grad_mag < gradient threshold, the corresponding initial feature point is discarded.

[0096] The purpose of motion rationality screening is to filter out feature points with obviously abnormal motion trajectories caused by noise, occlusion, or blurring due to rapid movement. For each initial feature point, a sparse optical flow method (such as LK optical flow) can be used to initially calculate the motion displacement vector of each remaining initial feature point from the previous frame to the current frame, and calculate its magnitude, i.e., the motion distance. This distance is compared with a preset distance threshold (for example, 10 pixels / frame considering the normal scanning speed of the dictionary pen). If the motion distance of an initial feature point is greater than this threshold, the motion of that point is considered to be outside the physical scope of normal scanning, and is very likely an unreliable point caused by matching errors or other reasons, and therefore should also be removed from the feature point set.

[0097] The core idea behind motion consistency filtering is that during normal scanning (e.g., from left to right) by the dictionary pen, the motion direction of the vast majority of background feature points should be consistent (e.g., from right to left). This step aims to filter out feature points whose motion direction does not match the main motion direction. These points may originate from independent moving objects (though rare in document scanning) or incorrect matches. When filtering based on motion consistency, a specified direction must first be determined, which can be preset based on the dictionary pen's main scanning direction. For example, for a left-to-right scan, the specified direction should be horizontal to the left. An acceptable angle range can be defined, for example, an angle within ±30 degrees from the standard horizontal to the left direction. Then, the motion direction (i.e., the angle of its motion displacement vector) of each remaining initial feature point is calculated. If the motion direction of a feature point does not fall within this specified direction range, its motion is considered inconsistent with the overall trend and is an outlier, which should be removed from the feature point set.

[0098] After the triple screening of gradient, distance, and direction described above, the remaining initial set of feature points is what we define as "effective feature points" in this embodiment. These feature points are not only located on meaningful content in the image, but their motion trajectories also conform to the reasonable expectations of the physical scanning process in both amplitude and direction, exhibiting extremely high stability and reliability. Subsequent steps (such as calculating the average motion distance of the region) will be based on these high-quality effective feature points, thereby ensuring the accuracy and robustness of the entire correction algorithm.

[0099] This embodiment filters out content noise through gradient filtering, motion noise through distance filtering, and directional noise through direction filtering. This progressive "purification" process ensures extremely high quality data sources for the final calculation, thereby significantly improving the accuracy of subsequent tilt angle determination based on motion differences. Even under challenges such as slight jitter, changes in scanning speed, or complex paper content, it can output stable and reliable correction results.

[0100] Based on any of the above embodiments, whether each image region is a blank region is determined based on the gradient magnitude of each image region or the number of non-zero pixels in the corresponding binary image of each image region.

[0101] Considering that image regions containing effective content such as text and lines usually have high pixel grayscale variation, i.e., drastic gradient changes, while blank regions have simple content and gentle gradients, close to zero, this embodiment determines whether each image region is a blank region based on the gradient magnitude of each image region.

[0102] Specifically, for each pixel within each image region (e.g., the left ROI), its gradient magnitude is calculated, which can be achieved through standard gradient operators such as the Sobel operator. The gradients of the pixel in the x and y directions are calculated separately, and then a total gradient magnitude grad_mag is synthesized. Next, a gradient threshold T is set. If grad_mag < T, it indicates that the pixel has a low gradient and is in a blank or textureless area; if grad_mag > T, it indicates that the pixel has a high gradient and is in a text or textured area. The pixel ratio low_pixels of the number of pixels with grad_mag < T to all pixels in the ROI region is statistically calculated. A blank pixel ratio ratio_g is set. If low_pixels > ratio_g, it indicates that the ROI region is a blank area; otherwise, it is a non - blank area.

[0103] In addition, considering that after binarization, the regions containing text or content in the image will leave a large number of non - zero pixels (usually shown as the foreground, such as white), while the flat blank background region will almost all become zero pixels (background, such as black). Therefore, in this embodiment, it is determined whether each image region is a blank area based on the number of non - zero pixels in the binary map corresponding to each image region.

[0104] Specifically, each image region is binarized, the number of non - zero pixels in the binary map is calculated, and then the ratio ratio_b of the number of non - zero pixels to the total number of pixels in the image region is obtained. The ratio ratio_b is compared with the set threshold T b If ratio_b < T b , it is a non - blank area; otherwise, it is a blank area.

[0105] Based on any of the above embodiments, the moving distance of each image region relative to the previous frame image is calculated based on the moving distance of the feature points of each image region relative to the previous frame image, including: Taking the average value of the moving distances of the feature points of each image region relative to the previous frame image as the moving distance of each image region relative to the previous frame image.

[0106] This embodiment provides a specific method for aggregating the discrete motion information of multiple effective feature points within a region into a single, stable value that can represent the overall motion state of the region. Within a preset image region, there may be dozens or even hundreds of screened effective feature points. Due to minute perspective changes or algorithm errors, the moving distances of each feature point may have slight differences.

[0107] Directly selecting the motion distance of any single feature point may introduce random errors. However, using the average motion distance of all effective feature points can effectively smooth out and suppress random errors introduced by tracking errors of a single feature point or local noise in the image. This avoids the risks associated with relying on a single, potentially unreliable feature point, and allows the final calculated regional motion distance to more accurately and representatively reflect the overall macroscopic motion of the region. This provides a more reliable data foundation for subsequent calculation of the precise target smoothing angle, ultimately improving the accuracy of image correction and the quality of the final output image.

[0108] Based on any of the above embodiments, determining the target smoothing angle of the current frame image based on the motion distance of each image region includes: The ratio of optical flow velocities in each image region is determined based on the motion distance of each region. The target smoothing angle of the current frame image is determined based on the mapping relationship between the ratio of optical flow velocities and the smoothing angle.

[0109] Considering the perspective distortion caused by the tilt of the dictionary pen, a specific proportional relationship will be established between the parallax motion speeds on the left and right sides of the image. This proportional relationship has a direct and quantifiable mapping with the tilt angle itself. Therefore, by calculating this proportion and querying the preset mapping relationship, the target correction angle required for the current frame can be obtained directly.

[0110] Specifically, the motion distance of each preset image region (e.g., the left ROI and the right ROI) is first obtained. In this embodiment, this motion distance (i.e., the amount of pixel movement per unit time) can be directly regarded as the optical flow velocity of that region. For example, if the motion distances of the left and right ROIs are 5.0 pixels and 7.0 pixels respectively, their optical flow velocities can also be regarded as 5.0 and 7.0.

[0111] Next, the ratio of optical flow velocities for each image region is calculated. In an embodiment where the left and right ROIs are preset image regions, the ratio of the optical flow velocity of the left ROI to that of the right ROI can be calculated. For example, the ratio of optical flow velocities = left optical flow velocity / right optical flow velocity = 5.0 / 7.0 ≈ 0.714. To avoid division by zero errors when the optical flow velocity on the right side is close to zero (e.g., during a brief pause in the device), a very small positive number (epsilon) can be added to the denominator, or when the denominator is less than a certain threshold, the ratio can be set to a default value (e.g., 1.0, representing no tilt).

[0112] Finally, using a pre-established mapping relationship stored in the device, the ratio of optical flow velocities calculated in the previous step is directly converted into the final target smoothing angle. This mapping relationship directly maps different ratios of optical flow velocities to the corresponding correction angle values.

[0113] The above mapping relationship can be pre-calibrated experimentally to construct a table with two columns. The first column represents the discrete ratios of optical flow velocities (e.g., 0.5, 0.6, 0.7...), and the second column represents the corresponding target smoothing angles (e.g., 25°, 20°, 15°...). During actual operation, by matching the currently calculated ratio of optical flow velocities with the values ​​in the lookup table, the precise target smoothing angle can be directly obtained or obtained through linear interpolation between adjacent entries. For example, the mapping relationship can store the correspondence between an optical flow velocities ratio of speed_ratio = 5.57 and a smoothing angle (angle of tilt between the dictionary pen and the medium) of 15°; the correspondence between an optical flow velocities ratio of speed_ratio = 1.2 and a smoothing angle (angle of tilt between the dictionary pen and the medium) of 60°; and the correspondence between an optical flow velocities ratio of speed_ratio = 0.7 and a smoothing angle (angle of tilt between the dictionary pen and the medium) of 90°.

[0114] Furthermore, the above mapping relationship can also be a mathematical function expression, such as a simple linear function: target smoothing angle = A × (1 - ratio of optical flow velocities) + B, where A and B are pre-calibrated constants.

[0115] If a certain optical flow velocity ratio does not exist in the mapping relationship, the corresponding target smoothing angle can be calculated using interval linear interpolation. For example, if the currently calculated optical flow velocity ratio is 1.0, and the mapping relationship (lookup table) only contains two adjacent data points: 60° when the optical flow velocity ratio is 1.2, and 90° when the optical flow velocity ratio is 0.7, then linear interpolation can be used to calculate that the target smoothing angle corresponding to an optical flow velocity ratio of 1.0 is 72°.

[0116] The formula for interval linear interpolation is as follows: y = y0 + (x- x0) × (y1 - y0) / (x2-x1) Where y is the target smoothing angle, x is the ratio of optical flow velocities calculated from the current frame image, x0 is the ratio of optical flow velocities that is numerically adjacent to the current ratio of optical flow velocities in the mapping relationship, x1 is the ratio of optical flow velocities that is numerically adjacent to the current ratio of optical flow velocities in the mapping relationship, y0 is the smoothing angle corresponding to the first ratio of optical flow velocities in the mapping relationship, and y1 is the smoothing angle corresponding to the second ratio of optical flow velocities in the mapping relationship.

[0117] This embodiment employs a calculation method based on the ratio of optical flow velocities and a direct mapping relationship to achieve a highly efficient and responsive target angle determination mechanism. This method eliminates the need for a time-dimensional smoothing filter that relies on historical frame information, allowing the correction angle for each frame to be entirely determined by the motion information of the current frame itself. This design reduces the algorithm's memory footprint (eliminating the need to store the angle state of the previous frame) and achieves the fastest response speed, enabling it to instantly reflect rapid changes in device posture.

[0118] Based on any of the above embodiments, the target smoothing angle of the current frame image is determined based on the motion distance of each image region, and then the process further includes: If the difference between the target smoothing angle and the smoothing angle of the previous frame is greater than a threshold, increase the smoothing angle of the previous frame. If the difference between the target smoothing angle and the smoothing angle of the previous frame is less than a threshold, the smoothing angle of the previous frame is reduced.

[0119] In actual testing, the smoothing angle calculated between consecutive frames of a real image may not change linearly. For example, the smoothing angle calculated for frame 30 might be 30°, for frame 31 it might be 36°, for frame 32 it might be 32°, and for frame 33 it might be 38°. This could result in significant image jitter during the actual correction process, leading to poor text analysis performance.

[0120] This embodiment aims to introduce an adaptive smoothing control mechanism. Its core idea is that instead of directly using the "target smoothing angle" calculated in the current frame as the final correction angle, it dynamically determines the adjustment rate (i.e., adjustment magnitude) of the final output angle in the current frame based on the degree of change between the target smoothing angle and the "final smoothing angle used in the previous frame." This effectively avoids abrupt changes in the correction angle caused by single-frame calculation errors or rapid device movement, thereby achieving a visually smoother and more natural correction effect.

[0121] Figure 4 This is a schematic diagram of the smooth transition process provided by the present invention, such as... Figure 4As shown, if the difference between the target smoothing angle 'a' and the smoothing angle 'b' of the previous frame is greater than the threshold 't', it indicates that the dictionary pen may have undergone a significant posture change (e.g., the user quickly rotates the dictionary pen). In this case, to quickly respond to the user's actual operating intentions and ensure that the corrected image can keep up with the actual posture change of the device, this embodiment increases the target smoothing angle of the current frame image, such as adjusting the target smoothing angle a' = a + 1. If the difference between the target smoothing angle 'a' and the smoothing angle 'b' of the previous frame image is less than the threshold 't', it indicates that the dictionary pen is in a stable scanning state or has only undergone minor posture adjustments. In this case, to prioritize ensuring the high stability of the output image and effectively suppress meaningless image shaking caused by computational noise or minor hand tremors, this embodiment decreases the target smoothing angle of the current frame image, such as adjusting the target smoothing angle a' = a - 1.

[0122] This embodiment introduces an adaptive smoothing adjustment mechanism based on threshold comparison to achieve "speed-limited smoothing" of the angle, ensuring that during the actual correction of the video stream, there will be no abrupt changes in the image due to abnormal angles or large changes in the output of intermediate frames. If the angle of the dictionary pen does change during the actual scanning process, the corrected image will smoothly transition to the new angle frame by frame according to the actual angle, without any abrupt jumps.

[0123] Based on any of the above embodiments, the transformation parameters used to correct the current frame image are determined according to the target smoothing angle, including: From multiple candidate smoothing angles, determine two candidate smoothing angles that are numerically adjacent to the target smoothing angle; Based on the perspective transformation point coordinates corresponding to the two candidate smoothing angles, determine the perspective transformation point coordinates corresponding to the target smoothing angle. The transformation parameters are determined based on the coordinates of the perspective transformation point corresponding to the target smoothing angle.

[0124] Specifically, the above mapping relationship corresponds to multiple candidate smoothing angles, and the perspective transformation point coordinates corresponding to each candidate smoothing angle are pre-calibrated. Based on this, if the target smoothing angle cannot precisely match any candidate smoothing angle in the mapping relationship, its corresponding perspective transformation point coordinates can be determined by linear interpolation. Here, the perspective transformation point coordinates can be understood as defining a set of vertex coordinates of a quadrilateral of the source image region to be corrected. For a rectangular image region, the perspective transformation point coordinates are a set (usually four) that define the vertex coordinates of the region to be corrected in the original image. The quadrilateral formed by these four points will be "stretched" into a standard rectangular image by the subsequent perspective transformation operation. For example, the perspective transformation point coordinates may include p1(x1, y1), p2(x1, y2), p3(x2, y3), and p4(x2, y4).

[0125] Specifically, the process begins by identifying two candidate smoothing angles that are numerically adjacent to the target smoothing angle from a pool of candidate smoothing angles. For example, if the candidate smoothing angles in the mapping relationship include 15°, 30°, 45°, 60°, 75°, and 90°, and the target smoothing angle is 32°, then the two corresponding candidate smoothing angles are 30° and 45°. Next, based on the two candidate smoothing angles identified in the previous step, interpolation is used to calculate the coordinates of the perspective transformation point corresponding to the target smoothing angle between them. Finally, based on the coordinates of the perspective transformation point corresponding to the target smoothing angle and the coordinates of the expected perspective transformation point after correction, the corresponding perspective transformation matrix is ​​calculated to obtain the final transformation parameters.

[0126] in, Figure 5 This is a schematic diagram of a frame image obtained when the dictionary pen is tilted at 30° according to the present invention, as shown in the figure. Figure 5 As shown, the regions in the current frame image that need to be corrected are four vertices p1(x1, y1), p2(x1, y2), p3(x2, y3), and p4(x2, y4). After correction, the expected coordinates of these four vertices are p1'(x1, y1), p2'(x1, y2), p3'(x2, y1), and p4'(x2, y2), in order to accurately correct the trapezoidal distortion caused by tilted shooting and restore the image content to a square and horizontal normal shape.

[0127] Based on any of the above embodiments, determining the perspective transformation point coordinates corresponding to the target smoothing angle based on the perspective transformation point coordinates corresponding to the two candidate smoothing angles includes: The interpolation ratio is determined based on the relative position of the target smoothing angle within the interval formed by the two candidate smoothing angles. Based on the interpolation ratio, the coordinates of the perspective transformation points corresponding to the two candidate smoothing angles are weighted and summed to obtain the coordinates of the perspective transformation points corresponding to the target smoothing angle.

[0128] Specifically, the interpolation ratio refers to the normalized position of the target smoothing angle within the numerical interval formed by two candidate smoothing angles. It is used to characterize the relative distance between the target smoothing angle and the starting point of the interval, and its value range is usually [0, 1]. The interpolation ratio can be calculated using the following formula: Interpolation ratio = (Target smoothing angle - First candidate smoothing angle) / (Second candidate smoothing angle - First candidate smoothing angle).

[0129] After obtaining the interpolation ratio, the interpolation ratio is used as a weight to perform a weighted summation on the two candidate smoothing angles, thereby obtaining the precise coordinates corresponding to the target smoothing angle. Among them, the candidate smoothing angle that is closer to the target smoothing angle has a greater weight corresponding to its coordinates.

[0130] The perspective transformation point coordinates corresponding to the target smoothing angle can be determined based on the following formula: Target coordinates = (1 - interpolation ratio) × first candidate coordinates + interpolation ratio × second candidate coordinates. Here, the target coordinates are the perspective transformation point coordinates corresponding to the target smoothing angle, the first candidate coordinates are the perspective transformation point coordinates corresponding to the first candidate smoothing angle, and the second candidate coordinates are the perspective transformation point coordinates corresponding to the second candidate smoothing angle.

[0131] Based on any of the above embodiments Figure 6 This is a flowchart illustrating the image correction method provided by the present invention, as shown below. Figure 6 As shown, the method includes: First, the current frame image is converted to grayscale and then subjected to Gaussian filtering. Next, it is determined whether the current frame image is the first frame image. If so, its target smoothing angle is set to 45°. If not, the left and right sides of the current frame image are cropped to obtain the left and right ROI image regions.

[0132] Next, it is determined whether the left and right ROI image regions are both non-blank regions. If so, the target smoothing angle of the current frame image is set to the smoothing angle of the previous frame image. If not, feature point detection is performed on the left and right ROI image regions to obtain multiple initial feature points for each image region.

[0133] If the gradient magnitude of any initial feature point is less than the gradient threshold, the corresponding initial feature point is discarded. It is then determined whether the number of remaining initial feature points is less than a preset number. If so, the target smoothing angle of the current frame image is set to the smoothing angle of the previous frame image. If not, further filtering is performed based on the motion distance and direction between the initial feature points and the previous frame image to obtain valid feature points.

[0134] Then, based on the motion distance of the effective feature points, it is determined whether the current frame image is in motion. If not, the target smoothing angle of the current frame image is set to the smoothing angle of the previous frame image. If so, the ratio of optical flow velocity of the left and right image regions is calculated, and the target smoothing angle of the current frame image is determined based on the ratio of optical flow velocity.

[0135] Next, based on the difference between the target smoothing angle and the smoothing angle of the previous frame image, the target smoothing angle is smoothed to obtain the updated target smoothing angle. Then, based on the perspective transformation point coordinates corresponding to the two candidate smoothing angles, the perspective transformation point coordinates corresponding to the target smoothing angle are determined using linear interpolation. Finally, based on the perspective transformation point coordinates corresponding to the target smoothing angle, the current frame image is corrected.

[0136] in, Figure 7This is a schematic diagram of image correction when the dictionary pen is tilted at 30°, as provided by the present invention. Figure 7 As shown, the left side is the current frame image taken by tilting the dictionary pen at 30°, the middle side is the tracking of feature points in the current frame image (yellow dots are the positions of feature points in the previous frame image, and red dots are the positions of feature points in the current frame image), and the right side is the corrected current frame image.

[0137] Based on any of the above embodiments, the electronic viewing window is set to the entire display area of ​​the display screen, or the electronic viewing window is set to a partial display area of ​​the display screen, where the partial display area is a segmented screen area or a floating window on the display screen.

[0138] In one alternative implementation, the electronic viewing window is set to the entire display area. In this mode, the calibrated target image fills the entire screen. This full-screen mode provides users with a maximized field of view and a highly immersive scanning experience, especially suitable for scenarios where users need to focus on aligning and scanning long paragraphs of text, helping to reduce scanning skew or interruptions caused by limited field of view.

[0139] In another alternative implementation, the electronic viewing window is set as a partial display area of ​​the screen, allowing the rest of the screen to simultaneously display other auxiliary information, such as real-time recognized text, translation results, battery level, or network status, effectively improving the device's information presentation efficiency and interactivity. Specifically, the partial display area can be a segmented screen area or a floating window on the display screen.

[0140] When the display area is a segmented screen, the screen is divided into at least two fixed, non-overlapping regions. For example, the display can be divided horizontally into upper and lower regions. The upper region (usually the side adjacent to the scanning device's pen tip) serves as an electronic viewing window, displaying the corrected scanned image in real time, while the lower region is used to display the instantaneously generated translation results or dictionary definitions. This layout is clear, with well-defined information separation, making it easy for users to view the processing results while scanning.

[0141] When a portion of the display area is a floating window, the electronic viewing window can be presented as a window that can hover above the main interface or other application content. In some further embodiments, users can also drag the floating window to change its position or adjust its size according to their personal usage habits. This approach provides extremely high interactive flexibility, allowing users to customize the interface layout as needed and avoid the viewing window obscuring other key information they are interested in.

[0142] This embodiment provides multiple electronic viewing window display modes, such as full-screen, split-screen, or floating window, enabling the scanning device to meet user preferences in different scenarios. Users can freely choose or switch between "immersive scanning" and "efficient information acquisition" modes as needed, which not only significantly improves the flexibility of human-computer interaction but also greatly optimizes the overall user experience of the product.

[0143] Based on any of the above embodiments, a portion of the display area is adjacent to the pen tip of the scanning device.

[0144] Specifically, the layout design in this embodiment aims to simulate the user experience of traditional writing or marking tools (such as pens or highlighters). When operating the scanning device, the user's attention naturally focuses on the physical pen tip to ensure the accuracy of the scanning path. Placing the electronic viewing window here visually links the physical scanning action with the image feedback on the screen, establishing a direct and immediate feedback loop for the user. For example, for a vertically held dictionary pen, this display area is typically located in the lower half or bottom edge of the display screen, i.e., the end closest to the user's hand and the scanned medium.

[0145] This design avoids the inconvenience and cognitive burden caused by the user's gaze frequently switching between the physical pen tip (for positioning) and the screen display area further away from the pen tip (for confirming the image). Users can focus their gaze on a very small area, much like looking through the physical transparent panel of a traditional dictionary pen, simultaneously guiding the pen tip and confirming the scanned image, thus providing a seamless and intuitive operating experience.

[0146] This embodiment effectively replicates the intuitive advantages of a physical viewing window by strategically placing the electronic viewing window, which serves as a partial display area, near the pen tip. This reduces the cognitive load and operational difficulty for users during the scanning process, thereby significantly improving scanning accuracy and overall human-computer interaction friendliness.

[0147] The image display device provided by the present invention will be described below. The image display device described below can be referred to in correspondence with the image display method described above.

[0148] Based on any of the above embodiments Figure 8 This is a schematic diagram of the structure of the image display device provided by the present invention, as shown below. Figure 8 As shown, the device includes: The acquisition unit 810 is used to acquire the current frame image and the previous frame image of the current frame image; The determining unit 820 is used to determine the transformation parameters for correcting the current frame image based on the relative motion between the current frame image and the previous frame image; Application unit 830 is used to apply transformation parameters to correct the current frame image to obtain the target image; Display unit 840 is used to display the target image in an electronic viewing window on the display screen.

[0149] Based on any of the above embodiments, the present invention also provides a scanning device, including: an image display device as described in the above embodiments, and an image acquisition element disposed on a scanning window, the image acquisition element being used to acquire the content selected by the scanning window.

[0150] Specifically, the scanning device includes a main body, and an image display device and an image acquisition element integrated on the main body. The image display device contains hardware such as a processor and a memory, and runs a program that implements the image display method described in any of the foregoing embodiments, for adaptive correction and display of the acquired image.

[0151] An image acquisition element, such as a camera, is positioned on the scanning window of the scanning device. The scanning window provides the user with a clear physical boundary for selecting the content to be scanned on the scanning medium (such as paper). The optical lens of the image acquisition element is aligned with the scanning window, ensuring that its field of view substantially coincides with the area selected by the scanning window.

[0152] When the scanning device is in operation, the user holds the device and aligns it with the physical scanning window, tracing a line of text. During this process, the image acquisition element located in the scanning window acquires optical images of the selected content in real time and converts them into a continuous digital video stream (i.e., the "current frame image" and "previous frame image" in the aforementioned embodiment). This video stream is then transmitted to the image display device. Upon receiving the original, potentially distorted image frames, the image display device executes the aforementioned image display method, dynamically calculating transformation parameters by analyzing the relative motion between frames, correcting the image, and finally displaying a clear, distortion-free target image on the electronic viewing window of the device screen.

[0153] The scanning device provided in this embodiment organically combines an image display device with adaptive correction function with front-end image acquisition hardware to form a complete closed-loop system from image acquisition to correction and display. This not only fundamentally solves the image distortion problem caused by changes in scanning angle, but also provides users with an intuitive and efficient scanning experience of "what you see is what you get", which significantly improves the ease of use and display effect of the scanning device.

[0154] Figure 9 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 9As shown, the electronic device may include a processor 910, a communications interface 920, a memory 930, and a communication bus 940, wherein the processor 910, the communications interface 920, and the memory 930 communicate with each other through the communication bus 940. The processor 910 can call logical instructions in the memory 930 to execute an image display method.

[0155] Furthermore, the logical instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0156] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the image display methods provided by the above methods.

[0157] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the image display methods provided by the methods described above.

[0158] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0159] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0160] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image display method, characterized in that, include: Obtain the current frame image and the previous frame image of the current frame image; Based on the relative motion between the current frame image and the previous frame image, transformation parameters for correcting the current frame image are determined; The target image is obtained by applying the transformation parameters to the current frame image; The target image is displayed in an electronic viewing window on the screen.

2. The image display method according to claim 1, characterized in that, The relative motion includes the motion distance of each of at least two preset image regions within the current frame image relative to the previous frame image; The step of determining the transformation parameters for correcting the current frame image based on the relative motion between the current frame image and the previous frame image includes: The target smoothing angle of the current frame image is determined based on the motion distance of each image region. Based on the target smoothing angle, the transformation parameters used to correct the current frame image are determined.

3. The image display method according to claim 2, characterized in that, The motion distance of each image region relative to the previous frame image is determined based on the following steps: Feature point detection is performed on each image region to obtain the effective feature points of each image region; If the effective feature point of a specified image region in at least two preset image regions moves a distance greater than a preset distance relative to the previous frame image, the movement distance of each image region relative to the previous frame image is calculated based on the movement distance of the effective feature point of each image region relative to the previous frame image. If the movement distance of the valid feature points in the specified image region relative to the previous frame image is less than or equal to the preset distance, the smoothing angle of the previous frame image is taken as the target smoothing angle of the current frame image, and the next frame image is taken as the current frame image. Then, the process returns to perform feature point detection on each image region in the current frame image.

4. The image display method according to claim 3, characterized in that, The feature point detection for each image region includes: Feature point detection is performed on each image region when all image regions are non-blank regions. If any image region is a blank region, the smoothing angle of the previous frame image is taken as the target smoothing angle of the current frame image, and the next frame image is taken as the current frame image, and at least two preset image regions in the current frame image are determined.

5. The image display method according to claim 3, characterized in that, The step of detecting feature points in each image region to obtain effective feature points for each image region includes: Feature point detection is performed on each image region to obtain multiple initial feature points in each image region; If the gradient value of any initial feature point is less than the gradient threshold, then that initial feature point is discarded. If the motion distance of any initial feature point relative to the previous frame image is greater than a distance threshold, then that initial feature point is discarded. If the motion direction of any initial feature point relative to the previous frame image is not specified, then the initial feature point is discarded. The remaining initial feature points of each image region are taken as valid feature points.

6. The image display method according to any one of claims 2 to 5, characterized in that, Determining the target smoothing angle of the current frame image based on the motion distance of each image region includes: The ratio of optical flow velocities in each image region is determined based on the motion distance of each region. The target smoothing angle of the current frame image is determined based on the mapping relationship between the ratio of optical flow velocities and the smoothing angle.

7. The image display method according to any one of claims 2 to 5, characterized in that, The process of determining the target smoothing angle of the current frame image based on the motion distance of each image region further includes: If the difference between the target smoothing angle and the smoothing angle of the previous frame image is greater than a threshold, the smoothing angle of the previous frame image is increased. If the difference between the target smoothing angle and the smoothing angle of the previous frame image is less than the threshold, the smoothing angle of the previous frame image is reduced.

8. The image display method according to any one of claims 2 to 5, characterized in that, The step of determining the transformation parameters for correcting the current frame image based on the target smoothing angle includes: From a plurality of candidate smoothing angles, determine two candidate smoothing angles that are numerically adjacent to the target smoothing angle; Based on the perspective transformation point coordinates corresponding to the two candidate smoothing angles, the perspective transformation point coordinates corresponding to the target smoothing angle are determined. The transformation parameters are determined based on the coordinates of the perspective transformation point corresponding to the target smoothing angle.

9. The image display method according to claim 8, characterized in that, The step of determining the perspective transformation point coordinates corresponding to the target smoothing angle based on the perspective transformation point coordinates corresponding to the two candidate smoothing angles includes: The interpolation ratio is determined based on the relative position of the target smoothing angle within the interval formed by the two candidate smoothing angles. Based on the interpolation ratio, the perspective transformation point coordinates corresponding to the two candidate smoothing angles are weighted and summed to obtain the perspective transformation point coordinates corresponding to the target smoothing angle.

10. The image display method according to claim 1, characterized in that, The electronic observation window is set to the entire display area of ​​the display screen, or the electronic observation window is set to a partial display area of ​​the display screen, wherein the partial display area is a segmented screen area or a floating window on the display screen.

11. The image display method according to claim 10, characterized in that, The portion of the display area is adjacent to the pen tip of the scanning device.

12. An image display device, characterized in that, include: The acquisition unit is used to acquire the current frame image and the previous frame image of the current frame image; The determining unit is configured to determine transformation parameters for correcting the current frame image based on the relative motion between the current frame image and the previous frame image; The application unit is used to apply the transformation parameters to correct the current frame image to obtain the target image; The display unit is used to display the target image in an electronic viewing window on the display screen.

13. A scanning device, characterized in that, include: The image display device as claimed in claim 12, and the image acquisition element disposed on the scanning window, wherein the image acquisition element is used to acquire the content selected by the scanning window.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the image display method as described in any one of claims 1 to 11.

15. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the image display method as described in any one of claims 1 to 11.

16. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the image display method as described in any one of claims 1 to 11.

Images