Document scanning method and device

Abstract

This application provides a document scanning method and apparatus. After obtaining the current document border, the method compares the current border with the previous border to determine if the border has deformed (e.g., part of the current document's border is obscured by the previous page). Then, it tracks feature points on a preset number of document images, re-detects the document's position information, and finally obtains the scanned image of the document. If the document is determined to be stable, it can be directly scanned to obtain the scanned image. This method determines whether the document is obscured by detecting whether the document border is deformed. If so, it tracks the document for a preset number of frames before performing document detection and image capture, thereby reducing the false detection rate and false capture rate, and thus improving the accuracy of the document scanning results.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to document scanning methods and apparatus. Background Technology

[0002] Users can use their smartphones to scan documents to obtain electronic scanned files. However, current document scanning methods suffer from low accuracy in detecting the actual outline boundaries of documents due to factors such as paper obstruction. Summary of the Invention

[0003] In view of the above, this application provides a document scanning method and apparatus to solve at least some of the above problems, and the disclosed technical solution is as follows:

[0004] Firstly, this application provides a document scanning method applied to an electronic device. The method includes: in a continuous document scanning mode, responding to an operation of a document scanning control by performing the following steps: acquiring first vertex position information and a scanned image of a first document to be scanned; acquiring a second image of a second document to be scanned, and obtaining second vertex position information of the second document to be scanned based on the second image; determining that the boundary of the second document to be scanned is deformed according to the first vertex position information and the second vertex position information; tracking feature points of each acquired image of the second document to be scanned based on document feature points of the second image, and counting the number of successful trackings; after the number of successful trackings is greater than or equal to a first preset number, capturing a third image of the second document to be scanned, and obtaining third vertex position information of the second document to be scanned based on the third image; and obtaining a scanned image of the second document to be scanned based on the latest captured image of the second document to be scanned and the third vertex position information. It can be seen that this solution determines whether the document is occluded by detecting whether the document border is deformed. If so, it tracks the document for a preset number of frames before performing document detection and document image capture, thereby reducing the false detection rate and false capture rate and improving the accuracy of the document scanning results. Furthermore, once it is determined that the document border changes significantly, real-time feature point tracking is performed, eliminating the need for real-time document detection. The time consumed by the feature point tracking process is far less than that consumed by the document detection process, thus improving the document scanning speed.

[0005] In one possible implementation of the first aspect, determining that the boundary of the second document to be scanned is deformed based on the first vertex position information and the second vertex position information includes: obtaining first boundary information of the first document to be scanned based on the first vertex position information, and obtaining second boundary information of the second document to be scanned based on the second vertex position information, wherein the first boundary information includes the length and position of each boundary of the first document to be scanned; obtaining the length difference of the same boundary between the first document to be scanned and the second document to be scanned; and determining that the boundary of the second document to be scanned is deformed if the length difference of the same boundary is greater than or equal to the first threshold. Thus, if a document is occluded, the occlusion can usually be quickly detected based on the length of the document boundary; therefore, determining changes in the length of the document boundary is simple and effective.

[0006] In one possible implementation of the first aspect, after capturing a third image of the second document to be scanned and obtaining the third vertex position information of the second document to be scanned based on the third image, the method further includes: capturing a latest image of the second document to be scanned, and based on the document feature points of the third image of the second document to be scanned, performing feature point tracking on each captured image of the second document to be scanned, and counting the number of successful trackings to obtain the document boundary stabilization count; after the document boundary stabilization count is greater than or equal to a second preset count, the step of obtaining the scanned image of the second document to be scanned based on the latest captured image of the second document to be scanned and the third vertex position information is performed. It can be seen that this scheme can track a preset number of frames in real time after determining that the document state is stable, and then capture the document image to finally obtain the scanned image of the document. That is, it waits for the document state to remain stable for a period of time before capturing the document image, avoiding the problem of unclear document scanned images due to electronic device jitter or movement, ultimately improving the accuracy of document scanned images and reducing the collaborative requirements of users.

[0007] In one possible implementation of the first aspect, the electronic device includes a camera application and a document tracking module. The step of tracking document feature points on each acquired image of the second document to be scanned based on the document feature points of the third image of the second document to be scanned, and counting the number of successful tracking attempts to obtain the document boundary stability count, includes: the document tracking module tracking feature points on each acquired image of the second document to be scanned based on the document feature points of the third image, and returning the tracking result to the camera application; after receiving the successful tracking result, the camera application increments the document boundary stability count by 1; after receiving the failed tracking result, the camera application resets the document boundary stability count to zero; and, through an image transmission channel, transmits the latest image of the second document to be scanned to the document detection module, enabling the document detection module to detect the document in the latest image of the second document to be scanned and obtain the vertex position information of the document. It can be seen that this scheme allows the camera application to maintain the document boundary stability count based on the tracking result fed back by the document tracking module, so that the camera application only triggers the step of obtaining the scanned image of the document after determining that the document boundary stability count has reached the required level. This eliminates the need to modify the underlying algorithm, requiring only modification of the camera application's processing logic, reducing code modification workload and improving program development efficiency.

[0008] In one possible implementation of the first aspect, the process of determining successful tracking includes: determining the latest boundary information of the second document to be scanned based on feature point tracking information; determining successful tracking if the proportion of the latest boundary information exceeding the viewfinder of the electronic device's shooting interface is less than or equal to a second threshold; or, determining successful tracking if the ratio of the number of feature points in the image of the second document to be scanned acquired in this acquisition to the number of feature points contained in the second image is greater than a third threshold based on the feature point tracking information; or, determining tracking vertex information in the image of the second document to be scanned acquired in this acquisition based on the feature point tracking information, and determining successful tracking if the angle difference of the same vertex is less than a fourth threshold based on the second vertex position information corresponding to the same vertex and the tracking fixed point information.

[0009] In one possible implementation of the first aspect, after obtaining the first vertex position information and the scanned image of the first document to be scanned, the method further includes: determining that the vertex position parameter is empty, and assigning the vertex position parameter to the first vertex position information.

[0010] In one possible implementation of the first aspect, the electronic device includes a camera application, a re-detection judgment module, and a document tracking module. The step of determining that the boundary of the second document to be scanned is deformed based on the first vertex position information and the second vertex position information, and then performing feature point tracking on each acquired image of the second document to be scanned based on the document feature points of the second image, and counting the number of successful tracking attempts, includes: the re-detection judgment module, after determining that the boundary of the second document to be scanned is deformed based on the first vertex position information and the second vertex position information, returning boundary deformation status information to the camera application and setting the tracking count to 0; the camera application, after receiving the boundary deformation status information, transmitting each acquired image of the second document to be scanned to the document tracking module through an image transmission channel; the document tracking module, based on the document feature points of the second image, performing feature point tracking on each acquired image of the second document to be scanned and returning the tracking result to the reconstruction judgment module; and the re-detection judgment module, upon receiving the document tracking success result returned by the document tracking module, incrementing the tracking count by 1.

[0011] In one possible implementation of the first aspect, the electronic device further includes a document detection module, and the method further includes: after the document tracking module determines that image tracking of the second document to be scanned has failed, it returns tracking failure status information to the camera application; after receiving the tracking failure status information, the camera application transmits the latest image of the second document to be scanned to the document detection module through an image transmission channel, so that the document detection module performs document detection on the latest image of the second document to be scanned to obtain the latest vertex position information of the second document to be scanned.

[0012] In one possible implementation of the first aspect, after the number of successful tracking attempts is greater than or equal to a first preset number, the method further includes: determining, through the re-detection judgment module, that the number of tracking attempts is greater than or equal to the first preset number, and returning a scanning command to the camera application; the camera application responds to the scanning command and obtains a scanned image of the second document to be scanned based on the image of the last successfully tracked second document to be scanned.

[0013] In a second aspect, this application also provides an electronic device, the electronic device comprising: one or more processors, a memory, and a touch screen; the memory being used to store program code; and the processor being used to run the program code, such that the electronic device implements the document scanning method as described in any of the first aspects.

[0014] Thirdly, this application also provides a computer-readable storage medium having program code stored thereon, which, when run on an electronic device, causes the electronic device to perform the document scanning method as described in any of the first aspects.

[0015] Fourthly, this application also provides a computer program product having stored on it an executable that, when the computer program product is run on an electronic device, causes the electronic device to perform the document scanning method as described in any of the first aspects.

[0016] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0019] Figure 2 This is a schematic diagram of the software architecture of an electronic device provided in an embodiment of this application;

[0020] Figure 3 This is a schematic diagram of an interface for starting a document scanning function provided in an embodiment of this application;

[0021] Figure 4 This is a schematic diagram of a continuous document scanning process provided in an embodiment of this application;

[0022] Figure 5 This is a schematic diagram of a document error detection process provided in an embodiment of this application;

[0023] Figure 6 This is a flowchart of a document scanning method provided in an embodiment of this application. Detailed Implementation

[0024] The terms "first", "second" and "third" in the specification, claims and drawings of this application are used to distinguish different objects rather than to limit a specific order.

[0025] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0026] Figure 1 This is a schematic diagram of an electronic device provided in an embodiment of this application. The electronic device is used to run the document scanning method provided in this application.

[0027] The electronic device can be an electronic device equipped with a camera, such as a mobile phone, tablet computer, desktop / laptop computer, ultra-mobile personal computer (UMPC), handheld computer, netbook, personal digital assistant (PDA), or wearable electronic device. This application does not impose any specific limitations on the form of the electronic device.

[0028] like Figure 1 As shown, the electronic device may include a processor, camera, display screen, touch sensor, memory, etc.

[0029] A processor may include one or more processing units. For example, a processor may include an application processor (AP), a graphics processing unit (GPU), an image signal processor (ISP), a digital signal processor (DSP), etc. Different processing units may be independent devices or integrated into one or more processors.

[0030] The memory can be used to store computer executable program code, which may include an operating system, at least one application program required for a function (such as sound playback, image and video playback, etc.). The executable program code includes instructions, and the processor executes these instructions stored in the memory to cause the electronic device to perform various functional applications and data processing. For example, in this application, the processor executes the instructions stored in the memory to cause the electronic device to perform the document scanning method provided in this application.

[0031] The display screen is used to display images, videos, and a series of graphical user interfaces (GUIs). An electronic device may include one or N display screens, where N is a positive integer greater than 1. For example, in this embodiment, the display screen may display controls for a camera application.

[0032] A camera is used to capture still images or videos. An object passes through a lens to generate an optical image that is projected onto a photosensitive element. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP (Internet Service Provider) for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP (Digital Signal Processor) for processing. The DSP converts the digital image signal into image signals in standard formats such as RGB and YUV. In an exemplary embodiment, the electronic device may include one or N cameras, where N is a positive integer greater than 1.

[0033] A touch sensor, also known as a "touch device," is a device that can be mounted on a display screen. The touch sensor and the display screen together form a touchscreen, also called a "touchscreen." The touch sensor detects touch operations applied to or near it. It then transmits the detected touch operation to an application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen. In some embodiments, the touch sensor may also be located on the surface of the electronic device, in a different position than the display screen.

[0034] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0035] In addition, an operating system runs on the aforementioned components, on which applications can be installed and run.

[0036] The operating system of an electronic device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application's embodiment uses a layered architecture. Taking the system as an example, the software structure of the electronic device is illustrated.

[0037] Figure 2 This is a software structure block diagram of an electronic device according to an embodiment of this application.

[0038] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. Taking the Android system as an example, in an exemplary embodiment, the Android system is divided into four layers, from top to bottom: the application layer (APP), the application framework layer (Framework), the system library, the hardware abstraction layer (HAL), and the system kernel layer (Kernel).

[0039] The application layer can include a series of application packages, such as applications for camera, gallery, calendar, phone, navigation, etc.

[0040] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes predefined functions such as a camera service, window manager, content provider, view system, phone manager, resource manager, and notification manager. For example, in an exemplary embodiment of this application, the camera service provides an interface for communication between the upper-layer camera application and the camera HAL layer.

[0041] The Camera HAL allows you to connect the Camera Framework API to the underlying camera drivers and hardware.

[0042] In an exemplary embodiment, the camera HAL may include: a document detection module, a re-detection judgment module, and a document tracking module.

[0043] The document detection module is used to detect whether an image contains a document. If a document is found, it further determines the four boundaries of the document and finally determines the position information (or vertex coordinates) of the four corner points of the document based on the four boundaries.

[0044] The re-inspection judgment module is used to obtain the document's border information (or boundary information, such as the length and position of each side of the document) based on the corner positions of the document obtained by the document detection module. Furthermore, it determines whether the document is occluded based on the document's border information. If occlusion exists, the document tracking module is triggered to track the document in real time; if no occlusion exists, the camera application can be directly triggered to automatically capture an image of the document.

[0045] The document tracking module is used to track the feature points of a document and obtain the position information of the document's corner points.

[0046] The kernel layer is the layer between hardware and software, and includes at least camera drivers, touch drivers, etc.

[0047] It should be noted that although the embodiments of this application are illustrated using the Android system as an example, the basic principles are equally applicable to electronic devices based on other operating systems.

[0048] The following is a schematic diagram of the document scanning process using a mobile phone as an example. The interface for document scanning on other types of electronic devices is similar and will not be described in detail here.

[0049] Please see Figure 3 The diagram shows a schematic of the interface for launching the document scanning function provided in an embodiment of this application.

[0050] like Figure 3 As shown in (1), users can click the camera application icon 101 on the desktop interface to enter... Figure 3 The camera interface shown in (2) includes a "More" button 102 at the bottom. After clicking the "More" button 102, the user enters... Figure 3 The interface shown in (3) is as follows.

[0051] like Figure 3 As shown in (3), the interface includes various function buttons, such as the "Document Scan" button 103, the "Slow Motion" button, the "HDR" button, the "Dynamic Photo" button, the "Time-lapse" button, and the "Panorama" button. After the user clicks the "Document Scan" button 103, they enter... Figure 3 The document scanning interface 104 shown in (4) includes multiple shortcut function buttons in the function button area 105 at the top of the document scanning interface 104, such as the continuous scan button 106, the flash button, the AI ​​photography master button, the settings button, etc.

[0052] Figure 3 The status of button 106 shown in (4) indicates that the continuous scanning function is enabled. When the continuous scanning function is enabled, the user does not need to operate the shooting button 107 below the viewfinder. The mobile phone can automatically capture images of multiple pages of documents to obtain the document scan image of each page.

[0053] I understand. Figure 3 The interface shown is merely an exemplary figure. The document scanning entry can be displayed in other locations in the camera application (e.g., adding a document scanning button to the function buttons in the photo-taking interface), and this application does not limit this.

[0054] In scenarios where multiple documents need to be scanned, users can simply move the already scanned document pages manually without manually operating the camera button 107 to obtain scanned images of each document sequentially.

[0055] For example, multiple overlapping paper documents need to be scanned, such as Figure 4 As shown in (1), clicking the camera button 107 before scanning the first document will start the automatic document scanning process. Figure 4 As shown in (2), the mobile phone will automatically detect the outline boundary of the first document 201 (i.e., the black solid line frame 200 shown in the figure), and then automatically take an image of the first document 201 to obtain the following result: Figure 4 The scanned image 202 shown in (3) is as follows. Figure 4 As shown in (4), after the first document is successfully scanned, the user manually moves the first document 201 away. When the phone detects that the user has turned to the next document 203, it automatically processes the next document. Figure 4 As shown in (5), the outline boundary of the second document 203 is automatically detected and an image of the document is automatically captured, as shown in (5). Figure 4 As shown in (6), the final scanned image 204 of the second document is obtained. The scanning process for subsequent documents is the same as that for the second document, and will not be described again here.

[0056] Depend on Figure 4 As shown in the process, in automatic scanning mode, the user only needs to operate the shooting button 107 once before scanning the first document to automatically detect the outline boundaries of multiple documents in sequence and automatically obtain the scanned image of each document.

[0057] However, current continuous document scanning solutions may suffer from problems such as false detection and false grabbing due to factors such as slow paper movement by the user or paper obstruction. For example, Figure 5 As shown in (1), after the first document 201 is scanned, the user moves the first document 201 slowly, causing the second document 203 to not be fully exposed (i.e., the first document 201 occludes the second document 203). At this time, the mobile phone detects the outline boundary of the document 203, such as... Figure 5 As shown in (2), the outline boundary of the detected document 203 is incomplete at this time. Further processing of the document image within the detection box yields an incomplete scanned image of document 203, as shown in (2). Figure 5 As shown in (3), the final scanned image of document 203 is incomplete, that is, there are false detections and false captures.

[0058] To address the issue of low accuracy in detecting document occlusion boundaries due to slow document page turning speeds in continuous scanning mode, this application provides a document scanning method. This method, after obtaining the current document border, compares it with the previous border to determine if the border has deformed (e.g., part of the current document's border is occluded by the previous page). Then, it tracks feature points on the document image for a preset number of frames, re-detects the document's position information, and finally obtains the scanned image of the document. If the document is determined to be stable, it can be directly scanned to obtain the scanned image. This method determines whether the document is occluded by detecting whether the document border is deformed. If so, it tracks the document for a preset number of frames before performing document detection and image capture, thereby reducing the false detection and false capture rates and improving the accuracy of the document scanning results. Furthermore, this method can decide whether to track the document for a preset number of frames before capturing the image based on the actual situation of the document, rather than waiting a uniform period before capturing, shortening the overall time of scanning multiple documents and improving document scanning efficiency.

[0059] The following will combine Figure 6 This application provides a detailed description of the document scanning method, such as... Figure 6 As shown, the method may include the following steps:

[0060] S101, the camera app is in continuous document scanning mode, capturing images of documents.

[0061] For example, in one exemplary embodiment, the user clicks Figure 3 After clicking the "More" button 102 on the interface shown in (2), the following will be displayed. Figure 3 The interface shown in (3) allows you to enter the document scanning interface by clicking the document scanning button 103. Figure 3 As shown in (4), after the user clicks the continuous scan button 106, the camera APP enters the continuous document scanning mode.

[0062] In document scanning mode, the camera captures images within its shooting range at a certain frame rate. In this embodiment, the captured image includes the document to be scanned, i.e., the document image. The frame rate is the number of frames captured, transmitted, and displayed per second.

[0063] S102, the camera app transmits document images to the document detection module through the image data transmission channel to trigger the document detection module to perform document detection.

[0064] It is understandable that the captured document images contain other objects besides the document in the actual shooting scene. Therefore, document detection is required on the captured document images, that is, to detect whether the image contains a document. If a document is contained, the position and length of the document's four boundaries are further determined, and finally, the position coordinates of the four corner points of the document are determined based on the four boundaries.

[0065] In an exemplary embodiment of this application, the camera app sets a parameter (e.g., a global variable `track`) to indicate whether document detection or document tracking is performed. For example, a value of "track" of "0" indicates document detection, and a value of "1" indicates document tracking. In this example, the value of "track" defaults to "0", and the global variable is assigned the value of "1" when a document tracking condition is detected.

[0066] The camera app determines whether to send the captured current frame image to the document detection module or the document tracking module based on the value of the `track` parameter. In this example, if the value of the `track` parameter is "0", the current frame image is sent to the document detection module, triggering the document detection module to detect the corner positions (or vertex positions) of the document. If the value of the `track` parameter is "1", the current frame image is sent to the document tracking module, triggering the document tracking module to track the feature points of the document.

[0067] In one exemplary embodiment, when a document scan is performed for the first time, the "track" is set to 0 by default, which means that document detection is performed by default. Therefore, the original image of the document captured by the camera each time is transmitted to the document detection module through the image data transmission channel, and the document detection module further performs document detection on the document image.

[0068] S103, the document detection module obtains the corner positions of the document based on the document image.

[0069] The document detection module analyzes and processes the original image data of the document to detect whether the document image contains the document. If the document contains the document, it finally obtains the positions of the four corner points of the document in the image, that is, the position coordinates of the four corner points of the document.

[0070] S104, the document detection module returns the corner positions of the document to the camera APP through the camera service interface, and sends the corner positions of the document to the re-inspection judgment module.

[0071] After obtaining the coordinates of the corner points of the current document, the document detection module sends the corner point positions back to the camera app through the camera service interface. The camera app then draws a document detection box based on the corner point positions of the document.

[0072] At the same time, the document detection module sends the corner positions of the document to the re-inspection and judgment module so that the re-inspection and judgment module can determine whether the document's borders have been deformed.

[0073] The S105's camera app draws a document detection box based on the corner positions of the document and displays the document detection box on the shooting interface.

[0074] The camera app draws a detection box for the document based on the corner coordinates provided by the document detection module, and displays this detection box within the viewfinder of the shooting interface. For example, Figure 4 The black solid border 200 in the interface shown in (2) is the document detection box of document 201.

[0075] S106, the re-checking module checks whether the corner position of the previous document is stored in the cache. If yes, execute S107; if no, store the current document corner position in the cache and then execute S118.

[0076] In an exemplary embodiment of this application, a global parameter can be set to store corner positions. For example, the value of parameter c is the corner position of the document. If parameter c is empty (or zero), it is determined that the corner position of the document is not saved in the cache; if parameter c is not zero, it is determined that the corner position of the document is saved in the cache.

[0077] If there are no corner positions of the document in the cache, it means that the current document is not occluded, that is, the document is stable. The detected corner positions of the document are stored in the cache and then the stable document status information (or first state or state 1) is fed back to the camera APP.

[0078] S107, the re-inspection judgment module compares whether the change between the border of the current document and the border of the previous document is greater than or equal to the first threshold; if yes, then execute S108; if no, then execute S118.

[0079] If a corner point exists in the previous document, the re-checking module calculates the border information of the previous document based on that corner point. Similarly, the re-checking module calculates the border information of the current document based on the corner point obtained this time. The border information of this document includes the length and position of the four sides of the document.

[0080] Furthermore, the difference in border lengths between the two documents is compared to see if it exceeds a threshold. Based on the comparison result, a decision is made on whether to track a preset number of document images before re-triggering document detection. If the change in border lengths between the two documents exceeds a certain threshold, it is determined that the current document may be occluded.

[0081] The first threshold can be determined based on the statistical results of a finite number of experiments, and this application does not limit its specific value.

[0082] S108, the re-inspection judgment module transmits the status information of large changes in document borders to the camera APP through the camera service interface, and sets the document tracking count to 0.

[0083] If the position difference of at least one corner point between two documents is greater than the first threshold, the current document is determined to be in the third state (or state 3). Furthermore, the third state is fed back to the camera APP through the camera service interface.

[0084] The third state indicates that the document border changes significantly between the two detections, and the current document may be occluded, which triggers the document tracking module to track feature points in the image for a preset number of frames.

[0085] The re-detection module maintains a counter to record the number of frames of a document that have been successfully tracked. Therefore, when it is determined that a document needs to be re-detected, the counter is first cleared to zero, and then incremented by 1 after each successful document tracking.

[0086] S109, after receiving the third state, the camera APP transmits the original image of the document to the document tracking module through the image transmission channel to trigger the document tracking module to perform feature point tracking on the document, and sets the document stabilization count to 0.

[0087] After the camera app receives the third status returned by the re-inspection judgment module, it assigns the value of the parameter track to "1". After the camera app detects that the value of the parameter track is "1", it transmits the initial image of the currently acquired document to the document tracking module through the image transmission channel, thereby triggering the document tracking module to perform the document tracking process.

[0088] In one exemplary embodiment, the camera app maintains a counter to record the number of frames in which the document state is stable. When the camera app receives a third state indicating significant deformation of the document border, it resets the counter to zero; when the camera app receives a first state indicating stable document state, it increments the counter by 1.

[0089] S110, the document tracking module analyzes and processes the received document image to obtain the document's feature points, and then tracks these feature points.

[0090] In one exemplary embodiment, an optical flow tracing algorithm can be used to track feature points of a document.

[0091] The document feature point tracking process uses the document image containing the corner points detected by the document detection module as the original document image. It analyzes and obtains the feature points of this original document image, and then performs feature point tracking on subsequent received document images. For example, ... Figure 5 The document image corresponding to document 203 shown in (2) (the image within the thick black solid line box) is the original document image. Document tracking is performed on each acquired document image, that is, the feature points of the original document image are searched in the new document image.

[0092] During the document feature point tracking process, each frame of document image captured by the camera APP is transmitted to the document tracking module through the image transmission channel. The document tracking module analyzes the feature points contained in each frame of document image that are the same as the document feature points in the original document image.

[0093] S111, the document tracking module determines whether the document has been successfully tracked; if the tracking is successful, S112 is executed; if the tracking fails, S116 is executed.

[0094] In an exemplary embodiment of this application, the conditions for successful tracking may include at least one of the following:

[0095] (1) The document border (calculated based on the corner position of the document) does not exceed the viewfinder, or the proportion exceeding the viewfinder is less than or equal to a certain threshold (e.g., the second threshold).

[0096] (2) The ratio of the number of feature points in the current tracking document to the number of original feature points is greater than a certain threshold (such as the third threshold).

[0097] (3) The difference between the angle of any corner point in the current document and the angle of any corner point in the previous document is less than a certain threshold (such as the fourth threshold). The angle of a corner point refers to the included angle between the two sides passing through that corner point.

[0098] If the current feature point tracking meets any of the above conditions, the tracking is determined to have failed. Conversely, if the current feature point tracking does not meet the above tracking failure conditions, the tracking is determined to have succeeded.

[0099] The thresholds mentioned above are determined based on the statistical results of a limited number of experiments, and this application does not limit their specific values.

[0100] S112, the document tracking module returns a document tracking success message to the re-inspection judgment module.

[0101] In one exemplary embodiment, the document tracking success information includes the position coordinates of the four corner points contained in the currently successfully tracked document image. Based on the original position coordinates of the four corner points of the original document image and the homography matrix corresponding to the feature points obtained from feature point tracking, the latest position coordinates of the four tracked corner points are calculated.

[0102] S113 After receiving the document tracking success information, the re-inspection judgment module increments the document tracking count by 1 and feeds back the document tracking success information to the camera APP to draw the latest document detection box.

[0103] In one exemplary embodiment, the re-detection judgment module increments the tracking count counter by 1 each time it receives a tracking success message.

[0104] The re-detection judgment module returns the latest position coordinates of the tracked document corner points to the camera APP through the camera service interface, so that the camera APP can draw a new document detection box based on the latest position coordinates of the document corner points and display the latest document detection box in the viewfinder of the shooting interface.

[0105] S114, the re-inspection judgment module determines whether the current tracking number is greater than or equal to the first preset number; if not, it continues to track feature points of the document; if yes, it executes S115.

[0106] The first preset number of times can be determined by statistical data from a limited number of experiments, such as 15 times. This application does not limit the value of the first preset number of times.

[0107] S115, the re-inspection judgment module deletes the document corner point positions stored in the cache and transmits the document detection information to the camera APP through the camera service interface.

[0108] In an exemplary embodiment, after the number of consecutive successful tracking of a document reaches a first preset number, it is assumed that the current document is not obscured. The number of successful tracking attempts triggers the document detection module to re-detect the corner positions of the document. At the same time, the corner position information of the document stored in the cache is deleted, thus ensuring that the document position information stored in the cache is always the latest.

[0109] After receiving the document detection information returned by the re-detection judgment module, the camera APP sets the value of the parameter "track" to "0". Further, after detecting that the value of "track" is "0", the current frame document image is transmitted to the document detection module, thereby triggering the document detection module to perform document detection on the current frame document image, and finally obtains the positions of the four corner points of the document. That is, after executing S115, it returns to execute S102.

[0110] S116, the document tracking module transmits the document tracking failure status information (which can be referred to as the second state or state 2) to the camera APP through the camera service interface.

[0111] If the document tracking module determines that it has failed to track the feature points of the current frame document image, it sends a second state to the camera app, which indicates that the document tracking has failed.

[0112] S117, after receiving the second state, the camera APP sets the document stabilization count to 0 and returns to execute S102.

[0113] After receiving the second status feedback from the document tracking module, the camera APP determines that the current frame document image tracking has failed, transmits the current frame document image to the document detection module to trigger the document detection module to detect the document position, and then returns to execute S102.

[0114] In one exemplary embodiment, when the camera app receives a document tracking failure status (second status), it needs to reset its own document stabilization count counter to zero.

[0115] As can be seen from the content described in S109 and S117, after the camera APP receives the information of the second state or the third state, it controls the document stabilization count counter to be reset to zero.

[0116] If the re-inspection judgment module determines in S106 that the corner position of the previous document does not exist in the cache, or determines in S107 that the border change between the two documents does not exceed the first threshold, then S118 is executed.

[0117] S118, the re-inspection judgment module transmits the document stability status information (or first state or state 1) to the camera APP through the camera service interface.

[0118] S119, after the camera APP receives the first status, it increments the document stabilization count by 1.

[0119] Each time the camera app receives the first status information sent by the re-inspection judgment module, it increments the document stability count counter by 1.

[0120] In this embodiment of the application, after the camera APP receives the first state, it controls the document stabilization count counter to increment by 1, and when it receives the second or third state, it controls the document stabilization count counter to reset to zero.

[0121] S120: After receiving the first status information, the camera APP transmits the current frame document image to the document tracking module through the image transmission channel to trigger the document tracking module to track feature points of the document.

[0122] In an exemplary embodiment of this application, in order to ensure a high accuracy of document scanning results, after detecting that the document state is stable, the document image is tracked for a preset number of frames, and the document image is automatically captured only after the document state remains stable for a certain period of time.

[0123] S121, the document tracking module receives the document image and then performs document feature point tracking.

[0124] S122, the document tracking module determines whether document tracking is successful; if not, execute S123; if yes, execute S124.

[0125] The processes described in steps S121 and S122 are the same as those in S110 and S111, and will not be repeated here.

[0126] S123, the document tracking module returns a document tracking failure message (or second state or state 2) to the camera APP through the camera service interface. The camera APP receives the second state, sets the number of document stabilization attempts to 0, and returns to execute S102.

[0127] S124, the document tracking module returns a document tracking success result to the camera APP through the camera service interface. The document tracking success result includes the latest document location.

[0128] S125: The camera app draws and displays a new document detection border based on the latest document position, and increments the document stabilization count by 1.

[0129] In an exemplary embodiment of this application, the document stability count counter maintained by the camera APP increments by 1 after receiving the first state information or after successful document tracking.

[0130] S126, the camera app determines whether the number of document stabilization attempts is greater than or equal to the second preset number of attempts; if yes, then execute S127; if no, then continue to trigger the document tracking module to track the document, that is, continue to trigger the document tracking module to execute S121.

[0131] S127, The camera app triggers the process of obtaining a scanned image of the current document.

[0132] In one exemplary embodiment of this application, the algorithm layer further includes a document scanning module, which is used to obtain a scanned image of the document based on an image containing the document to be scanned.

[0133] For example, such as Figure 4 As shown in (2), an image of document 201 is captured, and then the image is processed to obtain the following: Figure 4 The scanned image 202 of document 201 shown in (3).

[0134] In this embodiment of the application, the scanned image of the document can be obtained from the image of the last frame in which the current document was successfully tracked.

[0135] In other embodiments of this application, the camera application can also directly trigger the process of obtaining a scanned image of the current document after receiving state information representing document stability. In this scenario, the scanned image of the current document can be obtained based on the document image used for document detection.

[0136] The document scanning method provided in this embodiment, after detecting the corner position of the document, first determines whether the position of the previous document exists. If it does, it compares whether the border changes of the two documents exceed a certain threshold to determine whether the document state is stable. If it exceeds the threshold, it indicates that the document state is unstable (e.g., the current document may be occluded). After tracking a preset number of document images in real time, the document position is detected again and a document image is captured to finally obtain the scanned image. It can be seen that this scheme, after detecting significant changes in the document border, tracks a preset number of document images and then detects the document position again and captures a document image, thereby avoiding the phenomenon of false captures (i.e., high false capture rate) caused by the slow movement speed of the previous page. Therefore, this scheme improves the document scanning accuracy. Moreover, after determining that the current document position is stable, this scheme directly captures an image of the current document to finally obtain the scanned image. This scheme can decide whether to track a preset number of frames before capturing a scanned image based on the actual situation of the document, rather than waiting for a period of time before scanning, thus shortening the overall time consumption of scanning multiple documents and improving document scanning efficiency.

[0137] Moreover, once the document status is stable, the solution can track a preset number of frames in real time before capturing the document image to obtain the final scanned image. This means that the document image is captured only after the document status has been stable for a period of time, avoiding the problem of unclear scanned images due to electronic device shaking or movement. Ultimately, this improves the accuracy of the scanned image and reduces the collaborative requirements for users, such as allowing users to shake or move the electronic device they are using to scan the document.

[0138] Furthermore, once it is determined that the document border changes significantly, real-time feature point tracking is performed, eliminating the need for real-time document detection. The time consumed by the feature point tracking process is far less than that consumed by the document detection process, thus improving the document scanning speed.

[0139] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0140] In the several embodiments provided in this example, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0141] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0142] Furthermore, in each embodiment of this invention, the functional units can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0143] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments. The aforementioned storage medium includes various media capable of storing program code, such as flash memory, portable hard disk, read-only memory, random access memory, magnetic disk, or optical disk.

[0144] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A document scanning method, characterized in that, Applied to electronic devices, the method includes: In continuous document scanning mode, the following steps are performed in response to operations on the document scanning control: Obtain the first vertex position information and scanned image of the first document to be scanned; Acquire a second image of the second document to be scanned, and obtain the second vertex position information of the second document to be scanned based on the second image; Based on the first vertex position information and the second vertex position information, it is determined that the boundary of the second document to be scanned is deformed. Based on the document feature points of the second image, feature point tracking is performed on the image of the second document to be scanned each time, and the number of successful tracking is counted. If the number of successful tracking attempts is greater than or equal to a first preset number, a third image of the second document to be scanned is captured, and the position information of the third vertex of the second document to be scanned is obtained based on the third image. The scanned image of the second document to be scanned is obtained based on the latest image of the second document to be scanned captured and the third vertex position information.

2. The method according to claim 1, characterized in that, The step of determining that the boundary of the second document to be scanned is deformed based on the first vertex position information and the second vertex position information includes: The first boundary information of the first document to be scanned is obtained based on the first vertex position information, and the second boundary information of the second document to be scanned is obtained based on the second vertex position information. The first boundary information includes the length and position of each boundary of the first document to be scanned. Obtain the length difference of the same boundary between the first document to be scanned and the second document to be scanned; If the length difference of the same boundary is greater than or equal to the first threshold, it is determined that the boundary of the second document to be scanned is deformed.

3. The method according to claim 1 or 2, characterized in that, After capturing a third image of the second document to be scanned and obtaining the third vertex position information of the second document to be scanned based on the third image, the method further includes: The latest image of the second document to be scanned is acquired, and based on the document feature points of the third image of the second document to be scanned, feature point tracking is performed on each acquired image of the second document to be scanned, and the number of successful tracking is counted to obtain the number of document boundary stabilizations. After the document boundary is stabilized more than or equal to a second preset number of times, the scanning image of the second document to be scanned is obtained based on the latest image of the captured second document to be scanned and the third vertex position information.

4. The method according to claim 3, characterized in that, The electronic device includes a camera application and a document tracking module; The document feature points based on the third image of the second document to be scanned are used to track feature points in each acquired image of the second document to be scanned, and the number of successful trackings is counted to obtain the number of document boundary stabilizations, including: The document tracking module tracks the feature points of the second document to be scanned in each acquisition based on the document feature points of the third image, and returns the tracking results to the camera application. After receiving the successful tracking result, the camera application increments the document boundary stabilization count by 1. After receiving the tracking failure result, the camera application resets the document boundary stabilization count to zero and transmits the latest image of the second document to be scanned to the document detection module through the image transmission channel, so that the document detection module can detect the document in the latest image of the second document to be scanned and obtain the vertex position information of the document.

5. The method according to claim 1, characterized in that, The process of determining successful tracking includes: The latest boundary information of the second document to be scanned is determined based on the feature point tracking information; if the proportion of the latest boundary information exceeding the viewfinder of the electronic device's shooting interface is less than or equal to the second threshold, the tracking is determined to be successful. Alternatively, if the ratio of the number of feature points in the image of the second document to be scanned to the number of feature points contained in the second image is greater than the third threshold, the tracking is determined to be successful. Alternatively, the tracking vertex information in the image of the second document to be scanned collected this time can be determined based on the feature point tracking information. If the angle difference of the same vertex is less than the fourth threshold based on the second vertex position information corresponding to the same vertex and the tracking vertex information, the tracking is determined to be successful.

6. The method according to claim 1, characterized in that, After obtaining the first vertex position information and the scanned image of the first document to be scanned, the method further includes: If the vertex position parameter is determined to be empty, the vertex position parameter is assigned the value of the first vertex position information.

7. The method according to claim 6, characterized in that, The electronic device includes a camera application, a re-detection module, and a document tracking module; After determining that the boundary of the second document to be scanned is deformed based on the first vertex position information and the second vertex position information, feature point tracking is performed on each acquired image of the second document to be scanned based on the document feature points of the second image, and the number of successful tracking is counted, including: After determining that the boundary of the second document to be scanned is deformed based on the first vertex position information and the second vertex position information, the re-detection judgment module returns the status information of the boundary deformation to the camera application and sets the tracking count to 0. After receiving the state information that the boundary is deformed, the camera application transmits the image of the second document to be scanned each time to the document tracking module through the image transmission channel. The document tracking module tracks the feature points of the second image for each collected image of the second document to be scanned, and returns the tracking results to the re-detection judgment module. The re-inspection judgment module receives the document tracking success result returned by the document tracking module and increments the tracking count by 1.

8. The method according to claim 7, characterized in that, The electronic device further includes a document detection module, and the method further includes: After the document tracking module determines that it has failed to track the image of the second document to be scanned, it returns tracking failure status information to the camera application. After receiving the tracking failure status information, the camera application transmits the latest image of the second document to be scanned to the document detection module through the image transmission channel, so that the document detection module performs document detection on the latest image of the second document to be scanned to obtain the latest vertex position information of the second document to be scanned.

9. The method according to claim 7 or 8, characterized in that, After the number of successful tracking attempts is greater than or equal to a first preset number, the method further includes: If the re-detection judgment module determines that the number of tracking times is greater than or equal to the first preset number of times, it returns a scanning command to the camera application. The camera application responds to the scanning command and obtains a scanned image of the second document to be scanned based on the last successfully tracked image of the second document to be scanned.

10. An electronic device, characterized in that, The electronic device includes: one or more processors, a memory, and a touch screen; the memory is used to store program code; the processor is used to run the program code, causing the electronic device to implement the document scanning method as described in any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that, It stores program code that, when run on an electronic device, causes the electronic device to perform the document scanning method as described in any one of claims 1 to 9.

Images