Text scanning method, terminal equipment and storage medium

Abstract

The embodiment of the invention provides a text scanning method, terminal equipment and a storage medium, and relates to the technical field of terminals. According to the text scanning method, after a text area in a preview flow image frame is detected, a text area in a subsequent preview flow image frame is determined through an optical flow tracking method, the calling frequency of a text area detection model can be reduced, and the time delay power consumption of single preview flow image frame processing can be effectively reduced; in addition, text area detection and text area tracking are performed asynchronously, so that real-time output of the text area can be ensured under the condition that the text area detection runs. In addition, the embodiment of the invention also provides a text scanning terminal device and a storage medium.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a text scanning method, a terminal device and a storage medium. Background Art

[0002] Users often encounter the need to input text in terminal devices, such as the text content of paper documents, cards, and electronic documents. Taking mobile phones as an example, terminal devices have provided entrances for convenient and fast text input in multiple applications. For example, a text scanning function is provided in a memo or notepad application. The user only needs to point the terminal device at the text content to be scanned to automatically input the text. Usually, when scanning text, it is necessary to detect the text area first, and then recognize the text in the text area into the input area. The detection of the text area requires calling the neural network model. Calling the neural network model frame by frame will result in high power consumption of the terminal device. Summary of the invention

[0003] The embodiments of the present application provide a text scanning method, a terminal device, and a storage medium, which can at least reduce the frequency of calling a text area detection model and effectively reduce the power consumption of the terminal device.

[0004] In a first aspect, an embodiment of the present application provides a text scanning method, which is applied to a terminal device and includes:

[0005] Acquire a preview stream image of the text to be scanned; if the preset time interval is satisfied from the last detection moment, detect the first text area in the preview stream image of the Mth frame; if the preset time interval is not satisfied from the last detection moment, acquire data in a buffer, wherein the data at least includes a second text area and the preview stream image of the Nth frame, wherein the second text area is detected in the preview stream image of the Nth frame; track the second text area, and determine a third text area in the preview stream image of the N+Kth frame, wherein the preset time interval is not satisfied between the N+Kth frame and the Nth frame; perform text recognition on one of the third text areas to obtain a scanning result.

[0006] In a possible implementation, if the preset time interval from the last detection moment is not satisfied and the data is not read in the buffer, the fourth text area in the preview stream image continues to be detected; or the fifth text area tracked in the last frame of the preview stream image is tracked.

[0007] In a possible implementation, if the tracking of the second text area fails in the N+Xth frame, the text area in the N+X+1th frame is detected, where X is less than K.

[0008] In one possible implementation, detecting the first text area / the fourth text area includes: obtaining a binary mask image of the Mth frame / N-1th frame preview stream image; obtaining text lines in the binary mask image; clustering the text lines to obtain text blocks; and expanding the text blocks to obtain the first text area.

[0009] In a possible implementation, obtaining the text line in the binary mask image includes: taking the circumscribed rectangle with the minimum area for the text connected domain in the binary mask image to obtain the coordinates of the text connected domain; determining the main directions of multiple text connected domains and rotating the text connected domains to a horizontal direction; and horizontally splicing the text connected domains in the same row to obtain the text line.

[0010] In a possible implementation, the text connected domain is a connected area in which the grayscale value of adjacent pixels in the binary mask image is 1.

[0011] In a possible implementation, clustering the text lines to obtain text blocks includes: vertically expanding the text lines; determining the IoU value in the horizontal direction and the IoU value in the vertical direction of the expanded text lines, and when the IoU value in the horizontal direction and the IoU value in the vertical direction meet preset conditions, clustering the text lines to obtain the text blocks.

[0012] In one possible implementation, the text block is expanded to obtain the first text area, including: determining an expansion ratio according to the maximum value of the shortest distance between a pixel point of the text line and its contour, expanding the text block with the expansion ratio, and taking a minimum-area circumscribed rectangle for the expanded text block to obtain the first text area.

[0013] In a possible implementation, the IoU value is the area where the intersection of two text regions is within the union of the two text regions.

[0014] In a possible implementation, the second text area / the fifth text area is tracked to determine the third text area in the N+Kth frame preview stream image, including: determining a bounding rectangle of the minimum area containing the second text area / the fifth text area; determining a first feature point of the bounding rectangle; determining a second feature point in the N+Kth frame preview stream image; when a first preset condition is met, determining a perspective transformation matrix between the N+Kth frame preview stream image and the Nth frame preview stream image, wherein the valid feature point refers to a feature point in which a motion difference between a feature point in the second feature point and a feature point corresponding to the first feature point is less than a threshold; when a second preset condition is met, applying a perspective transformation to the second text area / the fifth text area based on the perspective transformation matrix to determine the third text area.

[0015] In a possible implementation, the first feature point is determined by using one of the following algorithms: Harris corner detection algorithm, SIFT algorithm, SURF algorithm, or ORB algorithm.

[0016] In a possible implementation, the second feature point is determined by an optical flow method.

[0017] In one possible implementation, the first preset condition includes that the number of valid feature points in the second feature points is greater than a first preset threshold; the second preset condition includes that the number of local points is greater than a second preset threshold, wherein the local points are determined based on the RANSAC algorithm in the perspective transformation matrix process.

[0018] In a possible implementation, the first preset threshold is 90% of the number of the first feature points; and the second preset threshold is 80% of the number of the valid feature points.

[0019] In a possible implementation, the preset time interval is 2 seconds.

[0020] In a possible implementation, the text region detection process and the text region tracking process are performed asynchronously.

[0021] In a possible implementation, one of the third text regions includes a text region that is closest to a center position of the N+Kth frame preview stream image among the third text regions.

[0022] In a possible implementation, the nearest text area is determined by any of the following methods: obtaining the center point coordinates of each text area in the third text area and the distance value from the center point coordinates to the center position, and taking the text area with the smallest distance value as the nearest text area; obtaining the vertex coordinates of each text area in the third text area, respectively determining the distance value from the vertex coordinates of each text area to the center position, and sorting them, and taking the text area corresponding to the vertex with the smallest distance value as the nearest text area; expanding each text area in the third text area, determining the expansion ratio when at least one point in the outer contour of each text area coincides with the center position, and taking the text area corresponding to the minimum expansion ratio as the nearest text area.

[0023] In a possible implementation, performing text recognition on one of the third text regions to obtain a scanning result includes: displaying the one of the third text regions in the N+Kth frame preview stream image; and displaying a text recognition result of the one of the third text regions.

[0024] In a possible implementation, performing text recognition on one of the third text regions to obtain a scan result includes: using an OCR recognition algorithm to recognize the one of the third text regions to obtain a scan result.

[0025] In a possible implementation, if the third text area is not acquired in the N+Kth frame preview stream image, prompt information is displayed on the N+Kth frame image.

[0026] In a possible implementation, the prompt information includes moving too fast or please aim at the text.

[0027] In a possible implementation, before obtaining the preview stream image of the text to be scanned, the method further includes: receiving a first operation of the user in the application to start text scanning.

[0028] In a possible implementation, the application includes one of the following: memo, note, notepad, WeChat, and office software.

[0029] In a possible implementation, the first operation includes a click or a long press operation.

[0030] In a possible implementation, a second operation of the user in the application is received, and the scan result is inserted into the application.

[0031] In a second aspect, an embodiment of the present application provides a terminal device, comprising a memory and at least one processor, the memory being used to store code instructions, and the at least one processor being used to run the code instructions to execute the method in the first aspect or any possible implementation of the first aspect.

[0032] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program or instructions are stored. When the computer program or instructions are run on a computer, the computer executes the method in the first aspect or any possible implementation of the first aspect.

[0033] In a fourth aspect, an embodiment of the present application provides a device for text scanning, which may be a terminal device, or a chip or chip system in a terminal device. The device may include a processing unit and a display unit. The processing unit is used to implement any method related to processing performed by the electronic device in the first aspect or any possible implementation of the first aspect. The display unit is used to implement any method related to display performed by the terminal device in the first aspect or any possible implementation of the first aspect. When the device is a terminal device, the processing unit may be a processor. The device may also include a storage unit, which may be a memory. The storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit so that the terminal device implements the method described in the first aspect or any possible implementation of the first aspect. When the device is a chip or chip system in a terminal device, the processing unit may be a processor. The processing unit executes the instructions stored in the storage unit so that the terminal device implements the method described in the first aspect or any possible implementation of the first aspect. The storage unit may be a storage unit within the chip (eg, a register, a cache, etc.), or a storage unit within the terminal device that is located outside the chip (eg, a read-only memory, a random access memory, etc.).

[0034] In a fifth aspect, an embodiment of the present application provides a computer program product comprising a computer program, which, when executed on a computer, enables the computer to execute the method in the first aspect or any possible implementation manner of the first aspect.

[0035] It should be understood that the technical solutions in the various embodiments of the present application may correspond to each other, and the beneficial effects achieved by various aspects and corresponding feasible implementation methods are similar and will not be described in detail. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1A-1I A schematic diagram of an application scenario for text scanning provided by the related technology;

[0037] Figure 2A schematic diagram of the operation flow of text scanning provided for related technologies;

[0038] Figure 3 Schematic diagram of possible situations that may occur in text scanning provided by related technology;

[0039] Figure 4 A schematic diagram of the hardware structure of a terminal device provided in an embodiment of the present application;

[0040] Figure 5 A schematic diagram of the software structure of a terminal device provided in an embodiment of the present application;

[0041] Figure 6 A schematic diagram of a text area provided in an embodiment of the present application;

[0042] Figure 7 A flowchart of a text region detection and matching method provided in an embodiment of the present application;

[0043] Figure 8A-B A schematic diagram of a binary mask image and a circumscribed rectangle of the binary mask image with the minimum area provided in an embodiment of the present application;

[0044] Figure 9A-9B A schematic diagram of rotating multiple text connected regions to a horizontal direction and horizontally splicing them provided in an embodiment of the present application;

[0045] Fig.10 A schematic diagram of vertically expanding a text line provided in an embodiment of the present application;

[0046] Fig.11 A schematic diagram of the text region matching process provided in an embodiment of the present application;

[0047] Fig.12 The intention of text region matching provided by the embodiments of the present application;

[0048] Fig.13 A flowchart of text region matching provided by another embodiment of the present application;

[0049] Fig.14 The intention of text region matching provided by another embodiment of the present application;

[0050] Figure 15A-Figure 15C A schematic diagram of a method for determining a text area closest to a center position provided in an embodiment of the present application;

[0051] Fig.16 A flowchart of a text area tracking method provided in an embodiment of the present application;

[0052] Fig.17 A schematic diagram of tracking feature points using the optical flow method provided in an embodiment of the present application;

[0053] Fig.18 A schematic diagram of the perspective transformation matrix before and after transformation provided in an embodiment of the present application;

[0054] Fig.19 A schematic diagram of calculating the perspective transformation matrix of the current preview stream image frame relative to the initial preview stream image frame for text tracking provided by an embodiment of the present application;

[0055] Fig. 20 A flowchart of a text scanning method provided in an embodiment of the present application;

[0056] Fig.21 A timing diagram of asynchronous execution of text region detection and matching and text region tracking provided in an embodiment of the present application;

[0057] Fig. 22 A schematic diagram of a specific implementation of the text scanning method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0059] The term "interface" in the embodiments of the present application refers to the medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The commonly used form of the interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be an interface element such as an icon, window, control, etc. displayed on the display screen of a terminal device, where the control can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc.

[0060] In the embodiments of the present application, "first" and "second" are used to distinguish different objects rather than to describe a specific order of objects. In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more, for example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc. In the embodiments of the present application, "at least one" refers to one or more, for example, at least one text area refers to one or more text areas.

[0061] In the embodiments of the present application, the words "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0062] In the embodiments of the present application, the terms “including”, “comprising”, “having” and their variations all mean “including but not limited to”, unless otherwise specifically emphasized.

[0063] Users will encounter a lot of important and interesting information in their daily work and life, but it is difficult to remember all this information by memory alone. Therefore, users hope to be able to record this information for later viewing. With the development of computer technology, terminal devices such as mobile phones, tablets, and laptops have become an indispensable part of people's daily lives. Users can use the applications of terminal devices to store the above information. For example, the information that users want to store may include paper documents, cards, electronic documents, etc. Users can store this information in applications such as memos, notes, notepads, WeChat file transfer assistants, office software (such as Office), etc. When needed, they can find and use it in the corresponding application.

[0064] The terminal device has provided the user with a method of scanning text content into an application program by using a camera of the terminal device, and this method is generally called text scanning.

[0065] For example, the terminal device is a mobile phone, and the user records relevant information in a memo.

[0066] like Figure 1A FIG. 1 is a schematic diagram of a main interface 110 of a mobile phone in an example, and a plurality of applications can be installed in the mobile phone 1. Figure 1A As shown, the main interface 110 of the mobile phone 1 may include application icons of application programs such as “email”, “settings”, “camera”, and “memo”.

[0067] Mobile phone 1 can receive the user's Figure 1A In response to the user's click operation on the "Memo" application icon, the mobile phone 1 can start the "Memo" application and enter the following steps: Figure 1B The “Memo” interface 120 is shown.

[0068] like Figure 1BAs shown, the interface 120 can display the memo information that the user has input. Exemplarily, the memos that have been input in the figure include memo 1, memo 2, ..., memo 5. The application interface 120 can also display a control 121 for creating a new memo.

[0069] The mobile phone 1 can receive a user's click operation on the control 121, and in response to the click operation, enter the following Figure 1C The interface 130 for adding a memo is shown. For example, the user can enter the information that he wants to record in the memo in the editing area 131 in the interface 130.

[0070] The above-mentioned method of manually inputting information content by users is inefficient, takes a lot of time for users to input, and has a high error rate. Therefore, the application also provides users with more convenient ways to input information. For example, users can use the camera to obtain image information of the text to be scanned in real time in the entrance provided by the application, and can input text in real time and automatically in the camera preview interface, so that users can input text information more conveniently, improving the user experience.

[0071] like Figure 1C As shown, a plurality of selectable controls may be provided below the interface 130. For example, the controls may include "list", "font", "voice", "picture" and "graffiti" controls. The mobile phone 1 may receive a user's click operation on the picture control 132. In response to the user's click operation on the picture control 132, a menu item may pop up near the picture control 132, such as Figure 1D In addition, the user can also long press on a blank area in the editing area 131 to call out a menu item.

[0072] like Figure 1D As shown in FIG. 1 , the pop-up menu item may include four optional items: “Photograph”, “Text Scan”, “Card Collection”, and “Select from Gallery”. The mobile phone 1 may receive a user's click operation on the scan menu item 133, and in response to the user's click operation on the scan menu item, enter the following options: Figure 1E A text scanning interface 140 is shown.

[0073] Figure 1E and Figure 1FIn the text scanning interface 140, the text scanning interface 140 may include two upper and lower areas, the upper part is a text editing area 141, and the lower part is a real-time preview image area 142 of the text to be scanned obtained by using the camera module of the mobile phone 1. An insertion control 143 may be provided below the real-time preview image area 142. If the text area of ​​the text to be scanned is not detected in the real-time preview image, or the user moves the mobile phone too fast so that the text area cannot be detected, the user may be prompted. For example, prompt information 145 such as "Please aim at the text" and "Moving too fast" is displayed above the real-time preview image area. At this time, the insertion control 143 is gray, and the user cannot perform the insertion operation.

[0074] like Figure 1G As shown, after the text area of ​​the text to be scanned is identified, the text area can be highlighted, for example, a first focus frame 146 can be displayed around the text area. At this time, the insert control 143 becomes highlighted, and the text in the text area is identified and pre-displayed in the text editing area 141. It should be noted that the identified text at this time is only pre-displayed in the text editing area 141, which does not mean that the text input is successful, and the pre-displayed text content will change in real time with the text area detected in the real-time preview image.

[0075] The mobile phone 1 can also receive the user's operation on other areas of the real-time preview image except the first focus frame 146, and respond to the user's click operation on other areas, such as Figure 1H As shown, a second focus frame 147 is displayed around the text area corresponding to the area clicked by the user, and the text in the second focus frame 147 is automatically recognized and pre-displayed in the text editing area 141.

[0076] In some embodiments, a text recognition control 144 may be provided near the insert control 143. The text recognition control 144 is used to generate a picture containing the text area in the focus frame, and to suspend and display it on the real-time preview image area. The user may click on the suspending picture or slide on the suspending picture to select the text to be input, and the text selected by the user will be pre-displayed in the text editing area 141.

[0077] Further, after the user has selected the text area or text information to be input, the mobile phone 1 can receive a click operation of the user on the insert button 143. In response to the click operation, the pre-displayed text content is confirmed. Fig. 1I As shown in the interface 150, the pre-displayed text content is input into the text editing area 151, and the text scanning is exited at the same time.

[0078] It is understandable that the embodiment of the present application only takes the memo application as an example, and the text scanning entry can also be provided in other applications. For example, a text scanning icon is added to the office menu bar, and a text scanning menu is added to the WeChat chat window. This is not limited in the embodiment of the present application.

[0079] The specific operation process when users use the text scanning function is as follows: Figure 2 Specifically including steps S110-S180.

[0080] Step S110 , in response to a first operation of the user, obtaining a preview stream image frame of the text to be scanned.

[0081] The terminal device may receive a first operation from the user to start the text scanning function. For example, the first operation may include a click operation on a menu item, a long press operation on an editing area, etc. Figure 1D As shown, the terminal device can detect the user's touch operation on the text scanning menu item 133, and in response to the operation, enter the text scanning function and start to obtain the preview stream image frame of the text to be scanned.

[0082] like Figure 1G As shown, the terminal device turns on the camera module to scan the text and collects the preview stream image frames of the text to be scanned. The preview stream image frames are a sequence of image frames collected continuously at a certain frame rate, and each image frame is usually an RGB image. Exemplarily, the text to be scanned may include paper documents, cards, electronic documents, etc. Among them, paper documents may include: ordinary paper documents, books, papers, contracts, reports, official documents, newspapers, certificates and awards, etc.; cards may include: identity cards, bank cards, household registration books, driver's licenses, real estate certificates, business licenses, graduation certificates, degree certificates, etc.; electronic documents may include: PPT, Word, Excel, Keynote and other electronic documents displayed or projected on monitors, mobile phone screens, tablet screens, projection screens, and electronic screens at the venue.

[0083] Step S120, detecting the text area.

[0084] Step S130, determining whether there is data in the text area. If there is no data in the text area, executing step S140; if there is data in the text area, executing step S150.

[0085] Step S140: If there is no data in the text area, the user may be prompted. Figure 1E and 1F As shown, “Moving too fast” or “Please aim at the text” is displayed on the interface 150 to prompt the user.

[0086] Step S150, which includes step S1501 and step S1502, and step S1501 and step S1502 are executed simultaneously.

[0087] Step S1501, highlighting the text area according to the coordinates of the acquired text area. The highlighting of the text area may be performed by rendering the preview stream image frame with a user interface (UI) according to the coordinates of the text area, so that the text area is displayed on the preview stream image. For example, a focus frame 146 is displayed around the detected text area, such as Figure 1G In some embodiments, the text area can also be highlighted on the preview stream image in other ways, such as highlighting. In this step, the preview stream image frame can also be rendered and highlighted by the user interface according to the text area corresponding to the position clicked by the user, such as Figure 1H As shown, a focus frame 147 is displayed around the text area corresponding to the user click area.

[0088] Step S1502, perform text recognition on the text area, and pre-display the recognized text in the editing area. A variety of recognition algorithms can be used to recognize the text in the text area. For example, an optical character recognition (OCR) algorithm can be used for recognition.

[0089] Step S160, detecting whether the user performs a click operation on the preview stream image frame. If a click operation is detected, executing step S150; if no click operation is detected, executing step S170.

[0090] Step S170, detecting whether the user performs a click operation on the inserted control. If a click operation on the inserted control is detected, executing step S180; if no click operation is detected, returning to step S150.

[0091] Step S180, in response to the user's click operation on the insert control, the text pre-displayed in the editing area is input and confirmed, such as Fig. 1I shown.

[0092] In the related art, when detecting text regions, the detection results are usually inaccurate and unstable. For example, the detection results of text regions often change on the preview stream image frames, such as detecting a text region in the previous frame but detecting another text region in the next frame. Figure 3 As shown, the text area presented to the user on the real-time preview image changes from 147 to text area 147'. In this case, the insert control is gray, so that the user cannot operate to insert the required information into the text editing area, resulting in a poor user experience.

[0093] In view of this, the present application proposes a text scanning method applicable to a terminal device, which can accurately output a stable text area.

[0094] The text scanning method in the embodiment of the present application can be implemented in a terminal device 100 having the following hardware structure and software architecture system.

[0095] Exemplarily, the terminal device in the present application may be a mobile phone, or other types of intelligent terminal devices, for example, the terminal device may also include a tablet computer, a desktop computer, a handheld computer, a laptop computer, a smart screen, an artificial intelligence (AI) device, or an Internet of Things (IoT) device, etc. The embodiments of the present application do not impose any special restrictions on the specific form of the terminal device.

[0096] like Figure 4 , which is a schematic diagram of the structure of a terminal device 10 provided in an embodiment of the present application. The terminal device 10 may include a processor 11, an internal memory 12, a universal serial bus (USB) interface 13, a charging management module 14, a battery 15, an antenna 1, an antenna 2, a mobile communication module 16, a wireless communication module 17, an audio module 18, a sensor module 19, a camera 20, and a display screen 21, etc.

[0097] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the terminal device 10. In other embodiments of the present application, the terminal device 10 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0098] The processor 11 may include one or more processing units, for example, the processor 11 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a memory, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and / or a neural-network processing unit (neural-network processing unit, NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors. For example, in an embodiment of the present application, the processor 11 may be an application processor AP.

[0099] The processor 11 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 11 is a cache memory. The memory may store instructions or data that the processor 11 has just used or cyclically used. If the processor 11 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 11, and thus improves the efficiency of the system.

[0100] The charging management module 14 is configured to receive a charging input from a charger.

[0101] The wireless communication function of the terminal device 10 can be implemented through the antenna 1, the antenna 2, the mobile communication module 16, the wireless communication module 17, the modem processor and the baseband processor.

[0102] The mobile communication module 16 can provide wireless communication solutions including 2G / 3G / 4G / 5G etc. applied on the terminal device 10 .

[0103] The terminal device 10 implements the display function through a GPU, a display screen 21, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 21 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 11 may include one or more GPUs, which execute program instructions to generate or change display information.

[0104] The display screen 21 is used to display images, videos, etc. The display screen 21 includes a display panel. In some embodiments, the terminal device 10 may include 1 or N display screens 21, where N is a positive integer greater than 1. In the embodiment of the present application, the display screen 21 may display the application interface of "Memo", as well as various memo information stored by the user in the memo application, etc.

[0105] The terminal device 10 can realize the shooting function through the ISP, the camera 190, the video codec, the GPU, the display screen 21 and the application processor. The ISP is used to process the data fed back by the camera 20. In some embodiments, the ISP can be set in the camera 20. The camera 20 is used to capture a static image or a video. In some embodiments, the terminal device 10 may include 1 or N cameras 20, where N is a positive integer greater than 1.

[0106] The internal memory 12 can be used to store computer executable program codes, and the executable program codes include instructions. The processor 11 executes various functional applications and data processing of the terminal device 10 by running the instructions stored in the internal memory 12. In addition, the internal memory 12 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0107] The terminal device 10 can implement audio functions such as music playing and recording through the audio module 18 and the application processor.

[0108] The software system of the terminal device 10 can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the terminal device 10. Figure 5 It is a software structure block diagram of the terminal device 10 according to an embodiment of the present application.

[0109] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom: application layer, system framework layer, native layer, hardware abstraction layer (HAL) and kernel layer.

[0110] The application layer can include a series of application packages.

[0111] like Figure 5 As shown, the application package may include applications such as camera, calendar, phone, map, settings, mailbox, video, music and memo, among which the memo may provide an entry for the visual input method.

[0112] The system framework layer provides API and programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0113] like Figure 5 As shown, the application framework layer may include a window manager, a content provider, a resource manager, a view system, a notification manager, an activity manager, and the like.

[0114] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, touch the screen, drag the screen, capture the screen, etc.

[0115] Content providers are used to store and retrieve data and make it accessible to applications. The data can include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0116] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0117] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0118] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0119] The activity manager is responsible for managing the startup, status, life cycle and other activities of the application. Among them, activity is an application component that can provide an interface for users to interact with the electronic device through the interface to complete a task.

[0120] The Native layer can also be called the local framework layer, local algorithm layer or local service layer. The Native layer can include the algorithm engine layer, C library and class, and Android runtime library (Android runtime). Among them, the algorithm engine layer includes text block detection algorithm, text extraction algorithm, text block tracking algorithm and natural language understanding algorithm, etc.; C library and class are mainly used to build native code, which are public libraries and system libraries that the algorithm depends on, and provide related functions for some applications, such as OpenCV function library, MediaPipe function library, Faith function library, camera service, etc.; Android runtime is mainly responsible for the scheduling and management of the Android system.

[0121] The hardware abstraction layer is an abstract interface of the device kernel driver, which is used to provide an application programming interface for accessing the underlying device to a higher-level API framework. In this embodiment, the hardware abstraction layer can provide a virtual hardware platform for the system, which can include sensors, cameras, audio, etc.

[0122] The kernel layer is the layer between hardware and software. The kernel layer can include display driver, sensor driver, audio driver, sensor driver, power management, etc.

[0123] To facilitate understanding of the embodiments of the present application, some terms used in the embodiments of the present application are explained below to facilitate understanding of the embodiments of the present application.

[0124] 1. Binary mask image

[0125] In the field of image processing, a value is assigned to the area where the target object in the image is located, so that the area is assigned a different value from other areas in the image, so that the target object can be distinguished from other areas in the image, which facilitates subsequent rendering of the image. In the embodiment of the present application, the area where the text in the preview stream image frame is located is assigned a value of 1, and other areas in the preview stream image frame are assigned a value of 0.

[0126] 2. IoU (Intersection over Union)

[0127] IoU stands for intersection over union, and is a measure of the accuracy of detecting corresponding objects in a specific data set. IoU is the result of dividing the overlapping part of two regions by the combined part of the two regions. The IoU calculation results are compared by a set threshold. Usually, the set threshold will not be lower than 0.5. The higher the IoU value, the greater the overlap between the two regions.

[0128] 3. Perspective Transformation

[0129] Perspective Transformation refers to projecting an image onto a new plane. The general transformation formula is:

[0130]

[0131] Among them, (u,v) is the original image pixel coordinates, is the coordinate of the image pixel after transformation. The perspective transformation matrix is ​​illustrated as follows:

[0132]

[0133] in, Represents image linear transformation; T2 = [a 13 a 23 ] T Used to generate image perspective transformation; T3 = [a 31 a 32 ] indicates image translation.

[0134] The mathematical expression of perspective transformation is:

[0135]

[0136]

[0137] Given the four pixel coordinates corresponding to the perspective transformation, the perspective transformation matrix can be obtained; conversely, given the perspective transformation matrix, the perspective transformation of the image or pixel coordinates can be completed.

[0138] 4. Optical flow or optic flow

[0139] Optical flow is a concept related to the detection of object motion in the field of view. It is used to describe the sense of motion of an observed object, surface or edge caused by the motion relative to the observer. Optical flow is very useful in pattern recognition, computer vision and other image processing fields. It can be used for motion detection, object cutting, calculation of collision events and object expansion, motion compensation coding, or stereo measurement of object surfaces and edges.

[0140] 5. OCR (Optical Character Recognition)

[0141] OCR, also known as optical character recognition, refers to the process in which a terminal device checks characters printed on paper and then uses character recognition methods to translate the shapes into computer text. In other words, it is the process of analyzing and processing image files of text materials to obtain text and layout information. In other words, the text in the image is recognized and returned in the form of text. Commonly used methods include CTC (Connectionist Temporal Classification, neural network-based temporal classification) and attention-based methods.

[0142] The method provided in the embodiments of the present application will be described in detail below in combination with the embodiments and drawings.

[0143] It is understandable that if all the content displayed in the preview stream image is recognized and input, the user will need to spend a lot of time to reselect and edit the text content that he needs to input later, and the time for recognizing all the content will also be very long. In most cases, the user may only want to scan and input part of the text content. For example, part of the text content that belongs to the same paragraph or has a high correlation. Therefore, in the embodiment of the present application, it is necessary to perform text detection on the preview stream image frame. By detecting one or more text areas in the text to be scanned, it can facilitate user operation and improve user experience. At the same time, it can also improve the efficiency of text area recognition and input. It should be noted that the text area is an area in the preview stream image frame that contains part of the text content, which is usually represented by the vertex coordinate data of the circumscribed rectangle of the minimum area of ​​the text area. In the embodiment of the present application, the preview stream image frame can be rendered according to the coordinate data of the text area and displayed on the preview stream image frame. As shown in FIG. Figure 6 As shown, the text to be scanned is a paper, and text region 1 and text region 2 represent two text regions detected in the preview stream image frame.

[0144] like Figure 7 The figure is a flow chart of a text area detection method provided in one embodiment of the present application.

[0145] Step A: The steps before text area detection in the embodiment of the present application are the same as those in the related art, which may be Figure 2 Step S110 in will not be repeated here.

[0146] Step S200, based on the preview stream image frame data, detecting the text area in each frame of the preview stream image frame data. In this step, specifically, steps S201 to S208 may be included.

[0147] Step S201: Obtain a binary mask image of the preview stream image frame.

[0148] In order to obtain the text part in the preview stream image frame, it is first necessary to obtain the binary mask image of the preview stream image frame. In this step, a neural network model based on a segmentation scheme can be used to obtain the binary mask image of the preview stream image frame. Exemplarily, the DBNet (Differentiable Binarization Net) algorithm can be used to perform text detection on the acquired preview stream image frame to obtain a binary mask image of the text area. The DBNet algorithm uses a convolutional neural network (CNN) to extract image features, and then uses these features for classification. Specifically, the image features of the preview stream image frame can be extracted using the backbone network of DBNet; the extracted image features are upsampled to obtain a feature map with the same size as the preview stream image frame; the feature map is predicted to obtain a probability map and a threshold map; and the probability map and the threshold map are binarized to obtain a binary mask image of the text area. Among them, the binarization process refers to converting the probability map into a bounding box and a text area, and binarization is achieved by comparing the threshold. For example, the backbone network of DBNet can use ResNet18 or ResNet50, and deformable convolution can be introduced to improve the network's feature extraction ability. After the backbone network outputs at least one feature map, the standard FPN (feature pyramid networks) network structure is used to generate probability maps and threshold maps, and the probability map obtained by segmentation network training is converted into a binary map by setting a fixed threshold. In a binary mask image, the grayscale value of a pixel is 0 or 1, such as Fig. 8A shown.

[0149] In addition to using the DBNet algorithm to obtain the binary mask image, other methods can also be used to obtain the binary mask image of the preview stream image frame, which is not limited in the present application.

[0150] In a binary mask image, pixels form connected regions with a certain distribution. If two pixels are adjacent and have the same pixels, then the two pixels are in the same interconnected region. Visually, the connected points form a region, and the set of all connected points in the region is called a connected domain. The text connected domain can be understood as the region where the text is located, that is, the region where the gray value of the pixel is 1 after binarization. Fig. 8A As shown in FIG. 1 , the connected areas of adjacent pixels whose grayscale value is 1 constitute a text connected domain.

[0151] Step S202: for each text connected domain in the binary mask image, take the circumscribed rectangle with the minimum area to obtain the coordinates of the text connected domain.

[0152] In order to accurately characterize each text connected domain, it is necessary to take the minimum area of ​​the bounding rectangle for each text connected domain. The method of taking the minimum area of ​​the bounding rectangle can be: Get the initial X min , Y min , X max , Y ma To determine the initial outer rectangle, find the area of ​​the rectangle, and then rotate the point set according to the following rotation formula:

[0153] X′=Xcos(angle)-Xsin(angle)

[0154] Y′=Xsin(angle)+Ycos(angle)

[0155] Calculate the coordinates of the point set after rotation angle, and then find the X′ of the rotated rectangle min , Y′ min , X′ max , Y′ max , calculate the current rectangular area until the area is the smallest, which is the minimum enclosing rectangle. It can be understood that the minimum enclosing rectangle is the minimum rectangular area that includes all point sets.

[0156] For each text connected domain, the minimum area of ​​the bounding rectangle can be obtained according to the above method, such as Figure 8B As shown in the figure. The four vertex coordinates corresponding to each minimum area bounding rectangle are the coordinates of the text connected domain. For example, the vertex coordinates of the bounding rectangle are arranged in a clockwise direction: (X i1 ,Y i1 );(X i2 ,Y i2 );(X i3 ,Y i3 );(X i4 ,Y i4 ), then the coordinates of the text connected domain are (X i1 ,Y i1 ;X i2 ,Y i2 ;X i3 ,Y i3 ;X i4 ,Y i4 ), where i represents the i-th text connected domain.

[0157] In an embodiment of the present application, the coordinates of the text connected area can be obtained using the relevant methods of OpenCV. Specifically, all text connected domains in the preview stream image frame can be found. By setting a threshold, the connected domains with an area that is too large or too small are filtered out, and each of the remaining text connected domains forms a valid connected area. The boundingRect function is used to surround the outline of the valid connected area with a minimum rectangle to obtain four values ​​of X, Y, W, and H. These four values ​​represent the horizontal and vertical coordinates of the upper left corner vertex of the minimum bounding rectangle, and the width and height of the bounding rectangle of the minimum area, respectively, so that the bounding rectangle of the minimum area of ​​each valid connected area and its specific position can be obtained, that is, the coordinates of each text connected domain are obtained.

[0158] Step S203, obtaining the main direction of the text connected domain and rotating it to the horizontal direction.

[0159] When users scan text, the preview stream image frame collected may not be horizontal, or due to the neural network model, the text area in the binary mask image may be tilted, for example Fig. 8A and Figure 8B The text connected domain shown is tilted. In order to facilitate the subsequent tracking and recognition of the text region, the text region needs to be rotated to the horizontal direction.

[0160] Exemplarily, the direction of each text connected domain can be calculated based on the coordinates of the multiple text connected domains obtained in step S202. For example, the angle between each text connected domain and the horizontal direction can be obtained by the arctan function. After obtaining the angle between each text connected domain and the horizontal direction, the main directions of the multiple text connected domains can be obtained. For example, 0°-180° is divided into 18 intervals at an interval of 10°, and the number of text connected domains falling into each interval is counted, and the angle range in which the text connected domain falls into the interval the most is taken as the main direction of the multiple text connected domains.

[0161] According to the main direction of the obtained multiple text connected domains, if the main direction is not horizontal, the multiple text connected domains need to be rotated to the horizontal direction. Fig.9A As shown, the text connected domain A, the text connected domain B, and the text connected domain C respectively represent text connected domains rotated to the horizontal direction. Exemplarily, the rotation matrix can be calculated to rotate multiple text connected domains to the horizontal direction.

[0162] Step S204: horizontally splice the text connected domains in the same line to obtain a text line.

[0163] Since the detection accuracy of the neural network model is not high enough, when binarizing the preview stream image frames, text that should belong to the same text connected domain may be detected as multiple text connected domains. Fig.9A As shown in FIG. 1 , after detection by the neural network model, the text area in the same row is identified as three text connected domains. In order to avoid inaccurate results of subsequent text recognition, the text connected domains in the same row need to be spliced ​​in the horizontal direction.

[0164] Before horizontal splicing of multiple text connected domains in the same row, the height value of each text connected domain can be calculated based on the vertex coordinate data of each text connected domain, and the middle value of the height values ​​of multiple text connected domains in each row is taken as the first height value H. Then, the coordinate difference in the horizontal direction of adjacent text connected domains in the same row is calculated. When the difference is less than a preset threshold, it is considered that the multiple text connected domains should belong to the same text row, and the multiple text connected domains are horizontally spliced ​​to form a text row. For example, according to the vertex coordinates of text connected domain A, text connected domain B, and text connected domain C, the difference in the horizontal direction between the vertex coordinates of text connected domain B and text connected domain A, and text connected domain C and text connected domain B is calculated respectively. Specifically, the difference between the X coordinate of the lower left vertex of text connected domain B and the X coordinate of the lower right vertex of text connected domain A can be calculated. When the difference is less than the preset threshold, it is considered that text connected domain A and text connected domain B belong to the same text row, and text connected domain A and text connected domain B are spliced ​​into a whole rectangular block. The coordinates of the entire rectangular block may be the coordinates of the upper left vertex and the lower left vertex of the text connected domain A and the coordinates of the upper right vertex and the lower right vertex of the text connected domain B. Similarly, the text connected domain C and the text connected domain B may also be spliced ​​using the same method.

[0165] It should be noted that when at least three text connected domains (text connected domain A, text connected domain B and text connected domain C) are identified in the same row, the difference in the horizontal direction between the vertex coordinates of the text connected domain B and the text connected domain A, and between the text connected domain C and the text connected domain B can be calculated simultaneously, and then the splicing is performed after comparison with a preset threshold; or the difference in the horizontal direction between the vertex coordinates of the text connected domain B and the text connected domain A can be calculated first, and after the text connected domain B and the text connected domain A are spliced ​​into a rectangular block, the difference in the horizontal direction between the vertex coordinates of the text connected domain C and the spliced ​​rectangular block is calculated, and then the splicing is performed after comparison with a preset threshold, and finally the spliced ​​text connected domain D is obtained, as shown in FIG. Fig. 9B As shown in FIG. 1 , the connected domain of each line of text after splicing is a text line. The preset threshold value can be set to twice the first height value H.

[0166] Step S205: vertically expand each text line.

[0167] In order to cluster the text lines, the text lines need to be expanded vertically. For example, a fixed expansion ratio can be preset based on empirical values, for example, the expansion ratio can be set to 1.6 times the first height value H. The expanded text lines are as follows: Fig.10 shown.

[0168] Step S206: cluster the text lines according to the IoU values ​​between the text lines to obtain text blocks.

[0169] Calculate the IoU value between the two text lines after vertical expansion. The IoU (Intersection over Union) value is usually used to measure the relationship between two regions. For example, the IoU value of the match between text line A (X1, Y1, W1, H1) and text line B (X2, Y2, W2, H2) is calculated as follows:

[0170] IoU = (text line A ∩ text line B) / (text line A ∪ text line B)

[0171] The IoU value of the two text lines is obtained by dividing the intersection area of ​​text line A and text line B by the union area of ​​text line A and text line B. The larger the IoU value of the two text lines, the closer the two text lines are. In an embodiment of the present application, the IoU value of the two text lines in the horizontal direction and the IoU value in the vertical direction can be calculated respectively, and the IoU threshold in the horizontal direction and the IoU threshold in the vertical direction can be set. When the calculated IoU value of the two text lines meets the preset conditions, it is considered that the distance between the two text lines is close, and the two text lines are classified as the same text block using a clustering algorithm. Among them, a clustering algorithm refers to an algorithm that aggregates similar data together. For example, when the IoU value of the two text lines in the horizontal direction is greater than 0.8, and the IoU value in the vertical direction is greater than 0, it is considered that the two text lines belong to the same text block. If the IoU of two adjacent text lines does not meet the above preset conditions, it is considered that the two text lines belong to different text blocks.

[0172] Step S207: Expand the text lines in the text block according to the maximum value of the shortest distance between the pixel points in each text line and its contour.

[0173] Since the text connected domain in the binary mask image is compressed, it is necessary to perform an expansion process on the obtained text block. Adjacent text lines in the text block may be adhered to each other. For example, the lower boundary of the previous text line coincides with the upper boundary of the next text line. At this time, the two adhered text lines will be treated as one text line. If the expansion is still performed at a fixed ratio, the expansion area will become larger, resulting in inaccurate text lines after expansion. Therefore, in an embodiment of the present application, each text line in the text block can be expanded according to a preset expansion ratio, wherein the preset expansion ratio can be determined according to the maximum value of the shortest distance between the pixel point in the text line and its contour. For example, the shortest distance between the pixel point in each text line and the contour of the text line is calculated, each pixel point in the text line is traversed, and the maximum value of the above-mentioned shortest distance is found, and the expansion ratio is determined according to the maximum value, such as the expansion ratio can be twice the maximum value.

[0174] Step S208, taking the circumscribed rectangle with the smallest area for the expanded text block to obtain the text area.

[0175] For the convenience of calculation and processing, the circumscribed rectangle of the expanded text block is taken as the minimum area to obtain a rectangular text area. For a frame of preview stream image, there may be one or more text areas. In the embodiment of the present application, the four vertex coordinates of each text area can be stored in the buffer in the form of an array. Different text areas can be marked with different labels, for example, text area 1, text area 2, ...

[0176] In an embodiment of the present application, the text area in each preview stream image frame is detected to obtain a binary mask image of the preview stream image frame, and then the text lines in the binary mask image are obtained, the text lines are clustered to obtain text blocks, and the text blocks are expanded to obtain the text area. Through the above steps, the text area in the preview stream image frame can be detected, so that the text area in each preview stream image frame can be obtained.

[0177] Due to the instability of the detection results of the neural network model, the text areas detected on different preview stream image frames may change. For example, when the content of the preview stream image frame does not change, the current frame is detected as one text area, and the next frame may be detected as two text areas. The text area presented to the user is unstable, which makes it impossible for the user to accurately input the required text content into the application. In order to avoid the above situation, a method is proposed in an embodiment of the present application for matching the detected text area in multiple preview stream image frames after detecting the text area of ​​the preview stream image frame, so that a stable text area can be output to the user in the preview stream image frame.

[0178] Step S300, matching the text region according to the detected text region. This step may specifically include steps S301 to S304.

[0179] Step S301, obtaining the text area detected in the preview stream image frame.

[0180] Step S200 is used to obtain the text region detected in the preview stream image frame, wherein the text region may be one or more.

[0181] It should be noted that in one or more embodiments of the present application, the preview stream images are obtained frame by frame, the current frame refers to the preview stream image frame obtained last in the preview stream image frame sequence, and the previous frame refers to the preview stream image frame obtained last in the preview stream image frame sequence. For example, the current frame is the Nth frame, and the previous frame is the N-1th frame.

[0182] Step S302, obtaining a text region detected in the current frame and matching it with a corresponding text region in the previous frame to obtain a matching result, and determining the state of each text region.

[0183] In one embodiment of the present application, the corresponding text area in the previous frame can be determined based on the text area detected in the previous frame image and the matching results of the previous multiple frames, wherein the initial frame is the first frame for starting text area detection. This step may specifically include the following steps: Fig.11 Step S3021 ′ to step S3023 ′ in step S302 ′ shown.

[0184] Step S3021′, obtaining the text area detected in the current frame.

[0185] Step S3022 ′, matching the text region detected in the current frame with the text region corresponding to the previous frame to obtain matching status information of the current frame.

[0186] In an embodiment of the present application, the corresponding text area in the previous frame can be a text area obtained by matching the text area detected in the previous frame image with the corresponding text area in the previous frame of the previous frame image. For example, assuming that the current frame is the Nth frame and the previous frame is the N-1th frame, then the corresponding text area in the N-1th frame is the text area obtained based on the matching result between the text area detected in the N-1th frame and the corresponding text area in the N-2th frame. It should be noted that the corresponding text area in the initial frame is the text area detected in the initial frame.

[0187] In matching the text region detected in the current frame with the text region corresponding to the previous frame, it is necessary to obtain the maximum match between the text region detected in the current frame and the text region corresponding to the previous frame, and determine the text region matching pair. Specifically, the intersection over union (IoU) value between the text region detected in the current frame and the text region corresponding to the previous frame is calculated, wherein the intersection over union (IoU) value is calculated by dividing the intersection area of ​​the text region detected in the current frame and the text region corresponding to the previous frame by the union area of ​​the text region detected in the current frame and the text region corresponding to the previous frame. A cost matrix is ​​constructed based on the above-mentioned intersection over union (IoU) value, and the Hungarian Algorithm is used to perform optimal allocation by minimizing the cost matrix, thereby obtaining the maximum match between the text regions corresponding to the current frame and the previous frame.

[0188] The Hungarian algorithm is usually used to find the maximum number of matches or the minimum number of point covers in a bipartite graph. Among them, a bipartite graph means that the vertex set can be divided into two non-intersecting subsets, and the two vertices attached to each edge in the graph belong to these two complementary and intersecting subsets, and the vertices in the two subsets are not adjacent. If a number of matches are formed in a given bipartite graph, and the number of matches reaches the maximum value of all possible matches in this bipartite graph, then we believe that the match corresponding to this bipartite graph has reached the maximum match. In an embodiment of the present application, the text area detected by the current frame and the text area corresponding to the previous frame can be regarded as two groups of data, and there is a matching relationship between the two groups of data. The number of matches in the two groups of data is obtained to reach the maximum value of all possible matches, that is, the matches corresponding to the two groups of data have reached the maximum match. Based on the maximum match, the largest number of text area matching pairs between the text area detected by the current frame and the text area corresponding to the previous frame can be determined.

[0189] Since the larger the IoU value, the higher the degree of matching. Therefore, in order to improve the accuracy of matching, an IoU threshold can also be preset in the embodiment of the present application. When the IoU value between the two text areas constituting the matching pair is greater than or equal to the preset IoU threshold, it is determined that the corresponding two text areas in the text area matching pair are matched successfully; when the IoU value between the two text areas constituting the matching pair is less than the preset IoU threshold, it is determined that the two text areas are matched unsuccessfully. Among them, the IoU threshold can be set based on experience, for example, it can be 0.7.

[0190] Record the matching status information of each text region corresponding to the current frame. In the embodiment of the present application, the matching status information of each text region can be stored in a queue, wherein the recorded information may include the identification number ID of the text region, whether the match is successful, and the like.

[0191] like Fig.12As shown, it is assumed that the NMth frame is the initial frame, and M is a positive integer greater than or equal to 4. Taking M as 4 as an example, three text regions A, B, and C are detected in the N-4th frame, and their identification numbers are ID1, ID2, and ID3, respectively. Then, the corresponding text regions in the N-4th frame are ID1 (corresponding to text region A), ID2 (corresponding to text region B), and ID3 (corresponding to text region C).

[0192] Two text regions D and E are detected in the N-3th frame, and identification numbers of the two text regions are to be assigned. The maximum match between the text regions D and E in the N-3th frame and the corresponding text regions A, B, and C in the N-4th frame is calculated to obtain a text region matching pair.

[0193] Specifically, a cost matrix C is constructed using the IoU values ​​between the corresponding text region in the N-4th frame and the text region detected in the N-3th frame. For example, the IoU values ​​calculated for the three text regions A, B, and C in the N-4th frame can be used as the rows of the cost matrix, and the IoU values ​​calculated for the two text regions D and E in the N-3th frame can be used as the columns of the cost matrix. Based on the cost matrix, the Hungarian matching algorithm is used to obtain the maximum match between the text region detected in the N-3th frame and the corresponding text region in the N-4th frame, for example, text region A->text region D, text region B->text region E, where -> represents a matching pair between the text regions.

[0194] Then, the IoU value between the two corresponding text areas in the matching pair is calculated and compared with the preset IoU threshold to determine the text area that matches successfully between the N-3 frame and the N-4 frame. Assume that the IoU value between text area A and text area D is 0.79, and the IoU value between text area B and text area E is 0.90, and both of the above IoU values ​​are greater than the preset IoU threshold (0.7). Therefore, text area A matches text area D, and text area B matches text area E. Then, the identification number ID1 of text area A is assigned to text area D, and the identification number ID2 of text area B is assigned to text area E. Therefore, the corresponding text areas in the N-3 frame are text areas ID1 (D), ID2 (E), and ID3 (C).

[0195] It can be understood that if the IoU value between the text regions with the maximum match obtained by the Hungarian matching algorithm is less than the preset IoU threshold, the match is considered unsuccessful. For example, assuming that the text region A and the text region E are matched by the Hungarian matching algorithm, and the IoU value between the text region A and the text region D is 0.65, which is less than the preset IoU threshold, then it is considered that the match between the text region A and the text region D is unsuccessful. Therefore, the identification number of the text region A cannot be assigned to the text region D, and a new identification number needs to be reassigned to the text region D.

[0196] Record the identification numbers (ID1(D), ID2(E) and ID3(C)) of the corresponding text regions in the N-3th frame and the corresponding matching status information of the three text regions, wherein the matching status information includes the number of successful matches, such as Fig.12 As shown, the text regions ID1(D) and ID2(E) are recorded as 1 in the latest position of the corresponding queue respectively, while the text region ID3(C) is not matched successfully and is recorded as 0 in the latest position of its corresponding queue.

[0197] For the N-2 frame, the text regions F and G detected in the N-2 frame are matched with the corresponding text regions (ID1(D), ID2(E) and ID3(C)) in the N-3 frame (the matching process is the same as the steps of the N-3 frame). Assuming that the text region G matches the text region ID3 successfully, and the text region F does not match successfully, the identification number of the text region G is assigned to ID3, and the text region F is assigned a new identification number ID4. Then, for the N-2 frame, the corresponding text regions are the text regions ID1(D), ID2(E), ID3(G) and ID4(F). The matching status information of the text regions ID1(D), ID2(E), ID3(G) and ID4(F) is recorded respectively. For example, if the text regions ID1(D), ID2(E) and ID4(F) do not match successfully, they are recorded as 0 in the latest position of the corresponding queue; and if the text region ID3(G) matches successfully, it is recorded as 1 in the latest position of the corresponding queue.

[0198] For the N-1th frame, the text regions H and I detected in the N-1th frame are matched with the corresponding text regions (ID1(D), ID2(E), ID3(G) and ID4(F)) in the N-2th frame. Assuming that the text region H successfully matches the text region ID1 and the text region I successfully matches the text region ID2, the identification number of the text region H is assigned to ID1, the identification number of the text region I is assigned to ID2, and the text regions ID3 and ID4 retain the original identification numbers. Then, for the N-1th frame, the corresponding text regions are the text regions ID1(H), ID2(I), ID3(G) and ID4(F). The matching status information of the text regions ID1(H), ID2(I), ID3(G) and ID4(F) is recorded respectively. For example, if the text regions ID1(H) and ID2(I) are successfully matched, they are recorded as 1 in the latest position of the queue; if the text regions ID3(G) and ID4(F) are not successfully matched, they are recorded as 0 in the latest position of the queue.

[0199] For the Nth frame, the text regions J and K detected in the Nth frame are matched with the corresponding text regions (ID1(D), ID2(E), ID3(G) and ID4(F)) in the N-1th frame. Assuming that the text region K matches the text region ID1 successfully and the text region L matches the text region ID3 successfully, the identification number of the text region K is assigned to ID1, the identification number of the text region L is assigned to ID3, and the text regions ID2 and ID4 retain the original identification numbers. For the Nth frame, the corresponding text regions are the text regions ID1(J), ID2(I), ID3(K) and ID4(F). The status information of the text regions ID1, ID2 and ID3 is recorded respectively. For example, if the text regions ID1(J) and ID3(K) match successfully, they are recorded as 1 in the latest position of the array; if the text regions ID2(I) and ID4(F) do not match successfully, they are recorded as 0 in the latest position of the array.

[0200] Furthermore, in order to reduce the number of matches between the text region detected in the current frame and the text region corresponding to the previous frame, the text region that has not been successfully matched in two consecutive frames can be removed from the corresponding text region. Fig.12 As shown, text area ID4 (F) is not matched successfully in two consecutive frames, so starting from the Nth frame, text area ID4 (F) can be removed, that is, the corresponding text areas in the Nth frame image are text areas ID1 (J), ID2 (I) and ID3 (K).

[0201] In another embodiment of the present application, the corresponding text area in the previous frame may be a collection of all text areas detected before the current frame. This step may specifically include: Fig.13 Step S3021"-step S3022" in step S302" shown.

[0202] Step S3021", obtaining the text area detected in the current frame.

[0203] Step S3022', matching the text region detected in the current frame with the text region in the corresponding matching set in the previous frame, and recording the number of successful matches for each text region in the current frame.

[0204] Exemplarily, the corresponding matching set in the last preview stream image may be a set of all text regions detected from the initial frame to the last preview stream image. Successful matching may mean that the IoU value between the two text regions is greater than a preset IoU threshold. The preset IoU threshold may be set based on experience, for example, to 0.7.

[0205] like Fig.14As shown, assuming that the NMth frame is the initial frame, taking M as 4 as an example, in the N-4th frame image, two text areas A and B are detected, then the matching set corresponding to the N-4th frame image is the set of text areas A and B.

[0206] In the N-3 frame image, two text areas C and D are detected, so the corresponding matching set in the N-3 frame image is the set of text areas A, B, C, and D. Match text areas C and D with the corresponding text areas in the N-4 frame (i.e., text areas A and B), and record the number of successful matches of text areas C and D detected in the N-3 frame. Exemplarily, the IoU value between text areas C and D and text areas A and B can be calculated. When the IoU value between the two text areas is greater than a preset value, it is determined that the two text areas are successfully matched, and the number of successful matches of text areas C and D is recorded. For example, in Fig.14 In the example, text regions C and D did not successfully match text regions A and B, so text region C successfully matched 0 times and text region D also successfully matched 0 times.

[0207] In the N-2 frame image, three text regions E, F, and G are detected. Then the corresponding matching set in the N-2 frame image is the set of text regions A, B, C, D, E, F, and G. Text regions E, F, and G are matched with the corresponding text regions in the N-3 frame (i.e., text regions AD), and the number of successful matches of text regions E, F, and G detected in the N-2 frame is recorded. For example, Fig.14 In the example, text region E matches A once, text region F matches C once, and text region G matches B once.

[0208] In the N-1th frame image, two text regions H and I are detected, then the corresponding matching set in the N-1th frame image is the set of text regions A, B, C, D, E, F, G, H and I. Text regions H and I are matched with the corresponding text regions (i.e., text regions AG) in the N-2th frame respectively, and the number of successful matches between text regions H and I detected in the N-1th frame is recorded. Fig.14 In the figure, text region H successfully matches A and E twice, and text region I successfully matches C and F twice.

[0209] In the Nth frame image, two text regions J and K are detected, then the corresponding matching set in the Nth frame image is the set of text regions A, B, C, D, E, F, G, H, I, J and K. Text regions J and K are matched with the corresponding text regions in the N-1th frame (i.e., text region AI) respectively, and the number of successful matches of text regions J and K detected in the Nth frame is recorded. Fig.14In the figure, text region J successfully matches A, E, and H three times, and text region K successfully matches B and G twice.

[0210] Furthermore, in order to reduce the number of matches between the text region detected in the current frame and the matching set corresponding to the previous frame, the text region that has not been successfully matched in two consecutive frames can be removed from the matching set. Fig.14 As shown, text area D has not been matched successfully in two consecutive frames. Then, starting from the Nth frame, text area D can be removed from the matching set. That is to say, the corresponding matching set in the Nth frame image is the set of text areas A, B, C, E, F, G, H, I, J and K.

[0211] Step S303, determining whether the current frame contains a stable text area.

[0212] In the embodiment of the present application, text region matching is performed continuously on M preview stream image frames. If the text region has at least L successful matches, and the text region can also be matched with the text region corresponding to the Nth frame and the N-1th frame, then it is considered that the Nth frame contains a stable text region and meets the conditions for text region output. Wherein, M is a positive integer greater than or equal to 4, and L is a rounded integer value of 0.75 times M. For example, text region matching is performed on 4 consecutive preview stream image frames. If a text region has 3 successful matches, and the text region can also be successfully matched in the current preview stream image and the previous preview stream image, then the text region is determined to be a stable text region in the current frame. Fig.12 As shown in the figure, for the text region ID1(J) in the Nth frame, the matching is successful 3 times in the text region matching from the N-3th frame to the Nth frame, and the text region ID1(J) can also match the corresponding text region in the N-1th frame, so the text region ID1(J) is a stable text region. Fig.14 As shown, for the text region J in the Nth frame, it is successfully matched with the matching set in the N-1th frame for 3 times, so the text region J is a stable text region.

[0213] If the current frame includes a stable text region, step S304 is executed; if the current frame does not include a stable text region, the process returns to step S301 to re-match the text region until a stable text region detection result is obtained.

[0214] Step S304: output a stable text area.

[0215] There can be one or more stable text areas.

[0216] In some embodiments of the present application, the coordinates of all stable text areas may be output to a buffer.

[0217] In some other embodiments of the present application, the coordinates of only one text region in the stable text region may be output to the buffer. Exemplarily, the coordinates of the text region closest to the focal position of the camera may be selected for output, where the focal position of the camera may be the central position of the preview stream image frame.

[0218] One of the following methods may be used to determine the text region closest to the central position:

[0219] The first method: Calculate the central point coordinates of each text region and the distance value from the central point coordinates of each text region to the central position, and use the text region with the smallest distance value as the text region closest to the central position. As Fig.15A shown, the distance value from the central point coordinates of text region 1 to the central position is R1, and the distance value from the central point coordinates of text region 2 to the central position is R2. Among them, R2 < R1, then it is considered that text region 2 is the text region closest to the focal position, and the coordinates of text region 2 are output.

[0220] The second method: Obtain the 4 vertex coordinates of each text region, calculate the distance values from all vertices of multiple text regions to the central position respectively, and sort them. Use the text region with the vertex having the smallest distance value as the text region closest to the central position. As Fig. 15B shown, the 4 vertices of text region 1 are A1, A2, A3, A4 respectively, and the 4 vertex coordinates of text region 2 are (B1, B2, B3, B4) respectively. Calculate the distance values R1 to R8 from these 8 vertices to the central position respectively, and sort them. For example, if the distance value from vertex B1 of text region 2 to the central position is the smallest, then text region 2 is the text region closest to the central position, and the coordinates of text region 2 are output.

[0221] The third method: Expand each text region until at least one point on the outer contour of the text region coincides with the central position, record the expansion ratio coefficient of each text region at this time, and use the text region corresponding to the minimum expansion ratio coefficient as the text region closest to the central position. As Fig. 15C shown, expand text region 1 and text region 2 respectively until at least one point on the outer contours of text region 1 and 2 coincides with the central position. At this time, the expansion ratio coefficient of text region 1 is a, and the expansion ratio coefficient of text region 2 is b. The expansion ratio coefficient b is less than the expansion ratio coefficient a, then use text region 2 corresponding to the expansion ratio coefficient b as the text region closest to the central position, and output the coordinates of text region 2.

[0222] After obtaining the coordinates of the stable text region, step B can be executed. Among them, step B is the same as Figure 2Steps S130 to S180 are the same as those in the above description and will not be described in detail here.

[0223] In the embodiment of the present application, after obtaining the text region of the preview stream image frame, the number of successful matches of the text region detected in the current frame in the previous multiple frames is determined by matching the text region detected in the current frame with the corresponding text region in the previous frame, wherein successful matching means that the overlap between the two text regions is greater than a first preset value. When the number of successful matches of the text region detected in the current frame is greater than a second preset value, the text region is determined to be a stable text region, and one of the text regions can be displayed on the preview stream image frame, and the text in one of the text regions is recognized to obtain a text scanning result.

[0224] In the process of scanning the text, the preview stream image data is acquired frame by frame. For each frame of the image, a neural network model needs to be called for processing when detecting the text area. The inventors have found that if the neural network model is called frame by frame to obtain the text area, the detection result will have a time delay, the real-time presentation of the text area and text recognition results to the user will be poor, and the power consumption of the terminal device will also be relatively high. Therefore, in one embodiment of the present application, a method for tracking the detected text area is also proposed. By simultaneously tracking multiple detected text areas, the number of calls to the text area detection algorithm can be reduced, thereby reducing the power consumption of the terminal device.

[0225] like Fig.16 The figure is a flow chart of a text area tracking method provided in one embodiment of the present application.

[0226] Step A, this step is related to Figure 2 The process is the same as step S110 in , and will not be repeated here.

[0227] Step S400, checking whether there is data in the buffer.

[0228] The data may include preview stream image frame data, coordinate data of text area, etc. If the buffer contains the above data, execute step S500. If the buffer does not contain the above data, execute step C, wherein step C may include the following: Figure 7 In some other embodiments of the present application, step C may also adopt other text area detection methods.

[0229] Step S500, tracking the text area. The method may specifically include steps S501 to S511.

[0230] Step S501, reading data in the buffer.

[0231] Read the preview stream image frame data and the coordinate data of the detected text area in the buffer.

[0232] Step S502, obtaining a bounding rectangle with a minimum area covering all text regions, and obtaining coordinates of the bounding rectangle covering all text regions.

[0233] Step S503, determining feature points within the circumscribed rectangular area in step S502.

[0234] For the circumscribed rectangular area, Harris corner detection, SIFT (Scale invariant feature transform), SURF (Speeded Up Robust Features), ORB (Oriented FAST and Rotated BRIEF) and other algorithms can be used to extract feature points.

[0235] Taking the Harris corner detection algorithm to extract feature points as an example, the bounding rectangle can be divided into multiple regions, and the corner points of each of the multiple regions are determined as at least one feature point of the bounding rectangle. For example, the bounding rectangle can be divided into multiple grid regions of the same size, and the Harris corner points H of the grid regions are determined as at least one feature point of the bounding rectangle. j As the feature point of the grid area, 1≤j≤M, M is the total number of divided grid areas, all feature points can be recorded as H, and all feature points H are composed of M Hj. In addition to the regional corner points, Harris corner points can also be isolated points with the maximum or minimum intensity in some attributes, such as the end point of a line segment or the point with the maximum local curvature on a curve.

[0236] It can be understood that in the embodiment of the present application, other feature point extraction algorithms besides the Harris corner point detection algorithm can also be used to extract feature points within the circumscribed rectangular area.

[0237] After the feature points within the circumscribed rectangular area are extracted, the preview stream image frame data, the coordinate data of the text area, and the extracted feature point data may be stored in a buffer.

[0238] Step S504: determining feature points of the current preview stream image frame.

[0239] In the embodiment of the present application, based on the feature points obtained in step S503, the coordinates of at least one feature point in the current preview stream image frame can be calculated using the optical flow method. Fig.17 As shown, the feature points 11, 12, 13 and 14 in the previous preview stream image frame are tracked using the optical flow method, thereby obtaining the corresponding feature points 11', 12', 13', 14' in the current preview stream image frame.

[0240] Step S505 , determining whether the number of valid feature points in the current preview stream image frame is greater than a first preset threshold.

[0241] In an embodiment of the present application, after obtaining the feature point coordinates of the current preview stream image frame, the feature point motion difference between each feature point in the current preview stream image frame and the corresponding feature point in the previous preview stream image frame can be determined. Exemplarily, the feature point motion difference can be the Euclidean distance between the feature points. Based on the Euclidean distance, valid feature points are determined from the feature points. Specifically, the feature point motion difference is inversely correlated with the effectiveness of the feature point, that is, the greater the feature point motion difference of the feature point, the lower the effectiveness of the feature point. The effectiveness of the feature point can indicate the quality of the feature point tracking, that is, the effectiveness of the feature point can characterize the stability of the feature point. The higher the effectiveness of the feature point, the more accurate the tracking of the feature point is and the better the stability of the feature point is.

[0242] In the embodiment of the present application, for the first feature point among the feature points, if the feature point motion difference between the feature point of the current preview stream image frame and the feature point corresponding to the previous preview stream image frame is less than a preset threshold, the first feature point can be determined as a valid feature point. The valid feature point is a feature point with a high degree of validity, that is, a feature point with relatively good stability.

[0243] The method for determining effective feature points in the embodiment of the present application is not limited thereto, and effective feature points can also be determined by other possible methods based on the principle that the feature point motion difference of the feature point is inversely correlated with its effectiveness. For example, for each feature point, the sum of the feature point motion differences between any two adjacent frames can also be counted, and the sum is inversely correlated with the effectiveness of the feature point, that is, the larger the sum, the worse the effectiveness of the feature point. When the sum does not exceed the set threshold, the feature point can be determined to be a valid feature point, etc.

[0244] After determining the valid feature points, it is determined whether the number of valid feature points in the current preview stream image frame is greater than a first preset threshold value to determine whether the feature points in the previous preview stream image frame are correctly tracked. The first preset threshold value can be set according to an empirical value, for example, the first preset threshold value is 90% of the number of all feature points. If the number of valid feature points meets the above preset conditions, step S506 is executed; if the above preset conditions are not met, step S508 is executed.

[0245] Step S506, obtaining the perspective transformation matrix between the last preview stream image and the current preview stream image.

[0246] Perspective transformation is a processing method that performs spatial geometric transformation on image pixels without changing the image content. Specifically, perspective transformation maps the coordinate position in one image to the new coordinate position in another image. According to the perspective transformation matrix T, the previous preview stream image can be transformed and projected to the plane of the current preview stream image.

[0247] It should be noted that, in step S505, when determining valid feature points, excessive reliance is placed on a preset threshold, but the size of the preset threshold cannot be accurately determined under normal circumstances. When the preset threshold is set to a larger value, more erroneous valid feature points will appear; when the preset threshold is set to a smaller value, although the number of erroneous valid feature points will be reduced, the number of valid feature points will be significantly reduced, which seriously affects the selection of the perspective transformation matrix T. Therefore, in the embodiment of the present application, the RANSAC (Random Sample Consensus) algorithm is also introduced to further remove erroneous valid feature points, thereby obtaining a more accurate perspective transformation matrix T.

[0248] The RANSAC algorithm refers to fitting an estimated model by repeatedly randomly selecting a set of random subsets from the data. The selected random subsets are assumed to be inliers and verified by the following method: 1) There is a model for the assumed inliers, that is, all unknown parameters can be calculated from the assumed inliers; 2) Use the model obtained in 1) to test all other data. If a point is suitable for the estimated model, it is considered to be an inlier; 3) If enough points are classified as assumed inliers, then the estimated model is reasonable enough; 4) Use all assumed inliers to re-estimate the model; 5) Evaluate the model by estimating the error rate of inliers and models. This process is repeated a fixed number of times, and the model generated each time is either discarded because there are too few inliers or selected because it is better than the existing model.

[0249] For example, 4 valid feature points can be randomly selected from the valid feature points of the previous frame, and a perspective transformation matrix T can be calculated based on the 4 valid feature points and the positions tracked by the 4 valid feature points in the current frame. For other valid feature points in the previous frame except the above 4 points, the corresponding feature points in the current frame are obtained after passing through the perspective transformation matrix T. The feature points obtained after the perspective transformation matrix T of the current frame are compared with the feature points tracked by the optical flow method. If the difference between the two meets the preset conditions, for example, the difference is within 3 pixels, then the corresponding feature point in the previous frame is considered to be an internal point; if the above preset conditions are not met, then the corresponding feature point in the previous frame is considered to be an external point. In an embodiment of the present application, the internal points can be calculated using the RANSAC algorithm through the relevant functions of OpenCV, and the perspective transformation matrix T can be obtained based on the internal points, where the internal points are a subset of the valid feature points.

[0250] Step S507, determining whether the number of intra-office points is greater than a second preset threshold.

[0251] In order to ensure that there are enough feature points to be correctly tracked, the embodiment of the present application further determines whether the number of inliers is greater than a second preset threshold. The second preset threshold can be set according to an empirical value, for example, the second preset threshold can be 80% of the number of valid feature points. If the number of inliers meets the above condition, step S509 is executed, and if the above condition is not met, step S508 is executed.

[0252] Step S508: If tracking fails, re-detect the text area.

[0253] When the number of valid feature points is less than the first preset threshold, or the number of points within the bureau is less than the second preset threshold, the tracking is considered to have failed. In an embodiment of the present application, the value of the tracking flag can be set to characterize the success or failure of the tracking. For example, when the value of the tracking flag is true, it indicates that the tracking is successful; and when the value of the tracking flag is false, it indicates that the tracking has failed, and the text area detection step will be re-executed. The text area detection step can be step C or other text area detection methods.

[0254] Step S509, obtaining the perspective transformation matrix of the initial frame preview stream image when the current frame preview stream image is tracked relative to the text area, applying perspective transformation to each text area in the initial frame preview stream image, thereby obtaining the text area of ​​the current frame.

[0255] It should be noted that the text area will change from the original rectangle to an irregular quadrilateral after perspective transformation, such as Fig.18 As shown. Subsequently, it is necessary to calculate the bounding rectangle of the minimum area of ​​the text area after the perspective transformation. If the bounding rectangle of the minimum area of ​​the text area is calculated for each preview stream image frame after the perspective transformation, the bounding rectangle of the text area finally obtained will become larger and larger, resulting in inaccurate output and recognition results of subsequent text areas. Therefore, in an embodiment of the present application, it is necessary to calculate the perspective transformation matrix of the current preview stream image frame relative to the initial preview stream image frame of the text tracking, wherein the initial frame preview stream image can be the preview stream image in which the text area is detected, that is, the preview stream image frame corresponding to the text area in the buffer.

[0256] like Fig.19As shown, the preview stream image frame sequence for text area tracking includes the N-2th frame, the N-1th frame, and the Nth frame. Assume that the initial preview stream image frame is the N-2th frame, the current preview stream image frame is the Nth frame, and the N-1th frame is the preview stream image frame located between the initial frame and the current frame. The perspective transformation matrix T between the N-2th frame and the N-1th frame is calculated respectively. 12 , the perspective transformation matrix between the N-1th frame and the Nth frame is T 23 , then the perspective transformation matrix of the current preview stream image frame relative to the initial preview stream image frame is T 12 *T 23 .

[0257] The coordinates of each text area in the initial preview stream image frame are compared with the perspective transformation matrix T 12 *T 23 By multiplying them, you can get the coordinates of the corresponding text area in the current preview stream image frame.

[0258] Step S510, obtaining the minimum bounding rectangle of each text region in the current frame preview stream image.

[0259] After obtaining the minimum bounding rectangle of each text region in the preview stream image of the current frame, one or more text regions can be obtained.

[0260] Step S511, output the text area.

[0261] For each preview stream image frame, one or more text regions may be tracked. In an embodiment of the present application, the coordinates of the text region closest to the camera focus position may be selected and output, wherein the camera focus position may be the center position of the preview stream image frame. The text region closest to the center position may be determined using the method described in step S304, which will not be described in detail herein.

[0262] In some other embodiments of the present application, the coordinates of the text area corresponding to the user's touch point position can also be output in response to the user's point selection operation.

[0263] After obtaining the coordinates of the text area, execute step B. Figure 2 Steps S130 to S180 are the same as those in the above description and will not be described in detail here.

[0264] In an embodiment of the present application, after obtaining the preview stream image frame data and the corresponding text area from the buffer, the first feature point of the circumscribed rectangle is determined by determining the minimum area of ​​all text areas, and the second feature point in the subsequent preview stream image frame is determined. When the number of valid feature points in the second feature point is greater than the first preset threshold, the perspective transformation matrix between the subsequent preview stream image frame and the preview stream image frame in the buffer is determined, and the perspective transformation is applied to the text area in the buffer to obtain the tracked text area. By tracking the detected text area instead of calling the neural network model frame by frame to detect the text area, the power consumption of the terminal device can be reduced.

[0265] In one embodiment of the present application, a method for text scanning is proposed.

[0266] like Fig. 20 FIG. 4 is a flow chart of a text scanning method according to an embodiment of the present application.

[0267] Step A, this step is related to Figure 2 The process is the same as step S110 in , and will not be repeated here.

[0268] Step S600, determining whether a preset time interval has been reached.

[0269] In the embodiment of the present application, it is possible to set the text detection algorithm to be automatically called at regular intervals to detect new text areas in the preview stream image frames. In this step, it is necessary to determine whether the preset time interval has been reached. The preset time interval may be 2 seconds, and the first frame of the preview stream may be regarded as a case where the preset time interval has been reached. When the preset time interval has been reached, step C is executed, wherein step C may include Figure 7 Steps S200 and S300 in the embodiment; if the preset time interval is not reached, execute step S400.

[0270] Step S400, check whether there is data in the buffer. If the buffer contains data, execute step D; if the buffer does not contain data, execute step S700; wherein the data includes preview stream image frame data and text areas detected in the preview stream image frame.

[0271] Step D is as follows Fig.16 Step S500 shown in FIG.

[0272] Step S700, checking whether there is a tracking result of the previous frame.

[0273] If the buffer does not contain data, it means that there is no newly detected text area in the preview stream image frame, and then check whether there is a tracking result of the previous frame. If there is a tracking result of the previous frame, execute step S800; if there is no tracking result of the previous frame, execute step C.

[0274] Step S800, continue tracking the tracking result of the previous frame.

[0275] In the embodiment of the present application, in this step, the text area tracked in the previous frame continues to be tracked.

[0276] Step S900, determining whether the tracking is successful.

[0277] Whether the tracking is successful can be determined by reading the value of the tracking flag. In the process of tracking the text area, when the text area tracking is successful, the value of the tracking flag is set to true; when the text area tracking fails, the value of the tracking flag is set to false. When the value of the tracking flag is true, execute step B; when the value of the tracking flag is false, execute step C.

[0278] Wherein, step B can be as follows Figure 2 Steps S130 to S180 shown in FIG. 1 ; Step C may be as follows Figure 7 Steps S200 and S300 shown in FIG.

[0279] In the embodiment of the present application, the text scanning method can be divided into a text area detection and matching process and a text area tracking process, and the two processes are performed asynchronously. In the text area detection and matching process, the text area detection and text area matching algorithms can be performed on the collected preview stream image frames, specifically as the above steps S200 and S300; in the text area tracking process, the text area tracking algorithm is performed, specifically as the above step S500. Fig.21 A timing diagram showing the asynchronous execution of the text region detection and matching process and the text region tracking process.

[0280] like Fig.21 As shown, in the embodiment of the present application, when the text area detection result includes a stable text area, the text detection result is written into the buffer. For example, if 4 preview stream images are detected continuously, if a text area is successfully matched 3 times, and the text area can be successfully matched on both the 4th preview stream image frame and the 3rd preview stream image frame, the data of the text area is written into the buffer.

[0281] After obtaining stable text region data, the text region detection and matching process stops from frame 5. Meanwhile, if text region data is detected to be stored in the buffer during text region tracking, the detected text region is tracked from frame 5.

[0282] If the tracking fails at the 8th frame in the text area tracking process, the text area detection and matching is performed again from the 9th frame in the text area detection and matching process, and the text area tracking process is stopped. If a stable text area is detected again at the 12th frame, the data of the text area is written into the buffer again for the text area tracking process to read and continue tracking, and the text area detection and matching process is stopped from the 13th frame. Repeat the above text area detection and matching and text area tracking process.

[0283] It should be noted that in the process of text region detection and matching and text region tracking, new text regions may appear in the middle frame. In order to ensure that the newly appeared text regions can also be detected and tracked, the embodiment of the present application can be set to automatically execute the text region detection and matching process at regular intervals. Fig.21 As shown, when the set time interval is reached at the 74th frame, the text area detection and matching process is automatically executed, and the detection result is written into the buffer after the stable text area is obtained at the 77th frame. The text area tracking process is also synchronously executed from the 73rd frame to the 77th frame. When the text area tracking process reads the data with the detection result in the buffer at the 78th frame, the text area tracking process tracks the text area in the buffer instead of the tracking result of the 77th frame.

[0284] Fig. 22 A schematic diagram showing a specific implementation of the text scanning method in an embodiment of the present application is shown.

[0285] VisionInputImp (visual input image), InputView (input display), and VisionInputEngine (visual input engine) are functional modules in the system framework layer, and the AiPluginEngine plug-in is a functional module in the algorithm engine layer. The specific process of the text scanning method in the embodiment of the present application is as follows:

[0286] Step S1001, in response to a first operation of the user, calling the VisionInputImp function module to start a text scanning function.

[0287] Step S1002, the VisionInputImp function module connects to the service and loads the model in the AiPluginEngine plug-in.

[0288] Step S1003, the VisionInputImp function module calls the InputView function module to load the preview stream page and open the preview stream page, and obtains the Camera2API callback frame data and the preview stream image.

[0289] Step S1004, the VisionInputImp function module calls the text area detection, tracking, and text recognition algorithms in the AiPluginEngine plug-in through the VisionInputEngine function module, performs text area detection, tracking, and text recognition, and returns the detection results to the VisionInputEngine function module for processing. After the processing is completed, the processing results are returned to the VisionInputImp function module for result judgment, and the judgment result is whether there is text or no text. According to the judgment result, step S1008 or step S1009 is executed.

[0290] Step S1005 , in response to a user click operation on the preview stream image frame, the InputView function module is called to obtain the coordinates of the user click, and the click coordinates are returned to the VisionInputImp function module.

[0291] Step S1006: Process the image of the preview stream and the click coordinate data in the VisionInputImp function module.

[0292] Step S1007 is the same as step S1004.

[0293] Step S1008, when the judgment result is that there is text, the VisionInputImp function module calls the InputView function module to render the preview stream image frame according to the coordinate result of the text area, displays the text area on the preview stream image frame, and displays the recognized text at the same time. The VisionInputImp function module pre-submits the recognized text to the application.

[0294] Step S1009, when the judgment result is that there is no text, the VisionInputImp function module calls the InputView function module to display user prompt information, and obtains the Camera2API callback frame data and the preview stream image.

[0295] When the user clicks the text area again, steps S1005-S1009 may be repeated.

[0296] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. In order to achieve the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily appreciate that, in combination with the method steps of each example described in the embodiment disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.

[0297] The embodiment of the present application can divide the functional modules of the device implementing the method according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0298] Exemplarily, the present application also provides a text scanning device, which includes an acquisition module for acquiring a preview stream image frame; a text region detection module for detecting one or more text regions in the preview stream image frame; a text region matching module for performing multi-frame matching on the text region and outputting a stable text region; a text region tracking module for tracking a stable text region and outputting a successfully tracked text region; a text recognition module for recognizing text in a successfully tracked text region; and a UI rendering module for rendering the text region. This device can be specifically a chip, a component or a module.

[0299] In an embodiment of the present application, a terminal device is further provided, which may include: a memory and at least one processor. The memory is coupled to the at least one processor and is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the terminal device can perform the functions and steps in the above-mentioned method embodiments.

[0300] In the embodiment of the present application, a computer-readable storage medium is also provided, in which a computer instruction is stored, and when the computer instruction is run on an electronic device, the electronic device performs each function and step in the above-mentioned related method. Among them, any reference to a memory, a database or other medium used in each embodiment provided in the present application may include at least one of a non-volatile and a volatile memory. Non-volatile memory may include a read-only memory (ROM), a magnetic tape, a floppy disk, a flash memory, an optical memory, a high-density embedded non-volatile memory, a resistive random access memory (ReRAM), a magnetic random access memory (MRAM), a ferroelectric random access memory (FRAM), a phase change memory (PCM), a graphene memory, etc. Volatile memory may include a random access memory (RAM) or an external cache memory, etc. As an illustration and not limitation, RAM may be in various forms, such as a static random access memory (SRAM) or a dynamic random access memory (DRAM), etc. The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0301] This embodiment also provides a computer program product, which, when executed on a computer, enables the computer to execute each function or step in the above method embodiment. The computer may be a terminal device, such as a mobile phone.

[0302] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0303] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0304] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0305] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0306] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0307] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A text scanning method, the method being applied to a terminal device, comprising: Get the preview stream image of the text to be scanned; If the time interval from the last detection moment satisfies the preset time interval, the first text area in the preview stream image of the Mth frame is detected; If the time interval from the last detection moment does not meet the preset time interval, acquiring data in the buffer, wherein the data at least includes the second text area and the Nth frame preview stream image, wherein the second text area is detected in the Nth frame preview stream image; Tracking the second text area to determine a third text area in the preview stream image of the N+Kth frame, wherein the N+Kth frame does not satisfy the preset time interval with the Nth frame; Performing text recognition on one of the third text regions to obtain a scanning result; Among them, M, N, and K are all positive integers.

2. The text scanning method according to claim 1, characterized in that: The method further comprises: If the time interval from the last detection moment does not satisfy the preset time interval, and the data is not read in the buffer, continue to detect the fourth text area in the preview stream image; or The fifth text region tracked in the last frame of the preview stream image is tracked.

3. The text scanning method according to claim 1, characterized in that: The method further comprises: If the tracking of the second text area fails in the N+Xth frame, the text area in the N+X+1th frame is detected, where X is less than K.

4. The text scanning method according to claim 1 or 2, characterized in that: Detecting the first text area / the fourth text area includes: Obtaining a binary mask image of the Mth frame / N-1th frame preview stream image; Obtaining text lines in the binary mask image; Clustering the text lines to obtain text blocks; The text block is expanded to obtain the first text area.

5. The text scanning method according to claim 4, characterized in that: The obtaining of the text line in the binary mask image comprises: Taking the circumscribed rectangle of the minimum area for the text connected domain in the binary mask image to obtain the coordinates of the text connected domain; determining the main directions of the plurality of text connected domains, and rotating the text connected domain to the horizontal direction; The text connected domains in the same line are horizontally spliced ​​to obtain the text line.

6. The method for text scanning according to claim 5, characterized in that: The text connected domain is a connected area in which the grayscale value of adjacent pixels in the binary mask image is 1.

7. The method for text scanning according to claim 4, characterized in that: The text lines are clustered to obtain text blocks, including: vertically expanding the text line; The IoU value in the horizontal direction and the IoU value in the vertical direction of the expanded text line are determined, and when the IoU value in the horizontal direction and the IoU value in the vertical direction meet a preset condition, the text line is clustered to obtain the text block.

8. The text scanning method according to claim 4, characterized in that: Expanding the text block to obtain the first text area includes: Determine an expansion ratio according to the maximum value of the shortest distance between the pixel point of the text line and its contour, expand the text block according to the expansion ratio, and take the minimum area of ​​the circumscribed rectangle of the expanded text block to obtain the first text area.

9. The text scanning method according to claim 7, characterized in that: The IoU value is the intersection area of ​​two text regions within the union area of ​​the two text regions.

10. The text scanning method according to claim 1 or 2, characterized in that: Tracking the second text area / the fifth text area to determine a third text area in the N+Kth frame preview stream image includes: Determine a bounding rectangle with a minimum area that includes the second text area / the fifth text area; Determine a first feature point of the circumscribed rectangle; Determine a second feature point in the preview stream image of the N+Kth frame; When the first preset condition is met, determining the perspective transformation matrix between the preview stream image of the N+Kth frame and the preview stream image of the Nth frame, wherein the valid feature point refers to a feature point whose motion difference between the feature point in the second feature point and the feature point corresponding to the first feature point is less than a threshold; When the second preset condition is met, perspective transformation is applied to the second text region / the fifth text region based on the perspective transformation matrix to determine the third text region.

11. The method for text scanning according to claim 10, characterized in that: The first feature point is determined by using one of the following algorithms: Harris corner detection algorithm, SIFT algorithm, SURF algorithm, and ORB algorithm.

12. The text scanning method according to claim 10, characterized in that: The second feature point is determined by an optical flow method.

13. The method for text scanning according to claim 10, characterized in that: The first preset condition includes that the number of valid feature points in the second feature points is greater than a first preset threshold; the second preset condition includes that the number of inliers is greater than a second preset threshold, wherein the inliers are determined based on the RANSAC algorithm in the perspective transformation matrix process.

14. The method for text scanning according to claim 13, characterized in that: The first preset threshold is 90% of the number of the first feature points; the second preset threshold is 80% of the number of the valid feature points.

15. The method for text scanning according to claim 1, characterized in that: The preset time interval is 2 seconds.

16. The method for text scanning according to claim 1, characterized in that: The process of detecting the text region is performed asynchronously with the process of tracking the text region.

17. The method for text scanning according to claim 1, characterized in that: One of the third text regions includes a text region that is closest to the center position of the N+Kth frame preview stream image among the third text regions.

18. The method for text scanning according to claim 17, characterized in that: The nearest text area is determined by any of the following methods: Obtaining the center point coordinates of each text area in the third text area and the distance value from the center point coordinates to the center position, and taking the text area with the smallest distance value as the nearest text area; Obtaining the vertex coordinates of each text area in the third text area, determining the distance value from the vertex coordinates of each text area to the center position respectively, and sorting them, and taking the text area corresponding to the vertex with the smallest distance value as the nearest text area; Each text area in the third text area is expanded, and an expansion ratio when at least one point in the outer contour of each text area coincides with the center position is determined, and a text area corresponding to a minimum expansion ratio is used as the nearest text area.

19. The method for text scanning according to claim 1, characterized in that: Performing text recognition on one of the third text regions to obtain a scanning result includes: Displaying the third text area in the N+Kth frame preview stream image; The text recognition result of the third text is displayed.

20. The method for text scanning according to claim 19, characterized in that: Displaying the third text area in the N+Kth frame preview stream image includes: A focus frame is displayed around the one of the third text areas.

21. The method for text scanning according to claim 1, characterized in that: The method further includes: A click operation of the user on the N+Kth frame preview stream image is received, a text area corresponding to the user click position is displayed on the N+Kth frame preview stream image, and a text recognition result of the corresponding text area is displayed.

22. The text scanning method according to claim 1, characterized in that: Performing text recognition on one of the third text regions to obtain a scanning result includes: The third text area is recognized by using an OCR recognition algorithm to obtain a scanning result.

23. The method for text scanning according to claim 1, characterized in that: The method further includes: If the third text area is not acquired in the N+Kth frame preview stream image, prompt information is displayed on the N+Kth frame image.

24. The method for text scanning according to claim 23, characterized in that: The prompt information includes moving too fast or please aim at the text.

25. The method for text scanning according to claim 1, characterized in that: Before obtaining the preview stream image of the text to be scanned, the method further includes: Receive the user's first action in the application and start text scanning.

26. The method for text scanning according to claim 25, characterized in that: The application includes one of the following: memo, notes, notepad, WeChat, office software.

27. The method for text scanning according to claim 25, characterized in that: The first operation includes a click or a long press operation.

28. The method for text scanning according to claim 25, characterized in that: The method further includes: A second operation of the user in the application is received, and the scan result is inserted into the application.

29. A terminal device, comprising: A memory and at least one processor, wherein the memory is used to store a computer program, and the at least one processor is used to execute the computer program, so that the terminal device executes the text scanning method as described in any one of claims 1-28.

30. A computer-readable storage medium storing instructions, which, when executed, enable the computer to execute the text scanning method according to any one of claims 1 to 28.

Images