Text extraction method and device
By responding to touch screen operations by terminal devices, obtaining and adjusting text information, the problem of OCR technology's complicated operation of identifying fuzzy letters and controls is solved, and fast and accurate text extraction is achieved.
Patent Information
- Application Number
- CN202011362776.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-11-27
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2040-11-27
AI Technical Summary
The existing OCR technology is difficult to accurately identify fuzzy or similar letters, such as lowercase L and uppercase I, and the text information obtained by the control word collection technology requires users to search manually, which is cumbersome.
The terminal device responds to the touch screen operation, acquires text information in the touch area, determines the target control, and adjusts the first text information based on the second text information to improve the accuracy of the text position and content.
It realizes quick and convenient access to text information with accurate location and correct content, improving the accuracy and user experience of text extraction.
Smart Images

Figure CN114564141B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of optical character recognition (OCR) in the field of terminal artificial intelligence (AI), and in particular to a text extraction method and device. Background Art
[0002] In life, words are everywhere, and words are an important means for people to perceive the world. Artificial intelligence technology can simulate, extend and expand human consciousness and thinking. Obtaining text information is an important part of artificial intelligence technology.
[0003] OCR technology and control word extraction technology are two common ways to obtain text information. OCR technology can read characters on pictures or paper and convert them into computer text. However, OCR technology cannot accurately recognize text that is difficult for the human eye to distinguish. For example, OCR cannot accurately distinguish between lowercase L (i.e. l) and uppercase i (i.e. I). OCR technology cannot accurately recognize characters in links and password characters. Although the text obtained by control word extraction technology is exactly the same as the original text, control word extraction obtains the entire text in the entire control, requiring users to find the required part in it, which is cumbersome. Summary of the invention
[0004] In view of this, a text extraction method and device are proposed, which can conveniently, quickly and accurately obtain the text information required by the user.
[0005] In a first aspect, an embodiment of the present disclosure provides a text extraction method, including: a terminal device obtains a touch area in response to a touch operation on a touch screen, and extracts text information in the touch area through OCR technology, which is recorded as first text information. The terminal device determines a target control that matches the previously obtained touch area from one or more text controls on the touch screen that can obtain text content, and obtains text information from the target control, which is recorded as second text information. The terminal device adjusts the first text information based on the second text information to obtain a final third text information. In this way, the first text information with accurate text position is adjusted based on the second text information with correct text content, so that the third text information with accurate position and correct content can be obtained conveniently and quickly.
[0006] According to the first aspect, in a first implementation of the text extraction method, the terminal device can obtain the intersection-and-joint ratio of each text control displayed on the touch screen and the touch area, and determine the target control based on the intersection-and-joint ratio. In this way, the control from which the user wants to extract text information can be more accurately determined, thereby improving the accuracy of the text information finally obtained in terms of text content.
[0007] According to the first aspect, or the first implementation of the first aspect, in the second implementation of the text extraction method, the adjustment of the first text information based on the second text information to obtain the third text information may include: comparing the characters corresponding to the same position on the touch screen in the first text information and the second text information; determining the characters in the first text information that correspond to the same position on the touch screen and have inconsistent contents with the characters in the second text information as target characters; replacing the target characters in the first text information with the characters in the second text information that correspond to the same position on the touch screen as the target characters, to obtain the third text information. In this way, the text information extracted by the OCR technology can be made into text information with more correct text content.
[0008] According to the second implementation of the first aspect, in the third implementation of the text extraction method, replacing the target character in the first text information with the character in the second text information corresponding to the same position on the touch screen as the target character may include: determining a matching rate according to the number of the target characters and the number of characters in the first text information; and when the matching rate is greater than a first threshold, replacing the target character in the first text information with the character in the second text information corresponding to the same position on the touch screen as the target character, to obtain the third text information. In this way, by performing character replacement when the matching rate is relatively large, the accuracy can be improved.
[0009] According to the first aspect, or the first implementation of the first aspect, in the fourth implementation of the text extraction method, the adjusting the first text information based on the second text information to obtain the third text information may include: detecting whether there is a character set that meets the preset format in the second text information; if there is a character set that meets the preset format in the second text information, extracting the character set that meets the preset format from the second text information; replacing the first text information with the extracted character set to obtain the third text information. Determining the control to which the link or password belongs through OCR technology, and then automatically extracting the link or password from the control can ensure the integrity and position accuracy of the link or password, as well as the correctness of the link or password, while being quick and convenient to operate.
[0010] According to the fourth implementation of the first aspect, in the fifth implementation of the text extraction method, the terminal device can provide a service corresponding to the character set in the preset format according to the third text information. In this way, service efficiency can be improved, which is conducive to improving user satisfaction.
[0011] According to the first aspect, or any implementation of the first aspect above, in a sixth implementation of the text extraction method, in response to the touch operation, obtaining the touch area may include: in response to the touch operation, obtaining the position information of the starting touch point and the position information of the ending touch point; and determining the touch area according to the position information of the starting touch point and the position information of the ending touch point. In this way, the position where the user needs to obtain text information can be effectively and accurately determined.
[0012] According to the first aspect, or according to any one of the second to fifth implementations of the first aspect, in a seventh implementation of the text extraction method, the text extraction method may further include: in response to the touch operation, loading a region selection mark layer; and determining the touch region based on a confirmation operation of the region selection mark layer. In this way, the selection of the touch region can be made more accurate, thereby further improving the accuracy of the text information location.
[0013] According to the sixth implementation of the first aspect, in the eighth implementation of the text extraction method, determining the touch area according to the position information of the starting touch point and the position information of the ending touch point may include: when the starting touch point and the ending touch point correspond to the same text line, determining the touch area according to a first area between the starting touch point and the ending touch point. In this way, text information can be accurately obtained within the same text line.
[0014] According to the sixth implementation of the first aspect, in the ninth implementation of the text extraction method, determining the touch area according to the position information of the starting touch point and the position information of the ending touch point may include: when the starting touch point and the ending touch point correspond to adjacent text lines, determining the touch area according to a second area between the starting touch point and the right border of the touch screen, and a third area between the ending touch point and the left border of the touch screen. In this way, text information can be accurately obtained within adjacent text lines.
[0015] According to the sixth implementation of the first aspect, in the tenth implementation of the text extraction method, the determining the touch area according to the position information of the starting touch point and the position information of the ending touch point includes: when the text lines corresponding to the starting touch point and the ending touch point are separated by one or more text lines, the determining the touch area according to the fourth area between the starting touch point and the right border of the touch screen, the fifth area between the text line corresponding to the starting touch point and the text line corresponding to the ending touch point, and the sixth area between the ending touch point and the left border of the touch screen. In this way, text information can be accurately obtained in a larger range.
[0016] According to any one of the eighth to tenth implementations of the first aspect, in the eleventh implementation of the text extraction method, determining the touch area according to the position information of the starting touch point and the position information of the ending touch point may include: moving the starting touch point to the positive direction of the y-axis and the negative direction of the x-axis of the touch screen by a first distance to obtain an adjusted starting touch point; moving the ending touch point to the positive direction of the x-axis and the negative direction of the y-axis of the touch screen by a second distance to obtain an adjusted ending touch point; determining the touch area according to the position information of the adjusted starting touch point and the position information of the adjusted ending touch point. In this way, the touch area can be slightly enlarged, the influence of missing text selection caused by the inconsistency between the user's visual touch point and the actual touch point can be reduced, and the accuracy of position marking can be improved.
[0017] In a second aspect, an embodiment of the present disclosure provides a text extraction device, comprising: a first acquisition module, used to acquire a touch area in response to a touch operation on a touch screen; an extraction module, used to extract first text information in the touch area acquired by the first acquisition module through optical character recognition (OCR) technology; a determination module, used to determine a target control matching the touch area from one or more text controls displayed on the touch screen; a second acquisition module, used to acquire second text information from the target control determined by the determination module; and an adjustment module, used to adjust the first text information extracted by the extraction module based on the second text information acquired by the second acquisition module to obtain third text information.
[0018] According to the second aspect, in a first implementation of the text extraction device, the determination module includes: a first acquisition unit, used to obtain the intersection and union ratio of each textual control displayed on the touch screen and the touch area; a first determination unit, used to determine the target control based on the intersection and union ratio.
[0019] According to the second aspect, or the first implementation manner of the second aspect, in the second implementation manner of the text extraction device, the adjustment module includes: a comparison unit, used to compare the characters in the first text information and the second text information corresponding to the same position on the touch screen; a second determination unit, used to determine the characters in the first text information that correspond to the same position on the touch screen as the characters in the second text information and have inconsistent contents as target characters; and a first replacement unit, used to replace the target characters in the first text information with characters in the second text information that correspond to the same position on the touch screen as the target characters, to obtain the third text information.
[0020] According to the second implementation method of the second aspect, in the third implementation method of the text extraction device, the first replacement unit is also used to: determine the matching rate based on the number of the target characters and the number of characters in the first text information; when the matching rate is greater than the first threshold, replace the target characters in the first text information with characters in the second text information corresponding to the same position on the touch screen as the target characters, to obtain the third text information.
[0021] According to the second aspect, or the first implementation manner of the second aspect, in a fourth implementation manner of the text extraction device, the adjustment module also includes: a detection unit, used to detect whether there is a character set that satisfies a preset format in the second text information; an extraction unit, used to extract the character set that satisfies the preset format from the second text information when there is a character set that satisfies the preset format in the second text information; and a second replacement unit, used to replace the first text information with the extracted character set to obtain the third text information.
[0022] According to a fourth implementation manner of the second aspect, in a fifth implementation manner of the text extraction device, the device further includes: a service module, configured to provide a service corresponding to the character set in the preset format according to the third text information.
[0023] According to the second aspect, or any implementation of the second aspect above, in a sixth implementation of the text extraction device, the first acquisition module includes: a second acquisition unit, used to obtain the position information of the starting touch point and the position information of the ending touch point in response to the touch operation; a third determination unit, used to determine the touch area based on the position information of the starting touch point and the position information of the ending touch point.
[0024] According to the second aspect, or according to any one of the second to fifth implementations of the second aspect, in the seventh implementation of the text extraction device, the first acquisition module also includes: a loading unit, used to load the area selection mark layer in response to the touch operation; and a fourth determination unit, used to determine the touch area based on the confirmation operation of the area selection mark layer.
[0025] According to the sixth implementation method of the second aspect, in the eighth implementation method of the text extraction device, the third determination unit is also used to: when the starting touch point and the ending touch point correspond to the same text line, determine the touch area according to the first area between the starting touch point and the ending touch point.
[0026] According to the sixth implementation method of the second aspect, in the ninth implementation method of the text extraction device, the third determination unit is also used to: when the starting touch point and the ending touch point correspond to adjacent text lines, determine the touch area according to the second area between the starting touch point and the right border of the touch screen, and the third area between the ending touch point and the left border of the touch screen.
[0027] According to the sixth implementation manner of the second aspect, in the tenth implementation manner of the text extraction device, the third determination unit is further used to: when the text lines corresponding to the starting touch point and the ending touch point are separated by one or more text lines, determine the touch area according to the fourth area between the starting touch point and the right border of the touch screen, the fifth area between the text line corresponding to the starting touch point and the text line corresponding to the ending touch point, and the sixth area between the ending touch point and the left border of the touch screen.
[0028] According to any one of the eighth to tenth implementations of the second aspect, in the tenth implementation of the text extraction device, the third determination unit is further used to: move the starting touch point a first distance toward the positive y-axis and negative x-axis of the touch screen to obtain an adjusted starting touch point; move the ending touch point a second distance toward the positive x-axis and negative y-axis of the touch screen to obtain an adjusted ending touch point; and determine the touch area based on the adjusted position information of the starting touch point and the adjusted position information of the ending touch point.
[0029] In a third aspect, an embodiment of the present disclosure provides a terminal device, which can execute the text extraction method of the first aspect or one or more of the multiple possible implementations of the first aspect.
[0030] In a fourth aspect, an embodiment of the present disclosure provides a computer program product, comprising a computer-readable code, or a non-volatile computer-readable storage medium carrying a computer-readable code. When the computer-readable code runs in an electronic device, a processor in the electronic device executes the text extraction method of the above-mentioned first aspect or one or more of the multiple possible implementations of the first aspect.
[0031] In the disclosed embodiment, the first text information with accurate text position is accurately extracted through OCR technology, the second text information with correct text content is obtained through control word acquisition technology, and the first text information with accurate text position is adjusted based on the second text information with correct text content, so that the third text information with accurate position and correct content can be obtained conveniently and quickly.
[0032] These and other aspects of the present disclosure will become more apparent from the following description of the embodiment(s). BRIEF DESCRIPTION OF THE DRAWINGS
[0033] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.
[0034] Figure 1 A schematic diagram showing an implementation environment of the text extraction method provided by an embodiment of the present disclosure;
[0035] Figure 2a , Figure 2b and Figure 2c Schematic diagrams showing exemplary application interfaces displayed on a touch screen;
[0036] Figure 3a , Figure 3b , Figure 3c , Figure 3d and Figure 3e Exemplary schematic diagrams showing touch control areas respectively;
[0037] Figure 4 A schematic diagram of the structure of a terminal device 200 according to an embodiment of the present disclosure is shown;
[0038] Figure 5 A flowchart showing a text extraction method according to an embodiment of the present disclosure;
[0039] Figure 6 A schematic diagram showing the structure of a text extraction device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0040] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.
[0041] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.
[0042] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.
[0043] Figure 1 A schematic diagram of the implementation environment of the text extraction method provided by the embodiment of the present disclosure is shown. Figure 1 , the implementation environment includes a touch medium 100 and a terminal device 200. The touch medium may include a stylus 101 and a user's finger 102, etc. The terminal device 200 may be any terminal device with a touch screen, including but not limited to a mobile phone, a tablet computer, a notebook computer, a television, a laptop computer, a desktop computer, a mobile phone, a multimedia player, an e-reader, a smart car device, a smart home appliance, an artificial intelligence device, a wearable electronic device (such as a smart watch, etc.), an Internet of Things device, a virtual reality / augmented reality / mixed reality device, etc.
[0044] The terminal device 200 may be installed with a variety of applications, such as instant messaging applications, e-commerce applications, game applications, social applications, community applications, news applications, audio playback applications, video playback applications, live broadcast applications, browser applications, travel applications, financial applications, sports applications, shooting applications, image processing applications, audio and video processing applications, reading applications, food delivery applications, recipe applications, navigation applications, transportation ticketing applications, information recording applications, mailbox applications, health care applications, resource management applications, etc. The applications installed in the terminal device 200 may be independent applications or embedded applications, i.e., mini-programs.
[0045] In some possible implementations, the touch screen of the terminal device 200 can display the interface of the application, and the interface of the application may include one or more text controls. Among them, the text control can be used to represent a control that can obtain text content, for example, a text display control, a text input control, etc. For example, the text control can be a chat box in an instant messaging application, an input box in an information recording application, an area for displaying an e-book in a reading application, an area for displaying the text content of a recipe application, an area for displaying the text area in a news application, an area for displaying the text content in a browser application, etc. The text information extracted from the text control includes but is not limited to user names, passwords, links, words, sentences, paragraphs, articles, and other contents. The text information extracted from the text control includes but is not limited to symbols, numbers, Chinese, English, Japanese, Korean, Spanish, German, French, and other forms. The disclosed embodiment does not limit the application displayed in the terminal device 200, the text control included in the application, and the text information extracted from the text control.
[0046] Figure 2a , Figure 2b and Figure 2c 2 and 3 respectively show exemplary schematic diagrams of application interfaces displayed on the touch screen. In one example, the touch screen of the terminal device 200 may display Figure 2a The interface of the instant messaging application shown in Figure 1. Figure 2a , the textual controls in the interface of the instant messaging application may include a chat information display box and a chat information input box. For example, the text content obtained from the chat information display box may include "I bought a ticket for the play on the 10th, let's go and watch it together", "Great, great", "Who are the actors", "The link above has a detailed introduction, take a look", etc.; the text information obtained from the chat information input box may include "OK, I'll take a look", etc. In another example, the touch screen of the terminal device 200 may display Figure 2b The interface of the community application shown in Figure 1. Figure 2b , the textual controls in the interface of the community application may include a text display area and a search box in the display box. For example, the text information obtained from the text display area may include "Help recommend a three-day tour guide for Hangzhou", "The first day... ends", "Attraction introductions can refer to...", "Attraction ticket purchases can click...", etc. The text information obtained from the search box may include "Hangzhou Travel Notes" (not shown), etc. In another example, the touch screen of the terminal device 200 may display Figure 2c The browser application interface shown in . Figure 2c The text control in the interface of the browser application may include a text display area. For example, the text information obtained from the text display area may include "XL... adjustable" and "chip introduction".
[0047] In a possible implementation, the user can perform a touch operation (e.g., click, double-click, slide, double-finger press, etc.) on the touch screen of the terminal device 200 through a touch medium such as a stylus 101 and a user's finger 102. The terminal device 200 may include obtaining a touch area in response to the above touch operation.
[0048] Figure 3a , Figure 3b , Figure 3c , Figure 3d and Figure 3e Exemplary schematic diagrams of touch control areas are shown respectively.
[0049] In a possible implementation, the touch operation performed by the user may be used to determine a starting touch point and an ending touch point, and the terminal device may determine a touch area based on the starting touch point and the ending touch point.
[0050] See also Figure 3a ,based on Figure 2c In the application interface shown, the user can perform a two-finger press operation on the touch screen of the terminal device 200. In response to the two-finger press operation, the terminal device 200 can obtain the position information of two touch points (recorded as the starting touch point and the ending touch point). The terminal device 200 divides the interface into multiple text lines through OCR technology. The terminal device 200 can determine the text line corresponding to the starting touch point (recorded as the first text line) and the text line corresponding to the ending touch point (recorded as the second text line) based on the position information of the starting touch point and the position information of the ending touch point. Figure 3a As shown, the first text line and the second text line are the same text line, and the terminal device 200 can determine the touch area according to the area between the starting touch point and the ending touch point (recorded as the first area).
[0051] See also Figure 3b ,based on Figure 2c In the application interface shown, the user can perform a sliding operation on the touch screen of the terminal device 200 by using a finger. The terminal device 200 can determine the starting touch point and the ending touch point according to the starting point and the ending point of the sliding operation, and determine the first text line and the second text line. Figure 3b As shown, the first text line and the second text line are the same text line, and the terminal device 200 can determine the touch area according to the first area between the starting touch point and the ending touch point.
[0052] See also Figure 3c ,based on Figure 2cIn the application interface shown, the user can perform two clicks (single click or double click, etc.) on the touch screen of the terminal device 200 with a finger. The terminal device 200 can determine the starting touch point and the ending touch point based on the two click points detected, and determine the first text line and the second text line. Figure 3c As shown, the first text line and the second text line are adjacent text lines, and the terminal device 200 can determine the touch area based on the area between the starting touch point and the right border of the touch screen (recorded as the second area) and the area between the ending touch point and the left border of the touch screen (recorded as the third area).
[0053] See also Figure 3d ,based on Figure 2c In the application interface shown, the user can perform a sliding operation on the touch screen of the terminal device 200 by using a finger. The terminal device 200 can determine the starting touch point and the ending touch point according to the starting point and the ending point of the sliding operation, and determine the first text line and the second text line. Figure 3d As shown, the first text line and the second text line are separated by two text lines. The terminal device 200 can determine the touch area based on the area between the starting touch point and the right border of the touch screen (recorded as the fourth area, which can refer to the second area), the area between the first text line and the second text line (recorded as the fifth area), and the area between the ending touch point and the left border of the touch screen (recorded as the sixth area, which can refer to the third area).
[0054] In a possible implementation, the first area, the second area, the third area, the fourth area, the fifth area and the sixth area may be rectangular areas. Of course, the first area to the sixth area may also be areas of other shapes, such as elliptical areas, trapezoidal areas, hexagonal areas and octagonal areas, etc., which are not limited in the present disclosure.
[0055] It should be noted that in the embodiments of the present disclosure, the start touch point and the end touch point are determined based on their relative positions on the touch screen, which is consistent with the order of the text content. For example, when the first text line and the second text line are the same text line, the start touch point is located to the left of the end touch point; when the first text line and the second text line are different text lines, the start touch point is located above the end touch point.
[0056] In a possible implementation, the touch operation can also be used to trigger area selection. In response to the touch operation, the terminal device can load an area selection mark layer, and determine the touch area based on a confirmation operation of the area selection mark layer.
[0057] See also Figure 3e ,based on Figure 2cIn the application interface shown, the terminal device 200 displays the area selection mark layer in response to touch operations such as long pressing the screen and triggering the mark control. The user can adjust the position of the area selection mark layer. After the adjustment is completed, the user can click the "√" control to perform a confirmation operation to confirm the area selection mark layer. In response to the confirmation operation of the area selection mark layer, the terminal device 300 can determine the touch area according to the position of the area selection mark layer. In this way, the selection of the touch area can be made more accurate, thereby further improving the accuracy of the text information position.
[0058] Reference Figure 2a and Figure 2b It can be seen that the touch screen of the terminal device 200 can display one text control or multiple text controls at the same time. When the terminal device 200 detects a touch operation on the touch screen, it can determine all text controls currently displayed on the touch screen. Afterwards, the terminal device 200 can determine the target control that matches the touch area from these text controls. In one example, the terminal device 200 can obtain the intersection and union ratio of each text control displayed on the touch screen and the touch area, and determine the target control based on the intersection and union ratio. For example, the text control with the largest corresponding intersection and union ratio is determined as the target control. It can be understood that the position of the text control displayed on the touch screen of the terminal device 200 can be obtained from the application currently displayed on the touch screen.
[0059] After the terminal device 200 obtains the touch area, the first text information in the touch area can be extracted by OCR technology. After the terminal device 200 determines the target control, the second text information can be directly obtained from the target control. Since the first text information has more accurate position information and the second text information has more accurate text content, the terminal device 200 can adjust the first text information based on the second text information and update the less accurate part of the first text information, thereby obtaining the third text information with accurate position and content.
[0060] For example, based on Figure 3a , the terminal device 200 extracts the first text information "integrated 100mΩ MOSFET switch power", and obtains the second text information "XL 1509-3.3E1 5G base station power chip features 2A continuous output current 8-30V wide operating voltage input integrated 100mΩ MOSFET switch power tube output 18-28V adjustable". The terminal device 200 uses the second text information to adjust the first text information and obtains the third text information "integrated 100mΩ MOSFET switch power".
[0061] Figure 4 A schematic structural diagram of a terminal device 200 according to an embodiment of the present disclosure is shown.
[0062] The terminal device 200 may include a processor 210, an external memory interface 220, an internal memory 221, a USB interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 251, a wireless communication module 252, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, an earphone interface 270D, a sensor module 280, a button 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a SIM card interface 295, etc. The sensor module 280 may include a touch sensor 280K (of course, the terminal device 200 may also include other sensors, such as a gyroscope sensor, an acceleration sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a pressure sensor, a distance sensor, a magnetic sensor, an ambient light sensor, an air pressure sensor, a bone conduction sensor, etc., which are not shown in the figure).
[0063] The processor 210 may include one or more processing units, for example: the processor 210 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a memory, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and / or a neural network processor (Neural-network Processing Unit, NPU), etc. Among them, different processing units can be independent devices or integrated in one or more processors. Among them, the controller can be the nerve center and command center of the terminal device 200. The controller can generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions.
[0064] The processor 210 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. The memory may store instructions or data that the processor 210 has just used or cyclically used. If the processor 210 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.
[0065] The processor 210 can run the text extraction method provided by the embodiment of the present disclosure to obtain the text information required by the user conveniently, quickly and accurately. The processor 210 can include different devices. For example, when a CPU and a GPU are integrated, the CPU and the GPU can cooperate to execute the text extraction method provided by the embodiment of the present disclosure. For example, part of the algorithm in the text extraction method is executed by the CPU, and another part of the algorithm is executed by the GPU to obtain faster processing efficiency.
[0066] The display screen 294 is used to display images, videos, etc. The display screen 294 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the terminal device 200 may include 1 or N display screens 294, where N is a positive integer greater than 1. The display screen 294 can be used to display information input by a user or information provided to a user and various graphical user interfaces (GUI). For example, the display 294 can display photos, videos, web pages, or files, etc. For another example, the display 294 can display a graphical user interface. Among them, the graphical user interface includes a status bar, a hidden navigation bar, a time and weather widget, and application icons, such as a browser icon. The status bar includes the operator name (such as China Mobile), the mobile network (such as 4G), the time, and the remaining power. The navigation bar includes a back key icon, a home key icon, and a forward key icon. In addition, it can be understood that in some embodiments, the status bar may also include a Bluetooth icon, a Wi-Fi icon, an external device icon, etc. It can also be understood that in other embodiments, the graphical user interface may also include a Dock bar, and the Dock bar may include commonly used application icons, etc. When the processor 210 detects a touch event of a user's finger (or a stylus, etc.) for a certain application icon, in response to the touch event, the user interface of the application corresponding to the application icon is opened, and the user interface of the application is displayed on the display 294.
[0067] In the embodiment of the present disclosure, the display screen 294 may be an integrated flexible display screen, or a spliced display screen consisting of two rigid screens and a flexible screen located between the two rigid screens, etc., which is not limited in the embodiment of the present disclosure.
[0068] After the processor 210 runs the text extraction method provided by the embodiment of the present disclosure, the terminal device 200 can control the display screen 294 to display a corresponding graphical user interface, such as Figure 2a , Figure 2b and Figure 2c The application interface shown, Figure 3a , Figure 3b , Figure 3c and Figure 3d The touch area shown, and Figure 3e Select the marked layer in the area shown.
[0069] The camera 293 (a front camera or a rear camera, or one camera can be used as both a front camera and a rear camera) is used to capture still images or videos. Generally, the camera 293 may include a photosensitive element such as a lens group and an image sensor, wherein the lens group includes a plurality of lenses (convex lenses or concave lenses) for collecting light signals reflected by the object to be photographed and transmitting the collected light signals to the image sensor. The image sensor generates an original image of the object to be photographed based on the light signal.
[0070] The internal memory 221 can be used to store computer executable program codes, which include instructions. The processor 210 executes various functional applications and data processing of the terminal device 200 by running the instructions stored in the internal memory 221. The internal memory 221 may include a program storage area and a data storage area. Among them, the program storage area may store codes of operating systems, applications (such as camera applications, WeChat applications, etc.), etc. The data storage area may store data created during the use of the terminal device 200 (such as images, videos, etc. collected by camera applications), etc.
[0071] The internal memory 221 may also store one or more computer programs 1310 corresponding to the text extraction method provided in the embodiment of the present disclosure. The one or more computer programs 1304 are stored in the above-mentioned memory 221 and are configured to be executed by the one or more processors 210. The one or more computer programs 1310 include instructions, and the above-mentioned instructions can be used to execute the following steps: Figure 5In the respective steps of the corresponding embodiment, the computer program 1310 may include a first acquisition module, an extraction module, a determination module, a second acquisition module and an adjustment module, wherein the first acquisition module is used to obtain a touch area in response to a touch operation on the touch screen; the extraction module is used to extract the first text information in the touch area obtained by the first acquisition module through optical character recognition (OCR) technology; the determination module is used to determine a target control matching the touch area obtained by the first acquisition module from the textual controls displayed on the touch screen, wherein the textual control is used to represent a control that can obtain text content; the second acquisition module is used to obtain the second text information from the target control determined by the determination module; the adjustment module is used to adjust the first text information extracted by the extraction module based on the second text information obtained by the second acquisition module to obtain the third text information. When the code of the text extraction method stored in the internal memory 221 is executed by the processor 210, the processor 210 can control the display screen to display the third text information.
[0072] In addition, the internal memory 221 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.
[0073] Of course, the code of the text extraction method provided by the embodiment of the present disclosure may also be stored in an external memory. In this case, the processor 210 may run the code of the text extraction method stored in the external memory through the external memory interface 220.
[0074] The function of the touch sensor 280K in the sensor module 280 will be described below.
[0075] The touch sensor 280K is also called a "touch panel". The touch sensor 280K can be set on the display screen 294. The touch sensor 280K and the display screen 294 form a touch screen, also called a "touch screen". The touch sensor 280K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 294. In the embodiment of the present disclosure, the user can perform operations on the touch screen. Figure 3a , Figure 3b , Figure 3c and Figure 3d The touch operations shown are used by the processor to obtain the touch area based on these touch operations.
[0076] Exemplarily, the display screen 294 of the terminal device 200 displays a main interface, which includes icons of multiple applications (such as instant messaging applications, browser applications, etc.). The user clicks the icon of the instant messaging application in the main interface through the touch sensor 280K, triggering the processor 210 to start the instant messaging application. The display screen 294 displays the interface of the instant messaging application, such as a login interface or a chat interface.
[0077] The wireless communication function of the terminal device 200 can be implemented through antenna 1, antenna 2, mobile communication module 251, wireless communication module 252, modem processor and baseband processor.
[0078] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal device 200 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.
[0079] The mobile communication module 251 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the terminal device 200. The mobile communication module 251 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 251 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 251 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 251 can be set in the processor 210. In some embodiments, at least some of the functional modules of the mobile communication module 251 can be set in the same device as at least some of the modules of the processor 210. In the embodiment of the present disclosure, the mobile communication module 251 can also be used to interact with other terminal devices for information.
[0080] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 270A, a receiver 270B, etc.), or displays an image or video through a display screen 294. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 210 and be set in the same device as the mobile communication module 251 or other functional modules.
[0081] The wireless communication module 252 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the terminal device 200. The wireless communication module 252 can be one or more devices integrating at least one communication processing module. The wireless communication module 252 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and filters it, and sends the processed signal to the processor 210. The wireless communication module 252 can also receive the signal to be sent from the processor 210, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2. In the embodiment of the present disclosure, the wireless communication module 252 is used to transmit data between other terminal devices under the control of the processor 210. For example, when the processor 210 runs the text extraction method provided in the embodiment of the present disclosure, the processor can control the wireless communication module 252 to send service requests to other terminal devices, and can also receive service results provided by other terminal devices based on the above service requests. For example, a web page access request is sent to other terminal devices, and web page content provided by other terminal devices is received.
[0082] In addition, the terminal device 200 can implement audio functions such as music playing and recording through the audio module 270, the speaker 270A, the receiver 270B, the microphone 270C, the earphone interface 270D, and the application processor.
[0083] It should be understood that in actual applications, the terminal device 200 may include Figure 4 The embodiments of the present disclosure are not limited to the more or fewer components shown. The illustrated terminal device 200 is only an example, and the terminal device 200 may have more or fewer components than those shown in the figure, may combine two or more components, or may have different component configurations. The various components shown in the figure may be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and / or application specific integrated circuits.
[0084] Combine the following Figure 3a The application interface shown exemplifies the workflow of the software and hardware of the terminal device 200.
[0085] The terminal device 200 displays on the display screen 294 as follows Figure 3a In the case of the application interface shown, when the user touches both sides of the information "MOSFET switching power integrated with 100mΩ", the touch sensor 280K receives the touch operation, and the corresponding hardware terminal is sent to the kernel layer. The kernel layer processes the touch operation into raw input events (including touch coordinates, timestamp of the touch operation and other information). The raw input events are stored in the kernel layer. The application framework layer obtains the raw input events from the kernel layer and identifies the application corresponding to the input event. Taking the touch operation as a two-finger press operation, and the application corresponding to the two-finger press operation as an example, the text extraction application calls the interface of the application framework layer to start the text extraction application. The text extraction application responds to the above-mentioned two-finger press operation and obtains Figure 3a The touch area shown; extracting the first text information "integrated 100mQ MOSFET switch power" in the touch area by OCR technology; determining the text control displayed on the display screen 294 with Figure 3a The touch area shown matches the target control; the second text information "XL1509-3.3E1 5G base station power chip features 2A continuous output current 8-30V wide operating voltage input integrated 100mΩ MOSFET switch power tube output 18-28V adjustable" is obtained from the target control; the first text information is adjusted based on the second text information to obtain the third text information "integrated 100mΩ MOSFET switch power".
[0086] Figure 5 A flowchart of a text extraction method according to an embodiment of the present disclosure is shown. The method may be executed by a terminal device, for example Figure 4 The terminal device 200 shown in FIG. Figure 5 As shown, the method may include:
[0087] Step S601: acquiring a touch area in response to a touch operation on the touch screen.
[0088] Step S602: extracting first text information in the touch area by using optical character recognition (OCR) technology.
[0089] Step S603: determining a target control matching the touch area from one or more text controls displayed on the touch screen.
[0090] The text control is used to represent a control that can obtain text content.
[0091] Step S604: acquiring second text information from the target control.
[0092] Step S605: adjust the first text information based on the second text information to obtain third text information.
[0093] In the disclosed embodiment, the first text information with accurate text position is accurately extracted through OCR technology, the second text information with correct text content is obtained through control word acquisition technology, and the first text information with accurate text position is adjusted based on the second text information with correct text content, so that the third text information with accurate position and correct content can be obtained conveniently and quickly.
[0094] In step S601, the touch operation may include a click operation, a slide operation, and a press operation. Among them, the click operation may include two click operations, two double-click operations, one click operation and one double-click operation, or one double-click operation and one click operation, etc. The slide operation may include a single-finger slide operation, a multi-finger slide operation, etc. The press operation may include a single-finger press operation, a multi-finger press operation (such as a two-finger press operation), etc. The touch operation may be performed by a stylus, a user's finger, or a user's knuckles, etc. The terminal device may acquire a touch area in response to the touch operation on the touch screen. The touch area may accurately mark the location of the text information that the user needs to obtain. In one example, considering that the text has the characteristic of line breaks, the touch area may include one or more rectangular areas. The touch area may also be other areas that can accurately mark the location.
[0095] In a possible implementation, step S601 may include: in response to the touch operation, acquiring position information of a starting touch point and position information of an ending touch point; and determining the touch area according to the position information of the starting touch point and the position information of the ending touch point.
[0096] A touch operation may generate two or more touch points. In the embodiment of the present disclosure, it is necessary to determine the starting touch point and the ending touch point from these two or more touch points. Among them, the starting touch point can be used to mark the starting point of the text information that the user wants to obtain, and the ending touch point can be used to mark the ending point of the text information that the user wants to obtain. The starting touch point and the ending touch point are determined according to the order of the text content in the text information, rather than the order in which the touch points are generated. For a sliding operation from the lower left to the upper right, the starting touch point is the last touch point generated by the sliding operation, and the ending touch point is the first touch point generated by the sliding operation. For example, the terminal device determines the distance between the text line corresponding to each touch point generated by the touch operation and the upper boundary of the touch screen, and determines the touch point with the smallest distance as the first touch point. Similarly, the terminal device determines the distance between the text line corresponding to each touch point generated by the touch operation and the lower boundary of the touch screen, and determines the touch point with the smallest distance as the second touch point. In the case where there is one first touch point, the terminal device may determine the first touch point as the starting touch point; in the case where there are multiple first touch points, the terminal device may determine the first touch point that is closest to the left border of the touch screen as the starting touch point. In the case where there is one second touch point, the terminal device may determine the second touch point as the ending touch point; in the case where there are multiple second touch points, the terminal device may determine the second touch point that is closest to the right border of the touch screen as the ending touch point.
[0097] The position information of the starting touch point and the position information of the ending touch point are determined based on the touch screen. In one example, the lower left corner of the touch screen can be determined as the coordinate origin, the right side of the coordinate origin can be used as the positive direction of the x-axis, the left side of the coordinate origin can be used as the negative direction of the y-axis, the top of the coordinate origin can be used as the positive direction of the y-axis, and the bottom of the coordinate origin can be used as the negative direction of the y-axis. In this way, the position information of the starting touch point and the position information of the ending touch point can be represented by x and y.
[0098] In one example, when the starting touch point and the ending touch point correspond to the same text line, the terminal device can determine the touch area according to the first area between the starting touch point and the ending touch point. Taking the first area as a rectangular area as an example, the terminal device can determine the left boundary of the first area according to the starting touch point, determine the right boundary of the first area according to the ending touch point, determine the upper and lower boundaries of the rectangular area according to the lines divided by the OCR technology, and determine the rectangular area between the upper, lower, left and right boundaries as the touch area.
[0099] In one example, when the starting touch point and the ending touch point correspond to adjacent text lines, the terminal device can determine the touch area based on a second area between the starting touch point and the right border of the touch screen, and a third area between the ending touch point and the left border of the touch screen.
[0100] The method for determining the second area and the third area can refer to the method for determining the first area, which will not be repeated here. It can be understood that the text information in the second area should be arranged before the text information in the third area. In one example, the third area can be spliced after the second area to obtain the touch area.
[0101] In one example, when the text lines corresponding to the starting touch point and the ending touch point are separated by one or more text lines, the terminal device can determine the touch area based on the fourth area between the starting touch point and the right border of the touch screen, the fifth area between the text line corresponding to the starting touch point and the text line corresponding to the ending touch point, and the sixth area between the ending touch point and the left border of the touch screen.
[0102] Among them, the method for determining the fourth area, the fifth area and the sixth area can refer to the method for determining the first area, which will not be repeated here. In one example, the fifth area can be divided into one or more sub-areas, each sub-area corresponding to a text line. Then, the sub-areas of the fifth area are spliced after the fourth area in order, and the sixth area is spliced after the sub-area corresponding to the last text line of the fifth area to obtain the touch area.
[0103] In the embodiment of the present disclosure, the location of the text information that the user needs to obtain can be accurately marked through the above method.
[0104] In a possible implementation, before obtaining the position information of the starting touch point and the position information of the ending touch point, the terminal device may first adjust the positions of the starting touch point and the ending touch point. In one example, the terminal device may move the starting touch point to the positive direction of the y-axis and the negative direction of the x-axis of the touch screen by a first distance to obtain the adjusted starting touch point; and move the ending touch point to the positive direction of the x-axis and the negative direction of the y-axis of the touch screen by a second distance to obtain the adjusted ending touch point. Afterwards, the terminal device may determine the touch area according to the adjusted position information of the starting touch point and the adjusted position information of the ending touch point.
[0105] The first distance may include a first x-axis distance and a first y-axis distance. The first x-axis distance and the first y-axis distance may be the same or different. The first x-axis distance and the first y-axis distance may be set as needed, for example, according to one or more of the size of the touch screen, the height of the text line, and the size of the text, which is not limited in the present disclosure. For example, the first x-axis distance may be 0.5 cm, and the first y-axis distance may be 0.5 cm. The second distance may refer to the first distance, which will not be described in detail here.
[0106] In the embodiment of the present disclosure, by adjusting the position of the starting touch point toward the positive direction of the y-axis and the negative direction of the x-axis, and adjusting the position of the ending touch point toward the negative direction of the y-axis and the positive direction of the x-axis, the touch area can be slightly expanded, thereby reducing the impact of text omissions caused by the inconsistency between the user's visual touch point and the actual touch point, and improving the accuracy of position marking.
[0107] In a possible implementation, after determining the text lines corresponding to the start touch point and the end touch point, the terminal device can take an additional text line area upward and downward respectively and add it to the touch area. In this way, the touch area can be effectively expanded to better reduce the impact of text omission caused by the inconsistency between the user's visual touch point and the actual touch point, and improve the accuracy of position marking. In particular, the touch operation performed by fingers or knuckles has a more obvious effect.
[0108] In step S602, the first text information may represent text information in the touch area extracted by OCR technology. In one example, the terminal device may perform binarization, noise removal, tilt correction, line division, character segmentation, character recognition, and board restoration on the touch area (even if the recognized text content is still arranged as the text content displayed in the original touch area, the paragraph, position, and order of the text remain unchanged).
[0109] In step S603, the textual control can be used to represent a space where textual content can be obtained. In one example, the textual control can include a text display control and a text input control. For example, the textual control can be a short message display box, an instant messaging message display box, a memo, a notepad, etc. One or more textual controls can be displayed simultaneously on the touch screen. It should be noted that the textual controls displayed on the touch screen include textual controls that are not fully displayed on the touch screen.
[0110] In one possible implementation, the terminal device can obtain the intersection and union ratios of each textual control displayed on the touch screen and the touch area; based on the intersection and union ratios, the target control is determined. In one example, the terminal device can determine the textual control with the largest intersection and union ratio with the touch area as the target control. In another example, the terminal device can determine the textual control with the largest intersection and union ratio with the touch area, and the intersection and union ratio is greater than a specified threshold, as the target control. Among them, the specified threshold can be set as needed, for example, it can be set to 85%, 90%, etc., which is not limited by the present disclosure.
[0111] In step S604, since the target control is a control that can obtain text content, the terminal device can directly obtain the second text information from the target control. The second text information can be stored in the attribute information of the target control.
[0112] In step S605, the terminal device can adjust the first text information based on the second text information so that the incorrect text content in the first text information becomes the correct text content. Since the text position of the first text information itself is accurate, the text position in the third text information obtained after the first text information is adjusted is accurate and the text content is correct.
[0113] In a possible implementation, step S605 may include: comparing the characters in the first text information and the second text information that correspond to the same position on the touch screen; determining the characters in the first text information that correspond to the same position on the touch screen as the characters in the second text information and have inconsistent contents as target characters; replacing the target characters in the first text information with characters in the second text information that correspond to the same position on the touch screen as the target characters, to obtain the third text information.
[0114] In the disclosed embodiment, the first text information and the second text information are first aligned to find the characters in the first text information and the second text information that correspond to the same position on the touch screen. In one example, the first character of the first text information can be aligned with the first character of the second text information, and then the subsequent characters can be compared in sequence to determine the matching rate (for example, the ratio of the number of identical characters to the number of different characters); the first character of the first text information can be aligned with the second character of the second text information, and the matching rate can be determined again. And so on, until the first character of the first text information is aligned with the last character of the second text information, and the last matching rate is determined. Find the alignment position with the largest matching rate as the final alignment position. In another example, after determining a matching rate greater than a certain threshold (which can be set as needed, for example, 95%, 90%, etc.), the alignment position corresponding to the matching rate is used as the final alignment position, and no subsequent operations are performed. Then, the characters corresponding to the same position on the touch screen in the first text information and the second text information are compared. Based on the comparison results, replacement processing is performed for inconsistent situations. For example, based on Figure 3a , the terminal device 200 extracts the first text information "Integrated 100mQ MOSFET switch power", obtains the second text information "XL 1509-3.3E1 5G base station power chip features 2A continuous output current 8-30V wide operating voltage input integrated 100mΩ MOSFET switch power tube output 18-28V adjustable", and after alignment processing, the terminal device can determine that each character of "Integrated 100mQ MOSFET switch power" in the first text information corresponds to the same position with each character in the second text information "Integrated 100mΩ MOSFET switch power". The terminal device can compare the characters at the same position in the first text information and the second text information. After comparison, the terminal device finds that the character "Q" in the first text information corresponding to the same position is different from the character "Ω" in the second text information. At this time, the terminal device can determine the character "Q" as the target character, thereby replacing the target character "Q" in the first text information "Integrated 100mQ MOSFET switching power" with the character "Ω" at the same position in the second text information, and obtaining the final third text information "Integrated 100mΩ MOSFET switching power".
[0115] Compared with OCR technology, the text extraction method provided by the embodiment of the present disclosure can improve the correctness of the extracted text content. Compared with the control word extraction technology, the text extraction method provided by the embodiment of the present disclosure can improve the accuracy of the extracted text position, eliminating the process of users looking for the required text in the extraction results. In other words, the text extraction method provided by the embodiment of the present disclosure can conveniently, quickly and accurately obtain the text information required by the user.
[0116] In one example, before the terminal device replaces the target characters in the first text information with the characters in the second text information corresponding to the same position on the touch screen as the target characters to obtain the third text information, the terminal device may first determine the matching rate according to the number of the target characters and the number of characters in the first text information. Then, if the matching rate is greater than a first threshold, the terminal device replaces the target characters in the first text information with the characters in the second text information corresponding to the same position on the touch screen as the target characters to obtain the third text information.
[0117] Among them, the matching rate can be the ratio of the number of characters other than the target character in the first text information to the number of characters in the first text information. The first threshold value can be set as needed, for example, the first threshold value can be 92%, 95%, etc., and the first threshold value is not limited in the embodiment of the present disclosure. In the case where the matching rate is greater than the first threshold value, it indicates that there is a small amount of text content with extraction errors in the first text information, and the first text information can be adjusted to improve the correctness. Therefore, in the case where the matching rate is greater than the first threshold value, the terminal device replaces the target character in the first text information with the character in the second text information corresponding to the same position on the touch screen as the target character, and obtains the third text information. In the case where the matching rate is less than or equal to the first threshold value, it indicates that there may be a large amount of text content with extraction errors in the first text information, which may be caused by the misalignment or inaccurate alignment of the first text information and the second text information, or it may be caused by the extraction error of one or both of the first text information and the second text information. At this time, the terminal device can re-align the first text information and the second text information or re-acquire the first text information and the second text information.
[0118] In this way, by performing character replacement when the matching rate is high, the accuracy can be improved.
[0119] In a possible implementation, step S605 may include: detecting whether there is a character set that satisfies a preset format in the second text information; if there is a character set that satisfies the preset format in the second text information, extracting the character set that satisfies the preset format from the second text information; and replacing the first text information with the extracted character set to obtain the third text information.
[0120] The character set of the preset format may include a password or a link, etc., and the embodiments of the present disclosure do not limit the preset format. In one example, the terminal device may detect and extract the character set of the preset format for the second text information through regular expressions or natural language processing (Natural Language Processing, NLP) technology.
[0121] When obtaining a link or password through OCR technology, it is easy to make mistakes, and OCR technology will break the link or password when it wraps, making it difficult to accurately extract the link or password through OCR technology. When obtaining a link or password through control word extraction technology, the user needs to find a specific location from the obtained text, which is cumbersome. The text extraction method provided in the embodiment of the present disclosure can determine the control to which the link or password belongs through OCR technology, and then automatically extract the link or password from the control, which can ensure the integrity and position accuracy of the link or password, and can also ensure the correctness of the link or password, while the operation is quick and convenient.
[0122] In a possible implementation, after the first text information is replaced with the extracted character set to obtain the third text information, the terminal device can also provide a service corresponding to the character set in the preset format according to the third text information. For example, the terminal device can jump to the webpage corresponding to the link, open the application corresponding to the password and jump to the corresponding details page, or copy the password and automatically jump to the corresponding details page when the corresponding application is opened.
[0123] In this way, service efficiency can be improved, which is conducive to improving user satisfaction.
[0124] Figure 6 FIG. 2 is a schematic diagram showing the structure of a text extraction device according to an embodiment of the present disclosure. Figure 6 As shown, the device 80 may include:
[0125] A first acquisition module 81, configured to acquire a touch area in response to a touch operation on the touch screen;
[0126] An extraction module 82, configured to extract the first text information in the touch area acquired by the first acquisition module 81 by using an optical character recognition (OCR) technology;
[0127] A determination module 83, configured to determine a target control matching the touch area from one or more text controls displayed on the touch screen;
[0128] A second acquisition module 84, used to acquire second text information from the target control determined by the determination module 83;
[0129] The adjustment module 85 is configured to adjust the first text information extracted by the extraction module 82 based on the second text information acquired by the second acquisition module 84 to obtain third text information.
[0130] In the disclosed embodiment, the first text information with accurate text position is accurately extracted through OCR technology, the second text information with correct text content is obtained through control word acquisition technology, and the first text information with accurate text position is adjusted based on the second text information with correct text content, so that the third text information with accurate position and correct content can be obtained conveniently and quickly.
[0131] In a possible implementation, the determination module includes: a first acquisition unit, used to obtain the intersection-and-joint ratio of each textual control displayed on the touch screen and the touch area; and a first determination unit, used to determine the target control based on the intersection-and-joint ratio.
[0132] In a possible implementation, the adjustment module includes: a comparison unit, used to compare the characters in the first text information and the second text information that correspond to the same position on the touch screen; a second determination unit, used to determine the characters in the first text information that correspond to the same position on the touch screen as the characters in the second text information and have inconsistent contents as target characters; and a first replacement unit, used to replace the target characters in the first text information with characters in the second text information that correspond to the same position on the touch screen as the target characters, to obtain the third text information.
[0133] In a possible implementation, the first replacement unit is further used to: determine a matching rate based on the number of target characters and the number of characters in the first text information; and when the matching rate is greater than a first threshold, replace the target characters in the first text information with characters in the second text information corresponding to the same position on the touch screen as the target characters, to obtain the third text information.
[0134] In a possible implementation, the adjustment module also includes: a detection unit, used to detect whether there is a character set that satisfies a preset format in the second text information; an extraction unit, used to extract the character set that satisfies the preset format from the second text information when there is a character set that satisfies the preset format in the second text information; and a second replacement unit, used to replace the first text information with the extracted character set to obtain the third text information.
[0135] In a possible implementation, the device further includes: a service module, configured to provide a service corresponding to the character set in the preset format according to the third text information. In a possible implementation, the first acquisition module includes: a second acquisition unit, configured to acquire the position information of the starting touch point and the position information of the ending touch point in response to the touch operation; and a third determination unit, configured to determine the touch area according to the position information of the starting touch point and the position information of the ending touch point.
[0136] In a possible implementation, the first acquisition module further includes: a loading unit, configured to load an area selection mark layer in response to the touch operation; and a fourth determination unit, configured to determine the touch area based on a confirmation operation of the area selection mark layer.
[0137] In a possible implementation, the third determination unit is further used to determine the touch area according to a first area between the starting touch point and the ending touch point when the starting touch point and the ending touch point correspond to the same text line.
[0138] In a possible implementation, the third determination unit is also used to determine the touch area based on a second area between the starting touch point and the right border of the touch screen, and a third area between the ending touch point and the left border of the touch screen, when the starting touch point and the ending touch point correspond to adjacent text lines.
[0139] In a possible implementation, the third determination unit is also used to: when the text lines corresponding to the starting touch point and the ending touch point are separated by one or more text lines, determine the touch area based on a fourth area between the starting touch point and the right border of the touch screen, a fifth area between the text line corresponding to the starting touch point and the text line corresponding to the ending touch point, and a sixth area between the ending touch point and the left border of the touch screen.
[0140] In a possible implementation, the third determination unit is also used to: move the starting touch point a first distance toward the positive direction of the y-axis and the negative direction of the x-axis of the touch screen to obtain an adjusted starting touch point; move the ending touch point a second distance toward the positive direction of the x-axis and the negative direction of the y-axis of the touch screen to obtain an adjusted ending touch point; and determine the touch area based on the adjusted position information of the starting touch point and the adjusted position information of the ending touch point.
[0141] An embodiment of the present disclosure provides a text extraction device, comprising: a processor and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions.
[0142] An embodiment of the present disclosure provides a non-volatile computer-readable storage medium having computer program instructions stored thereon, and the computer program instructions implement the above method when executed by a processor.
[0143] An embodiment of the present disclosure provides a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.
[0144] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium includes, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (Random Access Memory, RAM), a read-only memory (Read Only Memory, ROM), an erasable programmable read-only memory (Electrically Programmable Read-Only-Memory, EPROM or flash memory), a static random access memory (Static Random-Access Memory, SRAM), a portable compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), a digital versatile disk (Digital Video Disc, DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a protruding structure in a groove on which instructions are stored, and any suitable combination thereof.
[0145] The computer-readable program instructions or codes described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.
[0146] The computer program instructions for performing the disclosed operation may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, by utilizing the state information of computer-readable program instructions to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA), the electronic circuit can execute the computer-readable program instructions to implement various aspects of the present disclosure.
[0147] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.
[0148] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0149] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0150] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the device, system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square frame in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the functions marked in the square frame can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square frames can actually be executed substantially in parallel, and they can also be executed in the opposite order sometimes, depending on the functions involved.
[0151] It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented by hardware (such as a circuit or ASIC (Application Specific Integrated Circuit)) that performs the corresponding function or action, or can be implemented by a combination of hardware and software, such as firmware.
[0152] Although the present invention is described herein in conjunction with various embodiments, in the process of implementing the claimed invention, those skilled in the art may understand and implement other variations of the disclosed embodiments by viewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "one" or "an" does not exclude multiple situations. A single processor or other unit may implement several functions listed in the claims. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.
[0153] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.
Claims
1. A text extraction method, characterized in that: The method comprises: In response to a touch operation on the touch screen, acquiring a touch area; Extracting first text information within the touch area by using optical character recognition (OCR) technology; Determining a target control matching the touch area from one or more textual controls displayed on the touch screen; Acquire second text information from the target control, where the second text information is text information with correct text content; Adjusting the first text information based on the second text information to obtain third text information, wherein the adjustment includes changing incorrect text content in the first text information to correct text content; In response to the touch operation, acquiring the touch area includes: In response to the touch operation, acquiring position information of a starting touch point and position information of an ending touch point; The touch area is determined according to the position information of the starting touch point and the position information of the ending touch point.
2. The method according to claim 1, characterized in that The step of determining a target control matching the touch area from one or more textual controls displayed on the touch screen comprises: Obtaining an intersection-and-joint ratio between each textual control displayed on the touch screen and the touch area; Based on the intersection-and-union ratio, the target control is determined.
3. The method according to claim 1 or 2, characterized in that: The adjusting the first text information based on the second text information to obtain the third text information includes: Comparing characters in the first text information and the second text information corresponding to the same position on the touch screen; Determine as a target character a character in the first text information that corresponds to a character in the second text information and has a same position on the touch screen but has different contents; The target character in the first text information is replaced with a character in the second text information corresponding to the same position on the touch screen as the target character, to obtain the third text information.
4. The method according to claim 3, characterized in that The step of replacing the target character in the first text information with a character in the second text information corresponding to the same position on the touch screen as the target character to obtain the third text information comprises: determining a matching rate according to the number of the target characters and the number of characters in the first text information; When the matching rate is greater than the first threshold, the target character in the first text information is replaced with a character in the second text information corresponding to the same position on the touch screen as the target character, so as to obtain the third text information.
5. The method according to claim 1 or 2, characterized in that: The adjusting the first text information based on the second text information to obtain the third text information includes: Detecting whether there is a character set satisfying a preset format in the second text information; If the second text information contains a character set that satisfies the preset format, extracting the character set that satisfies the preset format from the second text information; The first text information is replaced with the extracted character set to obtain the third text information.
6. The method according to claim 5, characterized in that The method further comprises: According to the third text information, a service corresponding to the character set in the preset format is provided.
7. The method according to any one of claims 1 to 6, characterized in that In response to the touch operation, acquiring the touch area includes: In response to the touch operation, loading the area selection mark layer; The touch area is determined based on the confirmation operation of the area selection mark layer.
8. The method according to any one of claims 1 to 6, characterized in that The determining the touch area according to the position information of the starting touch point and the position information of the ending touch point comprises: In a case where the starting touch point and the ending touch point correspond to the same text line, the touch area is determined according to a first area between the starting touch point and the ending touch point.
9. The method according to any one of claims 1 to 6, characterized in that The determining the touch area according to the position information of the starting touch point and the position information of the ending touch point comprises: When the starting touch point and the ending touch point correspond to adjacent text lines, the touch area is determined based on a second area between the starting touch point and the right border of the touch screen, and a third area between the ending touch point and the left border of the touch screen.
10. The method according to any one of claims 1 to 6, characterized in that The determining the touch area according to the position information of the starting touch point and the position information of the ending touch point comprises: When the text lines corresponding to the starting touch point and the ending touch point are separated by one or more text lines, the touch area is determined based on the fourth area between the starting touch point and the right border of the touch screen, the fifth area between the text line corresponding to the starting touch point and the text line corresponding to the ending touch point, and the sixth area between the ending touch point and the left border of the touch screen.
11. The method according to any one of claims 1 to 6, characterized in that The determining the touch area according to the position information of the starting touch point and the position information of the ending touch point comprises: Moving the starting touch point in the positive direction of the y-axis and the negative direction of the x-axis of the touch screen by a first distance to obtain an adjusted starting touch point; Moving the end touch point in the positive direction of the x-axis and the negative direction of the y-axis of the touch screen by a second distance to obtain an adjusted end touch point; The touch area is determined according to the adjusted position information of the starting touch point and the adjusted position information of the ending touch point.
12. A text extraction device, characterized in that: The device comprises: A first acquisition module, configured to acquire a touch area in response to a touch operation on the touch screen; An extraction module, configured to extract the first text information in the touch area acquired by the first acquisition module by using an optical character recognition (OCR) technology; A determination module, configured to determine a target control matching the touch area from one or more text controls displayed on the touch screen; A second acquisition module, used to acquire second text information from the target control determined by the determination module, wherein the second text information is text information with correct text content; an adjustment module, configured to adjust the first text information extracted by the extraction module based on the second text information acquired by the second acquisition module to obtain third text information, wherein the adjustment includes changing incorrect text content in the first text information into correct text content; The first acquisition module includes: A second acquisition unit, configured to acquire position information of a starting touch point and position information of an ending touch point in response to the touch operation; The third determining unit is used to determine the touch area according to the position information of the starting touch point and the position information of the ending touch point.
13. The device according to claim 12, characterized in that The determination module comprises: A first acquisition unit, configured to acquire an intersection-and-joint ratio between each textual control displayed on the touch screen and the touch area; The first determination unit is used to determine the target control based on the intersection-and-combination ratio.
14. The device according to claim 12 or 13, characterized in that The adjustment module comprises: a comparing unit, configured to compare characters in the first text information and the second text information corresponding to the same position on the touch screen; A second determining unit, configured to determine, as a target character, a character in the first text information that corresponds to a character in the second text information and has a same position on the touch screen but has different contents; The first replacement unit is used to replace the target character in the first text information with a character in the second text information corresponding to the same position on the touch screen as the target character, so as to obtain the third text information.
15. The device according to claim 14, characterized in that The first replacement unit is also used for: determining a matching rate according to the number of the target characters and the number of characters in the first text information; When the matching rate is greater than the first threshold, the target character in the first text information is replaced with a character in the second text information corresponding to the same position on the touch screen as the target character, so as to obtain the third text information.
16. The device according to claim 12 or 13, characterized in that The adjustment module also includes: A detection unit, configured to detect whether a character set satisfying a preset format exists in the second text information; an extraction unit, configured to extract the character set satisfying the preset format from the second text information when the character set satisfying the preset format exists in the second text information; The second replacing unit is configured to replace the first text information with the extracted character set to obtain the third text information.
17. The device according to claim 16, characterized in that The device also includes: A service module is used to provide a service corresponding to the character set in the preset format according to the third text information.
18. The device according to any one of claims 12 to 17, characterized in that The first acquisition module also includes: A loading unit, configured to load a region selection mark layer in response to the touch operation; The fourth determining unit is used to determine the touch area based on the confirmation operation of the area selection mark layer.
19. The device according to any one of claims 12 to 17, characterized in that The third determining unit is further configured to: In a case where the starting touch point and the ending touch point correspond to the same text line, the touch area is determined according to a first area between the starting touch point and the ending touch point.
20. The device according to any one of claims 12 to 17, characterized in that The third determining unit is further configured to: When the starting touch point and the ending touch point correspond to adjacent text lines, the touch area is determined based on a second area between the starting touch point and the right border of the touch screen, and a third area between the ending touch point and the left border of the touch screen.
21. The device according to any one of claims 12 to 17, characterized in that The third determining unit is further configured to: When the text lines corresponding to the starting touch point and the ending touch point are separated by one or more text lines, the touch area is determined based on the fourth area between the starting touch point and the right border of the touch screen, the fifth area between the text line corresponding to the starting touch point and the text line corresponding to the ending touch point, and the sixth area between the ending touch point and the left border of the touch screen.
22. The device according to any one of claims 12 to 17, characterized in that The third determining unit is further configured to: Moving the starting touch point in the positive direction of the y-axis and the negative direction of the x-axis of the touch screen by a first distance to obtain an adjusted starting touch point; Moving the end touch point in the positive direction of the x-axis and the negative direction of the y-axis of the touch screen by a second distance to obtain an adjusted end touch point; The touch area is determined according to the adjusted position information of the starting touch point and the adjusted position information of the ending touch point.
23. A text extraction device, characterized in that: It comprises a processor, a memory for storing processor executable instructions, and a touch screen for receiving touch operations, wherein when the processor calls the executable instructions, the terminal implements the method according to any one of claims 1 to 11.
24. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 11 is implemented.
Citation Information
Patent Citations
Text recognition method and device, mobile terminal, and storage medium
CN109002759A
Interface control testing method and system based on deep learning
CN111930622A
OCR apparatus and OCR result verification method
US20060210197A1
