A method for real-time selection and interaction of image text in Chinese

Abstract

The present application relates to a kind of real-time selection image Chinese text and the method of interaction, comprising the following steps:1) obtaining image content and display;2) text detection is carried out, the range of all existing characters in image is obtained and saved to recognition result list;3) waiting for user selection of image Chinese text, while text recognition is carried out to all existing characters in image and is updated to recognition result list;4) response user selection of image Chinese text, render the state when text is selected on image;5) according to the selection of user and the information of recognition result list, obtain the text content of user selection, provide copy option for user;Text content is identified and judged and according to the corresponding jump option of judging result is provided.Using the method of the present application to obtain image Chinese text, step 2 first carries out text detection step, saves the time consumption brought by direction classification and text recognition, so that there is no delay in the process from display image to selectable text.

Description

Technical Field

[0001] This invention relates to the technical field of text recognition in images, and in particular to a method for real-time selection and interaction of text in an image. Background Technology

[0002] With the widespread use of personal computers and smart devices, and the rapid development of multimedia technology, users frequently encounter images, videos, and various application interfaces when using operating systems. These interfaces often contain visible text. To better utilize the textual information within images, OCR image recognition technology has emerged. OCR image recognition refers to the process of recognizing the shapes of text in an image into computer text using character recognition methods. The recognition result typically includes all the characters identified in the image and the positional information of each character within the image.

[0003] Current OCR image recognition software typically involves first opening an image file, then the user selects to launch the OCR image recognition engine to perform the recognition task, waiting for the recognition result to be output to a text view after the task is completed, and finally the user selects and copies the desired text from the text view. Current technical solutions have the following shortcomings: OCR image recognition can only be performed by opening image files; it cannot directly recognize text from videos or user interfaces. For videos or user interfaces, the only option is to first take a screenshot, save it as an image file, and then open it, which is cumbersome and inconvenient. Users cannot directly select text from images, resulting in poor interactivity. OCR image recognition is often time-consuming, potentially taking tens of milliseconds to several seconds or even longer to recognize a single image. This means that users must wait for the OCR image recognition to complete the recognition task each time they want to use the text information in an image, resulting in poor immediacy. Furthermore, the lack of further processing of the output recognition results prevents the identification and utilization of specific information needed by the user, such as links, phone numbers, and addresses, leading to low efficiency in utilizing text information. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method for real-time selection and interaction of text in an image, which can acquire text in an image.

[0005] The technical solution adopted by the present invention to solve the above-mentioned technical problems is: a method for real-time selection and interaction of text in an image, comprising the following steps:

[0006] 1) Acquire and display the image content;

[0007] 2) Perform text detection, obtain the range of all existing characters in the image and save it to the recognition result list;

[0008] 3) Wait for the user to select text in the image, and at the same time perform text recognition on all ranges of characters in the image and update the recognition result list;

[0009] 4) Respond to the user's selection of text in the image, and render the state of the selected text on the image according to the user's selection range;

[0010] 5) Based on the user's selection and the information in the recognition result list, obtain the text content selected by the user and provide the user with a copy option; identify and judge the text content and provide corresponding jump options based on the judgment result.

[0011] Preferably, in step 1, the image content is obtained by opening and reading an image file or video file, or by directly obtaining the image content of the screen view;

[0012] In step 1, an image content acquisition module is constructed. This module can read and parse image files, forming an image view that includes a display view and zoom, move, and rotate tools. The same module can also read and parse video files, forming a video view that includes a video display view and play, pause, fast forward, rewind, and time-based video frame acquisition tools. When the video is paused or a specified time frame is acquired, the image content of the current video frame is acquired and displayed on the image view, which overlays the video display view. Furthermore, the module can acquire the current screen image content from the user interface and display it on the image view in a full-screen, borderless manner.

[0013] Preferably, in step 2, after acquiring the image content, text detection is performed on the image content, and the range information of each character-containing rectangle in the image is saved to the recognition result list; the recognition result list includes the following information: character, character range, line number, line end marker, selection marker; coordinates and size information of the rectangle containing the character within the character range; the list is arranged according to the line number from smallest to largest and the coordinates from left to right;

[0014] The row number is determined using a three-step loop method, as follows:

[0015] The image and text box are centered at the top left corner, with the horizontal x-axis increasing to the right and the vertical y-axis increasing downwards.

[0016] The first step is to start from Y1=0 and find the text box with the smallest vertex value. The vertex value of this text box is t1 and the bottom value is t2.

[0017] The second step is to find the text box with the smallest bottom point value in the range t1 to t2, which has a bottom point value of Y2.

[0018] The third step is to find all text boxes whose vertex values ​​are contained between Y1 and Y2, which are the text boxes in the first row, and arrange them from left to right, with the last one being the end of the row;

[0019] The second line uses Y1 as the Y2 of the previous step, and repeats the previous three steps; repeat these three steps until the line numbers of all text are determined.

[0020] Preferably, a row range list is created to store Y1, Y2 and the maximum value Ymax of each row, as well as the sequence numbers of the first and last text boxes of each row in the identification list, for subsequent determination of the rows and text boxes that may be included at a certain point.

[0021] Preferably, in step 2, the text detection uses the segmentation-based text detection algorithm DBNet.

[0022] Preferably, in step 3, while waiting for the user to select the text in the image, a task is created. This task is: based on the completion of text detection in step 2, to continue to perform orientation classification and text recognition on the image.

[0023] Orientation classification is used to determine the orientation of the text. If a box is determined to be reversed, it needs to be flipped. Text recognition uses the CRNN algorithm.

[0024] Preferably, in step 4, the selection status of the cells in the recognition list is set to no each time the selection is canceled or reselected; when the user presses and drags the mouse on the image view, the selection status of the cells is updated based on the coordinates of the mouse in the image view when the mouse is first pressed, the coordinates of the current mouse position, and the zoom value of the current view.

[0025] After updating the selection status of the cells in the recognition result list, when updating the view, iterate through the cells in the recognition list that are in the selected state. Based on the character range information of that cell, draw a semi-transparent rectangle or a rectangle with color overlay on the image to represent the state when the text is selected. This completes the rendering of the state when the text in the image is selected.

[0026] Preferably, in step 4, when the user completes the selection of text in the image, but the task in step 3 is not completed, and the text selection is not all selected, the task in step 3 is stopped.

[0027] Prioritize the character ranges of the selected states, continue to perform text recognition on the ranges of unrecognized characters in the image, and update the recognition result list;

[0028] The method of counting is used to determine whether the selected text content has been recognized. That is, first count the number of all selected status units and the number of selected status units with empty characters. The difference between the two is the number of units to be recognized. A variable is created based on this number. The variable is decremented until it reaches zero for each successful recognition. After completion, the characters of all units are concatenated to obtain the selected text content, and a newline character is added to the end of each line.

[0029] Preferably, in step 5, after the user completes the selection of text in the image, a copy option is provided to the user; when the user selects to copy, if the selected text content has already been obtained, the text content is written to the clipboard; if the selected text content has not yet been obtained, the text content is written to the clipboard after the selected text content is obtained.

[0030] After selecting and retrieving text content, the text content is identified and processed to determine whether it is a link, phone number, or address, and the corresponding judgment result is obtained. Based on the judgment result, the corresponding jump option is provided; regular expression matching is used to determine whether it is a link or phone number.

[0031] Preferably, for image files, the results of image recognition are persistently saved.

[0032] The beneficial effects of this invention are as follows: Using the method of this invention to obtain text in an image, step 2 involves a text detection step, eliminating the time consumption associated with direction classification and text recognition, significantly shortening the response time for selectable text in the image. The response time is between tens and hundreds of milliseconds, resulting in a seamless transition from image display to selectable text. In step 4, responding to the user's selection of text in the image, the state of the selected text is rendered on the image according to the user's selection range; this responsiveness enhances interactivity by addressing the user's need to select text in the image. Attached Figure Description

[0033] Figure 1 This is a flowchart of the method for real-time selection and interaction of text in an image according to the present invention;

[0034] Figure 2 This is a schematic diagram illustrating the determination of line numbers in this invention;

[0035] Figure 3 This is a schematic diagram illustrating the selection of text in this invention; Detailed Implementation

[0036] The present invention will now be described in further detail with reference to the accompanying drawings and preferred embodiments. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0037] like Figure 1As shown, a method for real-time selection and interaction of text in an image includes the following steps:

[0038] 1) Acquire and display the image content;

[0039] 2) Perform text detection, obtain the range of all existing characters in the image and save it to the recognition result list;

[0040] 3) Wait for the user to select text in the image, and at the same time perform text recognition on all ranges of characters in the image and update the recognition result list;

[0041] 4) Responds to the user's selection of text in the image, and renders the state of the selected text on the image according to the user's selection range; it can respond to the user's need to select text in the image, thus improving interactivity;

[0042] 5) Based on the user's selection and the information in the recognition result list, the text content selected by the user is obtained, and the user is provided with a copy option; the text content is identified and judged, and corresponding jump options are provided based on the judgment results, which improves the efficiency of information utilization.

[0043] Specifically, in one optional implementation, in step 1, image content is obtained by opening and reading an image file or video file, or by directly obtaining the image content of the screen view.

[0044] In step 1, an image content acquisition module is constructed. This module can read and parse image files, forming an image view that includes a display view and zoom, move, and rotate tools. It can also read and parse video files, forming a video view that includes a video display view and play, pause, fast forward, rewind, and time-based video frame acquisition tools. When the video is paused or a specified number of video frames are acquired, the image content of the current video frame is captured and displayed on the image view, which overlays the video display view. Furthermore, the module can acquire the current screen image content from the user interface and display it on the image view in a full-screen, borderless manner. The image content acquisition module acquires image content and displays it on the image view, which implements basic functions such as zooming, moving, and rotating. This expands the methods for acquiring image content, eliminating the need to first capture and save video files or user interfaces as image files, thus improving convenience.

[0045] Specifically, in one optional implementation, in step 2, after acquiring the image content, text detection is performed on the image content, and the range information of each character-containing rectangle in the image is saved to the recognition result list; the recognition result list includes the following information: character, character range, line number (determined according to information such as top to bottom and the size of the character box), line end marker (the rightmost unit of the same line number), selection marker; coordinates and size information of the rectangle containing the character within the character range; the list is arranged according to the line number from smallest to largest and the coordinates from left to right.

[0046] like Figure 2 As shown, a three-step loop method is used to determine the row number, as follows:

[0047] The image and text box are centered at the top left corner, with the horizontal x-axis increasing to the right and the vertical y-axis increasing downwards.

[0048] The first step is to start from Y1=0 and find the text box with the smallest vertex value. The vertex value of this text box is t1 and the bottom value is t2.

[0049] The second step is to find the text box with the smallest bottom point value in the range t1 to t2, which has a bottom point value of Y2.

[0050] The third step is to find all text boxes whose vertex values ​​are contained between Y1 and Y2, which are the text boxes in the first row, and arrange them from left to right, with the last one being the end of the row;

[0051] The second line uses Y1 as the Y2 of the previous step, and repeats the previous three steps; repeat these three steps until the line numbers of all text are determined.

[0052] Specifically, in one optional implementation, a row range list is created to store the Y1, Y2, and maximum value Ymax of each row, as well as the sequence numbers of the first and last text boxes of each row in the recognition list. This is used to subsequently determine the rows and text boxes that a certain point may be included in. Determining the row numbers facilitates the rapid determination of the cell selection status in the subsequent recognition list.

[0053] Specifically, in one optional implementation, in step 2, text detection uses the segmentation-based text detection algorithm DBNet.

[0054] Step 2 performs text detection first, eliminating the time consumed by orientation classification and text recognition. This significantly reduces the response time for text in the selectable image, with a response time between tens and hundreds of milliseconds, making the process from displaying the image to selecting the text imperceptible.

[0055] Specifically, in one optional implementation, in step 3, while waiting for the user to select the text in the image, a task is created. This task is: based on the completion of text detection in step 2, to continue to perform orientation classification and text recognition on the image.

[0056] Orientation classification is used to determine the orientation of text; if a box is determined to be reversed, it needs to be flipped. Text recognition uses the CRNN algorithm. The entire CRNN network structure consists of three parts, from bottom to top: CNN (convolutional layer), which extracts features from the input image to obtain a feature map; RNN (recurrent layer), which uses a bidirectional RNN (BLSTM) to predict the feature sequence, learns each feature vector in the sequence, and outputs the predicted label (true value) distribution; and CTC loss (transcription layer), which uses CTC loss to transform the series of label distributions obtained from the recurrent layer into the final label sequence. CRNN uses connection temporal classification (CTC) loss to avoid the problem of inconsistency between prediction and labeling.

[0057] Specifically, in one optional implementation, in step 4, the selection status of units in the recognition list is set to "no" each time a selection is canceled or reselected; when the user presses and drags the mouse on the image view, based on the coordinates of the mouse in the image view when it was first pressed, the current coordinates of the mouse hovering, and the zoom value of the current view, it is determined that some units in the recognition list are selected and the selection status of the units is updated. Figure 3 As shown, there are four possible relative positions of the mouse start and end points. It can be seen that distinguishing these based on the slope of the line determined by the start and end points can be simplified into two cases: the first case is when the slope is negative, and the second case is otherwise. The start point A and end point B are determined in two steps, with the point higher up in the image designated as point A, and the other point as point B. First, since the recognition list is sorted from left to right by row number and position, and the row range list allows for quick identification of rows that point A might contain, the text box units containing point A are searched within these rows. Second, if no text box is found, a unique row is determined based on the point and the row range. The text box unit closest to the right of point A is searched within this row. The same method is used to determine the text box unit corresponding to point B, except that the second step searches for the text box unit closest to the left of point B. This determines the start and end units in the selected state. This method eliminates the need to distinguish between different cases, providing a straightforward and concise way to determine the selection state of units in the recognition list.

[0058] After updating the selection status of the cells in the recognition result list, when updating the view, iterate through the cells in the recognition list that are in the selected state. Based on the character range information of that cell, draw a semi-transparent rectangle or a rectangle with color overlay on the image to represent the state when the text is selected. This completes the rendering of the state when the text in the image is selected.

[0059] Specifically, in one optional implementation, in step 4, when the user completes the selection of text in the image, but the task in step 3 has not been completed, and the text selection is not all selected, the task in step 3 is stopped.

[0060] Prioritize the character ranges of the selected states, continue text recognition of the ranges of unrecognized characters in the image, and update the recognition result list.

[0061] The method uses a counting approach to determine whether the selected text content has been fully recognized. First, it counts the total number of selected state units and the number of empty selected state units; the difference between these two is the total number of units to be recognized. A variable is created based on this count, and it is decremented until it reaches zero for each successfully recognized unit. After completion, the characters of all units are concatenated to obtain the selected text content, with a newline character added to the end of each line. Because the character range of the selected state is recognized first, the method can determine whether the selected text content has been fully recognized in the shortest time. After completion, the characters of all units are concatenated to obtain the selected text content, with a newline character added to the end of each line.

[0062] Specifically, in one optional implementation, in step 5, after the user completes the text selection in the image, a copy option is provided to the user; when the user selects to copy, if the selected text content has already been obtained, the text content is written to the clipboard; if the selected text content has not yet been obtained, the text content is written to the clipboard after the selected text content is obtained. Since actual user operations typically take time, this method responds promptly to the user's needs and simultaneously completes the copying task.

[0063] After selecting the text content to retrieve, the text content is identified and processed to determine whether it is a link, phone number, or address, and the corresponding judgment result is obtained. Based on the judgment result, the corresponding jump option is provided.

[0064] Specifically, in one optional implementation, regular expression matching is used to determine whether a message is a link or a phone number. Regular expressions have the following advantages: they are powerful and flexible, supporting combinations of multiple rules and can be used to match various complex text and data formats; they are efficient and convenient, as complex text patterns can be represented by simple characters or symbols, greatly reducing the amount of code.

[0065] For link identification, links can be categorized into several types: URL links, such as http: / / xxxx.com, www.xxx.com, etc.; file links, such as file: / / xxx / xx; and email links, such as mailto: xxx@xx.com, xxx@xx.com. By differentiating link types, different software is redirected to open them: URL links open in the default browser, file links open in the default file manager, and email links open in the default email client. For phone numbers, such as +8613xxxxxxxxx, or even 13x xxxx xxxx with spaces; or 07xxxxxxxxx, etc., the dialing function is used on mobile operating systems, and the address book is used on desktop operating systems. For address identification, the system first attempts to decompose and extract address elements from the text, such as province, city, district. If so, it combines this information with province, city, district, and street information from the database to reorganize it into a standardized address text, which is then opened using map software.

[0066] Specifically, in one optional implementation, for image files, the results of image recognition are persistently saved, which can further improve the immediacy of text recognition in the image when the image file is processed again.

[0067] The above description is only a specific embodiment of the present invention. Various examples and illustrations do not constitute a limitation on the substantive content of the present invention. Those skilled in the art can make modifications or variations to the above-described specific embodiments after reading the specification without departing from the substance and scope of the invention.

Claims

1. A method for real-time selection of text in an image and interaction, characterized in that: The method comprises the following steps: 1) obtaining image content and displaying; 2) performing text detection to obtain the range of all characters existing in the image and save to the recognition result list; 3) waiting for the user to select the image text, while performing text recognition on all character ranges existing in the image and updating to the recognition result list; 4) in response to the user's selection of the image text, rendering the state of the text being selected on the image according to the user's selection range; 5) obtaining the text content selected by the user according to the user's selection and the information of the recognition result list, and providing a copy option for the user; judging the text content and providing corresponding jump options according to the judgment result; In step 2, after obtaining the image content, text detection is performed on the image content, and the range information of each character-containing box in the image is saved to the recognition result list; the recognition result list contains the following information: character, character range, line number, end-of-line flag, selection flag; the character range contains the coordinate and size information of the rectangular box of the character; the list is arranged from small to large according to the line number and from left to right according to the coordinate; The line number is determined by a three-step loop method as follows: The image and the text box take the top-left corner as the origin, the horizontal x-axis increases to the right, and the vertical y-axis increases downward; Step 1: starting from Y1=0, find the text box with the smallest top point value, which is t1, and the bottom point value is t2; Step 2: find the text box with the smallest bottom point value in the range of t1 to t2, which is Y2; Step 3: find all text boxes whose top point values are contained between Y1 and Y2, which are the text boxes of the first line, and are arranged from left to right, with the last one being the end-of-line; The second line takes Y1 as Y2 of the previous step, and repeats the previous three steps; loop these three steps until all text line numbers are determined; In step 2, the text detection uses the DBNet algorithm based on segmentation; In step 3, while waiting for the user to select the image text, a task is created: on the basis of the text detection in step 2, continue to classify the direction of the image and perform text recognition; The direction classification is used to determine the direction of the text, and if a box is determined to be reversed, it needs to be flipped; the text recognition uses the CRNN algorithm.

2. The method of claim 1, wherein: In step 1, the image content is obtained by opening and reading an image file or a video file, or directly obtaining the image content of the screen view. In step 1, an image content acquisition module is constructed, which can read and parse image files to form an image view including a display view and including zoom, move and rotate tools; the image content acquisition module can read and parse video files to form a video view including a video display view and including play, pause, fast forward, fast backward and time-based video frame acquisition tools, when the video is paused or a specified time video frame is acquired, the image content of the current video frame is acquired and displayed on the image view, and the image view is overlaid on the video display view; the image content acquisition module can acquire the current screen image content of the user interface and display it on the image view in a full-screen manner without window boundaries.

3. The method of claim 1, wherein: A line range list is created to save Y1, Y2, the maximum value Ymax of each line, and the sequence number of the first text box and the last text box of each line in the recognition list, which is used to determine the line and text box that may contain a certain point in the subsequent.

4. The method of claim 1, wherein: In step 4, the selection state of each unit in the recognition list is set to false each time the selection is canceled or reselected; when the user presses the mouse and drags on the image view, the coordinates of the mouse when it is first pressed and the coordinates of the current mouse stop are determined according to the zoom value of the current view, and some units in the recognition list are determined to be selected and the selection state of the units is updated; After updating the selection state of the units in the recognition result list, when updating the view, the units in the recognition list that are in the selection state are traversed, and a semi-transparent rectangle is drawn on the image according to the character range information of the unit or a rectangle is drawn in a color superimposed manner as the state of the selected text, and the state rendering of the selected text in the image is completed.

5. The method of claim 4, wherein: In step 4, when the user completes the text selection in the image, the task in step 3 is not completed, and the text selection is not full selection, the task in step 3 is stopped; The character range in the selection state is arranged in priority, and text recognition is continued for the range of un-recognized characters in the image, and is updated to the recognition result list; A counting method is used to determine whether the selected text content is recognized, that is, the number of all units in the selection state is counted first, and the number of units with empty characters in the selection state is counted, and the difference between the two is the number of units to be recognized; a variable is created based on this number, and the variable is reduced by one for each successful recognition until it is zero; after completion, all units are connected to obtain the selected text content, and a line feed character is added at the end of each line.

6. The method of claim 1, wherein: In step 5, a copy option is provided for the user after the user completes the image text selection; when the user selects the copy, if the selected text content has been acquired, the text content is written to the clipboard; if the selected text content has not been acquired, the text content is written to the clipboard after the selected text content is acquired; After the selected text content is acquired, the text content is subjected to recognition and judgment processing to determine whether it is a link or a telephone number or an address to obtain a corresponding judgment result, and corresponding jump options are provided according to the judgment result; A regular expression matching method is used to determine whether it is a link or a telephone number.

7. The method of claim 1, wherein: For image files, the results of image recognition are persistently saved.

Images