Method, device and terminal device for selecting characters of curved text

Abstract

The embodiment of the present application is applicable to the field of artificial intelligence technology, and provides a method and device for selecting characters of curved text. It includes: displaying and detecting the original image, generating a picture to be identified containing straight text; then obtaining a connection time classification sequence corresponding to the text content, and calculating the first coordinate of each character in the connection time classification sequence in the picture to be identified; determining the segmented area where the first coordinate is located in the picture to be identified; multiplying the first coordinate of each character with the segmented perspective transformation matrix between the original image and the picture to be identified to obtain the second coordinate of each character in the original image; detecting the first operation of the user on the original image, and highlighting the selected character according to the second coordinate. By adopting the above method, when the user manually selects characters in the curved text, the positioning accuracy of the character position can be improved, as well as the efficiency and accuracy of manually selecting characters can be improved.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and in particular relates to a method, apparatus, and terminal device for selecting characters of curved text. Background Art

[0002] Optical Character Recognition (OCR) is a process that uses electronic devices such as mobile phones, scanners, or digital cameras to examine characters on paper, determine their shapes based on their dark and light patterns, and then use character recognition methods to translate these shapes into computer text. OCR is an important application scenario in the field of Computer Vision (CV) and also provides the foundational capabilities for the application of Augmented Reality (AR) technology in many fields, such as translation and image semantic understanding. Typically, OCR can include two steps: text region detection and text content recognition. The former step can detect where the text region is in the image, and the latter step can identify the specific content of the text in the text region. However, in natural scene text, the shape of the text is often not horizontal. The shape may be curved text with an arc or a wave. Curved text scenes pose great challenges to both OCR detection and recognition.

[0003] In the mobile phone smart lens object recognition scenario, the smart lens can perform multi-object detection and text detection and recognition tasks, and provide the location of the object and text box. The user can click on the text line on the phone screen, and the screen will freeze. The user can click on the detected and recognized text content, which is similar to the function of dragging the mouse to select text content character by character in a text document. The user can then choose to copy, translate, or search the selected content.

[0004] Text selection scenarios are divided into straight text scenarios and curved text scenarios. In straight text scenarios, text lines are all angled rectangular boxes, parallelogram boxes or quadrilateral boxes, and text lines are generally described by four vertices. In curved text scenarios, in order to describe the direction of the curve, the text box is generally composed of a polygon, which can be decomposed into many quadrilaterals, and these quadrilaterals are spliced ​​into the polygon. In straight text scenarios, since there is only a simple perspective transformation mapping relationship between the original image and the image to be identified, if the coordinates of each text on the image to be identified are obtained, the coordinates of each text in the straight text on the original image can be easily obtained. However, in curved text scenarios, due to the strong discontinuity and the more complex description of curved text itself, obtaining the coordinates of each text is much more difficult than in straight text scenarios. Summary of the Invention

[0005] The embodiments of the present application provide a method, apparatus, and terminal device for selecting characters in curved text, which are used to accurately obtain the coordinates of each character in the curved text, thereby achieving more precise character selection.

[0006] In a first aspect, an embodiment of the present application provides a method for selecting characters in curved text, which is applied to a terminal device. The method includes:

[0007] The terminal device displays an original image on a display interface, wherein the original image contains curved text; the terminal device then detects the original image and generates a picture to be identified containing straight text, wherein the text content of the straight text corresponds to the text content of the curved text; the terminal device then identifies the connection time classification corresponding to the text content of the straight text according to the picture to be identified. A connection temporal classification (CTC) sequence is provided, wherein the connection temporal classification sequence includes multiple characters; the terminal device then calculates the first coordinate of each of the multiple characters in the CTC sequence in the image to be recognized; the terminal device then determines the segmented area where the first coordinate of each of the multiple characters is located in the image to be recognized; the terminal device then determines, based on the original image and the image to be recognized, the perspective transformation matrix corresponding to each segmented area when the original image is processed and transformed into the image to be recognized; the first coordinate of each character in the multiple characters is multiplied by the perspective transformation matrix to obtain the second coordinate of each character in the multiple characters in the original image; finally, the terminal device detects the user's first operation on the original image, and highlights the selected character according to the second coordinate of each character in the multiple characters in the original image, wherein the first operation is used to select characters in the curved text on the original image.

[0008] The method for selecting characters of curved text provided in the embodiment of the present application calculates the coordinates of each character in the text content in the CTC sequence corresponding to the text content, and then obtains the first coordinate of each character in the image to be recognized based on the correspondence between the CTC sequence index and the text coordinates of the image to be recognized. Then, based on the piecewise perspective transformation relationship between the image to be recognized and the original image, a one-to-one correspondence calculation is performed on the first coordinate to obtain the second coordinate of each character in the original image, thereby improving the accuracy of the second coordinate calculation and also improving the accuracy of the character selection control drawn according to the second coordinate. When the user clicks on the character in the original image, the terminal device can accurately locate the character and output the selected character, thereby improving the efficiency and accuracy of character selection and recognition. At the same time, the OCR detection model configured in the terminal device can detect the original image and generate an image to be recognized containing the text content of the curved text in the original image. The above-mentioned image to be recognized can be used as input data of the OCR recognition model. The above-mentioned text content is recognized by the OCR recognition model to obtain a CTC sequence corresponding to the text content. The character selection method for curved text provided in the embodiment of the present application can directly adopt the OCR detection model and OCR recognition model configured in the terminal device and capable of detecting curved text, which helps to expand the application scope of the method and reduce the technical difficulty of the terminal device adopting the method.

[0009] Optionally, after obtaining the second coordinate of each of the multiple characters and before detecting the first operation, the terminal device may further generate a first prompt message in the original image based on the second coordinate corresponding to each of the multiple characters, wherein the first prompt message is used to indicate that the user can select a character in the original image. This makes it more convenient for the user to select a character.

[0010] Optionally, the CTC sequence corresponding to the text content may refer to a sequence after processing the initial CTC sequence output by the OCR recognition model, and the processing of the initial CTC sequence may be implemented by the character selection model provided in the embodiment of the present application. The above-mentioned text content is recognized by the OCR recognition model, and an initial CTC sequence may be output. Then, the character selection model may determine the length of the initial CTC sequence and the image width of the image to be recognized, and determine whether the product between the length of the initial CTC sequence and the downsampling multiple of the character selection model is greater than the image width of the image to be recognized. If the above product is greater than the image width of the image to be recognized, the initial CTC sequence needs to be cropped so that the product between the length of the cropped CTC sequence and the downsampling multiple of the model is less than or equal to the image width of the image to be recognized.

[0011] Optionally, the CTC sequence can be cropped by sequentially cropping the head element or tail element of the initial CTC sequence. For the initial CTC sequence that needs to be cropped, a head element of the sequence can be cropped first, and then a tail element can be cropped. After each cropping of a head element or a tail element, the product of the length of the cropped sequence and the preset downsampling multiple can be calculated again to see whether it is less than or equal to the image width of the image to be recognized. If after a certain cropping, the product of the length of the obtained sequence and the preset downsampling multiple is less than or equal to the image width of the image to be recognized, the cropping can be stopped and the currently obtained CTC sequence is output, which is the CTC sequence corresponding to the text content. If the product of the length of the sequence obtained after a certain cropping and the preset downsampling multiple is still greater than the image width of the image to be recognized, it is necessary to continue cropping in the above order until the product of the sequence length and the downsampling multiple is less than or equal to the image width of the image to be recognized.

[0012] The embodiment of the present application can reduce the amount of data for subsequent processing and improve processing efficiency by partially cropping the CTC sequence.

[0013] Optionally, after completing the cropping of the CTC sequence, parameters can be initialized. Parameters that need to be initialized may include the left boundary, the bounds, the current state, the previous state, the coordinate array, the content array, and so on.

[0014] Optionally, calculating the first coordinate of each character in the CTC sequence in the image to be recognized can be accomplished by determining the left and right boundary coordinates of each character in the CTC sequence. Combining the left and right boundary coordinates of each character in the CTC sequence, the first coordinate of the character in the image to be recognized can be obtained.

[0015] Optionally, the character boundary of each character in the CTC sequence can be obtained by determining the right boundary of the character and the left boundary of the next adjacent character. For any character in the CTC sequence, the original right boundary coordinates of the character and the original left boundary coordinates of the next character can be obtained. The average of the original right boundary coordinates and the original left boundary coordinates is then calculated. By fine-tuning the average, the right boundary coordinates of the character and the left boundary coordinates of the next character can be obtained.

[0016] Optionally, fine-tuning of the boundaries of each character can be performed according to different character types. Therefore, when fine-tuning the boundaries of a character, the first character type of the character and the second character type of the next character can be determined separately. If the first character type and the second character type are different, the two have different offsets. If the two character types are the same, the offsets are also the same. For the current character, the first difference can be obtained by calculating the above average value minus the offset corresponding to the first character type of the current character, and then the first difference can be used as the right boundary coordinate of the current character after fine-tuning; for the next adjacent character, the second sum can be obtained by calculating the average value plus the offset corresponding to the second character type, and then the second sum can be used as the left boundary coordinate of the next character after fine-tuning.

[0017] Since the first coordinate of each character in the CTC sequence corresponds to the coordinate of the character in the image to be recognized, the embodiment of the present application can prevent the problem of the character selection space generated according to the second coordinate not being able to completely cover the character when clicking on a character in the subsequent processing process by fine-tuning the boundaries of each character in the CTC sequence, thereby ensuring the accuracy of the character positioning position.

[0018] Optionally, since each element in the CTC sequence is obtained by downsampling the image to be recognized by a certain multiple through the convolutional neural network in the OCR recognition model and sliding the window through a 4×4 convolution kernel on the last layer feature map, one element in the CTC sequence corresponds to the range covered by the downsampling multiple pixel width in the image to be recognized. Then, when calculating the first coordinate of each character in the image to be recognized based on the character boundary coordinates of each character, the left boundary coordinate and the right boundary coordinate of each character can be multiplied by the preset downsampling multiple respectively to obtain the left boundary position and right boundary position of each character in the image to be recognized, and then the first vertex coordinate and the second vertex coordinate of the character in the image to be recognized are obtained based on the left boundary position and the right boundary position. On this basis, combined with the image height of the image to be recognized, the third vertex coordinate and the fourth vertex coordinate of each character in the image to be recognized can be determined, that is, the four vertex coordinates of each character in the image to be recognized are obtained. Each character in the image to be recognized is divided according to a rectangular box, and each character occupies a rectangular box. The four vertex coordinates are used to describe the rectangular box in which each character is located in the image to be recognized. Optionally, after obtaining the first vertex coordinate, the second vertex coordinate, the third vertex coordinate and the fourth vertex coordinate of each character, the terminal device can traverse the character, and then determine the first segmented area where the first vertex coordinate is located, the second segmented area where the second vertex coordinate is located, the third segmented area where the third vertex coordinate is located and the fourth segmented area where the fourth vertex coordinate is located. The segmented area is used to indicate the corresponding segmented perspective transformation area between the image to be identified and the original image. For example, the first vertex coordinate of a character is located in the rectangular box A in the image to be identified, and the rectangular box A corresponds to the area A in the original image, and the second vertex coordinate is located in the rectangular box B in the image to be identified, then the rectangular box B corresponds to the area B in the original image. Therefore, when the coordinates of the character in the image to be identified are mapped to the original image, the first vertex coordinate is multiplied by the perspective transformation matrix between the rectangular box A and the area A to obtain the coordinates of the first vertex coordinate in the original image, and the second vertex coordinate is multiplied by the perspective transformation matrix between the rectangular box B and the area B to obtain the coordinates of the second vertex coordinate in the original image.

[0019] Optionally, after obtaining the segmented area where each vertex coordinate is located, the terminal device obtains a third coordinate by multiplying the first vertex coordinate by the perspective transformation matrix of the first segmented area, obtains a fourth coordinate by multiplying the second vertex coordinate by the perspective transformation matrix of the second segmented area, obtains a fifth coordinate by multiplying the third vertex coordinate by the perspective transformation matrix of the third segmented area, and obtains a sixth coordinate by multiplying the fourth vertex coordinate by the perspective transformation matrix of the fourth segmented area; wherein the third coordinate, the fourth coordinate, the fifth coordinate, and the sixth coordinate serve as the second coordinate of the character in the original image. In this way, the segmented area where the four vertex coordinates of each character are located is determined, and the second coordinate is obtained according to the perspective transformation matrix of the corresponding segmented area, which improves the calculation accuracy of the first coordinate and ensures that when a character is selected, the character selection control can completely cover the area where the character is located.

[0020] Optionally, the character selection control can be displayed with different background colors. When generating a character selection control in the original image, the character area of ​​each character can be drawn as the first color according to the second coordinate. The first color can be any color, but it should be noted that after the character area is drawn as the first color, it should be avoided that the first color causes inconvenience to the user in recognizing the characters. By generating a character selection control in the original image, the embodiment of the present application can prompt the user which areas are recognized text areas, making it convenient for the user to click on the characters in the area.

[0021] Optionally, after the terminal device generates a character selection control in the original image, it can monitor the user's click event in the character area in real time and determine whether the user's click is within the text box. When it is monitored that the user clicks a certain character area (that is, the user clicks the area within the text box), the terminal device can redraw the background color of the area to distinguish it from other areas that have not been clicked. When it is monitored that the user's click area is not within any text box, no response is made.

[0022] In one possible implementation, the entire line of character area including the clicked character area can be painted in a second color different from the first color. Then continue to monitor whether there is a drag event in the character area corresponding to the second color. If so, the character area covered by the second color can be adjusted according to the drag event, and the adjusted character area is the character area that the user expects to click. For each character in the character area covered by the adjusted second color, the results obtained by the OCR recognition model can be output, and the results can be displayed on the display interface of the terminal device.

[0023] In another possible implementation, the area containing the clicked character (i.e., the area where the character closest to the click position is located) can be painted as a second color different from the first color. Then continue to monitor whether there is a drag event in the character area corresponding to the second color. If so, the character area covered by the second color can be adjusted according to the drag event, and the adjusted character area is the character area that the user expects to click. For each character in the character area covered by the adjusted second color, the results obtained by the OCR recognition model can be output, and the results can be displayed on the display interface of the terminal device.

[0024] In another possible implementation, the character area that the user slides and clicks on the screen (such as the area from the starting position to the ending position of the user's finger sliding on the screen) can be painted as a second color different from the first color. Then continue to monitor whether there is a drag event in the character area corresponding to the second color. If so, the character area covered by the second color can be adjusted according to the drag event, and the adjusted character area is the character area that the user expects to click. For each character in the character area covered by the adjusted second color, the results obtained by the OCR recognition model can be output, and the results can be displayed in the display interface of the terminal device. In this embodiment, the drag event can be understood as the terminal device drawing the background color of the second color for the selected text box, and draggable left and right controls will appear at the beginning and end of the text box, and the user can modify the text selection range by dragging the two controls.

[0025] The embodiment of the present application can prompt the user which characters in which areas are available for selection by drawing a prompt information of the first color in the original image. When monitoring the user's click event, the entire line of text area in the corresponding area can be drawn in the second color according to the different click positions to inform the user of the currently selected character contents. At the same time, according to actual needs, the user can adjust the selected character content by dragging in the character area drawn with the second color. The embodiment of the present application improves the operational convenience of character selection by realizing interaction with the user on the terminal interface.

[0026] In a second aspect, embodiments of the present application provide a character selection device that implements the terminal device behavior described in the first aspect. This functionality can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the aforementioned functionality.

[0027] In one possible implementation, the device includes a unit or module for executing each step of the first aspect above. For example, the device includes: a display module for displaying an original picture in a display interface of a terminal device, wherein the original picture includes curved text; a processing module for detecting the original picture, generating a picture to be identified including straight text, wherein the text content of the straight text corresponds one-to-one with the text content of the curved text; identifying the text content of the straight text according to the picture to be identified, obtaining a connection time series classification sequence corresponding to the text content of the straight text, wherein the connection time series classification sequence includes a plurality of characters; calculating the first coordinate of each of the plurality of characters in the picture to be identified; determining the position of each character in the picture to be identified; and The first coordinate corresponds to the segmented area where the first coordinate is located; according to the original picture and the picture to be identified, determining the perspective transformation matrix corresponding to each segmented area when the original picture is transformed into the picture to be identified; multiplying the first coordinate of each character in the multiple characters by the perspective transformation matrix to obtain the second coordinate of each character in the multiple characters in the original picture; detecting the first operation of the user on the original picture; then the display module is used to highlight the selected character according to the second coordinate of each character in the multiple characters in the original picture, and the first operation is used to select the character in the curved text on the original picture.

[0028] Optionally, the processing module is further used to generate first prompt information in the original image according to the second coordinate corresponding to each character in the multiple characters, and the first prompt information is used to indicate that the user can select a character in the original image.

[0029] Optionally, the processing module is specifically used to identify the text content of the straight text based on the picture to be identified, and obtain an initial connection time classification sequence; determine the length of the initial connection time classification sequence, and determine the picture width of the picture to be identified; if the product of the length of the initial connection time classification sequence and a preset downsampling multiple is greater than the picture width of the picture to be identified, then crop the initial connection time classification sequence to obtain a connection time classification sequence corresponding to the text content of the straight text; wherein, the product of the length of the connection time classification sequence obtained after cropping and the preset downsampling multiple is less than or equal to the picture width of the picture to be identified.

[0030] Optionally, the processing module is specifically used to sequentially crop the head element or tail element of the initial connection timing classification sequence; after cropping any head element or tail element, calculate whether the product of the length of the cropped initial connection timing classification sequence and a preset downsampling multiple is less than or equal to the image width of the image to be identified; if the product of the length of the cropped initial connection timing classification sequence and the preset downsampling multiple is less than or equal to the image width of the image to be identified, stop cropping and output the connection timing classification sequence corresponding to the text content of the straight text.

[0031] Optionally, the processing module is specifically used to determine the character boundary coordinates of each of the multiple characters in the connection time classification sequence, and the character boundary coordinates include left boundary coordinates and right boundary coordinates; according to the character boundary coordinates of each of the multiple characters, calculate the first coordinates of each of the multiple characters in the picture to be identified.

[0032] Optionally, the processing module is specifically used to obtain the original right boundary coordinate of any character in the connection timing classification sequence, and the original left boundary coordinate of the next character; calculate the average value of the original right boundary coordinate and the original left boundary coordinate; and determine the right boundary coordinate of the character and the left boundary coordinate of the next character based on the average value.

[0033] Optionally, the processing module is specifically used to respectively determine the first character type of the character and the second character type of the next character, the first character type and the second character type respectively having corresponding offsets; calculate the average value minus the offset corresponding to the first character type to obtain a first difference, and use the first difference as the right boundary coordinate of the character; calculate the average value plus the offset corresponding to the second character type to obtain a second sum, and use the second sum as the left boundary coordinate of the next character.

[0034] Optionally, the first coordinates of each character in the picture to be recognized include first vertex coordinates, second vertex coordinates, third vertex coordinates and fourth vertex coordinates; the processing module is specifically used to multiply the left boundary coordinates and right boundary coordinates of each character in the multiple characters by a preset downsampling multiple to obtain the first vertex coordinates and second vertex coordinates of each character in the multiple characters in the picture to be recognized; determine the third vertex coordinates and fourth vertex coordinates of each character in the picture to be recognized based on the first vertex coordinates, the second vertex coordinates and the picture height of the picture to be recognized.

[0035] Optionally, the processing module is specifically used to traverse the first coordinates of each character and the segmented areas of the image to be identified, and respectively determine the first segmented area where the first vertex coordinates are located, the second segmented area where the second vertex coordinates are located, the third segmented area where the third vertex coordinates are located, and the fourth segmented area where the fourth vertex coordinates are located.

[0036] Optionally, the processing module is specifically used to multiply the first vertex coordinates by the perspective transformation matrix of the first segmented area to obtain a third coordinate, multiply the second vertex coordinates by the perspective transformation matrix of the second segmented area to obtain a fourth coordinate, multiply the third vertex coordinates by the perspective transformation matrix of the third segmented area to obtain a fifth coordinate, and multiply the fourth vertex coordinates by the perspective transformation matrix of the fourth segmented area to obtain a sixth coordinate; wherein the third coordinate, the fourth coordinate, the fifth coordinate and the sixth coordinate serve as the second coordinate of the character in the original image.

[0037] Optionally, the processing module is specifically used to draw the character area of ​​each of the multiple characters in the original image as a first color according to the second coordinate corresponding to each of the multiple characters; or to draw a text box for the character area of ​​each of the multiple characters in the original image according to the second coordinate corresponding to each of the multiple characters.

[0038] Optionally, the processing module is further configured to, when a click event of the user on the character area is monitored, paint the entire row of character areas including the clicked character area in a second color; monitor a drag event of the user on the character area corresponding to the second color, and adjust the character area covered by the second color according to the drag event;

[0039] Recognize and display each character in the character area covered by the second color.

[0040] Optionally, the processing module is also used to, when monitoring a click event of the user in the character area, draw the character area closest to the click position indicated by the click event in a second color; monitor a drag event of the user in the character area corresponding to the second color, and adjust the character area covered by the second color according to the drag event; and identify and display each character in the character area covered by the second color.

[0041] Optionally, the processing module is also used to, when monitoring a sliding event of the user in the character area, draw the character area containing the sliding event indication in a second color; monitor a drag event of the user in the character area corresponding to the second color, and adjust the character area covered by the second color according to the drag event; and identify and display each character in the character area covered by the second color.

[0042] Optionally, a storage module is also included for storing necessary program instructions and data for the character selection device.

[0043] Optionally, an acquisition module is also included for acquiring the original image.

[0044] In one possible implementation, the apparatus includes a processor and a transceiver. The processor is configured to support the character selection apparatus in performing the corresponding functions of the method provided in the first aspect. The transceiver is configured to direct communication between the character selection apparatus and other devices in the communication system, such as transmitting the selected characters to other devices. Optionally, the apparatus may also include a memory coupled to the processor for storing program instructions and data necessary for the character selection apparatus.

[0045] In a possible implementation, when the character selection device is a chip within the character selection device, the chip includes: a processing module and a transceiver module. The transceiver module may be, for example, an input / output interface, a pin, or a circuit on the chip. The processing module may be, for example, a processor configured to detect the original image, generate an image to be recognized containing straight text, wherein the text content of the straight text corresponds one-to-one to the text content of the curved text; recognize the text content of the straight text according to the image to be recognized, and obtain a connection temporal classification sequence corresponding to the text content of the straight text, wherein the connection temporal classification sequence includes a plurality of characters; calculate the first coordinate of each of the plurality of characters in the image to be recognized; determine the segmented area where the first coordinate of each of the plurality of characters is located; determine the perspective transformation matrix corresponding to each segmented area when the original image is transformed into the image to be recognized according to the original image and the image to be recognized; multiply the first coordinate of each of the plurality of characters by the perspective transformation matrix to obtain the second coordinate of each of the plurality of characters in the original image; and detect a first operation of the user on the original image, wherein the first operation is used to select a character in the curved text on the original image. The processing module may execute computer-executable instructions stored in a storage unit to support the character selection device in performing the method provided by the first aspect. Optionally, the storage unit may be a storage unit within the chip, such as a register, a cache, etc., or a storage unit located outside the chip, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0046] In one possible implementation, the character selection device includes a processor, a radio frequency circuit, and an antenna. The processor is configured to control the functions of various circuit components and implement the method of the first aspect described above. The radio frequency circuit can perform analog conversion, filtering, amplification, and up-conversion on the information to be transmitted generated by the processor before transmitting it via the antenna to other devices in the communication system. Optionally, the device also includes a memory to store program instructions and data necessary for the character selection device.

[0047] In one possible implementation, the device includes a communication interface and a logic circuit, wherein the logic circuit is used to detect the original image and generate a picture to be identified containing straight text, wherein the text content of the straight text corresponds one-to-one to the text content of the curved text; based on the picture to be identified, the text content of the straight text is identified, and a connection time series classification sequence corresponding to the text content of the straight text is obtained, wherein the connection time series classification sequence includes multiple characters; calculate the first coordinate of each of the multiple characters in the picture to be identified; determine the segmented area where the first coordinate corresponding to each of the multiple characters is located; based on the original image and the picture to be identified, determine the perspective transformation matrix corresponding to each segmented area when the original image is transformed into the picture to be identified; multiply the first coordinate of each character in the multiple characters by the perspective transformation matrix to obtain the second coordinate of each character in the multiple characters in the original image; detect a first operation of the user on the original image, wherein the first operation is used to select a character in the curved text on the original image.

[0048] Among them, the processor mentioned in any of the above places can be a general-purpose central processing unit (CPU), a processor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of programs for the character selection method of curved text in the above aspects.

[0049] In a third aspect, an embodiment of the present application provides a terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for selecting characters of curved text as described in the first aspect above is implemented.

[0050] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor of a terminal device, it implements the character selection method for curved text described in the first aspect above.

[0051] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute the method for selecting characters of curved text as described in the first aspect above.

[0052] Some embodiments of the present invention may include one or more of the following benefits:

[0053] Some embodiments of the present application, by utilizing the existing Convolutional Neural Network (CNN) structure in the OCR detection and recognition model, establish a mapping relationship between the CTC sequence index output by the detection and recognition model and the original image coordinates based on the characteristics of CNN downsampling (8 times). The model used in the whole process is simple and the amount of data required to be annotated is small. During processing, compared with the existing technology that requires providing the original image or the corrected image, or requires traversing all pixels on the image, some embodiments of the present application can use real-time detected or collected images, with fast calculation speed, and can be quickly and conveniently adapted and deployed to the existing OCR model. Secondly, when determining the character boundaries, some embodiments of the present application can set hyperparameters according to different languages ​​to fine-tune the character boundaries to ensure the accuracy of character boundary recognition. It is not only applicable to most languages, but also supports the selection of symbols. Third, some embodiments of the present application record the four vertex coordinates of each character in the image to be identified, and map the four vertex coordinates to the original image according to the segmented area to obtain the coordinates of each character in the original image, so that the granularity of the coordinate output is finer, and when performing a manual selection operation, the recognition between text and text, text and symbols is accurate, and the segmentation of adjacent characters can be quickly achieved, which has stronger universality in practical applications. Fourth, thanks to the robustness of the OCR detection and recognition model, the character selection method provided by some embodiments of the present application is also more robust in positioning characters in natural scenes. Fifth, some embodiments of the present application associate the coordinates of each identified character with the coordinates of each character in the original image (also referred to as the second coordinates in this article for ease of distinction), and draw the corresponding identified text area according to the second coordinates, which can support users to make arbitrary selections and range adjustments in the above-mentioned identified text area. Sixth, the above-mentioned text recognition and selection scenarios can also be associated with functions such as translation and question scanning. After the user makes a selection, the selected text can be directly translated, or the recognized question can be answered; or, by parsing the character string selected by the user, if it contains text content in formats such as telephone and email, a separate card can be extracted to facilitate users to directly make calls or send emails, which is highly easy to use and practical. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1a to Figure 1bIt is a scene diagram of the text click selection scene;

[0055] Figures 2a to 2c This is a schematic diagram of the recognition result of a text point selection solution for curved text in the prior art;

[0056] Figure 3 This is a schematic diagram of the correspondence between a CTC sequence and text recognition or speech recognition results in the prior art;

[0057] Figure 4 A schematic diagram of the process and logical relationship between the OCR detection and recognition model and the character selection model in an embodiment of the present application;

[0058] Figure 5 Schematic diagram of a detection process for curved text in an embodiment of the present application;

[0059] Figure 6 A schematic diagram of a process flow for converting curved text into straight text in an embodiment of the present application;

[0060] Figure 7 This is a schematic diagram of an overall processing process for obtaining the coordinates of text on a picture to be recognized in an embodiment of the present application;

[0061] Figure 8 A schematic diagram of an algorithm flow for obtaining the coordinates of text on a picture to be recognized in an embodiment of the present application;

[0062] Figure 9 Schematic diagram of the hardware structure of a mobile phone to which the method for selecting characters for curved text in an embodiment of the present application is applicable;

[0063] Figure 10 Schematic diagram of the software structure of a mobile phone to which the method for selecting characters for curved text in an embodiment of the present application is applicable;

[0064] Figure 11 A schematic diagram of an embodiment of a method for selecting characters of curved text in an embodiment of the present application;

[0065] Figure 12 This is a schematic diagram showing the coordinates of each character on the image to be recognized according to an embodiment of the present application;

[0066] Figure 13 Schematic diagram of the effect of combining the coordinate display effect and the rectangular fragment on the image to be identified in an embodiment of the present application;

[0067] Figures 14a to 14b A schematic diagram of an embodiment of a character selection control provided in an embodiment of the present application;

[0068] Figure 15This is a schematic diagram of another embodiment of the method for selecting characters of curved text in an embodiment of the present application;

[0069] Figures 16a to 16c This is a schematic diagram showing an example effect of selecting a character in the method for selecting a curved text character according to an embodiment of the present application;

[0070] Figure 17 A schematic diagram of outputting a selected target character after the user selects the target character in an embodiment of the present application;

[0071] Figure 18 This is a schematic diagram of an interface for selecting multiple applications in an embodiment of the present application;

[0072] Figure 19 This is an exemplary structural block diagram of a character selection device in an embodiment of the present application;

[0073] Figure 20 This is another exemplary structural block diagram of the character selection device in an embodiment of the present application. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the following describes the embodiments of this application in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this application, rather than all the embodiments. It is known to those skilled in the art that with the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0075] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules clearly listed, but may include other steps or modules that are not clearly listed or that are inherent to these processes, methods, products or devices. The naming or numbering of steps in this application does not mean that the steps in the method flow must be executed in the time / logical sequence indicated by the naming or numbering. The process steps that have been named or numbered can be changed in the execution order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved. The division of units that appears in this application is a logical division. In actual application, there may be other division methods. For example, multiple units can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, and the indirect coupling or communication connection between units can be electrical or other similar forms, which are not limited in this application. In addition, the units or subunits described as separate components may or may not be physically separated, may or may not be physical units, or may be distributed into multiple circuit units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this application. The terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit this application. As used in the specification of this application and the appended claims, the singular expressions "a", "an", "said", "above", "the", and "the" are intended to also include expressions such as "one or more" unless the context clearly indicates otherwise. It should also be understood that in the embodiments of this application, "one or more" refers to one, two, or more than two; "and / or" describes the relationship between associated objects, indicating that three relationships can exist; for example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0076] For ease of understanding, the text selection scheme of the existing technology is first introduced: In the object recognition scene of mobile phone smart lenses (such as Google Lens, Baidu Smart Lens and Huawei HiVsion), only the lens can complete multi-object detection and text detection and recognition (OCR) tasks in the AR scene, and will provide the location of objects and text boxes. Users can click on the text line on the mobile phone screen. At this time, the screen will be framed and users can click on the detected and recognized text content. The effect is similar to the function of dragging the text content character by character with a mouse in a text document. For the selected content, the user can choose to copy, translate or search and other subsequent operations. The text selection scene is divided into straight text scenes (such as Figure 1a ) and curved text scenes (as shown in Figure 1b As shown). In straight text scenes, text lines are all angled rectangular boxes, parallelogram boxes or quadrilateral boxes, and text lines are generally described by four vertices. In curved text scenes, in order to describe the direction of the curve, the text box is generally composed of a polygon, which can be decomposed into many quadrilaterals, and these quadrilaterals are spliced ​​into the polygon. In straight text scenes, since there is only a simple perspective transformation mapping relationship between the original image and the image to be identified, if the coordinates of each text on the image to be identified are obtained, the coordinates of each text in the straight text on the original image can be easily obtained. However, in curved text scenes, due to the strong discontinuity and the more complex description of the curved text itself, obtaining the coordinates of each text is much more difficult than in straight text scenes.

[0077] like Figure 2a-2c The figure shows a schematic diagram of the recognition result of a text selection scheme in the prior art. However, this selection scheme has the following defects:

[0078] 1) There is serious overlap in the detection frames;

[0079] 2) The detection boxes are all stitching of straight text boxes;

[0080] 3) The highlighted areas in some scenes are incorrect, which can mislead users;

[0081] 4) Inaccurate text segmentation;

[0082] 5) Poor user visual effects;

[0083] 6) Poor user clicking experience.

[0084] The first three points show that when dealing with curved text, the logic of straight text is essentially used. This processing method cannot describe curved text well. There will inevitably be serious jagged and discontinuous phenomena at the intersection of straight text. Secondly, semantically coherent text will be cut off at the intersection, which will seriously affect the user experience. For example, in Figure 2aIn the results shown, since the curved text box is described by the logic of the straight text box, when displaying the text box, there are serious jagged edges in the text box. If the selected characters are located in the non-repeated area of the same straight line box, the recognized result is exactly the selected characters. For example, Figure 2a In the "技术" characters selected in Figure 2a , the recognized result is also the two characters "技术". If the selected characters are located within multiple straight text boxes, it is possible that the recognized text is split at the junction of semantically coherent texts. For example, Figure 2b The selected text in Figure 2b is "技术有限公司消费", and the recognized result is "技术合采有限公司公司消费". If we want to accurately recognize the result, we can expand the highlighted area according to the selected characters. For example, Figure 2c As shown in Figure 2c , the selected characters are "公司消费者BG智慧". At this time, the highlighted area can be the larger rectangle in Figure 2c , but the visual effect for the user is poor.

[0085] Therefore, in order to solve the problems existing in the above various selection schemes and improve the efficiency and accuracy of text selection, the embodiments of this application provide a method for character selection of curved text based on the CTC sequence output by the OCR recognition model.

[0086] The method for character selection of curved text provided by the embodiments of this application mainly uses the CTC sequence output after being recognized by the OCR recognition model. Based on the output CTC sequence, each character is obtained for the user to manually select character by character on the screen.

[0087] Connectionist Temporal Classification is a method for calculating the loss degree (Loss), mainly applied in sequence recognition models such as text recognition and speech recognition. Most sequence recognition models adopt the structure of CNN + Recurrent Neural Network (RNN) + CTC. Using CTC Loss instead of the loss degree calculation method based on logistic regression (Softmax Loss), the training samples do not need to be aligned. CTC Loss generally has the following two characteristics:

[0088] 1) Introduce a blank character to solve the problem that there are no characters in some positions;

[0089] 2) Calculate the gradient quickly through a recursive algorithm.

[0090] As Figure 3 shown, it is a schematic diagram of the correspondence between a CTC sequence and the result of text recognition or speech recognition in the prior art. In Figure 3 , the symbol "∈" represents blank, which is used as a placeholder for distinguishing repeated characters. That is, the repeated characters between two blanks will be merged into one character. For example, for Figure 3The corresponding text of the speech sequence input in is "wwo∈rrr∈lld". After merging the repeated "w" and "r" between the two blanks, the final recognition result, that is, the word "world", can be output.

[0091] like Figure 4 As shown, it is a schematic diagram of the process and logical relationship between the OCR detection and recognition model and the character selection model provided by an embodiment of the present application. Among them, the OCR detection model can detect the input picture (that is, the original picture containing curved text) to obtain a polygonal curved text box. In the correction process, the model converts the curved text box into a picture to be recognized containing straight text (the straight text corresponds one-to-one to the text content of the curved text in the original picture) through piecewise perspective transformation. The above-mentioned input picture can be an image taken or scanned by a mobile phone lens, scanner or other device, and the picture to be recognized can be regarded as input data for processing by the OCR recognition model. After using the CNN network to extract features from the picture to be recognized, the OCR recognition model can output a CTC sequence containing the recognized characters.

[0092] Optionally, during the text box detection process, the processing flow diagram of the curved text box can be as follows: Figure 5 As shown. Among them, Figure 5 a in the image is the curved text in the original image; then follow Figure 5 The center line is fitted as shown in b; further as Figure 5 Fit the text box and rotation angle as shown in c in the figure; after fitting the text box, follow Figure 5 As shown in d, the redundant boxes are removed by non-maximum suppression and smoothed to obtain Figure 5 The curved text box shown in Figure e.

[0093] The specific process of converting the curved text in the original image into the straight text in the image to be recognized through perspective transformation can be as follows: Figure 6 As shown. Among them, Figure 6 The a in the image is the curved text in the original image, and then the curved text detection is performed to obtain the following Figure 6 The curved text box shown in b; Figure 6 The c in the figure indicates that the curved text box is composed of multiple quadrilaterals, and the dotted line is the intersection between the quadrilaterals. Figure 6 The straight text box shown in d.

[0094] It should be noted that in the prior art, most of the OCR-based detection and recognition models use CNN+RNN+CTC for sequence recognition. Among them, CNN is used for feature extraction, and RNN+CTC is used for sequence recognition. The recognition model corresponding to the character selection method provided in the embodiment of the present application only uses CNN as the network structure, downsampling 8 times and then sliding the window directly on the last layer of feature map to obtain the CTC sequence. The entire recognition model does not involve any RNN structure.

[0095] The CTC sequence output by the OCR detection and recognition model can be used as the input sequence of the character selection model of an embodiment of the present application. By determining the index coordinates of different characters in the CTC sequence, and then mapping the index coordinates to the image to be recognized to obtain the coordinates of the image to be recognized, and then mapping the coordinates on the image to be recognized to the original image through piecewise inverse perspective transformation, the coordinates of each character in the original image can be obtained for the user to select character by character.

[0096] For example, a user uses a mobile phone camera to detect an image and manually selects the recognized text on the real-time mobile phone screen. The user can open the mobile phone camera and control the lens to focus on a certain image or object. Because the user may shake the phone while holding it, the image displayed on the screen may also shake. Therefore, the user can fix the image by clicking a certain position on the screen. At the moment the user clicks the screen, the image captured by the camera in real time, that is, the image displayed on the screen at that moment, is the input image that needs to be input into the OCR detection and recognition model.

[0097] See also Figure 7 and Figure 8 , which are respectively a schematic diagram of the overall processing process for obtaining the coordinates of the text on the image to be recognized and a schematic diagram of the algorithm flow provided by the embodiment of the present application. For the above-mentioned input image to be recognized, the OCR detection and recognition model can automatically detect the text area in the image, and adopt the CNN network as the network structure of the OCR recognition model. It can directly slide the window on the last layer feature map of CNN after downsampling by 8 times, and output the CTC sequence of the detection and recognition model. After cropping the above sequence, the character selection model can obtain a CTC sequence that meets the subsequent processing requirements, that is, Figure 7 The sequence in [bl,bl,company,bl,bl,bl,book,bl,bl,bl,bl,meeting,bl,bl,bl,bl,design,bl,bl,bl,person,bl,bl,bl,member,bl,bl].

[0098] according to Figure 8 The algorithm flow shown in the figure, the character selection model can be used for Figure 7The CTC sequence in the image is used to calculate the index coordinates of different characters, namely, the index coordinates of each character in the CTC sequence of "cost accountant", namely, {[2, 3], [6, 7], [10, 11], [14, 15], [18, 19], [22, 23]}. These index coordinates are then mapped back to the image to be recognized (i.e., the image to be recognized obtained by input image detection), obtaining the coordinates of each character in the image to be recognized, namely, the left and right boundaries of each character. Since the character selection model has already calculated the left and right boundaries of each character in the image to be recognized, combined with the fixed height of the image to be recognized, the character selection model can obtain the coordinates of the top, bottom, left, and right points of each character in the image to be recognized. On the other hand, after obtaining the coordinates of the characters in the above manner (which can also be understood as when the image display interface is in a fixed frame state), each recognized text box is drawn in a first color (e.g., a light highlighted background color). Then, after the terminal device detects a click event caused by the user clicking in the text box area, it draws the selected text in a second color (e.g., a dark highlighted background color). The character selection model can monitor the user's click events on the mobile phone screen. In one possible implementation, when the user clicks on the character area, by calculating the current clicked row position, the entire line of text clicked can be set to the selected state, and the line of text can be drawn with a dark highlighted background. At the same time, draggable handles can appear at the beginning and end of the entire line of text, and the user can modify the selected text range by dragging the handles. In another possible implementation, when the user clicks on the character area, by calculating the current clicked position, the character closest to the clicked position is selected, the character is drawn with a dark highlighted background, and then draggable handles appear on the left and right boundaries of the character, and the user can modify the selected text range by dragging the handles. In another possible implementation, the terminal device monitors the user's moving area, and by calculating the starting and ending positions of the moving area, the area contained in it is set to the selected state, and the area is drawn with a dark highlighted background. At the same time, draggable handles appear at the beginning and end of the area, and the user can modify the selected text range by dragging the handles.

[0099] If the user drags one of the handles while dragging, the character selection model can obtain the user's click location from the click event generated by the user dragging the handle and use the character closest to that location as the currently selected character. By repeating the above steps based on the user's dragging behavior, the user's final selected text area is obtained and a dark background is drawn.

[0100] After the user stops dragging the handle, for example, the system can wait for one second. If no user click event is detected, it can be assumed that the text currently selected by the user has met expectations. The character selection model can then pop up a text card on the phone screen to display the text selected by the user. At the same time, other action buttons can also be provided on the text card. For the displayed text, the user can click different buttons to instruct the corresponding application to perform corresponding actions based on the displayed text. For example, the displayed text can be translated, searched, and so on.

[0101] The character selection method for curved text of the present application is introduced below with reference to specific embodiments.

[0102] The method for selecting characters for curved text provided in the embodiments of the present application can be applied to electronic devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

[0103] Taking the electronic device as a mobile phone as an example, it should be understood that the electronic device 900 may have more or fewer components than shown in the figure, may combine two or more components, or may have a different component configuration. The various components shown in the figure may be implemented in hardware, including one or more signal processing and / or application-specific integrated circuits, software, or a combination of hardware and software.

[0104] The electronic device 900 may include: a processor 910, an external memory interface 920, an internal memory 921, a universal serial bus (USB) interface 930, a charging management module 940, a power management module 941, a battery 942, an antenna 1, an antenna 2, a mobile communication module 950, a wireless communication module 960, an audio module 970, a speaker 970A, a receiver 970B, a microphone 970C, an earphone interface 970D, a sensor module 980, a button 990, a motor 991, an indicator 992, a camera 993, a display screen 994, and a subscriber identification module (SIM) card interface 995, etc. The sensor module 980 may include a pressure sensor 980A, a gyroscope sensor 980B, an air pressure sensor 980C, a magnetic sensor 980D, an acceleration sensor 980E, a distance sensor 980F, a proximity light sensor 980G, a fingerprint sensor 980H, a temperature sensor 980J, a touch sensor 980K, an ambient light sensor 980L, a bone conduction sensor 980M, etc.

[0105] It should be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 900. In other embodiments of the present application, the electronic device 900 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0106] The processor 970 may include one or more processing units. For example, the processor 910 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0107] The controller may be the nerve center and command center of the electronic device 900. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0108] Processor 910 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 910 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 910. If processor 910 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 910 latency, and thus improves system efficiency.

[0109] In some embodiments, the processor 910 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0110] Electronic device 900 implements display functionality through a GPU, display screen 994, and an application processor. The GPU is a microprocessor for image processing that connects display screen 994 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 910 may include one or more GPUs that execute program instructions to generate or modify display information.

[0111] Display screen 994 is used to display images, videos, and the like. Display screen 994 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLED, or a quantum dot light-emitting diode (QLED). In some embodiments, electronic device 900 may include one or N display screens 994, where N is a positive integer greater than one.

[0112] The electronic device 900 can implement a shooting function through an ISP, a camera 993, a video codec, a GPU, a display screen 994, and an application processor.

[0113] The ISP processes data fed back by the camera 993. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within the camera 993.

[0114] The camera 993 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 900 may include 1 or N cameras 993, where N is a positive integer greater than 1.

[0115] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU enables intelligent cognitive applications in electronic device 900, such as image recognition, face recognition, voice recognition, and text comprehension.

[0116] Figure 10 4 is a software structure block diagram of the electronic device 900 according to an embodiment of the present invention.

[0117] A layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0118] The application layer can include a series of application packages.

[0119] like Figure 10 As shown, the application package may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications (also referred to as applications).

[0120] The application framework layer provides an application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0121] like Figure 10 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, a local profile assistant (LPA), and the like.

[0122] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0123] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0124] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.

[0125] The phone manager is used to provide communication functions of the electronic device 900, such as management of call status (including answering, hanging up, etc.).

[0126] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0127] The Notification Manager allows applications to display notifications in the status bar. These messages can be displayed briefly and then disappear automatically, without requiring user interaction. For example, the Notification Manager can be used to notify users of completed downloads, message reminders, and so on. The Notification Manager can also display notifications in the top status bar of the system as icons or scrolling text, such as notifications from background applications, or as dialog interfaces on the screen. Examples include text messages in the status bar, beeps, vibrations on electronic devices, and flashing indicator lights.

[0128] The Android Runtime consists of the core library and the virtual machine. The Android runtime is responsible for scheduling and management of the Android system.

[0129] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0130] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0131] The system library can include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0132] The surface manager is used to manage the display subsystem and provide the fusion of two-dimensional (2D) and three-dimensional (3D) layers for multiple applications.

[0133] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0134] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0135] A 2D graphics engine is a drawing engine for 2D drawings.

[0136] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, sensor driver, and virtual card driver.

[0137] The following describes the workflow of the software and hardware of the electronic device 900 in conjunction with capturing a photo scene.

[0138] When the touch sensor 980K receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, touch operation timestamp and other information). The raw input event is stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer and identifies the control corresponding to the input event. For example, if the touch operation is a touch single-click operation and the control corresponding to the single-click operation is the control of the camera application icon, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer to capture a still image or video through the camera 993.

[0139] The following embodiments can be implemented on a mobile phone 900 having the above hardware structure / software structure. The following embodiments will take the mobile phone 900 as an example to illustrate the character selection method based on character recognition provided by the embodiments of the present application.

[0140] Reference Figure 11 , shows a schematic flowchart of the steps of a method for selecting characters in curved text provided by an embodiment of the present application. As an example and not a limitation, the method can be applied to the above-mentioned mobile phone 900. The method may specifically include the following steps:

[0141] 1101. Detect the original image and generate an image to be recognized containing straight text. The original image includes curved text, and the text content of the straight text corresponds to the text content of the curved text.

[0142] It should be noted that this embodiment is aimed at processing the CTC sequence output by the OCR detection and recognition model, so as to present prompt information for the user to manually select on the screen, to introduce the character selection method of the curved text embodiment of the present application, that is, this embodiment describes Figure 4 The character selection model in receives the input CTC sequence and processes the input CTC sequence.

[0143] In the embodiment of the present application, the image to be recognized can be obtained by detecting an original image. The original image can be an image displayed in real time on the phone screen after the user opens the phone camera, and the image contains the text content to be recognized, which is curved text. The image to be recognized is the image after the curved text is converted into straight text.

[0144] 1102. Identify the text content and obtain a CTC sequence corresponding to the text content.

[0145] In an embodiment of the present application, the CTC sequence corresponding to the text content may refer to a sequence after preliminary processing of the above-mentioned input CTC sequence. The CTC sequence is the intermediate output result of the OCR recognition model. After the recognition model outputs the CTC sequence, the final recognition result can be obtained by parsing the CTC sequence through certain rules. Any CTC sequence can contain multiple CTC sequence elements. In this application, the element obtained by the sequence to be recognized through the last layer feature map of CNN through a 4×4 convolution kernel sliding window may be a character of the sequence to be recognized, or blank; the value of the element may be a recognized character or an index of a character.

[0146] The preliminary processing in this embodiment may include processing processes such as cropping the input CTC sequence and initializing parameters.

[0147] In a specific implementation, the input CTC sequence can be cropped based on the CNN downsampling factor in subsequent processing, so that the product of the cropped CTC sequence and the downsampling factor is no larger than the image width of the image to be recognized. When cropping, the characters at the beginning of the input CTC sequence can be cropped first, then the characters at the end, and finally the characters at the beginning.

[0148] After each character is cropped, calculate whether the product of the remaining CTC sequence length and the CNN downsampling factor is less than or equal to the image width of the image to be recognized. If the product is less than or equal to the image width of the image to be recognized, stop cropping and proceed to the next step; otherwise, continue cropping the next character as above.

[0149] For example, suppose the input CTC sequence in the image to be recognized is:

[0150] [bl, bl, bl, into, bl, bl, bl, this, bl, bl, bl, will, bl, bl, bl, count, bl, bl, bl, person, bl, bl, bl, member, bl, bl]

[0151] Where bl is a blank character (blank), and the size of the image to be recognized is 32*199 (height*width) pixels.

[0152] 1. Crop the input CTC sequence

[0153] a) Rule: The CTC sequence length (26) multiplied by the CNN downsampling ratio (8) is not greater than the image width (199);

[0154] b) Specific steps: Cut in the order of cutting the head first, cutting the tail second, and then cutting the head again;

[0155] c) If any of the above steps meets the conditions in a), then return.

[0156] For the input CTC sequence in the example, the following cropping process can be obtained:

[0157] [bl, bl, bl, into, bl, bl, bl, this, bl, bl, bl, will, bl, bl, bl, count, bl, bl, bl, person, bl, bl, bl, member, bl, bl]

[0158] ↓(Head cutting)

[0159] [bl, bl, into, bl, bl, bl, this, bl, bl, bl, will, bl, bl, bl, count, bl, bl, bl, person, bl, bl, bl, member, bl, bl]

[0160] ↓(tail cut)

[0161] [bl, bl, into, bl, bl, bl, this, bl, bl, bl, will, bl, bl, bl, count, bl, bl, bl, person, bl, bl, bl, member, bl]

[0162] After trimming the head and tail, if the condition is met (remaining sequence length 24*downsampling multiple 8=192<image width 199), the current CTC sequence is returned.

[0163] Initializing parameters of the pruned CTC sequence may include:

[0164] a) Initialize the left boundary (-1), right boundary (-1), current state (0), and previous state (blank);

[0165] b) Initialize the coordinate array and content array, both of which are empty initially.

[0166] The character boundaries can include a left boundary and a right boundary, that is, the positions of the left and right ends of each element or character in the CTC sequence. The character boundaries can be divided into boundaries in the CTC sequence and boundaries in the original image. What needs to be initialized in this step is the boundary in the CTC sequence.

[0167] The current state can refer to the type of each element. For example, numbers, Chinese characters, Japanese characters, Korean characters, or English, etc.

[0168] The coordinate array can refer to an array storing all character index coordinates, and the content array is an array storing the content of each character or word. During initialization, both the coordinate array and the content array can be set to empty.

[0169] 1103. Determine the character boundary coordinates of each character in the CTC sequence.

[0170] In the embodiments of the present application, the left and right boundaries of a character of an element can be obtained by traversing each element in the CTC sequence.

[0171] For example, for the following CTC sequence:

[0172] [bl, bl, 成, bl, bl, bl, 本, bl, bl, bl, 会, bl, bl, bl, 计, bl, bl, bl, 人, bl, bl, bl, 员, bl]

[0173] First, the first character "成" can be processed, recording the right boundary of this character and the left boundary of the next character "本". In the above manner, after processing all characters, the array index ranges corresponding to each different character are obtained.

[0174] After initially determining the left and right boundaries of characters or words, since the processing rules of the CTC sequences output by the detection and recognition model for different languages are inconsistent, it may be necessary to fine-tune the left and right boundaries for different languages. Therefore, in the embodiments of the present application, for different types of characters, their boundaries can also be fine-tuned according to certain rules.

[0175] When fine-tuning the character boundaries, the following steps can be carried out:

[0176] a) First, calculate the mean of the coordinates of the left and right boundaries of adjacent characters;

[0177] b) Subtract or add a certain offset from the mean to obtain the new left and right boundaries of each character.

[0178] In a specific implementation, for two adjacent characters, the average value of the coordinates of the left and right boundaries can be calculated first based on the coordinates of the right boundary of the previous character and the coordinates of the left boundary of the next character. For example, in the CTC sequence of the above example, for the adjacent characters "成" and "本", the average value of their coordinates can be calculated based on the coordinates of the right boundary of the character "成" and the left boundary of the character "本".

[0179] Then, by subtracting or adding a certain offset based on the above average coordinate value, the new left and right boundaries of each character can be obtained. Among them, for the previous character of two adjacent characters, the calculated average coordinate value can be subtracted by the above offset to obtain the new right boundary coordinate of the previous character; for the next character, the average coordinate value can be added by the above offset to obtain the new left boundary coordinate of the next character.

[0180] In the embodiments of the present application, for characters of different languages or types, the offset offset can be determined according to the following rules:

[0181] For Chinese, Japanese or Korean, the offset offset can be 1 / 8; for Western languages (Latin), the offset offset can be 1 / 2; for numbers, the offset offset can also be 1 / 2.

[0182] Therefore, after fine-tuning the left and right boundaries of each character according to the language, for the CTC sequence of the above example, the following coordinate results can be obtained:

[0183] [(1.0, 4.375), (4.625, 8.375), (8.625, 12.375), (12.625, 16.375), (16.625, 20.375), (20.625, 23)]

[0184] The above coordinate results are the coordinates of the six characters "成本会计人员" in the CTC sequence.

[0185] 1104. Calculate the first coordinate of each character in the to-be-recognized picture according to the character boundary coordinates of each character.

[0186] Generally, OCR recognition takes an input size of 32 * 512 pixels, the output feature map size is 4 * 65 pixels, the downsampling factor is 8, and finally when performing recognition, a 4 * 4 pixel convolutional kernel is used for sliding window, that is, one element in the CTC sequence corresponds to the Figure 8 range covered by the width of one pixel.

[0187] Therefore, in the embodiments of the present application, multiplying the character boundaries of each character calculated in the previous step by the CNN downsampling factor can obtain the coordinates of each character in the image to be recognized.

[0188] For example, for the following coordinate results:

[0189] [(1.0, 4.375), (4.625, 8.375), (8.625, 12.375), (12.625, 16.375), (16.625, 20.375), (20.625, 23)]

[0190] After multiplying the left and right boundary coordinates of each character by the downsampling factor 8, the coordinates of each character in the image to be recognized can be obtained, and the unit of these coordinates is (one) pixel:

[0191] [(8, 35.0), (37.0, 67.0), (69.0, 99.0), (101.0, 131.0), (133.0, 163.0), (165.0, 184)]

[0192] Generally, if the display is performed according to the above coordinates, there may be some pixels of the first and last characters in each row sequence outside the pixel points corresponding to the coordinates. For example, for the first character "成", if it is displayed according to the left boundary coordinate 8, some pixels of this character may be to the left of coordinate 8; similarly, for the last character "员", if it is displayed according to the right boundary coordinate 184, some pixels of this character may be to the right of coordinate 184.

[0193] Therefore, in order to ensure that the area corresponding to the finally mapped coordinates can completely cover the pixel points of each character, the coordinates of the first and last characters can also be fine-tuned.

[0194] In a specific implementation, the corresponding fine-tuning can be the left boundary coordinate of the first character in the image to be recognized and the right boundary coordinate of the last character in the image to be recognized. The final coordinates of the left and right boundaries of each character in the image to be recognized after fine-tuning the coordinates of the first and last characters can be:

[0195] [(2, 35.0), (37.0, 67.0), (69.0, 99.0), (101.0, 131.0), (133.0, 163.0), (165.0, 197)]

[0196] That is, subtract a certain value from the left boundary coordinate of the first character and add a certain value to the right boundary coordinate of the last character. The value subtracted or added above can be the same or different, and this embodiment does not limit this.

[0197] The coordinates obtained above are only the left and right boundary coordinates of each character in the image to be recognized. Therefore, in the subsequent processing, the height of the image to be recognized can be combined to obtain the coordinates of the four vertices of each character, that is, the coordinates of the upper left, lower left, upper right, and lower right vertices of each character.

[0198] Since the OCR recognition uses a fixed size of 32 * 512 pixels as the input data, its height can be fixed at 32 pixels. Therefore, for the first character "成" in the above example, the coordinates of its four vertices can be expressed as [(2, 32), (2, 0), (35.0, 0), (35.0, 32)].

[0199] As Figure 12 shown, it is a schematic diagram of the display effect of the coordinates of each character on the image to be recognized.

[0200] 1105. Determine the segmented area where the first coordinate is located.

[0201] As Figure 6 shown, when the curved text generates straight text through segmented perspective transformation, there is a corresponding rectangular segment for each character. As Figure 13 shown in the schematic diagram of the rectangular demarcation line, the vertex coordinates of each character do not necessarily lie within the same rectangular segment. For example, the four vertex coordinates of "成" are all within the first rectangular segment. However, the lower left and upper left of "会" are within the third rectangular segment, while the lower right and upper right are within the fourth rectangular segment.

[0202] 1106. Determine the perspective transformation matrix corresponding to each segmented area when transforming the original image into the image to be recognized according to the original image and the image to be recognized.

[0203] 1107. Multiply the first coordinate of each character by the perspective transformation matrix to obtain the second coordinate of each character in the original image.

[0204] For each vertex, according to the perspective transformation relationship corresponding to the small rectangle where it is located, multiply by the inverse perspective transformation matrix to obtain the coordinates of each vertex of the character in the bending detection frame of the original image.

[0205] 1108. Generate a character selection control in the original image according to the second coordinate.

[0206] In the embodiment of the present application, for the second coordinates of each character in each text area of the recognized original image, a character selection control can be generated in the original image according to the second coordinate. The above character selection control can draw the character rows in each text area as a highlighted background to inform the user of the recognized text range.

[0207] On the other hand, you can also monitor user click events on the phone screen, that is, on the original image displayed on the phone screen. When the user clicks on a character area, by calculating the current clicked line position, you can set the entire clicked line of text to a selected state and draw this line of text with a dark, highlighted background. At the same time, draggable handles can appear at the beginning and end of the entire line of text, allowing users to modify the selected text range by dragging the handles.

[0208] like Figure 14a and Figure 14b As shown in FIG. 1 , a schematic diagram of a character selection control of an embodiment of the present application is shown. For the recognized text range, the background of each line of characters can be drawn as a unit of character line. Figure 14a When a user clicks within a certain line of characters, you can Figure 14b As shown, a draggable handle is drawn at the beginning and end of the clicked row of characters, and the row of characters is drawn as Figure 14a A different color to inform the user that they can select characters in the current line.

[0209] The embodiment of the present application, by utilizing the existing Convolutional Neural Network (CNN) structure in the OCR detection and recognition model, establishes a mapping relationship between the CTC sequence index output by the detection and recognition model and the original image coordinates according to the characteristics of CNN downsampling (8 times). The model used in the whole process is simple and the amount of data required to be annotated is small. During processing, compared with the prior art that requires providing the original image or the corrected image, or requires traversing all pixels on the image, the embodiment of the present application can use the real-time detected or collected image, with fast calculation speed, and can be quickly and conveniently adapted and deployed to the existing OCR model. Secondly, when determining the character boundary, the embodiment of the present application can fine-tune the character boundary according to the hyperparameters set in different languages ​​to ensure the accuracy of character boundary recognition. It is not only applicable to most languages, but also supports the selection of symbols. Third, the embodiment of the present application records the four vertex coordinates of each character in the image to be recognized, and maps the four vertex coordinates to the original image according to the segmented area to obtain the coordinates of each character in the original image, so that the granularity of the coordinate output is finer. When performing manual selection operations, the recognition between text and text, text and symbols is accurate, and the segmentation of adjacent characters can be quickly achieved, which has stronger universality in practical applications. Fourth, thanks to the robustness of the OCR detection and recognition model, the character selection method provided by the embodiment of the present application is also more robust in positioning characters in natural scenes. Fifth, the embodiment of the present application associates the coordinates of each recognized character with the second coordinates of each character in the original image, and draws the corresponding recognized text area according to the second coordinate, which can support users to make arbitrary selections and range adjustments in the above-mentioned recognized text area. Sixth, the above-mentioned text recognition and selection scenarios can also be associated with functions such as translation and question scanning. After the user makes a selection, the selected text can be directly translated, or the recognized question can be answered; or, by parsing the character string selected by the user, if it contains text content in formats such as telephone and email, a separate card can be extracted to facilitate users to directly make calls or send emails, which is highly easy to use and practical.

[0210] Reference Figure 15 , shows a schematic flowchart of the steps of a method for selecting characters of curved text provided by another embodiment of the present application. The method may specifically include the following steps:

[0211] 1501. Control a terminal device to collect an original image, where the original image contains curved text to be recognized.

[0212] It should be noted that this embodiment introduces the character selection method of the embodiment of the present application from the perspective of the interaction between the user and the mobile phone, that is, this embodiment describes the interaction process between the user and the mobile phone screen when the user uses the mobile phone lens to perform text selection operations.

[0213] The terminal device in this embodiment can be a mobile phone, a tablet computer, a smart lens or other devices. This embodiment does not limit the specific type of the terminal device.

[0214] Take the terminal device as a mobile phone as an example. The user can turn on the mobile phone camera and control the lens to aim at a certain picture or object to shoot. Usually, since the user may shake when holding the mobile phone, the image presented on the screen may also shake, so the user can fix the image by clicking a certain position on the screen, that is, framing it. At the moment the user clicks the screen, the image captured by the camera in real time, that is, the image presented on the screen at that moment, is the original picture that needs to be input into the OCR detection and recognition model. The original picture contains curved text content to be recognized, and the above-mentioned text area can be one or more lines. Each line of text area may contain different types of characters or symbols, which is not limited in this embodiment.

[0215] 1502. Determine a second coordinate of each character in the curved text content in the original image;

[0216] In this embodiment of the present application, for an original image obtained by framing, the OCR detection and recognition model can automatically identify each text region in the image and output a CTC sequence corresponding to each line of characters or symbols in each text region. This CTC sequence can be provided to the character selection model for processing to obtain the second coordinates of each character in the original image.

[0217] Since the OCR recognition model is used to recognize the characters in the text area and output the corresponding CTC sequence, the character selection model provided in this embodiment is then used to process the CTC sequence. The coordinates of each character in the CTC sequence are calculated and mapped to the second coordinates of each character in the original image. This process is similar to steps 1101-1108 in the aforementioned embodiment and can be referenced to each other. This embodiment will not repeat this process.

[0218] 1503. Generate a character selection control for each character in the original image according to the second coordinates;

[0219] In an embodiment of the present application, a character selection control can be generated in the original image based on the second coordinates of each character in each text region in the original image. The character selection control can be configured to highlight the character rows in each text region with a first color background to inform the user of the currently recognized text range.

[0220] In the specific implementation, for each character row corresponding to the CTC sequence, a highlighted background of the first color covering all the characters in the entire row can be drawn based on the vertex coordinates of each character in the row that have been obtained; or, a highlighted background covering the character row can be drawn based on the vertex coordinates of the first and last two characters in the row.

[0221] Draw the character selection control according to the second coordinate of each character in the original image, see Figure 14a shown.

[0222] 1504. When a selection event for a target character is monitored, output the selected target character, where the target character is any one or more characters in the text area.

[0223] In the embodiment of the present application, the selection event for the target character may refer to the event generated when the user clicks on a text area on the mobile phone screen where a character selection control is drawn. Figure 14a A click operation is performed within any text area that has a highlighted background drawn in .

[0224] If the user clicks on the text area, the current clicked row position can be calculated first, and then the clicked text is set to selected state and the text background of the second color is drawn. Figure 14b As shown, the terminal device monitors the user's movement area, calculates the starting and ending positions of the movement area, sets the area contained in it to the selected state, and draws the area with a dark, highlighted background. At the same time, draggable handles appear at the beginning and end of the area, and the user can modify the selected text range by dragging the handles. Of course, when drawing the background of the clicked text, the background color to be drawn can be determined according to actual needs, and this embodiment does not limit this.

[0225] On the other hand, corresponding draggable handle symbols may be generated at the beginning and end of the clicked text to prompt the user that the selected range can be modified by dragging the handle.

[0226] If the user clicks on one of the handles and drags it, the selected text range can be re-determined based on the position of the handle being dragged by the user.

[0227] For example, for a draggable handle that is initially at the head or tail position, if the user drags the left handle, the position of the dragged left handle can be recognized, and the position between the dragged left handle and the undragged right handle can be used as the reproduced and determined selected text range. That is, from the right side of the dragged left handle to the last character of the line is the selected text range.

[0228] Alternatively, if the user drags the right handle, the position of the dragged right handle can be recognized, and the position between the dragged right handle and the undragged left handle can be used as the reproduced and determined selected text range. That is, from the first character of the line to the character to the left of the dragged right handle is the selected text range.

[0229] Of course, the user can also drag the left handle and the right handle successively. At this time, the re-determined text range is all the characters between the first character to the right of the current position of the dragged left handle and the first character to the left of the current position of the dragged right handle.

[0230] The user's action of dragging the left or right handle can be repeated until the user believes that the currently selected text meets the expectation. After the user finishes the selection, a text card can be popped up on the mobile phone interface to display the selected text content.

[0231] In a specific implementation, whether the user has completed the selection operation can be determined after the user drags a certain handle and waits for a certain period of time, such as 1 second. If the action of the user dragging a certain handle is not monitored again, it can be considered that the user has completed the selection operation.

[0232] As Figure 16a-Figure 16c shown, it is a schematic diagram of the example effect when selecting characters using the above character selection method. In Figure 16a it is shown that the selected characters are "cost will". The user can drag the right handle and move it to the left of the character "will", presenting the effect as Figure 16b shown. At this time, the selected characters are "cost". If the user drags the right handle to the right and moves it to the right of the character "accounting", the effect as Figure 16c shown can be obtained. At this time, the selected characters are "cost accounting".

[0233] In the embodiment of the present application, other operation buttons can also be provided on the text card. For the displayed text, the user can click different buttons to instruct the application corresponding to the button to perform corresponding operations according to the displayed text.

[0234] As Figure 17 shown, it is a schematic diagram of outputting the selected target characters after the user selects the target characters. The above selected target characters can be Figure 14b or Figure 16a-Figure 16c obtained by dragging the left and right handles of the character selection control in Figure 16a-Figure 16c . For example, based on Figure 16c , the user can drag the left handle and move it to the left of the character "计", so as to obtain a selection effect as shown in Figure 17 , and the currently selected target character is "成本会". In the mobile phone interface shown in Figure 17 , for the output selected target characters, the user can click the corresponding button to perform operations such as translating and searching on the above target characters. Figure 16c Based on Figure 16c , the user can drag the left handle and move it to the left of the character "计", so as to obtain a selection effect as shown in Figure 17 , and the currently selected target character is "成本会". Figure 17 In the mobile phone interface shown in Figure 17 , for the output selected target characters, the user can click the corresponding button to perform operations such as translating and searching on the above target characters. Figure 17 In the mobile phone interface shown in Figure 17 , for the output selected target characters, the user can click the corresponding button to perform operations such as translating and searching on the above target characters.

[0235] In a specific implementation, when it is detected that the user clicks buttons such as translation and search buttons, it means that the user hopes that the mobile phone performs corresponding processing on the selected target text. At this time, the mobile phone can call an application program that can perform the above operations.

[0236] It should be noted that if there is only one application program installed in the mobile phone that can perform the action that the user hopes to process, the application program can be directly called for processing. For example, after the user clicks the "Translation" button, if there is only a certain dictionary application program in the mobile phone that can translate the above target characters, the program can be directly called. If there are multiple application programs installed in the mobile phone that can perform the action that the user hopes to process, for example, after the user clicks the "Search" button, there are three application programs (Application A, Application B, and Application C) that can all perform this operation. At this time, an interface as shown in Figure 18 can be popped up on the mobile phone interface for the user to select an application to perform this operation. For a selected application, the user can also set it as the default application program. Subsequently, when performing the same processing, the default application program can be directly called to perform this operation. Figure 18 At this time, an interface as shown in Figure 18 can be popped up on the mobile phone interface for the user to select an application to perform this operation. For a selected application, the user can also set it as the default application program. Subsequently, when performing the same processing, the default application program can be directly called to perform this operation.

[0237] The embodiment of the present application, by utilizing the existing Convolutional Neural Network (CNN) structure in the OCR detection and recognition model, establishes a mapping relationship between the CTC sequence index output by the detection and recognition model and the original image coordinates according to the characteristics of CNN downsampling (8 times). The model used in the whole process is simple and the amount of data required to be annotated is small. During processing, compared with the prior art that requires providing the original image or the corrected image, or requires traversing all pixels on the image, the embodiment of the present application can use the real-time detected or collected image, with fast calculation speed, and can be quickly and conveniently adapted and deployed to the existing OCR model. Secondly, when determining the character boundary, the embodiment of the present application can fine-tune the character boundary according to the hyperparameters set in different languages ​​to ensure the accuracy of character boundary recognition. It is not only applicable to most languages, but also supports the selection of symbols. Third, the embodiment of the present application records the four vertex coordinates of each character in the image to be recognized, and maps the four vertex coordinates to the original image according to the segmented area to obtain the coordinates of each character in the original image, so that the granularity of the coordinate output is finer. When performing manual selection operations, the recognition between text and text, text and symbols is accurate, and the segmentation of adjacent characters can be quickly achieved, which has stronger universality in practical applications. Fourth, thanks to the robustness of the OCR detection and recognition model, the character selection method provided by the embodiment of the present application is also more robust in positioning characters in natural scenes. Fifth, the embodiment of the present application associates the coordinates of each recognized character with the second coordinates of each character in the original image, and draws the corresponding recognized text area according to the second coordinate, which can support users to make arbitrary selections and range adjustments in the above-mentioned recognized text area. Sixth, the above-mentioned text recognition and selection scenarios can also be associated with functions such as translation and question scanning. After the user selects, the selected text can be directly translated, or the recognized question can be answered; or, by parsing the string selected by the user, if it contains text content in the format of phone number, email address, etc., a separate card can be extracted to facilitate the user to directly make a call or send an email, which has strong ease of use and practicality. By associating the coordinates of each character recognized by the character selection model with the second coordinates of each character in the original image,

[0238] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0239] Corresponding to the method for selecting characters of curved text described in the above embodiment, Figure 19 A structural block diagram of a character selection device according to an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0240] In the embodiment of the present application, character selection device 1900 includes: a display module 1901, a processing module 1902, and a storage module 1903, wherein display module 1901, processing module 1902, and storage module 1903 are connected via a bus. Character selection device 1900 can be the terminal device in the above-mentioned method embodiment, or it can be configured as one or more chips within the terminal device. Character selection device 1900 can be used to perform some or all of the functions of the terminal device in the above-mentioned method embodiment.

[0241] The display module 1901 is configured to display an original image on a display interface of a terminal device, wherein the original image includes curved text.

[0242] Processing module 1902 is used to detect the original image, generate an image to be recognized containing straight text, wherein the text content of the straight text corresponds one-to-one with the text content of the curved text; recognize the text content of the straight text according to the image to be recognized, and obtain a connection time series classification sequence corresponding to the text content of the straight text, wherein the connection time series classification sequence includes multiple characters; calculate the first coordinate of each of the multiple characters in the image to be recognized; determine the segmented area where the first coordinate of each character in the multiple characters is located; determine the perspective transformation matrix corresponding to each segmented area when the original image is transformed into the image to be recognized according to the original image and the image to be recognized; multiply the first coordinate of each character in the multiple characters by the perspective transformation matrix to obtain the second coordinate of each character in the multiple characters in the original image; detect a first operation of the user on the original image, wherein the first operation is used to select a character in the curved text on the original image;

[0243] The display module 1901 is configured to highlight a selected character according to a second coordinate of each character in the original image.

[0244] Optionally, the processing module 1902 is further used to generate a first prompt information in the original image according to the second coordinate corresponding to each character in the multiple characters, wherein the first prompt information is used to indicate that the user can select a character in the original image.

[0245] Optionally, the processing module 1902 is specifically used to identify the text content of the straight text based on the picture to be identified, and obtain an initial connection time classification sequence; determine the length of the initial connection time classification sequence, and determine the picture width of the picture to be identified; if the product of the length of the initial connection time classification sequence and a preset downsampling multiple is greater than the picture width of the picture to be identified, then crop the initial connection time classification sequence to obtain a connection time classification sequence corresponding to the text content of the straight text; wherein, the product of the length of the connection time classification sequence obtained after cropping and the preset downsampling multiple is less than or equal to the picture width of the picture to be identified.

[0246] Optionally, the processing module 1902 is specifically used to sequentially crop the head element or tail element of the initial connection timing classification sequence; after cropping any head element or tail element, calculate whether the product of the length of the cropped initial connection timing classification sequence and the preset downsampling multiple is less than or equal to the image width of the image to be identified; if the product of the length of the cropped initial connection timing classification sequence and the preset downsampling multiple is less than or equal to the image width of the image to be identified, stop cropping and output the connection timing classification sequence corresponding to the text content of the straight text.

[0247] Optionally, the processing module 1902 is specifically used to determine the character boundary coordinates of each of the multiple characters in the connection time classification sequence, and the character boundary coordinates include left boundary coordinates and right boundary coordinates; according to the character boundary coordinates of each of the multiple characters, calculate the first coordinates of each of the multiple characters in the picture to be identified.

[0248] Optionally, the processing module 1902 is specifically used to obtain the original right boundary coordinate of any character in the connection timing classification sequence, and the original left boundary coordinate of the next character; calculate the average value of the original right boundary coordinate and the original left boundary coordinate; and determine the right boundary coordinate of the character and the left boundary coordinate of the next character based on the average value.

[0249] Optionally, the processing module 1902 is specifically used to respectively determine the first character type of the character and the second character type of the next character, the first character type and the second character type respectively having corresponding offsets; calculate the average value minus the offset corresponding to the first character type to obtain a first difference, and use the first difference as the right boundary coordinate of the character; calculate the average value plus the offset corresponding to the second character type to obtain a second sum, and use the second sum as the left boundary coordinate of the next character.

[0250] Optionally, the first coordinates of each character in the picture to be recognized include first vertex coordinates, second vertex coordinates, third vertex coordinates and fourth vertex coordinates; the processing module 1902 is specifically used to multiply the left boundary coordinates and right boundary coordinates of each character in the multiple characters by a preset downsampling multiple to obtain the first vertex coordinates and second vertex coordinates of each character in the multiple characters in the picture to be recognized; determine the third vertex coordinates and fourth vertex coordinates of each character in the picture to be recognized based on the first vertex coordinates, the second vertex coordinates and the picture height of the picture to be recognized.

[0251] Optionally, the processing module 1902 is specifically used to traverse the first coordinates of each character and the segmented areas of the image to be identified, and respectively determine the first segmented area where the first vertex coordinates are located, the second segmented area where the second vertex coordinates are located, the third segmented area where the third vertex coordinates are located, and the fourth segmented area where the fourth vertex coordinates are located.

[0252] Optionally, the processing module 1902 is specifically used to multiply the first vertex coordinates by the perspective transformation matrix of the first segmented area to obtain a third coordinate, multiply the second vertex coordinates by the perspective transformation matrix of the second segmented area to obtain a fourth coordinate, multiply the third vertex coordinates by the perspective transformation matrix of the third segmented area to obtain a fifth coordinate, and multiply the fourth vertex coordinates by the perspective transformation matrix of the fourth segmented area to obtain a sixth coordinate; wherein the third coordinate, the fourth coordinate, the fifth coordinate and the sixth coordinate serve as the second coordinate of the character in the original image.

[0253] Optionally, the processing module 1902 is specifically used to draw the character area of ​​each of the multiple characters in the original image as a first color according to the second coordinate corresponding to each of the multiple characters; or to draw a text box for the character area of ​​each of the multiple characters in the original image according to the second coordinate corresponding to each of the multiple characters.

[0254] Optionally, the processing module 1902 is also used to, when monitoring a click event of the user in the character area, draw the entire line of character area including the clicked character area in a second color; monitor a drag event of the user in the character area corresponding to the second color, and adjust the character area covered by the second color according to the drag event; and identify and display each character in the character area covered by the second color.

[0255] Optionally, the processing module 1902 is also used to, when monitoring a click event of the user in the character area, draw the character area closest to the click position indicated by the click event in a second color; monitor a drag event of the user in the character area corresponding to the second color, and adjust the character area covered by the second color according to the drag event; and identify and display each character in the character area covered by the second color.

[0256] Optionally, the processing module 1902 is also used to, when a sliding event of the user in the character area is monitored, draw the character area containing the sliding event indication into a second color; monitor the dragging event of the user in the character area corresponding to the second color, and adjust the character area covered by the second color according to the dragging event; identify and display each character in the character area covered by the second color.

[0257] The storage module 1903 is used to store the necessary program instructions and data of the character selection device.

[0258] It should be understood that the above Figure 19 The processes executed between the modules of the terminal device in the corresponding embodiment are the same as those in the aforementioned Figures 3 to 18 The process executed by the terminal device in the corresponding method embodiment is similar and will not be repeated here.

[0259] Figure 20 A schematic diagram of a possible structure of a character selection device 2000 in the above embodiment is shown. This character selection device 2000 can be configured as the aforementioned user equipment. The first communication device 2000 may include: a processor 2002, a computer-readable storage medium / memory 2003, a transceiver 2004, an input device 2005, an output device 2006, and a bus 2001. The processor, transceiver, computer-readable storage medium, etc. are connected via a bus. The embodiments of this application do not limit the specific connection medium between the above components.

[0260] In one example, the output device 2006 displays an original image, wherein the original image includes curved text; the processor 2002 detects the original image and generates an image to be recognized including straight text, wherein the text content of the straight text corresponds to the text content of the curved text.

[0261] According to the picture to be identified, the text content of the straight text is identified, and a connection time series classification sequence corresponding to the text content of the straight text is obtained, wherein the connection time series classification sequence includes multiple characters; the first coordinate of each character in the multiple characters in the picture to be identified is calculated; the segmented area where the first coordinate corresponding to each character in the multiple characters is located is determined; according to the original picture and the picture to be identified, the perspective transformation matrix corresponding to each segmented area when the original picture is transformed into the picture to be identified is determined; the first coordinate of each character in the multiple characters is multiplied by the perspective transformation matrix to obtain the second coordinate of each character in the multiple characters in the original picture; a first operation of the user on the original picture is detected, wherein the first operation is used to select a character in the curved text on the original picture; the output device 2006 highlights the selected character according to the second coordinate of each character in the original picture.

[0262] In one example, the processor 2002 may include a baseband circuit, and the transceiver 2004 may include a radio frequency circuit.

[0263] In another example, the processor 2002 may run an operating system to control functions between various devices and components. The transceiver 2004 may include a baseband circuit and a radio frequency circuit.

[0264] The transceiver 2004 and the processor 2002 can implement the above Figures 3 to 18 The corresponding steps in any embodiment are not described in detail here.

[0265] It is understandable that Figure 20 Only a simplified design of the character selection device is shown. In actual applications, the character selection device may include any number of transceivers, processors, memories, etc., and all character selection devices that can implement the present application are within the scope of protection of the present application.

[0266] The processor 2002 involved in the above-mentioned device 2000 can be a general-purpose processor, such as a CPU, a network processor (NP), a microprocessor, etc., or it can be an ASIC, or one or more integrated circuits for controlling the execution of the program of the present application. It can also be a digital signal processor (DSP), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The controller / processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and so on. The processor usually performs logical and arithmetic operations based on program instructions stored in a memory.

[0267] The bus 2001 mentioned above may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 20 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0268] The computer-readable storage medium / memory 2003 mentioned above may also store an operating system and other application programs. Specifically, the program may include program code, and the program code includes computer operating instructions. More specifically, the above-mentioned memory may be ROM, other types of static storage devices that can store static information and instructions, RAM, other types of dynamic storage devices that can store information and instructions, disk storage, etc. The memory 2003 may be a combination of the above-mentioned storage types. In addition, the above-mentioned computer-readable storage medium / memory may be in the processor, external to the processor, or distributed across multiple entities including a processor or processing circuit. The above-mentioned computer-readable storage medium / memory may be specifically embodied in a computer program product. For example, a computer program product may include a computer-readable medium in packaging material.

[0269] Alternatively, the embodiment of the present application also provides a general processing system, such as a chip, which includes: one or more microprocessors that provide processor functions; and an external memory that provides at least a portion of a storage medium, all of which are connected to other supporting circuits through an external bus architecture. When the instructions stored in the memory are executed by the processor, the processor executes the first communication device in Figures 3 to 18 Part or all of the steps in the data transmission method in this embodiment, and / or other processes used for the technology described in this application.

[0270] The steps of the method or algorithm described in conjunction with the disclosure of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, mobile hard disk, CD-ROM or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a terminal. Of course, the processor and the storage medium can also be present in the character selection device as discrete components.

[0271] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0272] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0273] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0274] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0275] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0276] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for selecting characters in curved text, characterized in that: include: Displaying an original image on a display interface of a terminal device, wherein the original image includes curved text; Detecting the original image to generate an image to be recognized containing straight text, wherein the text content of the straight text corresponds one-to-one with the text content of the curved text; Identifying the text content of the linear text according to the image to be identified, and obtaining a connection time sequence classification sequence corresponding to the text content of the linear text, wherein the connection time sequence classification sequence includes a plurality of characters; Calculating a first coordinate of each of the plurality of characters in the image to be recognized, where the first coordinate includes coordinates of four vertices of the character; Determine a segmented area where the first coordinate corresponding to each character of the plurality of characters is located; Determining, based on the original image and the image to be identified, a perspective transformation matrix corresponding to each segmented region when the original image is transformed into the image to be identified; multiplying a first coordinate of each character in the plurality of characters by the perspective transformation matrix to obtain a second coordinate of each character in the plurality of characters in the original image; A first operation of the user on the original picture is detected, and a selected character is highlighted according to a second coordinate of each of the multiple characters in the original picture, wherein the first operation is used to select a character in the curved text on the original picture.

2. The method according to claim 1, characterized in that After multiplying the first coordinate of each of the plurality of characters by the perspective transformation matrix to obtain the second coordinate of each of the plurality of characters in the original image, and before detecting a first operation of the user on the original image, the method further includes: According to the second coordinate corresponding to each character in the plurality of characters, first prompt information is generated in the original image, where the first prompt information is used to indicate that a user can select a character in the original image.

3. The method according to claim 1, characterized in that The step of identifying the text content of the linear text according to the image to be identified and obtaining a connection time sequence classification sequence corresponding to the text content of the linear text includes: Identify the text content of the linear text according to the image to be identified, and obtain an initial connection time series classification sequence; Determining the length of the initial connection temporal classification sequence and determining the picture width of the picture to be identified; If the product of the length of the initial connection temporal classification sequence and the preset downsampling multiple is greater than the picture width of the picture to be recognized, the initial connection temporal classification sequence is cropped to obtain a connection temporal classification sequence corresponding to the text content of the linear text; The product of the length of the connected temporal classification sequence obtained after cropping and a preset downsampling multiple is less than or equal to the picture width of the picture to be identified.

4. The method according to claim 3, characterized in that The step of trimming the initial connection time sequence classification sequence to obtain a connection time sequence classification sequence corresponding to the text content of the linear text includes: Sequentially trimming the head element or the tail element of the initial connection time series classification sequence; After any head element or tail element is cropped, calculating whether the product of the length of the cropped initial connected temporal classification sequence and the preset downsampling multiple is less than or equal to the picture width of the picture to be identified; If the product of the length of the cropped initial connection timing classification sequence and the preset downsampling multiple is less than or equal to the image width of the image to be identified, the cropping is stopped and the connection timing classification sequence corresponding to the text content of the straight text is output.

5. The method according to any one of claims 1 to 4, characterized in that Calculating the first coordinate of each character in the image to be recognized includes: Determining character boundary coordinates of each character in the plurality of characters in the connection time sequence classification sequence, the character boundary coordinates including left boundary coordinates and right boundary coordinates; According to the character boundary coordinates of each character in the plurality of characters, a first coordinate of each character in the plurality of characters in the to-be-recognized picture is calculated.

6. The method according to claim 5, characterized in that The determining of the character boundary coordinates of each character in the plurality of characters in the connection time sequence classification sequence comprises: For any character in the connection time sequence classification sequence, obtaining the original right boundary coordinates of the character and the original left boundary coordinates of the next character; Calculate the average value of the original right boundary coordinate and the original left boundary coordinate; Based on the average value, the right boundary coordinate of the character and the left boundary coordinate of the next character are determined.

7. The method according to claim 6, characterized in that The determining of the right boundary coordinate of the character and the left boundary coordinate of the next character based on the average value includes: respectively determining a first character type of the character and a second character type of the next character, the first character type and the second character type having corresponding offsets; Calculate the average value minus the offset corresponding to the first character type to obtain a first difference, and use the first difference as the right boundary coordinate of the character; The average value is added to the offset corresponding to the second character type to obtain a second sum value, and the second sum value is used as the left boundary coordinate of the next character.

8. The method according to claim 5, characterized in that The first coordinates of each character in the plurality of characters in the image to be recognized include a first vertex coordinate, a second vertex coordinate, a third vertex coordinate, and a fourth vertex coordinate; The step of calculating the first coordinate of each character in the to-be-recognized image according to the character boundary coordinates of each character in the multiple characters includes: Multiplying the left boundary coordinate and the right boundary coordinate of each character in the plurality of characters by a preset downsampling multiple, respectively, to obtain a first vertex coordinate and a second vertex coordinate of each character in the plurality of characters in the image to be recognized; Determine third and fourth vertex coordinates of each character in the picture to be recognized based on the first vertex coordinates, the second vertex coordinates, and the picture height of the picture to be recognized.

9. The method according to claim 8, characterized in that Determining the segmented area where the first coordinate is located includes: Traverse the first coordinates of each character and the segmented areas of the image to be recognized, and determine the first segmented area where the first vertex coordinate is located, the second segmented area where the second vertex coordinate is located, the third segmented area where the third vertex coordinate is located, and the fourth segmented area where the fourth vertex coordinate is located.

10. The method according to claim 9, characterized in that The multiplying the first coordinate of each character in the plurality of characters by the perspective transformation matrix to obtain the second coordinate of each character in the plurality of characters in the original image includes: multiplying the first vertex coordinate by the perspective transformation matrix of the first segment region to obtain a third coordinate, multiplying the second vertex coordinate by the perspective transformation matrix of the second segment region to obtain a fourth coordinate, multiplying the third vertex coordinate by the perspective transformation matrix of the third segment region to obtain a fifth coordinate, and multiplying the fourth vertex coordinate by the perspective transformation matrix of the fourth segment region to obtain a sixth coordinate; The third coordinate, the fourth coordinate, the fifth coordinate and the sixth coordinate serve as the second coordinate of the character in the original image.

11. The method according to claim 2, characterized in that Generating first prompt information in the original image according to the second coordinate corresponding to each character in the plurality of characters includes: Drawing a character area of ​​each character in the plurality of characters in the original image as a first color according to a second coordinate corresponding to each character in the plurality of characters; or, A text box is drawn in the original image for a character area of ​​each character in the plurality of characters according to the second coordinate corresponding to each character in the plurality of characters.

12. The method according to claim 11, characterized in that Also includes: When a click event of the user on the character area is monitored, the entire row of character areas including the clicked character area is painted in a second color; monitoring a drag event of the user on the character area corresponding to the second color, and adjusting the character area covered by the second color according to the drag event; Recognize and display each character in the character area covered by the second color.

13. The method according to claim 11, characterized in that Also includes: When a click event of the user on the character area is monitored, the character area closest to the click position indicated by the click event is painted in a second color; monitoring a drag event of the user on the character area corresponding to the second color, and adjusting the character area covered by the second color according to the drag event; Recognize and display each character in the character area covered by the second color.

14. The method according to claim 11, characterized in that Also includes: When a sliding event of the user in the character area is monitored, the character area including the sliding event indication is painted in a second color; monitoring a drag event of the user on the character area corresponding to the second color, and adjusting the character area covered by the second color according to the drag event; Recognize and display each character in the character area covered by the second color.

15. A character selection device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 14 is implemented.

16. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.

Images