Image processing methods, electronic devices and storage media

Abstract

This application provides an image processing method, electronic device, and storage medium, relating to the field of data processing technology, to solve the technical problem of text adhesion in the detection results output by a segmentation model. In this application's solution, after obtaining a binary image using a segmentation model, the connected components in the binary image are preprocessed. Then, a adhesion processing method based on morphological filtering is used to break up adhesion regions with low adhesion in the binary image. Finally, an adhesion processing method based on convex defect analysis is used to break up adhesion regions with high adhesion in the binary image. Because the adhesion region disconnection operation is completed, it facilitates the execution of downstream text recognition tasks, thereby improving the accuracy of OCR recognition results.

Description

Technical Field

[0001] This application relates to the field of data processing, and more particularly to an image processing method, electronic device, and storage medium. Background Technology

[0002] Optical character recognition (OCR) refers to the process of analyzing and processing images of textual materials to obtain textual and layout information.

[0003] Currently, OCR tasks mainly consist of two stages: text detection and character recognition. In the text detection stage, the mainstream text detection algorithms commonly used are segmentation-based algorithms. However, segmentation-based algorithms may exhibit text concatenation, such as grouping multiple lines of text into the same text box. This can negatively impact the downstream character recognition task, ultimately affecting the accuracy of the OCR recognition results. Summary of the Invention

[0004] This application provides an image processing method, an electronic device, and a storage medium, which solves the technical problem of text adhesion in the text detection results output by the segmentation model.

[0005] To achieve the above objectives, this application adopts the following technical solution:

[0006] In a first aspect, embodiments of this application provide an image processing method. The method includes:

[0007] First, the original image is input into an optical character recognition and detection model based on a segmentation scheme to obtain a binary image. Then, a rectangular checksum opening operation is performed on the binary image to regenerate it. Next, based on the convexity defects of connected components in the regenerated binary image, connected components with adhesion are disconnected, and a binary image is regenerated again. Finally, text recognition is performed on the regenerated binary image to obtain the text recognition result.

[0008] The above scheme, in the scenario of OCR recognition of the original image, firstly, an optical character recognition and detection model based on a segmentation scheme obtains a binary image corresponding to the original image. Then, a morphological filtering-based adhesion processing method is used to break up the adhesion regions with low adhesion in the binary image. Subsequently, a convex defect analysis-based adhesion processing method is used to break up the adhesion regions with high adhesion in the binary image. Since the adhesion region breaking operation is completed, it facilitates the execution of downstream text recognition tasks, thereby improving the accuracy of the OCR recognition results.

[0009] In one possible implementation, the grayscale value of the connected components in the binary graph is 255, and the grayscale value of the background in the binary graph is 0.

[0010] In one possible implementation, a rectangular kernel is used for opening the binary graph, with the kernel's direction aligned with the text direction corresponding to the connected components. It should be understood that using a square kernel for morphological opening can easily disrupt small lines of text, while using a rectangular kernel aligned with the text direction corresponding to the connected components avoids disrupting small lines of text and breaks up connected regions.

[0011] In one possible implementation, a rectangular kernel is used to perform an opening operation on the binary image and regenerate the binary image. This includes: determining the first connected component set corresponding to all horizontal text in the binary image; generating a horizontal text binary image based on the first connected component set; and performing an opening operation on the horizontal text binary image using a kernel of size 1×k to obtain an updated horizontal text binary image. Similarly, a second connected component set corresponding to all vertical text in the binary image is determined; generating a vertical text binary image based on the second connected component set; and performing an opening operation on the vertical text binary image using a kernel of size k×1 to obtain an updated vertical text binary image. A pixel-by-pixel OR operation is then performed on the updated horizontal and vertical text binary images to regenerate the binary image. Here, k is an integer greater than or equal to 2. It should be understood that dividing the binary image into binary images representing horizontal text and binary images representing vertical text facilitates the use of kernels of different sizes for opening operations on the binary images, thereby breaking up text-attached regions.

[0012] In one possible implementation, before determining the first set of connected components and the second set of connected components, the method further includes: determining the minimum bounding rectangle of each connected component in the binary graph; and determining whether each connected component is a connected component for horizontal text or a connected component for vertical text based on the minimum bounding rectangle of each connected component.

[0013] In one possible implementation, determining whether a connected component is a horizontal or vertical text component based on its minimum bounding rectangle includes: for each connected component, if the width of the minimum bounding rectangle is greater than or equal to its length, then the connected component is determined to be a horizontal text component; or, if the width of the minimum bounding rectangle is less than its length, then the connected component is determined to be a vertical text component.

[0014] In one possible implementation, k is equal to 4, 5, 6, or 7 pixel units. It should be understood that when k is too small, it may be impossible to separate small, contiguous areas; when k is too large, it may easily damage fine lines of text. Setting k to 4, 5, 6, or 7 pixel units avoids damaging fine lines of text while still separating contiguous areas.

[0015] In one possible implementation, based on the convex defects of connected components in the regenerated binary graph, connected components with adhesion are disconnected, and the binary graph is regenerated. This includes: traversing each connected component in the set of connected components of the binary graph, and performing the following operations on each connected component: if the text corresponding to a connected component is curved text, obtain all convex defects of that connected component; if at least one illegal convex defect exists among all convex defects, disconnect a connected component based on at least one illegal convex defect, obtaining multiple connected components, and adding the multiple connected components to the connected component set; or, if no illegal convex defects exist among all convex defects, add a connected component to the return set; after adding all connected components from the connected component set to the return set according to the above operations, the binary graph is regenerated again based on the connected components in the return set. It should be understood that when at least one illegal convex defect exists in a connected component, it indicates that it is more likely to be an adhesion region, and in this case, the adhesion region needs to be disconnected.

[0016] In one possible implementation, the illegal convex defect satisfies conditions 1 and 2, or conditions 1, 3, and 4. Condition 1 is: the distance from the farthest point of the convex defect to its two endpoints is greater than or equal to a first preset distance. Condition 2 is: the angle between the two endpoints of the convex defect and the farthest point is less than a first angle. Condition 3 is: the angle between the two endpoints of the convex defect and the farthest point is greater than or equal to a second angle and less than or equal to a third angle. Condition 4 is: the average distance is less than or equal to a second preset distance; the average distance is the average distance from points on the connected region located between the farthest point of the convex defect and the target endpoint to the straight line formed by the farthest point of the convex defect and the target endpoint; the target endpoint is any one of the two endpoints of the convex defect. The first angle is less than the second angle, and the second angle is less than the third angle.

[0017] For example, the second preset distance is in the range of [5, 15] pixels.

[0018] For example, the first angle has a value range of [50°, 90°], the second angle has a value range of [60°, 90°], and the third angle has a value range of [100°, 130°].

[0019] It should be understood that when all average distances are greater than the second preset distance, the connected component can be determined to be normal curved text and not an illegal convex defect. When at least one average distance is less than the second preset distance, the connected component can be determined to be non-normal curved text, an illegal convex defect, and the connected region needs to be broken.

[0020] In one possible implementation, after obtaining all convex defects in a connected component, the following operations can be performed on each convex defect: Obtain the distance from the farthest point of a convex defect to its two endpoints. If the distance from the farthest point of a convex defect to its two endpoints does not satisfy condition 1, the convex defect is determined to be a valid convex defect; or, if the distance from the farthest point of a convex defect to its two endpoints satisfies condition 1, obtain the angle between the two endpoints of a convex defect and the farthest point of the convex defect. If the angle between the two endpoints of a convex defect and the farthest point of the convex defect satisfies condition 2, the convex defect is determined to be an invalid convex defect; or, if the angle between the two endpoints of a convex defect and the farthest point of the convex defect satisfies condition 3, obtain the two average distances corresponding to the two endpoints of the convex defect; or, if the angle between the two endpoints of a convex defect and the farthest point of the convex defect does not satisfy conditions 2 and 3, the convex defect is determined to be a valid convex defect. If either of the two average distances satisfies condition 4, the convex defect is determined to be an invalid convex defect; or, if neither average distance satisfies condition 4, the convex defect is determined to be a valid convex defect.

[0021] In one possible implementation, based on at least one illegal convex defect, a connected component is broken to obtain multiple connected components. This includes: traversing all points of a connected component outside the illegal convex defect, determining the point closest to the farthest point of the illegal convex defect; and using a line segment of a first preset width with a grayscale value of 0 to connect the farthest point of the illegal convex defect and the point closest to the farthest point of the illegal convex defect, thus obtaining multiple connected components. It should be understood that since the grayscale value of a connected component in a binary graph is 255, and the grayscale value of the background in the binary graph is 0, the connected component can be broken by drawing a line segment with a grayscale value of 0.

[0022] In one possible implementation, the method further includes adding a connected component to the return set if the text corresponding to the connected component is a straight text.

[0023] In one possible implementation, the method further includes: if a connected component satisfies a first rule, then the text corresponding to that connected component is determined to be straight text; or, if a connected component does not satisfy the first rule, then the text corresponding to that connected component is determined to be curved text. The first rule includes at least one of the following: the area of ​​a connected component is less than or equal to a preset area; the rectangularity of a connected component is greater than or equal to a preset rectangularity. The rectangularity of a connected component is equal to the area of ​​the connected component divided by the area of ​​its minimum bounding rectangle. It should be understood that for some connected components with smaller areas, directly classifying them as straight text can improve robustness. Furthermore, the greater the curvature of a text, the larger the area of ​​its corresponding minimum bounding rectangle, and the greater the rectangularity of the connected component. Therefore, when the rectangularity of a connected component is greater than or equal to the preset rectangularity, it can be classified as curved text; otherwise, it can be classified as straight text.

[0024] In one possible implementation, before performing the rectangular check binary image opening operation and regenerating the binary image, the method further includes: determining the level of each contour in the binary image and establishing a level tree structure for the levels of all contours. In the level tree structure, the level of the outer contour is 1; if the outer contour contains an inner contour, the level of the inner contour is 2; if the inner contour contains an embedded contour, the level of the embedded contour is greater than or equal to 3. Connected components enclosed by contours with levels greater than or equal to 3 are deleted. The annular region between a contour with level 1 and a contour with level 2 is considered as a connected component. The connected component between the first point of the contour with level 1 and the second point of the contour with level 2 is disconnected. It should be understood that in text recognition scenarios, annular connected components may exist in the binary image. This ring-within-a-ring situation may affect subsequent adhesion processing operations, preventing the coverage of some illegal convex defects and thus making it impossible to disconnect the adhered regions. Therefore, the binary image after threshold segmentation can be preprocessed before performing the operation to disconnect the adhered regions.

[0025] In one possible implementation, the connected component between a first point on a contour at level 1 and a second point on a contour at level 2 is disconnected by connecting the first and second points using a line segment of a second preset width with a grayscale value of 0. It should be understood that since the grayscale value of the annular connected component between the contours at level 1 and level 2 is 255, and the grayscale value of the area outside the contours at levels 1 and 2 is 0, changing the grayscale value of the area traversed by the line segment to 0 is equivalent to creating an opening in the annular connected component, thereby disconnecting it.

[0026] In one possible implementation, the first point and the second point are either of the following: the first point is the leftmost point of the contour with level 1 and the second point is the leftmost point of the contour with level 2; or, the first point is the rightmost point of the contour with level 1 and the second point is the rightmost point of the contour with level 2; or, the first point is the topmost point of the contour with level 1 and the second point is the topmost point of the contour with level 2; or, the first point is the bottommost point of the contour with level 1 and the second point is the bottommost point of the contour with level 2.

[0027] Secondly, this application provides an image processing apparatus, which includes units / modules for performing the method described in the first aspect above. This apparatus can correspond to performing the method described in the first aspect above. For a brief summary, the relevant descriptions of the units / modules in this apparatus are provided in the description of the first aspect above and will not be repeated here.

[0028] Thirdly, an electronic device is provided, including a processor coupled to a memory, the processor being configured to execute a computer program or instructions stored in the memory to cause the electronic device to implement the image processing method as described in any of the first aspects.

[0029] Fourthly, a chip is provided that is coupled to a memory for reading and executing a computer program stored in the memory to implement an image processing method as described in any of the first aspects.

[0030] Fifthly, a computer-readable storage medium is provided that stores a computer program, which, when run on an electronic device, causes the electronic device to perform an image processing method as described in any of the first aspects.

[0031] In a sixth aspect, a computer program product is provided that, when the computer program product is run on an electronic device, causes the electronic device to perform the image processing method as described in any of the first aspects.

[0032] It is understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0033] Figure 1 This is a flowchart illustrating a conventional OCR algorithm provided in an embodiment of this application;

[0034] Figure 2 A schematic diagram illustrating threshold segmentation using a segmentation model, provided for an embodiment of this application;

[0035] Figure 3A flowchart illustrating the OCR algorithm based on the processing scheme of interline adhesion in the output of the segmentation model provided in the embodiments of this application;

[0036] Figure 4 A schematic diagram illustrating a scenario for processing inter-line adhesion in the output of a segmentation model, as provided in an embodiment of this application.

[0037] Figure 5 A schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application;

[0038] Figure 6 A schematic diagram of the software structure of the electronic device provided in the embodiments of this application;

[0039] Figure 7 This is a schematic diagram of the preprocessing of a binary image after threshold segmentation, provided in an embodiment of this application.

[0040] Figure 8 A schematic diagram illustrating a preprocessing method provided in an embodiment of this application;

[0041] Figure 9 A schematic diagram illustrating another preprocessing method provided in an embodiment of this application;

[0042] Figure 10 A schematic diagram of the hierarchical tree structure provided in the embodiments of this application;

[0043] Figure 11 A schematic diagram of a connected region with a relatively low degree of adhesion, provided for an embodiment of this application;

[0044] Figure 12 A schematic diagram of a connected region with a high degree of adhesion provided in an embodiment of this application;

[0045] Figure 13 A flowchart illustrating the adhesion processing method based on morphological filtering provided in this application embodiment;

[0046] Figure 14 This is a schematic diagram illustrating the breaking of the adhesion region in a binary image, as provided in an embodiment of this application.

[0047] Figure 15 A schematic diagram illustrating the acquisition of the width and height of the minimum bounding rectangle of two connected components, provided for an embodiment of this application;

[0048] Figure 16 A schematic diagram illustrating the results of opening operations using a square kernel and a rectangular kernel, provided for embodiments of this application;

[0049] Figure 17 A schematic diagram of the adhesion processing method based on convex defect analysis provided in the embodiments of this application for the adhesion region;

[0050] Figure 18 A schematic diagram of convex hulls and convex defects detected using a hand as an example, provided for embodiments of this application;

[0051] Figure 19 A flowchart illustrating the process of determining the legality of a convex defect according to rule 2, provided for an embodiment of this application;

[0052] Figure 20 A schematic diagram illustrating the disconnection of a connected component with an illegal convexity defect, provided as an embodiment of this application;

[0053] Figure 21 Another schematic diagram illustrating the disconnection of a connected component with an illegal convexity defect, provided as an embodiment of this application;

[0054] Figure 22 A schematic diagram illustrating the detection of curved text provided in an embodiment of this application;

[0055] Figure 23 A comparative diagram showing the final detection results obtained using two OCR algorithms, provided for embodiments of this application;

[0056] Figure 24 This is a schematic diagram of the structure of the image processing apparatus provided in the embodiments of this application. Detailed Implementation

[0057] To ensure clarity and brevity in the description of the following embodiments, a brief introduction to the relevant concepts or technologies is given first.

[0058] OCR (Optical Character Recognition) algorithms are algorithms that identify text from images. For example... Figure 1 As shown, the OCR algorithm mainly includes the following steps: 1. Input the original color image into the OCR detection model, such as an OCR detection model based on a segmentation scheme (referred to as a segmentation model). 2. Perform threshold segmentation on the grayscale image output by the OCR detection model, splitting it into different connected components. 3. For the segmented connected components, perform post-processing on the characters according to the algorithm (including layout analysis, character cutting, and character correction, etc.), and recognize the text.

[0059] Text detection algorithms based on segmentation models are a type of deep learning approach. These methods model pixels and predict the confidence level of whether each pixel belongs to text or background. They obtain each text line instance through threshold segmentation and connected component extraction.

[0060] For example, Figure 2 A schematic diagram of threshold segmentation using a segmentation model is shown. Figure 2 (a) in the image is the original image, which contains characters and punctuation. The electronic device can input the original image into the segmentation model. Figure 2In the image, (b) is the grayscale image output by the segmentation model without thresholding. Figure 2 In (b), the background of the image is mostly black (grayscale value 0), the foreground is mostly white (grayscale value 255), and the color of the region corresponding to the intersection of the first letter "S" and the first three lines of text is between black and white (grayscale value greater than 0 and less than 255). Assuming a cut threshold of 100, the segmentation model can perform thresholding as follows: when the grayscale value of a pixel is greater than or equal to 100, update the grayscale value of that pixel to 255; when the grayscale value of a pixel is less than 100, update the grayscale value of that pixel to 0. If the grayscale value of the region corresponding to the intersection of the first letter "S" and the first three lines of text is greater than or equal to 100, then after thresholding, as shown... Figure 2 As shown in (c), the first letter "S" is stuck together with the first three texts and is framed in the same text box, which will affect the downstream text recognition task.

[0061] It should be noted that this embodiment uses the example of the black area of ​​the binary image corresponding to the background area of ​​the original image and the white area of ​​the binary image corresponding to the foreground area of ​​the original image. In actual implementation, it can also be set that the black area of ​​the binary image corresponds to the foreground area of ​​the original image and the white area of ​​the binary image corresponds to the background area of ​​the original image; this embodiment does not limit this.

[0062] Furthermore, this application does not limit the segmentation algorithm on which the segmentation model is based, and it can be set according to actual usage requirements. For example, the segmentation algorithm can be a deep learning-based segmentation algorithm, such as the PSENet algorithm, the PAN algorithm, or the BNet algorithm, etc.

[0063] Given the issue of text concatenation in the text detection results output by the segmentation model, the embodiments of this application address this problem. Figure 1 An improvement has been made to the traditional OCR algorithm shown, providing a solution for handling line adhesion in the output of the segmentation model.

[0064] For example, such as Figure 3As shown, after thresholding the grayscale image using a segmentation model to obtain a binary image composed of different connected components, a processing scheme targeting inter-line adhesion in the segmentation model output can be adopted: first, preprocess the binary image; then, use a morphological filtering-based adhesion processing method to break up adhesion regions with low adhesion in the binary image; subsequently, use a convex defect analysis-based adhesion processing method to break up adhesion regions with high adhesion in the binary image. After completing the disconnection of adhesion regions, post-processing and text recognition can be performed on the regenerated binary image. Because the above scheme completes the disconnection of adhesion regions, it facilitates the execution of downstream text recognition tasks, thereby improving the accuracy of OCR recognition results.

[0065] The processing scheme for interline adhesion of segmentation model output provided in this application embodiment can be applied to various electronic devices with OCR function.

[0066] The aforementioned electronic devices can also be referred to as terminals, terminal equipment, user equipment (UE), mobile stations (MS), mobile terminals (MT), etc. For example, electronic devices can be mobile phones, smart TVs, wearable devices, tablets, computers with wireless transceiver capabilities, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, or wireless terminals in smart homes, etc. The embodiments of this application do not limit the specific technologies or device forms used in the electronic devices.

[0067] The following example uses a mobile phone as an electronic device, combined with... Figure 4 The application scenarios of the processing scheme for inter-line adhesion of segmentation model output provided in the embodiments of this application are illustrated with examples.

[0068] For example, a user can point their phone's camera at various items containing characters to be recognized, such as books, journals, magazines, newspapers, posters, brochures, product manuals, or calendars, and take a picture. The phone then stores the captured image in its gallery. Figure 4As shown in (a), the phone's home screen UI1 displays a gallery icon. When a user wants to use the phone's OCR function to perform character recognition on a captured image, the user can tap the gallery icon on the home screen. In response to the user's tap, the phone displays the following... Figure 4 The gallery interface UI2 shown in (b) provides an "OCR Recognition" option. Users can click the "OCR Recognition" option, and the phone responds to this click by displaying the following: Figure 4 The gallery interface UI3 shown in (c) can include multiple recently captured text images. The user clicks on any one of the text images. In response to the user's click, the phone's OCR engine calls the OCR recognition module to perform text detection and character recognition on the text image. Especially when there are connected areas in the text images, a segmentation model-based approach to handle inter-line connections can be used to separate them. After the phone obtains the final character recognition result, it can display the result, for example, as shown in... Figure 4 The recognition result interface UI4 is shown in (d) in the figure.

[0069] In one example, such as Figure 4 As shown in (d), the recognition result interface UI4 mainly includes three functional areas: 1. A functional area located in the upper middle of the screen, which contains multiple regions segmented from the original image; 2. A functional area located in the lower middle of the screen, which contains multiple paragraphs, each paragraph containing the characters in a region, i.e., a paragraph is the recognition result of a region; 3. Multiple functional controls located at the bottom of the screen, such as copy control, export control, translation control, and re-recognition control. The copy control is used to copy the recognized text, the export control is used to export the recognized text to other applications, the translation control is used to translate the recognized text into other languages ​​(such as from English to Chinese), and the re-recognition control is used to re-recognize the original image.

[0070] The above embodiments are illustrated using the application scenario of OCR recognition of images in a gallery, and do not limit the embodiments of this application. The processing scheme for line-to-line adhesion in segmentation model output provided by this application can also be applied to other possible scenarios: for example, when the content of a document downloaded from the network is not editable, the processing scheme for line-to-line adhesion in segmentation model output provided by this application can be used to recognize the text and convert it into an editable document; another example is when the product descriptions provided by merchants when shopping on a mobile phone are in foreign language images, the processing scheme for line-to-line adhesion in segmentation model output provided by this application can be used to recognize the text and translate it into Chinese.

[0071] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application is shown. Taking a mobile phone as an example, as... Figure 5 As shown, the mobile phone 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, and a bone conduction sensor 180M, etc.

[0072] The processor 110 may include one or more processing units, such as a controller, application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU). The controller may serve as the central nervous system and command center of the mobile phone 100. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution. The processor 110 may also include memory for storing instructions and data.

[0073] In this embodiment of the application, the processor 110 is used to: perform threshold segmentation processing on the original image based on the segmentation model; then adopt a processing scheme for inter-line adhesion of the segmentation model output to disconnect the adhesion region; after completing all disconnection processing operations, regenerate the binary image, and perform post-processing and text recognition on the binary image.

[0074] The mobile phone 100 utilizes a GPU, a display 194, and an application processor to achieve its display function. The GPU is a microprocessor for image processing, connecting the display 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering, such as rendering OCR recognition results.

[0075] The display screen 194 is used to display images from the gallery, OCR recognition results, etc. The display screen can also display various operable controls provided by the mobile phone for the user, such as clickable buttons and sliding sliders. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), or a quantum dot light-emitting diode (QLED), etc. In some embodiments, the mobile phone 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0076] The mobile phone 100 can achieve shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0077] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0078] Camera 193 is used to capture still images or videos. An object passes through the lens to generate an optical image that is projected onto a photosensitive element. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP (Internet Service Provider) for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP (Digital Signal Processor) for processing. The DSP converts the digital image signal into image signals in standard formats such as RGB and YUV. In some embodiments, mobile phone 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0079] A digital signal processor (DSP) is used to process digital signals. Besides digital image signals, it can also process other digital signals. For example, when a mobile phone 100 is selecting a frequency, the DSP performs Fourier transforms on the frequency energy.

[0080] Video codecs are used to compress or decompress digital video. Mobile phone 100 can support one or more video codecs. Thus, mobile phone 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0081] NPU stands for Neural Network (NN) Computing Processor. By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs can be used to train OCR detection models based on segmentation schemes, enabling applications such as intelligent cognition on mobile phones. Examples of OCR detection models based on segmentation schemes include image recognition, face recognition, speech recognition, and text understanding.

[0082] The external memory interface 120 can be used to connect an external memory card, thereby expanding the storage capacity of the mobile phone 100. The external memory card communicates with the processor 110 through the external memory interface 120 to perform data storage.

[0083] The internal memory 121 can be used to store computer executable program code, which includes instructions. The processor 110 executes various functional applications and data processing of the mobile phone 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application (APP) required for a function (such as a camera APP, a gallery APP, and third-party video editing software, etc.). The data storage area may store data created during the use of the mobile phone 100 (such as photos or videos taken, screenshots, screen recordings, images downloaded from other devices, and OCR recognition results, etc.). In addition, the internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, and universal flash storage (UFS), etc.

[0084] Mobile phone 100 can implement audio functions through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor. For example, after text recognition is completed using OCR, audio module 170 can generate a signal corresponding to the text, and then speaker 170A, also known as a "speaker," converts the audio electrical signal into a sound signal. In this way, the user can hear the text recognition result.

[0085] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the mobile phone 100. In other embodiments of this application, the mobile phone 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0086] Figure 6 This is a schematic diagram of the software structure of an electronic device provided in an embodiment of this application. For example... Figure 6 As shown, electronic devices can adopt a layered architecture, dividing the software into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the software layers of the software structure are divided from top to bottom as follows: application (APP) layer, application framework (FWK) layer, system library, and kernel layer.

[0087] The application layer, or application layer for short, can include a series of application packages, such as camera, calendar, map, music, gallery, SMS, OCR engine, Bluetooth, and video software. When these application packages are run, they can access the various service modules provided by the application framework layer through the application programming interface (API) and perform corresponding intelligent business operations. In some embodiments, the OCR engine can be a system-level application or a third-party application. The OCR engine can be called by applications such as gallery, map, or SMS to perform OCR recognition on images. The images can be screenshots obtained by a screenshot application, photos taken by a camera application, or images stored in the gallery, etc.

[0088] The application framework layer provides APIs and a programming framework for applications within the application layer. It includes predefined functions such as an OCR recognition module, an activity manager, and a resource manager. The OCR recognition module uses a deep learning-based OCR model to recognize images. This model includes a text detection model and a text recognition model. The text detection model identifies multiple valid connected components in the text, while the text recognition model identifies the text within each connected component. The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, and video files. The activity manager manages the lifecycle of each application and the navigation / back functionality, and is responsible for creating the Android main thread and maintaining the lifecycle of each application.

[0089] The system library layer can include multiple functional modules, such as a surface manager, a 3D graphics processing library (e.g., OpenGL ES), a 2D graphics engine (e.g., SGL), and media libraries. The surface manager manages the display subsystem and provides fusion of 2D and 3D layers for multiple applications. The 3D graphics processing library implements 3D graphics drawing, image rendering, compositing, and layer processing. The 2D graphics engine is the drawing engine for 2D graphics. The media library supports playback and recording of various common audio and video formats, as well as still image files. The media library supports various audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0090] Within the system library layer, the Android Runtime comprises the core libraries and the virtual machine. The Android Runtime is responsible for the scheduling and management of the Android system. The core libraries consist of two parts: one part contains the functionalities that Java calls, and the other part comprises the core Android libraries. The application layer and application framework layer run within the virtual machine. The virtual machine executes the Java files from the application layer and application framework layer as binary files. The virtual machine is used for managing object lifecycles, stack management, thread management, security and exception management, and garbage collection, among other functions.

[0091] The kernel layer is the layer between hardware and software, belonging to the lower layers of the Android system. The kernel layer can contain various driver interfaces, such as display drivers, camera drivers, call drivers, Wireless Fidelity (Wi-Fi) drivers, and sensor drivers. The kernel layer can also contain power management, processes / threads, etc.

[0092] It should be noted that although the embodiments of this application are described using the Android system as an example, the basic principles are also applicable to electronic devices based on operating systems such as iOS or Windows.

[0093] It is understood that, in order to implement the inter-line adhesion processing scheme for the segmentation model output in the embodiments of this application, the electronic device includes hardware and / or software modules that perform various functions. Based on the algorithm steps of the examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments.

[0094] To facilitate understanding of the processing scheme for inter-line adhesion in the output of the segmentation model provided in the embodiments of this application, the preprocessing, adhesion processing method based on morphological filtering, and adhesion processing method based on convex defect analysis in the processing scheme for inter-line adhesion in the output of the segmentation model will be described in detail below with reference to Embodiment 1, Embodiment 2, and Embodiment 3.

[0095] Example 1

[0096] The smallest unit of an image is a pixel. Each pixel has a specific location and color value. A pixel is surrounded by eight neighboring pixels. Visually, connected pixels form a region, while disconnected pixels form different regions. A set consisting of all connected pixels is called a connected component.

[0097] In the text detection stage, a segmentation model can be used to threshold the original image to obtain a binary image. The black areas (grayscale value of 0) in the binary image correspond to the background area of ​​the original image (i.e., the area that does not contain the text to be recognized), and the white areas (grayscale value of 255) correspond to the foreground area of ​​the original image (i.e., the area that contains the text to be recognized). Therefore, a white area can be called a connected component. Electronic devices can disconnect the connected regions.

[0098] However, in some scenarios, ring-shaped connected components may exist in the binary graph. A ring-shaped connected component may include one or more sub-ring-shaped connected components. This ring-within-a-ring situation may affect subsequent adhesion processing operations, making it impossible to cover some illegal convex defects, resulting in the inability to break the adhesion regions. Therefore, this application proposes a scheme to first preprocess the threshold-segmented binary graph to generate a new binary graph, and then break the inter-row adhesion in the new binary graph. Of course, in other scenarios, it is also possible not to preprocess the threshold-segmented binary graph and directly break the inter-row adhesion in the binary graph. This can be determined according to actual usage requirements, and this application does not limit it.

[0099] Figure 7 This diagram illustrates the preprocessing steps for the threshold-segmented binary image. For example... Figure 7 As shown, the preprocessing may include the following steps S11-S15.

[0100] S11. Input binary image. This binary image is obtained by thresholding the original image using a segmentation model. All non-zero pixels in the binary image are considered equal.

[0101] S12. Find the contours of all connected components in the binary graph and determine the rank of each contour.

[0102] In some embodiments, an electronic device can use the function "findContours" to find the contours of all connected components in a binary graph, creating a hierarchical tree structure for all contours. In this hierarchical tree structure, outer contours contain inner contours, and inner contours can further contain one or more nested contours. This determines the rank of each contour within the hierarchical tree structure.

[0103] For example, one way to call cv::findContours() is as follows:

[0104] void cv::findContours(

[0105] cv::InputOutputArray image, / / 8-bit single-channel binary image as input

[0106] cv::OutputArrayOfArrays contours, / / A vector of vectors containing points

[0107] cv::OutputArray hierarchy, / / (optional) topology information

[0108] int mode, / / Contour retrieval mode

[0109] int method, / / approximate method

[0110] cv::Point offset = cv::Point() / / (optional) Offset of all points )

[0112] The first parameter, image, is the input single-channel image matrix, such as a binary image obtained by thresholding the original image using a segmentation model.

[0113] The second parameter, contours, is defined as a vector <vector>"contours" is a two-dimensional vector. Each element of the vector stores a set of points consisting of consecutive points, and each set of points constitutes a contour. The number of contours is equal to the number of elements in the vector "contours".

[0114] The third parameter, hierarchy, is defined as the structural information of the output contour tree. For each contour, hierarchy contains four integers, representing: the index of the next contour, the index of the previous contour, the index of the child contour, and the index of the parent contour. It should be noted that if the current contour has no corresponding next contour, previous contour, parent contour, or nested contour, then the corresponding positions of hierarchy[i][0] to hierarchy[i][3] are set to the default value of -1.

[0115] The fourth parameter, `int mode`, defines the contour retrieval mode. For example, if `int mode` is set to `CV_RETR_TREE`, all contours will be detected, and a hierarchical tree structure will be built for all contours.

[0116] The fifth parameter, int method, defines the approximation method for the contour.

[0117] The sixth parameter, Point offset, defines the offset of all contour information relative to the corresponding point in the original image. It is equivalent to adding this offset to each detected contour point. Point can be positive or negative.

[0118] In the hierarchical tree structure, the outermost contour has a level of 1, the sub-contours contained in the outermost contour have a level of 2, the embedded contours contained in the sub-contours have a level of 3, and so on. If each sub-contour contains an embedded contour, the level of the embedded contour is increased by 1.

[0119] S13. Delete the connected components enclosed by contours with a level greater than or equal to 3. That is, only retain the connected components enclosed by contours with a level of 1 and 2.

[0120] Regardless of whether a contour with a level greater than or equal to 3 contains sub-contours, the grayscale value of all pixels within the region enclosed by a contour with a level greater than or equal to 3 can be set to 0 (i.e., the region is set to black), thereby deleting the connected components enclosed by the contour with a level greater than or equal to 3.

[0121] In text recognition scenarios, when the level of a contour is greater than or equal to 3, the recognition of that contour is not very meaningful. Therefore, such connected components can be ignored to reduce the complexity of text recognition.

[0122] S14. Treat the annular region between the contour with level 1 and the contour with level 2 as a connected component; and disconnect the connected component between the first point of the contour with level 1 and the second point of the contour with level 2.

[0123] For example, the first and second points can be set in any of the following ways:

[0124] Method 1: The first point is the leftmost point of the contour with a level of 1, and the second point is the leftmost point of the contour with a level of 2.

[0125] Method 2: The first point is the rightmost point of the contour with a level of 1, and the second point is the rightmost point of the contour with a level of 2.

[0126] Method 3: The first point is the topmost point of the contour with a level of 1, and the second point is the topmost point of the contour with a level of 2.

[0127] Method 4: The first point is the bottom edge of the contour with a level of 1, and the second point is the bottom edge of the contour with a level of 2.

[0128] Of course, the first point can be any point on the contour with a level of 1, and the second point can be any point on the contour with a level of 2. This application does not limit the method of setting the two points.

[0129] In some embodiments, the electronic device may connect the first point and the second point using a line segment of the target width, and set the grayscale value of all pixels traversed by the line segment to 0. Since the grayscale value of the annular connected region between the contour at level 1 and the contour at level 2 is 255, while the grayscale value of the region outside the contour at level 1 and the contour at level 2 is 0, changing the grayscale value of the region traversed by the line segment to 0 is equivalent to creating an opening in the annular connected region, thereby breaking the annular connected region.

[0130] In some embodiments, an outer contour with a level of 1 may include multiple inner contours with a level of 2. In this case, it is necessary to set multiple line segments with a grayscale value of 0, as described in the embodiments above, so as to provide an opening connecting each contour with a level of 2 to the outside. The number of line segments used to break the loop connection is equal to the number of inner contours, that is, one inner contour corresponds to one opening to the outside.

[0131] For example, the target width can be N pixels, where N ranges from [N1, N2]. For instance, N1 = 4, N1 = 6.

[0132] S15. Regenerate the binary image.

[0133] The preprocessing methods provided in the embodiments of this application will be specifically explained below with reference to two examples.

[0134] Example 1

[0135] For circular or ring-shaped patterns such as school badges, trademarks, coins, and organizational logos, text adhesion issues may occur. To facilitate subsequent adhesion processing and cover various illegal convex defects, methods such as... Figure 8 The preprocessing method shown.

[0136] like Figure 8 As shown in (a), there are two contours in the binary graph: contour 1 and contour 2. Contour 1 is the parent contour of contour 2, and contour 2 is the child contour of contour 1. Contour 1 has no sibling contours or parent contours, but has one child contour 2, so the hierarchy of contour 1 is [-1 -1 2 -1]. Contour 2 has no sibling contours or child contours, but has one parent contour 1, so the hierarchy of contour 2 is [-1 -1 -1 1]. It can be determined that in the binary graph, the level of contour 1 is 1, the level of contour 2 is 2, and there are no contours with a level greater than or equal to 3.

[0137] Then, as Figure 8 As shown in (b), the annular region between contour 1 and contour 2 can be considered as a connected region. Determine point A, the leftmost point of contour 1, and point B, the leftmost point of contour 2.

[0138] Then, as Figure 8 As shown in (c), a line segment with a width of 5 pixels is used to connect points A and B, and the grayscale value of all pixels passed through by the line segment is set to 0, thereby breaking the circular connected domain.

[0139] Based on the description of the adhesion processing method based on convex defect analysis provided in Embodiment 3 below, after the annular connected domain is broken, a convex defect exists in the connected domain. If the angle between the two breakpoints and the farthest point of the convex defect is less than a first angle, then the convex defect is an illegal convex defect, and the connected domain needs to be further broken. Therefore, the preprocessing provided in Embodiment 1 can cover some illegal convex defects, thereby breaking the adhesion region.

[0140] Example 2

[0141] For long, circular patterns such as horizontal or vertical text, text adhesion issues may occur. To facilitate subsequent adhesion processing and cover various illegal convex defects, methods such as... Figure 9 The preprocessing method shown.

[0142] like Figure 9 As shown in (a) of the diagram, there are four contours in the binary graph: contour 1, contour 2, contour 3, and contour 4. Contour 1 is the parent contour of contours 2 and 3, and contour 2 is the parent contour of contour 4. Contour 1 has no sibling contours or parent contours, but has child contours 2 and 3, so its hierarchy is [-1-1 2 -1]. Contour 2 has no preceding contour, but has parent contour 1, child contour 4, and following contour 3, so its hierarchy is [3 -1 4 1]. Contour 3 has no following contour or child contours, but has parent contour 1 and preceding contour 2, so its hierarchy is [-1 2 -1 1]. Contour 4 has no sibling contours or child contours, but has parent contour 2, so its hierarchy is [-1 -1-1 2]. For example, the hierarchical tree structure of contours 1, 2, 3, and 4 can be represented as follows: Figure 10 As shown, it can be determined that in the binary graph, the level of contour 1 is 1, the levels of contours 2 and 3 are 2, and the level of contour 4 is 3.

[0143] Then, as Figure 9 As shown in (b), since there is a contour 4 with a level of 3 in the binary image, the grayscale value of all pixels within the region enclosed by contour 4 is set to 0, i.e., the connected component enclosed by contour 4 is deleted. Then, the annular region between contours 1, 2, and 3 is taken as a connected component (contour 1 is the outer contour of this connected component, and contours 2 and 3 are the inner contours of this connected component). After that, the leftmost point C of contour 1, the leftmost point D of contour 2, the rightmost point F of contour 1, and the rightmost point E of contour 3 are determined.

[0144] Then, as Figure 9 As shown in (c), point C and point D are connected by line segment 1, which is 5 pixels wide, and point E and point F are connected by line segment 2, which is 5 pixels wide. The grayscale value of all pixels passed through by line segment 1 and line segment 2 is set to 0, thus breaking the circular connected component.

[0145] Referring to the description of the adhesion processing method based on convex defect analysis provided in Embodiment 3 below, after the annular connected domain is broken, there are two convex defects in the connected domain. For each convex defect, if the angle between the two breakpoints and the farthest point of the convex defect is less than a first angle, then the convex defect is an illegal convex defect, and the connected domain needs to be further broken. Therefore, the preprocessing provided in Embodiment 1 can cover some illegal convex defects, thereby breaking the adhesion area.

[0146] Example 2

[0147] After preprocessing the binary image after threshold segmentation to regenerate the binary image, for some adhesion regions with a small degree of adhesion, the adhesion processing method based on morphological filtering provided in Example 2 can be used to break up the adhesion regions; while for adhesion regions with a large degree of adhesion, the adhesion processing method based on convex defect analysis provided in Example 3 can be used to break up the adhesion regions.

[0148] For example, Figure 11 A schematic diagram of a connected region with relatively low adhesion is shown. For example... Figure 11 As shown in the magnified view of two horizontal texts, a single horizontal text line includes at least 3 interconnected pixels. If a morphological filtering-based adhesion processing method (kernel size 1×5) is used, horizontal texts with low adhesion can be separated.

[0149] For example, Figure 12 A schematic diagram of a connected region with a high degree of adhesion is shown. For example... Figure 12 As shown in the magnified view corresponding to the intersection of the three horizontal texts, a column of vertical text includes at least 6 interconnected pixels. If a morphological filtering-based adhesion processing method (kernel size 5×1) is used, it will be unable to break up the vertical texts with a high degree of adhesion. It is necessary to further adopt an adhesion processing method based on convex defect analysis to break up the adhesion area.

[0150] The following is combined Figures 13 to 16 This paper introduces a method for processing adhesion based on morphological filtering.

[0151] Figure 13 A flowchart illustrating the adhesion processing method based on morphological filtering is shown. Accordingly, Figure 14 This diagram illustrates the breaking of the adhered regions in a binary image.

[0152] like Figure 13 As shown, this method may include the following S31-S38.

[0153] S31. Input a binary image, such as... Figure 14 The binary image with the adhesive region shown in the dashed box in (a) is shown.

[0154] S32. Find all connected components and the minimum bounding rectangle (MBR) of each connected component in a binary graph.

[0155] All connected components in a binary image can form a set of connected components. For each connected component in the set of connected components, the minimum bounding rectangle refers to the maximum range of the connected component represented in two-dimensional coordinates, that is, a rectangle defined by the maximum abscissa, minimum abscissa, maximum ordinate, and minimum ordinate among the vertices of the given two-dimensional shape.

[0156] Exemplarily, the electronic device can first find all connected components in the binary image, and then calculate the relevant parameters of the minimum bounding rectangle according to the contour coordinates of each connected component, such as the coordinates of the four vertices, the width of the minimum bounding rectangle, the height of the minimum bounding rectangle, the area of the minimum bounding rectangle, the perimeter of the minimum bounding rectangle, and so on.

[0157] S33. Determine whether it is horizontal or vertical text according to the aspect ratio of each minimum bounding rectangle.

[0158] For the OCR recognition scenario, the text to be recognized is usually horizontal text or vertical text.

[0159] The minimum bounding rectangle of the connected component corresponding to the horizontal text satisfies the condition:

[0160]

[0161] The minimum bounding rectangle of the connected component corresponding to the vertical text satisfies the condition:

[0162]

[0163] Where, b represents the width of the minimum bounding rectangle, and a represents the height of the minimum bounding rectangle.

[0164] That is to say, when the bounding rectangle of a connected component satisfies b > a, the connected component can be determined as the connected component of horizontal text; when the bounding rectangle of a connected component satisfies b < a, the connected component can be determined as the connected component of vertical text.

[0165] For example, Figure 15 in (b) is Figure 15 a partial enlarged view of the connected component corresponding to two vertically arranged texts that are adhered to each other shown in (a) in. In Figure 15 in (b), the dashed box is used to represent the minimum bounding rectangle of the connected component. b1 represents the width of the minimum bounding rectangle, and a1 represents the height of the minimum bounding rectangle. Since b1 < a1, the connected component can be determined as the connected component of vertical text.

[0166] For another example, Figure 15 in (c) is Figure 15 a partial enlarged view of the connected component corresponding to two horizontally arranged texts that are adhered to each other shown in (a) in. In Figure 15 In (c), the dashed box represents the minimum bounding rectangle of the connected component. b2 represents the width of the minimum bounding rectangle, and a2 represents its height. Since b2 > a2, this connected component can be determined as a connected component of horizontal text.

[0167] It should be noted that when the circumscribed rectangle of a connected component satisfies a = b, the connected component can be determined as either horizontal or vertical text, and this application does not limit the definition.

[0168] S34. Based on the set of connected components of the horizontal text, generate a binary graph representing the horizontal text, such as... Figure 14 The binary graph representing the horizontal text is shown in (b) above.

[0169] The set of connected domains for horizontal text includes all connected domains corresponding to horizontal text.

[0170] It should be noted that a connected component in, for example Figure 14 The coordinates in the binary graph shown in (b) are similar to those in... Figure 14 The coordinates are the same in the binary graph shown in (a). Therefore, Figure 14 The binary graph shown in (b) is equivalent to the graph from... Figure 14 The binary graph extracted from the binary graph shown in (a) contains only the set of horizontally connected text regions.

[0171] S35. Using a kernel of size 1×k, perform an opening operation on the binary image representing the horizontal text to break up the connected regions, for example, to obtain... Figure 14 The binary image of the disconnected adhesion region is shown in (c). Here, k is an integer greater than or equal to 2.

[0172] In some embodiments, k = 4, 5, 6, or 7 pixel units. It should be noted that when k is too small, it may fail to separate small, contiguous areas; when k is too large, it may damage fine lines of text. Setting k to a value between 4, 5, 6, or 7 pixel units avoids damaging fine lines of text while still separating contiguous areas.

[0173] Opening is a filter based on geometric operations. It typically involves erosion followed by dilation and can be used to eliminate discrete points and "burrs". It can smooth the boundaries of large objects without significantly changing their volume, thereby breaking up adhered regions.

[0174] In traditional OCR recognition scenarios, square kernels are typically used for morphological opening operations, but this can easily distort fine text lines. To address this issue, embodiments of this application provide two types of rectangular convolution kernels (hereinafter referred to as kernels): one is a 1×k kernel with its long side horizontal, used for opening binary images representing horizontally arranged text; the other is a k×1 kernel with its long side vertical, used for opening binary images representing vertically arranged text. In other words, regardless of the rectangular kernel used, the kernel's direction is consistent with the text direction.

[0175] For example, Figure 16 (a) shows the detection result mask of the segmentation model, with the adhesion regions to be broken marked by circular dashed boxes. If a 5×5 kernel (a square kernel) is used to perform the opening operation on the detection results, then... Figure 16 As shown in (b), besides breaking the adhered area marked by the circular dashed box, it also destroys the small text lines marked by the elliptical dashed box, thus affecting the accuracy of the OCR recognition result. If a kernel of size 1×5 (rectangular kernel) is used to perform opening operations on the detection result, then as... Figure 16 As shown in (c), only the adhered area marked by the circular dashed box will be broken, without destroying small lines of text.

[0176] S36. Based on the set of connected components of the vertically arranged text, generate a binary graph representing the vertically arranged text, such as... Figure 14 The binary graph representing vertical text is shown in (d) in the figure.

[0177] The set of connected domains for vertical text includes all connected domains corresponding to vertical text.

[0178] It should be noted that a connected component in, for example Figure 14 The coordinates in the binary graph shown in (d) are similar to those in... Figure 14 The coordinates are the same in the binary graph shown in (a). Therefore, Figure 14 The binary image shown in (d) is equivalent to the image from... Figure 14 The binary graph extracted from the binary graph shown in (a) contains only the set of vertically connected text regions.

[0179] S37. Using a kernel of size k×1, perform an opening operation on the binary image representing vertical text to break up connected regions, for example, to obtain... Figure 14 The binary image of the disconnected adhesion region is shown in (e). Here, k is an integer greater than or equal to 2.

[0180] For the opening operation, please refer to the description of S35 above, which will not be repeated here.

[0181] S38. Perform a pixel-by-pixel OR operation to regenerate the binary image.

[0182] The OR operation refers to the logical "OR" operator. If either or both operands are true, the logical "OR" operator (||) returns the boolean value true; otherwise, it returns false. Operands are implicitly cast to type bool before evaluation, and the result is of type bool. The logical "OR" operation has left-to-right associativity.

[0183] In the embodiments of this application, for example, Figure 14 The binary image of the broken adhesion area shown in (c) is as follows: Figure 14 The binary image of the disconnected and adhered area shown in (e) has a one-to-one correspondence between the pixel coordinates in the two binary images. The electronic device can perform an OR operation on the two pixels corresponding to each coordinate, row by row and pixel by pixel (or column by pixel). If the grayscale value of any pixel is 255, or if the grayscale values ​​of both pixels are 255, then the final grayscale value corresponding to that coordinate is 255. After performing the OR operation on the pixels corresponding to all coordinates, a regenerated binary image can be obtained, such as... Figure 14 As shown in (f) in the binary graph, both horizontal and vertical text have had their weakly connected areas broken.

[0184] Example 3

[0185] After the adhesion region is broken up using the adhesion processing method based on morphological filtering provided in Example 2, since the adhesion region with a high degree of adhesion has not yet been broken up, the adhesion processing method based on convex defect analysis provided in Example 3 can be further used to break up the adhesion region with a high degree of adhesion.

[0186] The following is combined Figures 17 to 22 This paper introduces the adhesion treatment method based on convex defect analysis for the adhesion region.

[0187] Figure 17 A schematic diagram of the adhesion treatment method based on convex defect analysis is shown for the adhesion region.

[0188] like Figure 17 As shown, this method may include the following S40-S49.

[0189] S40, Input binary image. This binary image is a newly generated binary image after the adhesion region is broken up using a morphological filtering-based adhesion processing method.

[0190] S41, find all connected components in the binary graph.

[0191] All connected components in a binary graph can form a set of connected components. Initially, this set includes all connected components in the binary graph. After executing steps S42-S48, this set may also include disconnected or adhered connected components.

[0192] S42, Traverse the connected components. For each connected component: Determine whether the text corresponding to the connected component is straight or curved according to rule 1. If a connected component satisfies rule 1, then the text corresponding to the connected component is straight, and the electronic device can continue to execute S43 below. If a connected component does not satisfy rule 1, then the text corresponding to the connected component is curved, and the electronic device can execute S44 below.

[0193] Rule 1 above may include at least one of the following:

[0194] ① The area of ​​a connected component is less than or equal to a preset area, for example, the preset area is 1000 pixel units;

[0195] ② The rectangularity of the connected components is greater than or equal to the preset rectangularity.

[0196] The area of ​​a connected region is the same as the area of ​​its outline, which can be determined based on the outline points of the connected region.

[0197] For example, an electronic device can use the following function to calculate the area of ​​a connected component:

[0198] area = contourArea(contours[t]); / / Calculate the area of ​​the contour

[0199] It should be understood that for some connected components with small areas, directly classifying them as straight text can improve robustness.

[0200] The rectangularity of a connected region is equal to the area of ​​the connected region divided by the area of ​​its smallest bounding rectangle.

[0201] For example, an electronic device can first draw the minimum bounding rectangle of the connected component, then calculate the area of ​​the minimum bounding rectangle based on the coordinates of its four vertices, then calculate the area of ​​the connected component based on its outline, and finally use the ratio of the area of ​​the connected component to the area of ​​its minimum bounding rectangle as the rectangularity of the connected component.

[0202] It should be understood that the greater the curvature of a text, the larger the area of ​​its minimum bounding rectangle, and the smaller the rectangularity of the connected components. When the rectangularity of a connected component is less than a preset rectangularity, it can be identified as curved text; otherwise, it can be identified as straight text.

[0203] S43, determine that the connected component is valid, and add the connected component to the return set.

[0204] The return set consists of at least one valid connected component. Whenever a valid connected component is determined, the electronic device can add it to the return set. After adding all valid connected components to the return set, the electronic device can regenerate the binary graph based on the return set and terminate the adhesion processing flow based on convex defect analysis.

[0205] S44, find the convex hull and convex defects of a connected component.

[0206] A convex hull is a convex polygon formed by connecting the outermost points of a connected region. After performing contour analysis on a binary image, the convex hull of each contour can be constructed, and the set of points contained in the convex hull is returned. Based on the returned convex hull point set, the convex hull corresponding to the contour can be drawn. Theoretically, a convex curve should be either convex or flat. If the convex hull has concave areas, it is called a convex defect or convexity flaw.

[0207] In some embodiments, an electronic device may use the following function to find the convex hull of a connected component:

[0208] hull=cv2.convexHull(points,clockwise,returnpoints);

[0209] Here, `points` represents the input set of contour points. `clockwise` is the direction flag; if set to `true`, the output convex hull is clockwise; otherwise, it is counter-clockwise. `returnPoints` indicates whether to return the point set. `Hull` represents the output convex hull result. When `returnPoints` is `true`, the returned convex hull vertex coordinates are the point set; when `returnPoints` is `false`, the returned values ​​are the indices corresponding to the points on the contour corresponding to the convex hull points.

[0210] In some embodiments, an electronic device may use the following function to find convex defects in a connected component:

[0211] convexityDefects=cv2.convexityDefect(contour,convexhull);

[0212] Here, `contour` represents the detected contour, which can be obtained by calling the `findContours` function. `Convexhull` represents the detected convex hull, which can be obtained by calling the `convexHull` function. `convexityDefects` is the output parameter, the final result of the detection. The final result is an array, with each row containing the values ​​[start point, end point, farthest point, approximate distance to the farthest point]. The first three values ​​represent: the start point of the convex defect, the end point of the convex defect, and the farthest point of the convex defect (i.e., the point with the maximum distance from the edge point to the convex hull). The start point and end point of the convex defect are also called the two endpoints of the convex defect. The approximate distance to the farthest point is the maximum distance from the edge point to the convex hull. Connecting the start point and end point of the convex defect yields the convex hull.

[0213] For example, Figure 18 A schematic diagram of convex hulls and convex defects detected using the palm as an example is shown. Figure 18 As shown, the palm outline can be represented by a bold black solid line, and the convex hull can be represented by a black dashed line segment. The area between the palm outline and the convex hull is the convex defect (e.g., including convex defect 1 to convex defect 6). Each convex defect corresponds to two endpoints and one farthest point of the convex defect. Taking convex defect 1 and convex defect 2 as examples, the two endpoints of convex defect 1 are endpoint 1 and endpoint 2, and the farthest point of convex defect 1 is farthest point 1. The two endpoints of convex defect 2 are endpoint 2 and endpoint 3, and the farthest point of convex defect 2 is farthest point 2.

[0214] It should be noted that, for ease of understanding, Figure 18 This explanation uses a hand as an example. In OCR recognition scenarios, text lines may also have convex defects. The above method can be used to determine the convex hull and convex defects of the connected components corresponding to the text line.

[0215] S45, Traverse the convex defects. For each convex defect: determine whether the convex defect is valid according to rule 2. If all convex defects of a connected component are valid, then execute S46 below; otherwise, execute S47.

[0216] In some embodiments, the process of determining whether a convex defect is valid according to rule 2 can be as follows: Figure 19 As shown.

[0217] For example, such as Figure 19 As shown, for each convexity, the process of determining whether the convex defect is legal according to rule 2 may include the following (1)-(9).

[0218] (1) Input the convex defect and related parameters.

[0219] For example, the relevant parameters can be an array detected using the function convexityDefects, which includes: the farthest point of the convex defect, the two endpoints of the convex defect (i.e., the starting point and the ending point of the convex defect), and the approximate distance to the farthest point (i.e., the distance from the farthest point of the convex defect to the two endpoints of the convex defect).

[0220] (2) Determine whether the distance from the farthest point of the convex defect to the two endpoints of the convex defect is greater than or equal to the first preset distance. If the distance from the farthest point of the convex defect to the two endpoints of the convex defect is greater than or equal to the first preset distance, it indicates that the convex defect is large and can proceed to (3); otherwise, proceed to (8).

[0221] (3) Determine the angle between the two endpoints of the convex defect and the farthest point of the convex defect. In this angle, the farthest point of the convex defect is the vertex of the angle, and the two endpoints of the convex defect are located on the two sides respectively.

[0222] (4) Determine whether the subdivided angle is less than the first angle thAngle1. If the subdivided angle is less than or equal to the first angle thAngle1, then proceed to (9) below. Otherwise, proceed to (5) below.

[0223] (5) Determine whether the subdivided angle is greater than or equal to the second angle thAngle2 and less than or equal to the third angle thAngle3. That is, determine whether the subdivided angle is between the second angle thAngle2 and the third angle thAngle3. If the subdivided angle is greater than or equal to the second angle thAngle2 and less than or equal to the third angle thAngle3, then proceed to (6) below; otherwise, proceed to (8) below.

[0224] In this context, the first angle thAngle1 is less than or equal to the second angle thAngle2, and the second angle thAngle2 is less than the third angle thAngle3. The methods for obtaining the first angle thAngle1, the second angle thAngle2, and the third angle thAngle3 are as follows: Multiple samples that may exhibit adhesion are collected for batch testing. Adhesive samples are selected as examples, along with curved text samples. Hyperparameters are repeatedly adjusted until a set of parameters is found that accurately identifies adhesion without misjudging curved text as adhesion. It should be understood that the more parameters collected, the more accurate the calculated three angles will be.

[0225] In some embodiments, the first angle thAngle1 takes values ​​in the range of [50°, 90°].

[0226] The second angle thAngle2 has a range of values ​​in the range of [60°, 90°].

[0227] The value range of the third angle thAngle3 is [100°, 130°].

[0228] (6) Obtain the average distance from a point in the first region of the connected domain to the line formed by the defect point and the endpoint; and obtain the average distance from a point in the second region of the connected domain to the line formed by the defect point and the endpoint. That is, obtain the average distance from a point in the connected domain between the defect point and the endpoint to the line formed by the defect point and the endpoint. Wherein, one end of the first region is the farthest point of the convex defect, and the other end is one endpoint of the convex defect. One end of the second region is the farthest point of the convex defect, and the other end is the other endpoint of the convex defect.

[0229] (7) Determine whether the average distance is greater than the second preset distance thDis. If the average distance is greater than the second preset distance thDis, then execute (8) below; if at least one average distance is less than the second preset distance thDis, execute (9) below.

[0230] The second preset distance thDis has a value range of [5, 15] pixels. The second preset distance is less than the first preset distance.

[0231] For example, the second preset distance thDis is 7 pixels.

[0232] (8) Determine that the convex defect is legal.

[0233] (9) Determine that the convex defect is illegal.

[0234] It should be understood that when all average distances are greater than the second preset distance, the connected component can be determined to be normal curved text and not an illegal convex defect. When at least one average distance is less than the second preset distance, the connected component can be determined to be not normal curved text and is an illegal convex defect. By performing (5)-(9), illegal defects and curved text can be distinguished.

[0235] S46, determine that the connected component is valid, and add it to the return set.

[0236] S47, according to rule 3, split the connected component at the point of maximum defect.

[0237] Rule 3 is: Traverse all points in the connected domain outside the convex defect, find the point D that is closest to the farthest point A of the convex defect, and draw a line segment with a certain width and a gray value of 0 between points A and D on the binary image. That is, set the gray value of the area through which the line segment passes to 0.

[0238] In some embodiments, a connected component may contain multiple illegal convex defects. In one approach, the electronic device can disconnect the maximum points of all illegal convex defects at once in S47, thereby splitting the component into all connected components. In another approach, the electronic device can disconnect the maximum point of one illegal convex defect at a time, thereby splitting the component into two connected components, and then disconnect the maximum point of the next illegal convex defect in the next splitting, thereby splitting the component into two more connected components, and so on, until all connected components are obtained.

[0239] S48, add the multiple connected components obtained after splitting to the connected component set, and continue to execute S42 above.

[0240] S49, determine whether all connected components have been added to the return set.

[0241] If all connected components in the connected component set are valid, or if the connected components are changed to valid connected components by disconnecting the adhesion region, all of these valid connected components will be added to the return set. The electronic device can then regenerate the binary graph based on the return set and end the adhesion processing flow based on convex defect analysis.

[0242] To facilitate understanding of the adhesion processing methods based on convex defect analysis provided in S40-S49 above, several specific examples of disconnecting adhesion regions based on convex defect analysis are provided below.

[0243] Example a

[0244] Figure 20 (a) shows a U-shaped connected component. Assuming the area of ​​this connected component is greater than a preset area and its rectangularity is greater than a preset rectangularity, it can be determined that the U-shaped connected component is not a straight text. Then, the convex hull and convex defects of the U-shaped connected component are searched. For example... Figure 20 As shown in (b), line segment BC is the convex hull of the U-shaped connected region. The region between the convex hull and the inner contour of the U-shaped connected region is the convex defect. Points B and C are the two endpoints of the convex defect, and point A is the farthest point of the convex defect. Connect points B and A, and connect points C and A. ∠BAC is the angle subtended from the two endpoints B and C of the convex defect to the farthest point A of the convex defect. Assuming that ∠BAC is less than the first angle thAngle1, it can be determined that the convex defect is invalid. Then, all points of the connected region outside the convex defect (i.e., the outer contour of the connected region) can be traversed, and the distance from each point outside the convex defect to the farthest point A of the convex defect can be obtained. Then, the minimum distance among all distances can be determined. Assuming that the minimum distance corresponds to point D, the point closest to the farthest point A of the convex defect can be determined as point D, as shown in the diagram. Figure 20 As shown in (c) above. Then, points A and D are connected by a line segment of preset width with a grayscale value of 0 (black), thus breaking the circular connected region, resulting in the following... Figure 20 The two new connected components are shown in (d) in the diagram.

[0245] Example b

[0246] Figure 21 (a) shows an L-shaped connected component. Assuming the area of ​​this connected component is greater than a preset area and its rectangularity is greater than a preset rectangularity, it can be determined that the L-shaped connected component is not a straight text. Then, the convex hull and convex defects of the L-shaped connected component are searched. For example... Figure 20 As shown in (a), line segment BC is the convex hull of an L-shaped connected region. The region between the convex hull and the inner contour of the L-shaped connected region is the convex defect. Points B and C are the two endpoints of the convex defect, and point A is the farthest point of the convex defect. Connect points B and A, and connect points C and A. ∠BAC is the angle subtended from the two endpoints B and C of the convex defect to the farthest point A of the convex defect. Assuming that ∠BAC is greater than or equal to the second angle thAngle2 and ∠BAC is less than or equal to the third angle thAngle3, the average distance h1 from points in the first region (i.e., the connected region between points A and C) to line AC can be obtained; and the average distance h2 from points in the second region (i.e., the connected region between points A and B) to line AB can be obtained. Assuming that at least one of the average distances h1 and h2 is less than a second preset distance, the convex defect can be determined to be illegal. Then, we can iterate through all points on the outside of the convex defect (i.e., the outer contour of the connected domain), obtain the distance from each point on the outside of the convex defect to the farthest point A of the convex defect, and then determine the minimum distance among all distances. Assuming the minimum distance corresponds to point D, we can determine that the point closest to the farthest point A of the convex defect is point D, such as... Figure 21 As shown in (b) above. Then, points A and D are connected by a line segment of preset width with a grayscale value of 0 (black), thus breaking the circular connected region, resulting in the following... Figure 21 The two new connected components are shown in (c) in the diagram.

[0247] Example c

[0248] Figure 22 A C-shaped connected component is shown. Assuming the area of ​​this connected component is greater than a preset area and its rectangularity is greater than a preset rectangularity, it can be determined that the C-shaped connected component is not a straight text. Then, the convex hull and convex defects of the C-shaped connected component are searched. For example... Figure 22 As shown, line segment BC is the convex hull of a C-type connected region. The region between the convex hull and the inner contour of the C-type connected region is the convex defect. Points B and C are the two endpoints of the convex defect, and point A is the farthest point of the convex defect. Connect points B and A, and connect points C and A. ∠BAC is the angle subtended from the two endpoints B and C of the convex defect to the farthest point A of the convex defect. Assuming that ∠BAC is greater than or equal to the second angle thAngle2, and ∠BAC is less than or equal to the third angle thAngle3, we can obtain the average distance h3 from points in the first region (i.e., the connected region between points A and C) to line AC; and obtain the average distance h4 from points in the second region (i.e., the connected region between points A and B) to line AB. Assuming that both average distance h3 and average distance h4 are greater than the second preset distance, we can determine that the convex defect is valid, and thus the connected region can be directly added to the return set.

[0249] After the adhesion region is broken up using morphological filtering-based adhesion processing and convex defect analysis-based adhesion processing, if all the disconnection processing operations for the adhesion region have been completed, the electronic device can regenerate the binary image and perform post-processing and text recognition on the binary image.

[0250] For example, Figure 23 (a) in the figure shows the use of, for example Figure 1 The diagram shown illustrates the final detection result obtained by the traditional OCR algorithm. Figure 23 (b) shows the use of, for example Figure 3 The diagram shows the final detection result obtained by the OCR algorithm based on the processing scheme of inter-line adhesion output by the segmentation model. Figure 23 In (a) of the text, the content of the first three lines, "stray birds of summer come to my window to sing and fly away. And yellowleaves of autumn which have no songs, flutter and," is ultimately assigned to segmentation region 1. Figure 23 In (b), the content of the first three text lines is divided into segmentation region 2, segmentation region 3, segmentation region 4, and segmentation region 5, respectively. (Comparison) Figure 23 (a) and Figure 23 As shown in (b), the OCR algorithm based on the processing scheme of interline adhesion in the output of the segmentation model can break the adhesion text, which is convenient for performing downstream text recognition tasks, thereby improving the accuracy of OCR recognition results.

[0251] It is understood that, in order to achieve the above-mentioned functions, electronic devices include hardware structures or software modules, or a combination of both, corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in a hardware-driven or software-driven manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0252] This application embodiment can divide an electronic device into functional modules based on the above method example. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation. The following description uses the example of dividing each functional module according to each function.

[0253] Figure 24 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application. Figure 24 As shown, the device 240 may include a text detection module 241, a preprocessing module 242, a morphological filtering-based adhesion processing module 243, a convex defect analysis-based adhesion processing module 244, and a text recognition module 245.

[0254] The text detection module 241 can be used in an optical character recognition and detection model based on a segmentation scheme to obtain a binary image corresponding to the original image.

[0255] The preprocessing module 242 can be used to preprocess the binary image, split the annular region, and obtain the updated binary image.

[0256] The adhesion processing module 243 based on morphological filtering can perform a rectangular check binary image opening operation and regenerate the binary image.

[0257] The adhesion processing module 244 based on convex defect analysis can be used to disconnect the adhered connected regions based on the convex defects of the connected regions in the regenerated binary graph, and then regenerate the binary graph again.

[0258] The text recognition module 245 can be used to perform text recognition on the regenerated binary image to obtain the text recognition result.

[0259] This application also provides an electronic device, including a processor coupled to a memory, the processor being used to execute computer programs or instructions stored in the memory, so that the electronic device implements the methods in the above embodiments.

[0260] This application also provides a computer-readable storage medium storing computer instructions. When the computer-readable storage medium is operated on an electronic device, it causes the electronic device to perform the method described above. The computer instructions can be stored in the computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access, or it can include one or more data storage devices such as servers or data centers that can be integrated using media. Available media can be magnetic media (e.g., floppy disks, hard disks, or magnetic tapes), optical media, or semiconductor media (e.g., solid-state drives (SSDs)).

[0261] This application also provides a computer program product, which includes computer program code that, when run on a computer, causes the computer to perform the methods described in the above embodiments.

[0262] This application also provides a chip coupled to a memory. This chip is used to read and execute computer programs or instructions stored in the memory to perform the methods described in the above embodiments. The chip can be a general-purpose processor or a special-purpose processor. It should be noted that the chip can be implemented using one or more field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gate logic, discrete hardware components, any other suitable circuits, or any combination of circuits capable of performing the various functions described throughout this application.

[0263] The electronic devices, apparatuses, computer-readable storage media, computer program products, and chips provided in the embodiments of this application are all used to perform the methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects corresponding to the methods provided above, and will not be repeated here.

[0264] It should be noted that the terms "first" and "second," etc., in the specification, claims, and drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0265] It should be understood that in this application, "at least one" means one or more, "more" means two or more, "at least two" means two or three or more, and "and / or" is used to describe the relationship between related objects, indicating that there can be three relationships. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the related objects before and after are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0266] It should be understood that in the embodiments of this application, "B corresponding to A" means that B is associated with A. For example, B can be determined based on A. It should also be understood that determining B based on A does not mean that B is determined solely based on A; B can also be determined based on A and / or other information. Furthermore, the term "connection" in the embodiments of this application refers to various connection methods, such as direct connection or indirect connection, to achieve communication between devices, and the embodiments of this application do not impose any limitations on this.

[0267] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0268] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0269] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0270] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0271] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device, such as a microcontroller, chip, or processor, to execute all or part of the steps of the methods provided in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0272] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.< / vector>

Claims

1. An image processing method, characterized in that, The method includes: The original image is input into an optical character recognition and detection model based on a segmentation scheme to obtain a binary image; The binary image is opened using a rectangular kernel, and the binary image is regenerated. Based on the illegal convexity defects of the connected components in the regenerated binary graph, disconnect the connected components that are stuck together, and regenerate the binary graph again. Text recognition is performed on the regenerated binary image to obtain the text recognition result; Wherein, the illegal convex defect satisfies conditions 1 and 2, or satisfies conditions 1, 3 and 4; Condition 1 is: the distance from the farthest point of the convex defect to the two endpoints of the convex defect is greater than or equal to the first preset distance; Condition 2 is: the angle between the two endpoints of the convex defect and the farthest point of the convex defect is less than the first angle; Condition 3 is: the angle between the two endpoints of the convex defect and the farthest point of the convex defect is greater than or equal to the second angle and less than or equal to the third angle; Condition 4 is: the average distance is less than or equal to the second preset distance; the average distance is the average distance from the point on the connected domain located between the farthest point of the convex defect and the target endpoint to the straight line formed by the farthest point of the convex defect and the target endpoint; the target endpoint is any one of the two endpoints of the convex defect. The first angle is smaller than the second angle, and the second angle is smaller than the third angle.

2. The method according to claim 1, characterized in that, In the binary graph where the opening operation is performed, the text direction corresponding to the connected component is consistent with the direction of the kernel.

3. The method according to claim 1, characterized in that, The step of performing an opening operation on the binary image using a rectangular kernel and regenerating the binary image includes: Determine the first connected component set corresponding to all horizontal text in the binary graph; generate a horizontal text binary graph based on the first connected component set; and perform an opening operation on the horizontal text binary graph using a kernel of size 1×k to obtain an updated horizontal text binary graph. Determine the second connected component set corresponding to all vertical text in the binary graph; generate a vertical text binary graph based on the second connected component set; and perform an opening operation on the vertical text binary graph using a kernel of size k×1 to obtain an updated vertical text binary graph. Perform a pixel-by-pixel OR operation on the updated horizontal text binary image and the updated vertical text binary image to regenerate the binary image; Where k is an integer greater than or equal to 2.

4. The method according to claim 3, characterized in that, Before determining the first set of connected components and the second set of connected components, the method further includes: Determine the minimum bounding rectangle for each connected component in the binary graph; Based on the minimum bounding rectangle of each connected component, determine whether each connected component is a connected component of horizontal text or a connected component of vertical text.

5. The method according to claim 4, characterized in that, The step of determining whether each connected component is a connected component for horizontal or vertical text based on its minimum bounding rectangle includes: If the width of the minimum bounding rectangle of each connected component is greater than or equal to the length of the minimum bounding rectangle, then each connected component is determined to be a connected component of horizontal text; or, If the width of the minimum bounding rectangle of each connected component is less than the length of the minimum bounding rectangle, then each connected component is determined to be a connected component of vertical text.

6. The method according to claim 3, characterized in that, k equals 4, 5, 6, or 7 pixel units.

7. The method according to claim 1, characterized in that, The step of disconnecting connected components based on illegal convexity defects in the connected components of the regenerated binary graph, and then regenerating the binary graph again, includes: For each connected component in the set of connected components of the binary graph, perform the following operation: In the case where the text corresponding to a connected component is curved text, obtain all convex defects of the connected component. If at least one illegal convex defect exists among all the convex defects, then based on the at least one illegal convex defect, the connected component is disconnected to obtain multiple connected components, and these multiple connected components are added to the connected component set; or, If there are no illegal convex defects among all the convex defects, then add the connected component to the return set; After adding all connected components from the connected component set to the returned set according to the above operations, the binary graph is regenerated based on the connected components in the returned set.

8. The method according to claim 7, characterized in that, The method further includes: After obtaining all convex defects of the connected component, the following operation is performed on each convex defect: Obtain the distance from the farthest point of a convex defect to the two endpoints of the convex defect; If the distance from the farthest point of a convex defect to the two endpoints of the convex defect does not satisfy condition 1, the convex defect is determined to be a legal convex defect; or, if the distance from the farthest point of a convex defect to the two endpoints of the convex defect satisfies condition 1, the angle between the two endpoints of the convex defect and the farthest point of the convex defect is obtained. If the angles between the two endpoints of a convex defect and the farthest point of the convex defect satisfy condition 2, the convex defect is determined to be an illegal convex defect; or, if the angles between the two endpoints of a convex defect and the farthest point of the convex defect satisfy condition 3, the two average distances corresponding to the two endpoints of the convex defect are obtained; or, if the angles between the two endpoints of a convex defect and the farthest point of the convex defect do not satisfy conditions 2 and 3, the convex defect is determined to be a legal convex defect. If either of the two average distances satisfies condition 4, the convex defect is determined to be an illegal convex defect; or, if neither of the two average distances satisfies condition 4, the convex defect is determined to be a legal convex defect.

9. The method according to claim 7, characterized in that, The step of disconnecting a connected component based on the at least one illegal convex defect to obtain multiple connected components includes: Traverse all points of the connected region outside the illegal convex defect, and determine the point closest to the farthest point of the illegal convex defect; Using line segments with a first preset width and a grayscale value of 0, the farthest point of the illegal convex defect and the point closest to the farthest point of the illegal convex defect are connected to obtain the multiple connected components.

10. The method according to claim 7, characterized in that, The method further includes: If the text corresponding to a connected component is a straight text, then the connected component is added to the returned set.

11. The method according to claim 10, characterized in that, The method further includes: If a connected component satisfies the first rule, then the text corresponding to the connected component is determined to be straight text; or, if a connected component does not satisfy the first rule, then the text corresponding to the connected component is determined to be curved text. The first rule includes at least one of the following: the area of ​​a connected component is less than or equal to a preset area, and the rectangularity of a connected component is greater than or equal to a preset rectangularity. The rectangularity of a connected component is equal to the area of ​​the connected component divided by the area of ​​its smallest bounding rectangle.

12. The method according to claim 1, characterized in that, The second preset distance has a value range of [5, 15] pixels.

13. The method according to claim 1, characterized in that, The first angle takes values ​​in the range [50°, 90°). The second angle takes values ​​in the range of [60°, 90°]. The value range of the third angle is [100°, 130°].

14. The method according to any one of claims 1 to 13, characterized in that, Before performing the opening operation on the binary image using a rectangular kernel and regenerating the binary image, the method further includes: In the binary graph, the level of each contour is determined, and a level tree structure is established for the levels of all contours; wherein, in the level tree structure, the level of the outer contour is 1; if the outer contour contains an inner contour, the level of the inner contour is 2; if the inner contour contains an embedded contour, the level of the embedded contour is greater than or equal to 3. Delete the connected components enclosed by contours with a level greater than or equal to 3; The annular region between the contour with level 1 and the contour with level 2 is treated as a connected component; and the connected component between the first point of the contour with level 1 and the second point of the contour with level 2 is disconnected.

15. The method according to claim 14, characterized in that, The step of disconnecting the connected region between the first point of the contour at level 1 and the second point of the contour at level 2 includes: A line segment with a second preset width and a grayscale value of 0 is used to connect the first point and the second point.

16. The method according to claim 14, characterized in that, The first point and the second point are either of the following: The first point is the leftmost point of the contour with level 1, and the second point is the leftmost point of the contour with level 2; or, The first point is the rightmost point of the contour with level 1, and the second point is the rightmost point of the contour with level 2; or, The first point is the topmost point of the contour with a level of 1, and the second point is the topmost point of the contour with a level of 2; or, The first point is the bottommost point of the contour with a level of 1, and the second point is the bottommost point of the contour with a level of 2.

17. The method according to any one of claims 1 to 13, characterized in that, The grayscale value of the connected components in the binary graph is 255, and the grayscale value of the background in the binary graph is 0.

18. An electronic device, characterized in that, The device includes a processor and a memory coupled to the processor; wherein the memory stores instructions that, when executed by the processor, cause the electronic device to perform the image processing method as described in any one of claims 1 to 17.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when run on an electronic device, causes the electronic device to perform the image processing method as described in any one of claims 1 to 17.

Images