Text area detection method, device and server

Through text area segmentation and morphological processing, pre-trained models and morphological expansion operations are used to solve the problem of low text detection efficiency in the prior art, and the text area in the image is quickly detected.

CN112749704BActive Publication Date: 2024-11-15BEIJING KINGSOFT CLOUD NETWORK TECH CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201911058697.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2019-10-31
Publication Date
2024-11-15
Estimated Expiration
2039-10-31

AI Technical Summary

Technical Problem

The text detection efficiency in the prior art is low, mainly due to the pixel-by-pixel iteration expansion method, making it difficult to quickly detect text areas in the image.

Method used

By acquiring the image to be processed, the text area segmentation process is performed, multiple areas to be selected are obtained, and morphologically processed. The text area in the image is quickly detected using the pre-trained text area division model and morphological expansion operation.

Benefits of technology

The efficiency of text detection is improved, the text area in the image can be detected quickly and accurately, and the detection efficiency is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112749704B_ABST
    Figure CN112749704B_ABST
Patent Text Reader

Abstract

The present invention provides a method, device and server for detecting text regions, wherein the method comprises: obtaining an image to be processed; performing text region segmentation processing on the image to be processed to obtain multiple regions to be selected of the text region in the image to be processed; wherein the scales of the multiple regions to be selected are multiple; performing morphological processing on the multiple regions to be selected to obtain the text region of the image to be processed. The present invention processes the regions to be selected obtained after the text region segmentation processing by morphology, and can quickly detect the text region of the image, thereby improving the efficiency of text detection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a text area detection method, device and server. Background Art

[0002] In the related art, in order to realize text detection in an image, an image segmentation algorithm can be used to perform multi-level segmentation on the text area in the image to obtain multi-scale segmented areas, and then the segmented areas are iteratively expanded pixel by pixel according to the contour size of the segmented areas. Finally, the text area in the image is detected. In this method, due to the use of a pixel-by-pixel iterative expansion method, text detection is time-consuming and the detection efficiency is low. Summary of the invention

[0003] The object of the present invention is to provide a text region detection method, device and server to quickly detect the text region of an image and improve the text detection efficiency.

[0004] In a first aspect, an embodiment of the present invention provides a method for detecting a text area, the method comprising: acquiring an image to be processed; performing text area segmentation processing on the image to be processed to obtain a plurality of areas to be selected for the text area in the image to be processed; wherein the scales of the plurality of areas to be selected are various; performing morphological processing on the plurality of areas to be selected to obtain the text area of ​​the image to be processed.

[0005] In a preferred embodiment of the present invention, the above-mentioned step of performing text area segmentation processing on the image to be processed includes: inputting the image to be processed into a pre-trained text area division model to obtain a plurality of to-be-selected areas of the text area in the image to be processed; wherein the text area division model is trained in the following manner: determining a target training image based on a preset training set; wherein the target training image carries text area identifiers of multiple scales; inputting the target training image into an initial model, and outputting pixel feature data of the target training image; the pixel feature data includes a feature value of each pixel in the target training image; calculating a model loss value based on the pixel feature data and the text area identifier; adjusting the network parameters in the initial model according to the model loss value, and continuing to perform the step of determining the target training image based on the preset training set, until the model loss value meets the preset specified value or the number of target training images input into the initial model meets the preset number, thereby obtaining a text area division model.

[0006] In a preferred embodiment of the present invention, the above-mentioned text area identification includes: a mask corresponding to the text area in the target training image; wherein the scale of the maximum-scale mask is the same as the scale of the text area in the target training image; the scales of the masks other than the maximum-scale mask are obtained by reducing the scale of the previous-scale mask by a preset percentage.

[0007] In a preferred embodiment of the present invention, the step of calculating the model loss value based on pixel feature data and text area identifier includes: for each scale, determining the non-text area corresponding to the current scale from the target training image according to the text area identifier of the current scale; extracting a preset number of pixels from the non-text area according to the feature values ​​of the pixels in the non-text area; calculating the loss value corresponding to the current scale according to the feature values ​​of the preset number of pixels extracted and the feature values ​​of the pixels in the text area corresponding to the text area identifier; calculating the model loss value according to the loss value corresponding to the current scale and the weighting coefficient corresponding to each scale.

[0008] In a preferred embodiment of the present invention, the step of calculating the loss value corresponding to the current scale based on the feature values ​​of the preset number of pixels extracted and the feature values ​​of the pixels in the text area corresponding to the text area identifier includes: the loss value Where x and y represent the horizontal and vertical coordinates of the pixel position respectively; when the pixel belongs to the text area, S x,y =1; when the pixel belongs to the preset number of pixels extracted, S x,y =0; G x,y Indicates the feature values ​​of a preset number of pixels or pixels in a text area extracted.

[0009] In a preferred embodiment of the present invention, among the above-mentioned multiple areas to be selected, areas to be selected of the same scale are identified in the same mask; the step of performing morphological processing on the multiple areas to be selected includes: extracting a mask corresponding to the area to be selected of the smallest scale from the multiple areas to be selected; determining a connected area from the mask corresponding to the area to be selected of the smallest scale; using a preset template to perform a morphological dilation operation on the connected area; and determining a text area of ​​the image to be processed based on the dilated connected area.

[0010] In a preferred embodiment of the present invention, the step of determining the text area of ​​the image to be processed based on the expanded connected area includes: determining the previous scale of the smallest scale as the current scale; wherein the masks corresponding to the areas to be selected at different scales are arranged in descending order according to the scales of the areas to be selected; performing bitwise operation on the pixel values ​​of the corresponding pixels in the expanded connected area and the mask of the current scale to obtain an operation result; continuing to perform morphological expansion operation on the expanded connected area until the operation result converges; determining the previous scale of the current scale as the new current scale, and continuing to perform bitwise operation on the pixel values ​​of the corresponding pixels in the mask of the current scale to obtain an operation result until the current scale is the maximum scale and the operation result converges; determining the text area of ​​the image to be processed based on the connected area after the last expansion.

[0011] In a preferred embodiment of the present invention, the step of determining the text area of ​​the image to be processed based on the connected area after the last expansion includes: extracting pixel feature values ​​corresponding to the connected area obtained by the last expansion operation from the pixel feature data of the image to be processed output by a pre-trained text area segmentation model; calculating the average value of the extracted pixel feature values; if the average value is greater than a preset average value threshold, determining that the connected area obtained by the last expansion operation is a text area.

[0012] In a preferred embodiment of the present invention, after the above-mentioned step of performing morphological processing on multiple areas to be selected to obtain the text area of ​​the image to be processed, the method also includes: inputting the text area and the image to be processed into the minAreaRect function of opencv, and outputting the image to be processed marked with the text area border.

[0013] In a second aspect, an embodiment of the present invention provides a device for detecting a text area, the device comprising: an image acquisition module for acquiring an image to be processed; a module for determining an area to be selected, for performing text area segmentation processing on the image to be processed to obtain a plurality of areas to be selected for text areas in the image to be processed; wherein the scales of the plurality of areas to be selected are various; and a text area determination module for performing morphological processing on the plurality of areas to be selected to obtain a text area of ​​the image to be processed.

[0014] In a preferred embodiment of the present invention, among the above-mentioned multiple areas to be selected, the areas to be selected of the same scale are identified in the same mask; the above-mentioned text area determination module is used to: extract the mask corresponding to the area to be selected with the smallest scale from the multiple areas to be selected; determine the connected area from the mask corresponding to the area to be selected with the smallest scale; use a preset template to perform a morphological dilation operation on the connected area; and determine the text area of ​​the image to be processed based on the dilated connected area.

[0015] In a preferred embodiment of the present invention, the above-mentioned device also includes: a border marking module, which is used to input the text area and the image to be processed into the minAreaRect function of opencv, and output the image to be processed marked with the text area border.

[0016] In a third aspect, an embodiment of the present invention provides a server, including a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the above-mentioned text area detection method.

[0017] In a fourth aspect, an embodiment of the present invention provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the above-mentioned text area detection method.

[0018] The embodiments of the present invention bring the following beneficial effects:

[0019] The present invention provides a text area detection method, device and server, which first obtains an image to be processed; then performs text area segmentation processing on the image to be processed to obtain multiple areas to be selected in the text area of ​​the image to be processed; finally, performs morphological processing on the multiple areas to be selected to obtain the text area of ​​the image to be processed. The present invention processes the areas to be selected obtained after the text area segmentation processing by morphology, and this method can quickly detect the text area of ​​the image, thereby improving the efficiency of text detection.

[0020] Other features and advantages of the present invention will be set forth in the following description, or some features and advantages may be inferred or unambiguously determined from the description, or may be learned by implementing the above-mentioned technology of the present invention.

[0021] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are specifically listed below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 A flowchart of a text area detection method provided by an embodiment of the present invention;

[0024] Figure 2 A flowchart of another text area detection method provided by an embodiment of the present invention;

[0025] Figure 3 A schematic diagram of masks of various scales in a target training image provided by an embodiment of the present invention;

[0026] Figure 4 A structural schematic diagram of a text area division model provided by an embodiment of the present invention;

[0027] Figure 5A flowchart of another text area detection method provided by an embodiment of the present invention;

[0028] Figure 6 A schematic diagram of a processing flow of a morphological dilation operation provided by an embodiment of the present invention;

[0029] Figure 7 A schematic diagram of the structure of a text area detection device provided by an embodiment of the present invention;

[0030] Figure 8 A schematic diagram of the structure of a server provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] With the development of intelligent high-definition technology, video processing technology based on image analysis is increasingly being applied to practical tasks. For example, in the live show video task scenario, in addition to analyzing the anchor character, the detection and analysis of the text area has practical application significance for video compression, image quality improvement, and image quality restoration.

[0033] In the related art, in order to realize text detection in an image, an image segmentation algorithm can be used to segment the text area in the image. When a traditional image segmentation algorithm segments an image text, it is usually unable to separate two texts that are very close to each other, resulting in poor text detection effect. Therefore, a step-by-step segmentation algorithm is usually used to perform multi-level segmentation on the text area to obtain multi-scale segmented areas, and then the segmented areas are iteratively expanded pixel by pixel according to the outline size of the segmented areas, and finally the text area in the image is detected. However, this method of text detection is time-consuming and has low detection efficiency.

[0034] Based on this, the embodiments of the present invention provide a text area detection method, device and server. This technology can be widely used for text detection in various scenarios, especially for text detection in complex scenarios such as live broadcast on the Internet, live broadcast on limited TV, games, videos, etc.

[0035] To facilitate understanding of this embodiment, a text region detection method disclosed in an embodiment of the present invention is first described in detail. Figure 1 As shown, the method comprises the following steps:

[0036] Step S102, obtaining an image to be processed.

[0037] The image to be processed can be a picture, or a video frame captured from a video file or live video, etc. The image to be processed usually includes characters, backgrounds, and CG (Computer generated) text, etc. Among them, CG text is usually text generated by a computer and can be applied to image or video post-processing. The most typical application is to add subtitles or bullet comments to the video. These subtitles and bullet comments added later are collectively referred to as CG texts, and the area corresponding to the CG text can be referred to as a text area.

[0038] Step S104, performing text region segmentation processing on the above-mentioned image to be processed to obtain multiple regions to be selected of the text region in the image to be processed; wherein the scales of the multiple regions to be selected are multiple.

[0039] By assigning a label to each pixel in the image to be processed and classifying each pixel according to the label, the segmentation result of a specific area in the image can be obtained, and the specific area can be a person area, a background area or a text area. In specific implementation, the segmentation result of the text area in the image can be obtained according to the characteristics of the pixels in the text area, and the segmentation result can be further segmented at multiple levels to obtain multiple areas to be selected in the text area.

[0040] There are multiple scales of the above-mentioned area to be selected, including the largest scale area to be selected and multiple small scale areas to be selected. Among them, the small scale area to be selected can be obtained by reducing the large scale area to be selected by a certain proportion, or it can be a preset fixed size; the text area in the largest scale area to be selected may include one or more lines of text, and the small scale area to be selected may only contain part of a line of text. Therefore, the large scale area to be selected can usually obtain the outline of the text area, but cannot separate dense text, while the small scale area to be selected can separate dense text, but cannot obtain the outline of the text area, so it is necessary to combine the areas to be selected of different scales to obtain a more accurate text area.

[0041] Step S106, performing morphological processing on the multiple regions to be selected to obtain text regions of the image to be processed.

[0042] This morphology can also be called mathematical morphology. Morphology can extract image components that are meaningful for expressing and depicting the shape of the region from the image, so that subsequent image recognition and image detection work can grasp the most essential or most distinguishing shape features of the target object. The shape features can be the boundaries of a certain area or connected areas, etc. Morphological processing is performed on the areas to be selected at different scales to obtain the text area under the constraints of the area to be selected at the largest scale. The obtained text area usually includes the segmentation results of each line of text, or the minimum bounding rectangle of each line of text, etc.

[0043] The present invention provides a method for detecting text regions, which comprises the following steps: firstly obtaining an image to be processed; then performing text region segmentation processing on the image to be processed to obtain a plurality of regions to be selected of the text region in the image to be processed; and finally performing morphological processing on the plurality of regions to be selected to obtain the text region of the image to be processed. The present invention processes the regions to be selected obtained after the text region segmentation processing by morphology, and this method can quickly detect the text region of the image, thereby improving the efficiency of text detection.

[0044] The embodiment of the present invention also provides another text region detection method, which is implemented on the basis of the method described in the above embodiment; the method focuses on describing the specific process of segmenting the text region of the image to be processed by the text region segmentation model, and the specific training process of the text region segmentation model, which is described in detail in the explanation of the following step S204; Figure 2 As shown, the detection method comprises the following steps:

[0045] Step S202, obtaining an image to be processed.

[0046] Step S204: input the image to be processed into a pre-trained text region segmentation model to obtain a plurality of regions to be selected in the text region of the image to be processed.

[0047] The above-mentioned text region segmentation model can be a neural network, other artificial intelligence networks or models, and can be trained through the following steps 01-04:

[0048] Step 01, determining a target training image based on a preset training set; wherein the target training image carries text region identifiers of multiple scales.

[0049] The preset training set usually includes multiple images (for example, 3,000, 5,000 or even more), and may also include images of various scenes, such as live scene images, game scene images, outdoor scene images, indoor scene images, etc.; the images in the training set may also include text lines of various colors, font sizes, shapes, fonts, and languages, so that the trained model can segment various types of text lines.

[0050] Each image in the training set contains a manually annotated text region, and the target training images determined from the preset training set carry text region identifiers of multiple scales, which can be a rectangular border of the text region or other borders that match the text.

[0051] The above-mentioned text area mark can be a rectangular frame marked on the image, or a mask corresponding to the text area in the target training image; the mask usually covers the selected area in the image in whole or in part to control the area of ​​image processing; the area covered by the mask is usually the area within the rectangular frame or other frame matching the text in the target training image. There are masks of multiple scales corresponding to the text area in the target training image, wherein the scale of the mask with the largest scale is the same as the scale of the text area in the target training image; the scales of the masks other than the mask with the largest scale are obtained by reducing the scale of the mask of the previous scale by a preset percentage.

[0052] In the specific implementation, the text area in the target training image F can be filled with 1, and other areas can be filled with 0 to form the maximum scale mask S n ,like Figure 3 As shown, the maximum scale mask S is then n The corresponding text area is reduced by a preset percentage (e.g., 10%) to obtain the mask S of the next scale. n-1 , and then the mask S of the next scale n-1 The corresponding text area is reduced according to a preset percentage to obtain the mask S of the next scale n-2 , continue to shrink according to the preset ratio until the minimum scale mask S is obtained 1 The size of the minimum-scale mask is usually set according to demand, that is, when the mask is reduced to the minimum scale, it will stop shrinking.

[0053] Step 02: input the target training image into the initial model, and output pixel feature data of the target training image; the pixel feature data includes the feature value of each pixel in the target training image.

[0054] The initial model is usually a convolutional neural network model or other types of neural networks, in which the convolutional neural network is usually a neural network designed specifically for image recognition problems, which imitates the multi-layer process of human image recognition, such as pupil intake of pixels, preliminary processing of certain cells in the cerebral cortex (such as finding shape edges and directions), abstract judgment of shapes (such as circles and squares), and further abstract judgment (such as judging that the object is a balloon). Before inputting the target training image into the initial model, the target training model can be adjusted to an RGB (Red Green Blue) image of a preset size. For example, adjust the target training image to a standard 320*320 RGB image.

[0055] Step 03, calculate the model loss value based on the above pixel feature data and text area identification.

[0056] Based on the pixel feature data and the text area identifier of each scale, the loss value of the corresponding scale can be calculated, and the weighted sum of the loss values ​​of each scale can be obtained to obtain the model loss value. Since the text area only accounts for a small proportion of the image, when calculating the model loss value, it is only necessary to select a non-text area that is very similar to the text area to test the model output performance. This can be achieved through the following steps 031-034:

[0057] Step 031, for each scale, according to the text area identifier of the current scale, determine the non-text area corresponding to the current scale from the target training image; in specific implementation, the text area identifier of the current scale can be inverted to obtain the identifier of the non-text area, for example, 1 in the text area identifier is changed to 0, and 0 is changed to 1.

[0058] Step 032, extracting a preset number of pixels from the non-text area according to the characteristic values ​​of the pixels in the non-text area; the preset number is usually related to the number of pixels in the current scale text area, and can be 2 times or 3 times the number of pixels in the text area. In specific implementation, the characteristic values ​​of the pixels in the non-text area can be sorted in descending order, and a preset number of pixels with top characteristic values ​​in the non-text area are selected.

[0059] Step 033, calculate the loss value corresponding to the current scale according to the feature values ​​of the preset number of pixels extracted and the feature values ​​of the pixels in the text area corresponding to the text area identifier; the loss value is related to the Dice coefficient, which is usually a set similarity measurement function, which is the same as the similarity index and is also called a coefficient.

[0060] The above loss value Where x and y represent the horizontal and vertical coordinates of the pixel position respectively; when the pixel belongs to the text area, S x,y =1; when the pixel belongs to the preset number of pixels extracted, S x,y =0; G x,y It represents the feature values ​​of a preset number of pixels extracted or the feature values ​​of pixels in the text area.

[0061] Step 034, calculate the model loss value according to the loss value corresponding to the current scale and the weighting coefficient corresponding to each scale.

[0062] The weighting coefficient corresponding to each scale can be set with different weights according to different scales, or it can be set as an empirical value. For example, the weighting coefficient corresponding to the maximum scale can be set to 0.7, and the weighting coefficients corresponding to other small scales can be set to 0.3. In the specific implementation, the weighting coefficient corresponding to each scale is multiplied by the loss value corresponding to the scale, and then the multiplication results of each scale are added to obtain the model loss value. For example, when there are three scales, assuming that the loss value corresponding to the maximum scale is 0.1 and the weighting coefficient is 0.7, the loss value corresponding to the next scale of the maximum scale is 0.2 and the weighting coefficient is 0.3, and the loss value corresponding to the minimum scale is 0.3 and the weighting coefficient is 0.3, then the model loss value is 0.7*0.1+0.3*0.2+0.3*0.3=0.22.

[0063] Step 04, adjust the network parameters in the initial model according to the above model loss value, and continue to perform the step of determining the target training image based on the preset training set until the model loss value meets the preset specified value or the number of target training images input into the initial model meets the preset number, thereby obtaining the text area segmentation model. Figure 4 A structural schematic diagram of a text area segmentation model is shown; a target training image is input into a convolutional neural network, pixel feature data can be output, a model loss value can be obtained based on the pixel feature data, a pixel score map of the target training image can be obtained by the pixel feature data through a sigmoid function, and multiple regions to be selected of the target training image can be obtained by the pixel feature data through a sign function. Threshold processing can be performed on the above pixel score map to obtain a maximum scale mask, for example, the area corresponding to pixels with a feature value greater than or equal to a preset threshold (for example, 0.5) is filled with 1, and the area corresponding to pixels with a feature value less than the preset threshold is filled with 0, so as to obtain a maximum scale mask.

[0064] The initial model adjusts the network parameters according to the model loss value to optimize the initial model. As the initial model is continuously optimized, the pixel feature values ​​of the text area in the target training image are getting closer to 1, and the pixel feature values ​​of the non-text area are getting closer to 0. When the model loss value is greater than the preset specified value, or the number of target training images input into the initial model is less than the preset number (equivalent to the network model training not meeting the number of iterations), the target training images will continue to be input into the initial model and the model loss value will be output.

[0065] Step S206: performing morphological processing on the multiple regions to be selected to obtain text regions of the image to be processed.

[0066] The above-mentioned text region detection method inputs the image to be processed into a pre-trained text region segmentation model to obtain multiple regions to be selected in the text region of the image to be processed, and performs morphological processing on the multiple regions to be selected to obtain the text region of the image to be processed. This method uses the text region segmentation model to segment the text region, and can quickly and accurately obtain the image segmentation result, which is conducive to improving the efficiency of text region detection.

[0067] The embodiment of the present invention also provides another text region detection method, which is implemented on the basis of the method described in the above embodiment; the method focuses on describing the specific process of morphological processing of multiple regions to be selected, and the details are explained in the following steps S506 to S512; Figure 5 As shown, the detection method comprises the following steps:

[0068] Step S502: obtaining an image to be processed.

[0069] Step S504, inputting the image to be processed into a pre-trained text region segmentation model to obtain multiple regions to be selected of the text region in the image to be processed; among the multiple regions to be selected, the regions to be selected of the same scale are marked in the same mask.

[0070] The above-mentioned multiple areas to be selected include multiple scales. The scale of the area to be selected is the same as the scale of the mask. It can also be understood that the areas to be selected of the same scale are marked in the same mask. For example, for an area to be selected of a certain scale, the area in the area to be selected of the scale is marked as 1, and the area outside the area to be selected is marked as 0, so that the mask at the scale can be obtained. Before inputting the image to be processed into the text area segmentation module, it is usually necessary to adjust the image to be processed to a standard 320*320 RGB image.

[0071] Step S506: extracting a mask corresponding to a region to be selected with the smallest scale from the multiple regions to be selected.

[0072] The multiple regions to be selected include multiple scales, each candidate region of each scale corresponds to a mask of a corresponding scale, and a mask of a minimum scale is selected from the masks of multiple scales, so that the mask of the minimum scale can gradually approach the mask of the maximum scale.

[0073] Step S508, determining a connected region from the mask corresponding to the area to be selected with the smallest scale. The connected region is usually a mask region that is horizontally connected in the mask corresponding to the area to be selected, wherein one connected region usually corresponds to one text line in the image to be processed.

[0074] The mask corresponding to the minimum scale of the selected area may include one or more connected areas, such as Figure 3As shown, the mask corresponding to the minimum scale of the selected area is equivalent to S 1 , Figure 3 Medium S 1 There are four connected regions.

[0075] Step S510: Using a preset template, perform a morphological dilation operation on the connected area.

[0076] The above preset template is usually a conventional morphological template. In specific implementation, an image matrix can be established according to the structural element, and then the origin of the structural element can be determined from the image matrix to obtain the morphological template. The morphological template can be in various shapes, such as square, rectangle or other irregular shapes; Figure 6 The matrix composed of four pixels in is the morphological template, which is a square.

[0077] A morphological dilation operation is performed on each connected region corresponding to the minimum scale area to be selected. The processing process of the morphological dilation operation is as follows: Figure 6 As shown, first, an image corresponding to a connected area with a text area identifier is obtained, wherein the text area within the connected area is filled with 1, and the area outside the connected area is filled with 0; then, a morphological template is used to slide from left to right and from top to bottom in the image corresponding to the connected area with a text area identifier (equivalent to the original image in the figure), so that the morphological template can traverse each pixel in the image, and when the pixel in the image to which the morphological template slides is multiplied by the filling number of the corresponding position of the pixel in the morphological template and the sum is greater than or equal to 1, the position of the pixel in the image to which the morphological template slides corresponding to the origin pixel in the morphological model is filled with 1, and the surroundings of the pixels filled with 1 in the image are also filled with 1 to obtain the expanded connected area.

[0078] Figure 6 The first image in the figure is the original image without morphological dilation, and the image in the lower right corner is the image after morphological dilation. Figure 6 It can be seen that the image area of ​​the text area filled with 1 in the expanded image is larger than that of the original image, that is, the expanded image is equivalent to filling some of the surrounding pixels with 1 in the pixels filled with 1 in the original image.

[0079] Step S512: determining the text area of ​​the image to be processed according to the expanded connected area.

[0080] After each minimum-scale connected region is expanded, each expanded connected region obtained corresponds to a text line in the image. The expanded connected regions obtained are combined to obtain the text region of the image to be processed. Then, according to the expanded connected regions, the step of determining the text region of the image to be processed can be implemented by the following steps 10-14:

[0081] Step 10, determining the previous scale of the smallest scale as the current scale; wherein the masks corresponding to the to-be-selected regions of different scales are arranged in descending order according to the scales of the to-be-selected regions, thereby obtaining masks whose scales are arranged in descending order.

[0082] Step 11, bitwise operation is performed on the pixel values ​​of the corresponding pixels in the expanded connected area and the mask of the current scale to obtain the operation result; the pixel value is usually the fill value of the pixel at the position, usually 1 or 0. In specific implementation, the pixel values ​​in the image corresponding to the expanded connected area and the pixel values ​​at the corresponding positions in the image corresponding to the mask of the current scale are bitwise operated (for example, AND) to obtain the operation result.

[0083] Step 12, continue to perform morphological dilation operation on the expanded connected area until the operation result converges; if the operation result obtained this time is different from the operation result obtained last time, then continue to perform morphological dilation operation on the expanded connected area until the operation result converges, which can also be understood as the operation result obtained this time is the same as the operation result obtained last time.

[0084] Step 13, determine the previous scale of the current scale as the new current scale, and continue the above step 11 until the current scale is the maximum scale and the operation result converges. For example, determine the previous scale of the previous scale of the minimum scale as the current scale, perform bitwise operation on the pixel values ​​of the corresponding positions in the mask of the expanded connected area and the current scale; determine whether to continue to expand the expanded connected area based on the bitwise operation result obtained this time and the bitwise operation result obtained last time; if the expansion is not to be continued, determine the previous scale of the current scale as the new current scale and perform bitwise operation on the expanded connected area until the maximum scale is determined as the current scale.

[0085] Step 14: Determine the text area of ​​the image to be processed based on the connected area after the last dilation.

[0086] The above steps 10-14 are performed for each connected area corresponding to the area to be selected of the minimum scale to obtain the connected area after the last expansion. In order to determine whether the connected area after the last expansion is a text area, it is necessary to extract the pixel feature values ​​corresponding to the connected area obtained by the last expansion operation from the pixel feature data of the image to be processed output by the pre-trained text area segmentation model; calculate the average value of the extracted pixel feature values; if the average value is greater than the preset average value threshold, determine that the connected area obtained by the last expansion operation is a text area.

[0087] For example, assuming that the average value threshold is 0.93, the connected area after the last expansion contains 4 pixels, and the eigenvalues ​​of the pixels are 1, 0.95, 0.99, and 0.9 respectively. The average value of the eigenvalues ​​of the 4 pixels is 0.96, which is greater than the average value threshold. Therefore, the connected area after the last expansion is a text area.

[0088] Step S514, input the above text area and the image to be processed into the minAreaRect function of opencv, and output the image to be processed marked with the text area border.

[0089] The above minAreaRect function can usually obtain the minimum circumscribed rectangle of the connected area corresponding to the text area (equivalent to the border of the text area marked above), and the rectangle can have a deflection angle and a border that is not parallel to the boundary.

[0090] The above-mentioned text area detection method inputs the image to be processed into a pre-trained text area segmentation model to obtain multiple areas to be selected for the text area in the image to be processed; then extracts the mask corresponding to the area to be selected with the smallest scale from the multiple areas to be selected, and determines the connected area therefrom; uses a preset template to perform a morphological expansion operation on the connected area; then determines the text area of ​​the image to be processed based on the expanded connected area, and finally inputs the text area and the image to be processed into the minAreaRect function of opencv to output the image to be processed with the text area border marked. This method can use morphology to perform fast text area detection with the contour maximization constraint as the boundary condition to improve the efficiency of text area detection and the accuracy of image segmentation.

[0091] Corresponding to the above method embodiment, the present invention provides a text area detection device, such as Figure 7 As shown, the device comprises:

[0092] An image acquisition module 70, used to acquire an image to be processed;

[0093] The to-be-selected region determination module 71 is used to perform text region segmentation processing on the above-mentioned to-be-processed image to obtain multiple to-be-selected regions of the text region in the to-be-processed image; wherein the scales of the multiple to-be-selected regions are multiple;

[0094] The text region determination module 72 is used to perform morphological processing on the multiple regions to be selected to obtain the text region of the image to be processed.

[0095] The above-mentioned text area detection device first obtains the image to be processed; then performs text area segmentation processing on the image to be processed to obtain multiple areas to be selected in the text area of ​​the image to be processed; finally, performs morphological processing on the multiple areas to be selected to obtain the text area of ​​the image to be processed. The present invention processes the multiple areas to be selected obtained by image segmentation processing through morphology to obtain the text area of ​​the image. This method can quickly detect the text area of ​​the image, thereby improving the efficiency of text detection.

[0096] Furthermore, the above-mentioned module for determining the area to be selected 71 is used to: input the image to be processed into a pre-trained text area division model to obtain multiple areas to be selected in the text area in the image to be processed; wherein the above-mentioned device also includes a model training module, which is used to: determine the target training image based on a preset training set; wherein the target training image carries text area identifiers of multiple scales; input the target training image into the initial model, and output pixel feature data of the target training image; the pixel feature data contains the feature value of each pixel in the target training image; based on the pixel feature data and the text area identifier, calculate the model loss value; adjust the network parameters in the initial model according to the model loss value, and continue to execute the step of determining the target training image based on the preset training set, until the model loss value meets the preset specified value or the number of target training images input into the initial model meets the preset number, and obtain the text area division model.

[0097] The above-mentioned text area identification includes: a mask corresponding to the text area in the target training image; wherein the scale of the maximum-scale mask is the same as the scale of the text area in the target training image; the scales of the masks other than the maximum-scale mask are obtained by reducing the scale of the previous-scale mask by a preset percentage.

[0098] Furthermore, the above-mentioned model training module is also used to: for each scale, determine the non-text area corresponding to the current scale from the target training image according to the text area identifier of the current scale; extract a preset number of pixels from the non-text area according to the feature values ​​of the pixels in the non-text area; calculate the loss value corresponding to the current scale according to the feature values ​​of the extracted preset number of pixels and the feature values ​​of the pixels in the text area corresponding to the text area identifier; calculate the model loss value according to the loss value corresponding to the current scale and the weighting coefficient corresponding to each scale.

[0099] The above loss value Where x and y represent the horizontal and vertical coordinates of the pixel position respectively; when the pixel belongs to the text area, S x,y =1; when the pixel belongs to the preset number of pixels extracted, S x,y =0; G x,y Indicates the feature values ​​of a preset number of pixels or pixels in a text area extracted.

[0100] Among the above-mentioned multiple areas to be selected, the areas to be selected of the same scale are identified in the same mask; the above-mentioned text area determination module 72 is used to: extract the mask corresponding to the area to be selected with the smallest scale from the multiple areas to be selected; determine the connected area from the mask corresponding to the area to be selected with the smallest scale; use a preset template to perform a morphological dilation operation on the connected area; and determine the text area of ​​the image to be processed based on the dilated connected area.

[0101] The above-mentioned text area determination module 72 is also used to: determine the previous scale of the smallest scale as the current scale; wherein the masks corresponding to the to-be-selected areas of different scales are arranged in descending order according to the scale of the to-be-selected areas; perform bitwise operation on the pixel values ​​of the corresponding pixels in the expanded connected area and the mask of the current scale to obtain the operation result; continue to perform morphological expansion operation on the expanded connected area until the operation result converges; determine the previous scale of the current scale as the new current scale, and continue to perform bitwise operation on the pixel values ​​of the corresponding pixels in the mask of the current scale and obtain the operation result until the current scale is the maximum scale and the operation result converges; determine the text area of ​​the image to be processed according to the connected area after the last expansion.

[0102] Furthermore, the above-mentioned text area determination module 72 is used to: extract the pixel feature values ​​corresponding to the connected area obtained by the last expansion operation from the pixel feature data of the image to be processed output by the pre-trained text area division model; calculate the average value of the extracted pixel feature values; if the average value is greater than a preset average value threshold, determine that the connected area obtained by the last expansion operation is a text area.

[0103] Furthermore, the above-mentioned device also includes: a border marking module, which is used to input the text area and the image to be processed into the minAreaRect function of opencv, and output the image to be processed marked with the text area border.

[0104] The text area detection device provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0105] The embodiment of the present invention also provides a server for executing the text area detection method; see Figure 8 As shown, the server includes a processor 101 and a memory 100. The memory 100 stores machine executable instructions that can be executed by the processor 101. The processor 101 executes the machine executable instructions to implement the above-mentioned text area detection method.

[0106] Further, Figure 8 The server shown also includes a bus 102 and a communication interface 103 , and the processor 101 , the communication interface 103 and the memory 100 are connected via the bus 102 .

[0107] The memory 100 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 103 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus 102 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0108] The processor 101 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 101 or the instruction in the form of software. The above processor 101 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present invention can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 100, and the processor 101 reads the information in the memory 100 and completes the steps of the method of the above embodiment in combination with its hardware.

[0109] An embodiment of the present invention also provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the above-mentioned text area detection method. The specific implementation can be found in the method embodiment, which will not be repeated here.

[0110] The computer program product of the text area detection method, device and server provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the method described in the previous method embodiments. The specific implementation can be found in the method embodiments, which will not be repeated here.

[0111] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting a text area, characterized in that: The method comprises: Get the image to be processed; Performing text region segmentation processing on the image to be processed to obtain a plurality of to-be-selected regions of the text region in the image to be processed; wherein the plurality of to-be-selected regions have a plurality of scales, including a to-be-selected region of a maximum scale and a plurality of to-be-selected regions of a small scale, the text region in the to-be-selected region of the maximum scale includes one or more lines of text, and the to-be-selected region of the small scale includes a partial text of a line of text; Performing morphological processing on the multiple areas to be selected to obtain a text area of ​​the image to be processed; After the step of performing morphological processing on the multiple areas to be selected to obtain the text area of ​​the image to be processed, the method further includes: Input the text area and the image to be processed into the minAreaRect function of opencv, and output the image to be processed marked with the text area border; Among the multiple areas to be selected, areas to be selected of the same scale are marked in the same mask; The step of performing morphological processing on the multiple areas to be selected comprises: Extracting a mask corresponding to a region to be selected with the smallest scale from the multiple regions to be selected; Determine a connected area from the mask corresponding to the area to be selected of the minimum scale; Using a preset template, a morphological dilation operation is performed on the connected area; wherein an image matrix is ​​established according to the structural element, and the origin of the structural element is determined from the image matrix to obtain the template; The text region of the image to be processed is determined according to the dilated connected region.

2. The method according to claim 1, characterized in that The step of performing text region segmentation processing on the image to be processed comprises: inputting the image to be processed into a pre-trained text region segmentation model to obtain a plurality of to-be-selected regions of the text region in the image to be processed; The text area segmentation model is trained in the following way: Determining a target training image based on a preset training set; wherein the target training image carries text region identifiers of multiple scales; Input the target training image into the initial model, and output pixel feature data of the target training image; the pixel feature data includes a feature value of each pixel in the target training image; Calculating a model loss value based on the pixel feature data and the text area identifier; The network parameters in the initial model are adjusted according to the model loss value, and the step of determining the target training image based on the preset training set is continued until the model loss value meets the preset specified value or the number of target training images input into the initial model meets the preset number, thereby obtaining the text area segmentation model.

3. The method according to claim 2, characterized in that The text region identification includes: a mask corresponding to the text region in the target training image; The scale of the maximum-scale mask is the same as the scale of the text area in the target training image; the scales of the masks other than the maximum-scale mask are obtained by reducing the scale of the previous-scale mask by a preset percentage.

4. The method according to claim 2, characterized in that: The step of calculating the model loss value based on the pixel feature data and the text area identifier comprises: For each scale, according to the text area identifier of the current scale, determining the non-text area corresponding to the current scale from the target training image; Extracting a preset number of pixels from the non-text area according to feature values ​​of the pixels in the non-text area; Calculate the loss value corresponding to the current scale according to the feature values ​​of the preset number of pixels extracted and the feature values ​​of the pixels in the text area corresponding to the text area identifier; The model loss value is calculated according to the loss value corresponding to the current scale and the weighting coefficient corresponding to each scale.

5. The method according to claim 4, characterized in that The step of calculating the loss value corresponding to the current scale according to the extracted feature values ​​of the preset number of pixels and the feature values ​​of the pixels in the text area corresponding to the text area identifier comprises: The loss value Where x and y represent the horizontal and vertical coordinates of the pixel position respectively; when the pixel belongs to the text area, S x,y =1; when the pixel belongs to the preset number of pixels extracted, S x,y =0; G x,y Represents the extracted feature values ​​of the preset number of pixels or pixels in the text area.

6. The method according to claim 1, characterized in that The step of determining the text area of ​​the image to be processed according to the expanded connected area comprises: The previous scale of the minimum scale is determined as the current scale; wherein the masks corresponding to the to-be-selected regions of different scales are arranged in descending order according to the scale of the to-be-selected regions; Performing a bitwise operation on the expanded connected area and the pixel values ​​of corresponding pixels in the mask of the current scale to obtain an operation result; Continue to perform morphological dilation operation on the dilated connected area until the operation result converges; The previous scale of the current scale is determined as a new current scale, and the step of performing bitwise operation on the expanded connected area and the pixel values ​​of corresponding pixels in the mask of the current scale to obtain the operation result is continued until the current scale is the maximum scale and the operation result converges; The text region of the image to be processed is determined according to the connected region after the last dilation.

7. The method according to claim 6, characterized in that The step of determining the text area of ​​the image to be processed according to the connected area after the last dilation comprises: Extracting pixel feature values ​​corresponding to the connected area obtained by the last dilation operation from the pixel feature data of the image to be processed output by the text area division model; Calculating the average value of the extracted pixel feature values; If the average value is greater than a preset average value threshold, it is determined that the connected region obtained by the last dilation operation is a text region.

8. A text area detection device, characterized in that: The device comprises: An image acquisition module, used for acquiring an image to be processed; A module for determining regions to be selected, configured to perform text region segmentation processing on the image to be processed, and obtain multiple regions to be selected of the text region in the image to be processed; wherein the multiple regions to be selected have multiple scales, including a region to be selected with a maximum scale and multiple regions to be selected with a small scale, the text region in the region to be selected with a maximum scale includes one or more lines of text, and the region to be selected with a small scale includes part of a line of text; A text region determination module, used for performing morphological processing on the multiple regions to be selected to obtain the text region of the image to be processed; A border marking module, used for inputting the text area and the image to be processed into the minAreaRect function of opencv, and outputting the image to be processed marked with the text area border; Among the multiple to-be-selected regions, the to-be-selected regions of the same scale are identified in the same mask; and the text region determination module is used to: Extracting a mask corresponding to a region to be selected with the smallest scale from the multiple regions to be selected; Determine a connected area from the mask corresponding to the area to be selected of the minimum scale; Using a preset template, a morphological dilation operation is performed on the connected area; wherein an image matrix is ​​established according to the structural element, and the origin of the structural element is determined from the image matrix to obtain the template; The text region of the image to be processed is determined according to the dilated connected region.

9. A server, characterized in that: The invention comprises a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the text area detection method according to any one of claims 1 to 7.

10. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the text area detection method according to any one of claims 1 to 7.