Text detection method, device, and storage medium

Abstract

A text detection method, device, and storage medium. The text detection method includes: obtaining a text feature image corresponding to the text image based on a text image; using a portion of the text feature image near a first edge of the text feature image as a base region, wherein the first edge of the text feature image corresponds to the first edge of the text image, the text to be detected is near the first edge of the text image, and at least some pixels in the base region are positive pixels; grouping at least some of the positive pixels in the base region to obtain at least one connected domain; extending the at least one connected domain in a direction away from the first edge of the text feature image to obtain at least one final connected domain; determining at least one feature box corresponding to the at least one final connected domain, and mapping the at least one feature box to the text image to obtain at least one text box, wherein the text box includes the text box of the text to be detected.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to a text detection method, a text detection device, and a storage medium. Background Art

[0002] With the development of science and technology, when users read foreign articles and encounter unfamiliar words that need to be looked up, they are no longer limited to using dictionaries, electronic dictionaries, mobile phone apps (applications), etc. to search. They can also use translation pens, for example. Dictionaries are not easy to carry, and the efficiency of flipping through and searching is low; mobile phone apps and electronic dictionaries use keyboard input, which is not only time-consuming and cumbersome to operate, but also easily interrupts thoughts and distracts attention. In contrast, translation pens have the advantages of being easy to use, easy to carry, and more in line with users' reading habits. They can provide users with a good translation and search experience when reading foreign articles. Summary of the Invention

[0003] At least one embodiment of the present disclosure provides a text detection method, comprising: obtaining a text feature image corresponding to a text image based on a text image; using a partial area of ​​the text feature image close to a first edge of the text feature image as a base area, wherein the first edge of the text feature image corresponds to the first edge of the text image, text to be detected in the text image is close to the first edge of the text image, and at least some pixels in the base area are positive pixels; grouping at least some of the positive pixels in the base area to obtain at least one connected domain; extending the at least one connected domain in a direction away from the first edge of the text feature image to obtain at least one final connected domain corresponding to the at least one connected domain; and determining at least one feature box corresponding to the at least one final connected domain, and mapping the at least one feature box to the text image to obtain at least one text box, wherein the at least one text box includes a text box of the text to be detected.

[0004] For example, in the text detection method provided in some embodiments of the present disclosure, when the text feature image includes h rows and w columns of pixels, the basic area includes h base Row w column pixels, where h, w, h base are all positive integers, and h base / h≤1 / 2.

[0005] For example, in the text detection method provided in some embodiments of the present disclosure, there is a connection probability between each pixel in the text feature image and the directly adjacent pixel; grouping at least some of the positive pixels in the basic area to obtain the at least one connected domain includes: based on a union-find algorithm, grouping the at least some of the positive pixels in the basic area according to the connection probability between each positive pixel in the at least some of the positive pixels in the basic area and the directly adjacent pixel to obtain the at least one connected domain.

[0006] For example, in the text detection method provided in some embodiments of the present disclosure, based on the union-find algorithm, the at least some positive pixels in the basic area are grouped according to the connection probability between each positive pixel in the at least some positive pixels in the basic area and the directly adjacent pixels, so as to obtain the at least one connected domain, including: constructing an index set based on the at least some positive pixels in the basic area, wherein the index set includes the at least some positive pixels in the basic area, and in the index set, each positive pixel corresponds to a root node, and the initial value of the root node of each positive pixel is itself; in response to any directly adjacent pixel of each positive pixel in the index set being a positive pixel and each positive pixel having a positive connection relationship with the directly adjacent pixel, setting the value of the root node of the directly adjacent pixel to the value of the root node of each positive pixel; and taking each group of positive pixels with the same root node value as a connected domain to obtain the at least one connected domain.

[0007] For example, in the text detection method provided in some embodiments of the present disclosure, when the connection probability between each positive pixel in the basic area and the directly adjacent pixel is greater than the connection probability threshold, it is determined that each positive pixel has the positive connection relationship with the directly adjacent pixel.

[0008] For example, in the text detection method provided in some embodiments of the present disclosure, the directly adjacent pixels of each positive pixel in the basic area include: pixels directly adjacent to each positive pixel in a first direction perpendicular to the first edge of the text feature image, and pixels directly adjacent to each positive pixel in a second direction parallel to the first edge of the text feature image.

[0009] For example, in the text detection method provided by some embodiments of the present disclosure, each positive pixel in the basic area has four directly adjacent pixels.

[0010] For example, in the text detection method provided in some embodiments of the present disclosure, the at least one connected domain is expanded along a direction away from the first edge of the text feature image to obtain the at least one final connected domain corresponding to the at least one connected domain, including: extracting the positive pixel in the current connected domain that is farthest from the first edge of the text feature image in a first direction perpendicular to the first edge of the text feature image as the first positive pixel; taking the pixel in the text feature image that is on the side of the first positive pixel away from the first edge of the text feature image and directly adjacent to the first positive pixel as the first neighboring pixel; in response to the first neighboring pixel being a positive pixel and the first positive pixel having a positive connection relationship with the first neighboring pixel, modifying the value of the root node of the first neighboring pixel to the value of the root node of the first positive pixel, and adding the first neighboring pixel to the first neighboring pixel set; expanding the first neighboring pixel set in a second direction parallel to the first edge of the text feature image; and expanding the current connected domain to include all pixels in the first neighboring pixel set, and continuing to expand the current connected domain along the direction away from the first edge of the text feature image until it can no longer be expanded.

[0011] For example, in the text detection method provided in some embodiments of the present disclosure, the first set of neighboring pixels is expanded in a second direction parallel to the first edge of the text feature image, including: adding positive pixels that are directly adjacent to any pixel in the first set of neighboring pixels in the second direction parallel to the first edge of the text feature image and have a positive connection relationship to the first set of neighboring pixels until the first set of neighboring pixels can no longer be expanded in the direction parallel to the first edge of the text feature image.

[0012] For example, in the text detection method provided by some embodiments of the present disclosure, the at least one final connected domain includes a connected domain within the basic area that cannot be extended in a direction away from the first edge of the text feature image.

[0013] For example, in the text detection method provided in some embodiments of the present disclosure, based on the text image, the text feature image corresponding to the text image is obtained, including: using a text detection neural network to process the text image to obtain the text feature image, and obtaining the connection probability between each pixel in the text feature image and the directly adjacent pixels.

[0014] For example, in the text detection method provided in some embodiments of the present disclosure, the text detection neural network includes first to sixth convolution modules, first to fifth downsampling modules, first to fourth upsampling modules, and a classifier; the text detection neural network is used to process the text image to obtain the text feature image, and the connection probability between each pixel in the text feature image and the directly adjacent pixels is obtained, including: using the first convolution module to convolve the text image to obtain a first convolution feature map group; using the first downsampling module to downsample the first convolution feature map group to obtain a first downsampled feature map group; using the second convolution module to convolve the first downsampled feature map group to obtain a second convolution feature map group; down-sample the second convolution feature map group using a second down-sampling module to obtain a second down-sampled feature map group, and use a fifth dimensionality reduction module to perform dimensionality reduction processing on the second convolution feature map group to obtain a fifth reduced dimensionality feature map group; convolve the second down-sampled feature map group using a third convolution module to obtain a third convolution feature map group; down-sample the third convolution feature map group using a third down-sampling module to obtain a third down-sampled feature map group, and use a fourth dimensionality reduction module to perform dimensionality reduction processing on the third convolution feature map group to obtain a fourth reduced dimensionality feature map group; convolve the third down-sampled feature map group using a fourth convolution module to obtain a fourth convolution feature map group; downsampling the fourth convolution feature map group using a fourth downsampling module to obtain a fourth downsampling feature map group, and performing dimensionality reduction processing on the fourth convolution feature map group using a third dimensionality reduction module to obtain a third reduced dimensionality feature map group; convolution processing on the fourth downsampling feature map group using a fifth convolution module to obtain a fifth convolution feature map group; downsampling the fifth convolution feature map group using the fifth downsampling module to obtain a fifth downsampling feature map group, and performing dimensionality reduction processing on the fifth convolution feature map group using the second dimensionality reduction module to obtain a second reduced dimensionality feature map group; convolution processing on the fifth downsampling feature map group using a sixth convolution module to obtain a sixth convolution feature map group; Using a first upsampling module to upsample the sixth convolutional feature map group to obtain a first upsampled feature map group; using a first dimensionality reduction module to perform dimensionality reduction processing on the first upsampled feature map group to obtain a first reduced dimensionality feature map group; fusing the first reduced dimensionality feature map group and the second reduced dimensionality feature map group to obtain a first fused feature map group; using a second upsampling module to upsample the first fused feature map group to obtain a second upsampled feature map group; fusing the second upsampled feature map group and the third reduced dimensionality feature map group to obtain a second fused feature map group; using a third upsampling module to upsample the second fused feature map group to obtain a third upsampled feature map group;The third upsampled feature map group and the fourth reduced-dimensionality feature map group are fused to obtain a third fused feature map group; the third fused feature map group is upsampled using a fourth upsampling module to obtain a fourth upsampled feature map group; the fourth upsampled feature map group and the fifth reduced-dimensionality feature map group are fused to obtain a fourth fused feature map group; the fourth fused feature map group is classified using a classifier to obtain a text classification prediction image and a connection probability prediction image; and the text feature image is obtained based on the text classification prediction image and the connection probability prediction image, and the connection probability between each pixel in the text feature image and its directly adjacent pixels is obtained.

[0015] For example, in the text detection method provided in some embodiments of the present disclosure, each pixel in the text classification prediction image has a type probability, and each pixel in the connection probability prediction image has a connection probability between the pixel and a directly adjacent pixel; based on the text classification prediction image and the connection probability prediction image, the text feature image is obtained, and the connection probability between each pixel in the text feature image and its adjacent pixels is obtained, including: taking pixels in the text classification prediction image whose type probability is greater than or equal to a type probability threshold as positive pixels, and taking pixels in the text classification prediction image whose type probability is less than the type probability threshold as negative pixels, to obtain the text feature image, and the connection probability between each pixel in the text feature image and a directly adjacent pixel can be correspondingly queried from the connection probability prediction image.

[0016] For example, in the text detection method provided in some embodiments of the present disclosure, determining the at least one feature box corresponding to the at least one final connected domain includes: performing contour detection on the at least one final connected domain using a contour detection algorithm to obtain the contour of the at least one final connected domain; and processing the contour of the at least one final connected domain using a minimum circumscribed rectangle algorithm to obtain the at least one feature box corresponding to the at least one final connected domain.

[0017] For example, some embodiments of the present disclosure provide a text detection method that further includes: determining a text box containing the text to be detected from the at least one text box.

[0018] For example, in the text detection method provided in some embodiments of the present disclosure, the text box of the text to be detected is determined from the at least one text box, including: constructing a virtual detection box in the text image; and calculating the overlapping area between the virtual detection box and each text box, and taking the text box with the largest overlapping area with the virtual detection box as the text box of the text to be detected.

[0019] For example, some embodiments of the present disclosure provide a text detection method that further includes: performing recognition processing on the text to be detected based on the text frame of the text to be detected.

[0020] For example, some embodiments of the present disclosure provide a text detection method that further includes: using an image acquisition element of a point translation pen to acquire the text image; wherein, when acquiring the text image, the tip of the point translation pen is pointed at a side of the text to be detected close to a first edge of the text image, and the text image includes the text to be detected.

[0021] At least one embodiment of the present disclosure further provides a text detection device, comprising: a memory for storing a text image and computer-readable instructions; a processor for reading the text image and executing the computer-readable instructions, wherein the computer-readable instructions, when executed by the processor, execute the text detection method provided in any embodiment of the present disclosure.

[0022] For example, the text detection device provided by some embodiments of the present disclosure further includes: an image acquisition component for acquiring the text image.

[0023] For example, in the text detection device provided in some embodiments of the present disclosure, the text detection device is a point translation pen, wherein the image acquisition element is provided on the point translation pen, and the point translation pen is used to select the text to be detected.

[0024] At least one embodiment of the present disclosure further provides a storage medium that non-temporarily stores computer-readable instructions, wherein when the computer-readable instructions are executed by a computer, the text detection method provided by any embodiment of the present disclosure can be executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, rather than limiting the present disclosure.

[0026] Figure 1 A schematic diagram of the working principle of a point translation pen;

[0027] Figure 2 An exemplary flow chart of a text detection method provided for at least one embodiment of the present disclosure;

[0028] Figure 3 A schematic diagram of a text image provided by at least one embodiment of the present disclosure;

[0029] Figure 4 A schematic diagram of a text detection neural network provided in at least one embodiment of the present disclosure;

[0030] Figure 5 A schematic diagram of a pixel adjacency relationship provided by at least one embodiment of the present disclosure;

[0031] Figure 6 A schematic diagram of a text feature image provided by at least one embodiment of the present disclosure;

[0032] Figure 7 A method corresponding to at least one embodiment of the present disclosure is provided Figure 2 An exemplary flow chart of step S400 shown in FIG.

[0033] Figure 8 A method corresponding to at least one embodiment of the present disclosure is provided Figure 2 An exemplary flow chart of step S600 shown in FIG.

[0034] Figure 9 A method corresponding to at least one embodiment of the present disclosure is provided Figure 2 The operation diagram of step S600 shown in FIG.

[0035] Figure 10 A schematic block diagram of a text detection device provided for at least one embodiment of the present disclosure; and

[0036] Figure 11 A schematic diagram of a storage medium provided in at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0038] Unless otherwise defined, the technical or scientific terms used in this disclosure should have the usual meanings understood by persons of ordinary skill in the field to which this disclosure belongs. The words "first", "second" and similar terms used in this disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Words such as "include" or "comprise" mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Words such as "connect" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0039] The present disclosure is described below using several specific embodiments. To keep the following description of the embodiments of the present disclosure clear and concise, detailed descriptions of known functions and components may be omitted. When any component of an embodiment of the present disclosure appears in more than one drawing, the component is represented by the same or similar reference numeral in each drawing.

[0040] Translation pens typically include scanning pens (abbreviated as "scanning pens") and point-to-point pens (abbreviated as "point-to-point pens"). When using a scanning pen, you need to hold the pen upright and slide it over the text to be translated (i.e., scan it). This usage mode is different from the usual pen-holding habits, so users usually need to adapt to using a scanning pen. Unlike scanning pens, when using a point-to-point pens, you only need to align the pen tip with the bottom of the text to be translated and lightly tap it to perform the corresponding recognition and translation. Therefore, the usage method is more flexible and more in line with the user's pen-holding habits, providing a better user experience.

[0041] The working principle of the current point translation pen is mainly as follows: first, the tip of the point translation pen is clicked below the text to be detected (for example, English words, but not limited to this), and the pen body camera of the point translation pen is used to capture the text image, for example, Figure 1 Then, the traversal text detection process is performed on each pixel position of the entire text image to obtain all text boxes on the text image (such as Figure 1 The pen then locates the text box near the pen tip, i.e., the text box surrounding the text to be detected (i.e., the text box surrounding the text to be detected), and recognizes and translates the text within it. Text detection requires traversing the entire text image, but most of the text boxes detected in the text image are redundant (i.e., unrelated to the text to be detected). This limits the pen's response speed and reduces its efficiency.

[0042] If it is possible to focus only on the area near the pen tip position (i.e. Figure 1 If the point translator can detect and recognize text in the lower area of ​​the text image shown in the figure, it can greatly improve processing speed and reduce response time and computing resource usage. However, because the point translator needs to recognize text of different font sizes, if the area to be detected in the text image is manually limited, the following problems may arise: On the one hand, if the manually limited area to be detected is too large, the beneficial effects (i.e., improving processing speed, reducing response time and computing resource usage, etc.) may not be obvious; on the other hand, if the manually limited area to be detected is too small, it may not be able to cover large-font text, resulting in the inability to fully detect and recognize large-font text, which in turn limits the scope of use of the point translator.

[0043] At least one embodiment of the present disclosure provides a text detection method. The detection method includes: obtaining a text feature image corresponding to the text image based on a text image; using a portion of the text feature image near a first edge of the text feature image as a base region, wherein the first edge of the text feature image corresponds to the first edge of the text image, the text to be detected in the text image is near the first edge of the text image, and at least some pixels in the base region are positive pixels; grouping at least some of the positive pixels in the base region to obtain at least one connected domain; extending the at least one connected domain in a direction away from the first edge of the text feature image to obtain at least one final connected domain corresponding to the at least one connected domain; and determining at least one feature box corresponding to the at least one final connected domain, and mapping the at least one feature box to the text image to obtain at least one text box, wherein the at least one text box includes a text box of the text to be detected.

[0044] Some embodiments of the present disclosure also provide a text detection device and a storage medium corresponding to the above-mentioned text detection method.

[0045] The text detection method provided by the embodiments of the present disclosure can perform text detection based on pre-defined basic regions and the concept of connected domains. This reduces the computational complexity of text detection (i.e., the number of traversals) and the response time. This text detection method is applicable to point translation pens, etc., and can improve the processing speed of point translation pens and the user experience.

[0046] Some embodiments and examples of the present disclosure are described in detail below with reference to the accompanying drawings.

[0047] Figure 2This is an exemplary flow chart of a text detection method provided by at least one embodiment of the present disclosure. For example, the text detection method provided by the embodiment of the present disclosure can be applied to text images obtained by a point translation pen, but is not limited thereto. For example, Figure 2 As shown, the text detection method includes but is not limited to steps S100 to S600.

[0048] Step S100: Based on the text image, a text feature image corresponding to the text image is acquired.

[0049] For example, in step S100, the text image may include an image captured by an image acquisition device or component. For example, in some embodiments, the text detection method further includes step S000: acquiring a text image before step S100.

[0050] For example, in some examples, a text image can be captured using a point-to-point translation pen. For example, the point-to-point translation pen may include an image capture element, such as a camera; for example, the camera may be provided on the pen body of the point-to-point translation pen. Thus, the point-to-point translation pen (the camera on the point-to-point translation pen) may be used to perform step S000, i.e., to capture a text image. For example, when the image capture element of the point-to-point translation pen is used to capture a text image, the tip of the point-to-point translation pen is generally located below the text to be detected, so that relative to the text image, the tip of the point-to-point translation pen is equivalent to being located on the side of the text to be detected close to the edge of the text image. In order to distinguish it from other edges of the text image, this edge is called the first edge of the text image (reference Figure 3 The first edge FE of the text image is shown in FIG.

[0051] For example, the text image can be a grayscale image or a color image. The shape of the text image can be rectangular, diamond-shaped, circular, etc., and the embodiments of the present disclosure are not limited to this. In the embodiments of the present disclosure, the text image is described as a rectangle, but this should not be regarded as a limitation of the present disclosure.

[0052] For example, the text image can be an original image directly captured by an image acquisition device or component, or it can be an image obtained after preprocessing the original image. For example, in order to avoid the influence of data quality, data imbalance, etc. of the text image on text recognition, before performing text detection on the text image, the text detection method provided by the embodiment of the present disclosure can also include the operation of preprocessing the text image. Preprocessing can eliminate irrelevant information or noise information in the text image to facilitate better processing of the text image. Preprocessing can, for example, include scaling, cropping, gamma correction, image enhancement or noise reduction filtering of the text image.

[0053] For example, a text image includes at least one text, and the at least one text includes the text to be detected. For example, the text to be detected is usually close to the first edge (e.g., the bottom edge) of the text image. It should be noted that the text to be detected is the text that the user wants to detect. A text image refers to a form of presenting text in a visual manner, such as a picture or video containing text.

[0054] For example, the text to be detected may include: a word in one of the languages ​​such as English, French, German, Spanish, etc., or a character or word in one of the languages ​​such as Chinese, Japanese, Korean, etc., but is not limited thereto.

[0055] Figure 3 A schematic diagram of a text image provided by at least one embodiment of the present disclosure. For example, Figure 3 As shown, the text image includes multiple texts. For example, one text may be an English word (for example, Figure 3 "Tecent", "the", etc.), a number or a series of numbers (for example, Figure 3 ) etc., but not limited to this. For example, in Figure 3 In the text image shown, the text to be detected may be "Tecent". For example, in some examples, when "Tecent" is selected as the text to be detected using the point translation pen, the tip of the point translation pen is placed below "Tecent" (near the first edge FE), and the camera provided on the pen body of the point translation pen is used to take a picture to obtain Figure 3 The text image shown.

[0056] For example, in some embodiments, in step S100, a text detection neural network may be used to process the text image to obtain a text feature image, and obtain the connection probability between each pixel in the text feature image and its directly adjacent pixels.

[0057] Figure 4 A schematic diagram of a text detection neural network provided in at least one embodiment of the present disclosure. Figure 4 As shown, the text detection neural network includes first to sixth convolution modules, first to fifth downsampling modules, first to fourth upsampling modules, and a classifier.

[0058] For example, each of the first to sixth convolution modules may include a convolution layer. The convolution layer is the core layer of the convolutional neural network. The convolution layer can apply several convolution kernels (also called filters) to the input image to extract various types of features of the input image. Each convolution kernel can extract one type of feature. The convolution kernel is generally initialized in the form of a random decimal matrix, and during the training process of the convolutional neural network, the convolution kernel will be learned to obtain reasonable weights. The result obtained after applying a convolution kernel to the input image is called a feature map, and the number of feature images is equal to the number of convolution kernels. For example, in an embodiment of the present disclosure, if Figure 4 As shown, a text image is used as an input image. It should be noted that the embodiments of the present disclosure do not limit the number of convolutional layers included in the first to sixth convolutional modules.

[0059] For example, in some embodiments, each of the convolution modules may further include an activation layer. The activation layer includes an activation function, which is used to introduce nonlinear factors into the convolutional neural network so that the convolutional neural network can better solve more complex problems. The activation function may include a rectified linear unit (ReLU) function, a leaky rectified linear unit function (LeakyReLU), an S-shaped function (Sigmoid function), or a hyperbolic tangent function (tanh function). The ReLU function and the LeakyReLU function are non-saturated nonlinear functions, while the Sigmoid function and the tanh function are saturated nonlinear functions.

[0060] For example, in some embodiments, each of the convolution modules described above may further include, for example, a batch normalization (BN) layer. For example, the batch normalization layer is used to perform batch normalization on the feature images of a mini-batch of samples (i.e., the input image) so that the grayscale values ​​of the pixels of each feature image vary within a predetermined range, thereby reducing computational difficulty and improving contrast. For example, the predetermined range may be [-1, 1], but is not limited thereto. For example, the batch normalization layer may perform batch normalization on each feature image based on the mean and variance of the feature images of each mini-batch of samples.

[0061] For example, each of the first through fifth downsampling modules can include a downsampling layer. On the one hand, the downsampling layer can be used to reduce the size of the input image, simplify computational complexity, and, to a certain extent, mitigate overfitting. On the other hand, the downsampling layer can also perform feature compression and extract the key features of the input image. The downsampling layer can reduce the size of the feature image without changing the number of feature images. For example, a 12×12 input image is downsampled through a 2×2 downsampling layer filter, resulting in a 6×6 feature image. This means that four pixels in the input image are merged into one pixel in the feature image.

[0062] For example, the downsampling layer can use downsampling methods such as max pooling, average pooling, strided convolution, decimation (e.g., selecting fixed pixels), and demuxout (splitting the input image into multiple smaller images) to perform downsampling processing. For example, in some embodiments, the downsampling factors of the downsampling layers in the first to fifth downsampling modules are all 1 / (2×2), and the present disclosure includes but is not limited to this.

[0063] For example, each of the first to fourth upsampling modules may include an upsampling layer. For example, the upsampling layer may perform upsampling processing using upsampling methods such as strided transposed convolution and interpolation algorithms. Interpolation algorithms may include, for example, interpolation, bilinear interpolation, and bicubic interprolation. The upsampling process is used to increase the size of the feature image, thereby increasing the amount of data of the feature image. For example, in some embodiments, the upsampling factors of the upsampling layers in the first to fourth upsampling modules are all 2×2, and the present disclosure includes but is not limited to this.

[0064] For example, each of the first to fifth dimensionality reduction modules may include a convolution layer using a 1×1 convolution kernel. For example, each of the above dimensionality reduction modules may use a 1×1 convolution kernel to reduce the dimensionality of the data, reducing the number of feature images, thereby reducing the number of parameters in subsequent processing, reducing the amount of computation, and improving processing speed. For example, in some embodiments, each of the first to fifth dimensionality reduction modules may include 10 1×1 convolution kernels, so that each dimensionality reduction module can output 10 feature images.

[0065] For example, the classifier may include two softmax classifiers, namely a first softmax classifier and a second softmax classifier. The first softmax classifier is used to perform text classification prediction on whether each pixel is a text pixel (i.e., a positive pixel) or a non-text pixel (i.e., a negative pixel), and the second softmax classifier performs connection classification prediction on whether each pixel has a connection (link) relationship with its four directly adjacent pixels. It should be noted that in the present disclosure, any other feasible method can also be used to perform text classification prediction and connection classification prediction, including but not limited to the above-mentioned first and second softmax classifiers.

[0066] It should be noted that in the present disclosure, each of the layers such as convolution layer, downsampling layer, upsampling layer, etc. refers to the corresponding processing operation, namely convolution processing, downsampling processing, upsampling processing, etc., and will not be repeated below.

[0067] For example, a text detection neural network is used to process a text image to obtain a corresponding text feature image, including: using a first convolution module to perform convolution processing on the text image to obtain a first convolution feature map group; using a first downsampling module to perform downsampling processing on the first convolution feature map group to obtain a first downsampling feature map group; using a second convolution module to perform convolution processing on the first downsampling feature map group to obtain a second convolution feature map group; using a second downsampling module to perform downsampling processing on the second convolution feature map group to obtain a second downsampling feature map group, and using a fifth dimensionality reduction module to perform dimensionality reduction processing on the second convolution feature map group to obtain a fifth reduced dimensionality feature map group; using a third convolution module to perform convolution processing on the second downsampling feature map group. The third convolution feature map group is downsampled using the third downsampling module to obtain a third downsampling feature map group, and the third convolution feature map group is reduced in dimension using the fourth dimensionality reduction module to obtain a fourth reduced dimensionality feature map group; the third downsampling feature map group is convolved using the fourth convolution module to obtain a fourth convolution feature map group; the fourth convolution feature map group is downsampled using the fourth downsampling module to obtain a fourth downsampling feature map group, and the fourth convolution feature map group is reduced in dimension using the third dimensionality reduction module to obtain a third reduced dimensionality feature map group; the fourth downsampling feature map group is convolved using the fifth convolution module to obtain a fifth convolution feature map group; the fourth downsampling feature map group is downsampled using the fourth downsampling module to obtain a fourth downsampling feature map group, and the fourth convolution feature map group is reduced in dimension using the third dimensionality reduction module to obtain a third reduced dimensionality feature map group; the fourth downsampling feature map group is convolved using the fifth convolution module to obtain a fifth convolution feature map group; The sampling module downsamples the fifth convolution feature map group to obtain a fifth downsampled feature map group, and uses the second dimensionality reduction module to perform dimensionality reduction processing on the fifth convolution feature map group to obtain a second reduced dimensionality feature map group; uses the sixth convolution module to convolve the fifth downsampled feature map group to obtain a sixth convolution feature map group; uses the first upsampling module to upsample the sixth convolution feature map group to obtain a first upsampled feature map group; uses the first dimensionality reduction module to perform dimensionality reduction processing on the first upsampled feature map group to obtain a first reduced dimensionality feature map group; performs fusion processing on the first reduced dimensionality feature map group and the second reduced dimensionality feature map group to obtain a first fused feature map group; uses the second upsampling module to upsample the first fused feature map group, to obtain a second up-sampled feature map group; fusing the second up-sampled feature map group and the third reduced dimensionality feature map group to obtain a second fused feature map group; up-sampling the second fused feature map group using the third up-sampling module to obtain a third up-sampled feature map group; fusing the third up-sampled feature map group and the fourth reduced dimensionality feature map group to obtain a third fused feature map group; up-sampling the third fused feature map group using the fourth up-sampling module to obtain a fourth up-sampled feature map group; fusing the fourth up-sampled feature map group and the fifth reduced dimensionality feature map group to obtain a fourth fused feature map group; classifying the fourth fused feature map group using the classifier to obtain a text classification prediction image and a connection probability prediction image;and obtaining a text feature image based on the text classification prediction image and the connection probability prediction image, and obtaining the connection probability between each pixel in the text feature image and its directly adjacent pixels.

[0068] For example, in an embodiment of the present disclosure, each feature map group generally includes multiple feature images.

[0069] For example, in the embodiments of the present disclosure, Figure 4 As shown, the fusion process may include a parallel addition process ADD. For example, the parallel addition process ADD generally refers to adding the values ​​of each row and each column of the image matrix of each channel of a set of input images to the values ​​of each row and each column of the image matrix of the corresponding channel of another set of input images. For example, the two sets of images that serve as inputs to the parallel addition process ADD have the same number of channels, and for example, the number of channels of the image output by the parallel addition process ADD is also the same as the number of channels of any set of input images. Therefore, the "fusion process" means adding the value of each pixel in each feature image in one feature map group to the value of the corresponding pixel of the corresponding feature image in another feature map group to obtain a new feature image. The "fusion process" does not change the number and size of the feature images.

[0070] For example, in some embodiments, the text classification prediction image includes two feature images, and the connection probability prediction image includes eight feature images. It should be noted that the values ​​of the pixels in each feature image in the text classification prediction image and the connection probability prediction image are both greater than or equal to 0 and less than or equal to 1, and represent the text prediction probability or the connection prediction probability. The feature image in the text classification prediction image represents a probability map of whether each pixel is text, and the feature image in the connection probability prediction image represents a probability map of whether each pixel is connected to its directly adjacent pixels.

[0071] For example, the two feature images in the text classification prediction image include a text probability image and a non-text probability image. The text probability image represents the predicted probability that each pixel belongs to text (i.e., the type probability of each pixel), and the non-text probability image represents the predicted probability that each pixel belongs to non-text. The sum of the values ​​of the corresponding pixels of the two feature images is 1. For example, in some embodiments, a type probability threshold can be set, for example, 0.75; if the predicted probability of a pixel belonging to text is greater than or equal to the type probability threshold, it indicates that the pixel belongs to text, that is, the pixel is a positive pixel (positive pixel); if the predicted probability of a pixel belonging to text is less than the type probability threshold, it indicates that the pixel belongs to non-text, that is, the pixel is a negative pixel (negative pixel).

[0072] Figure 5A schematic diagram of a pixel adjacency relationship provided in at least one embodiment of the present disclosure. For example, in some embodiments, Figure 5 As shown, in direction R1, pixel PX3 and pixel PX4 are directly adjacent to pixel PX0, and in direction C1, pixel PX1 and pixel PX2 are directly adjacent to pixel PX0, that is, pixels PX1 to PX4 are four pixels directly adjacent to pixel PX0, and are respectively located above, below, to the left, and to the right of pixel PX0. For example, in some embodiments, the pixel array in each feature image is arranged into multiple rows and columns. For example, direction C1 can represent a first direction perpendicular to the first edge (including the first edge of the text image and the first edge of the text feature image), such as a column direction; direction R1 can represent a second direction parallel to the first edge (including the first edge of the text image and the first edge of the text feature image), such as a row direction.

[0073] For example, the eight feature images in the connection probability prediction image may include a first connection classification image, a second connection classification image, a third connection classification image, a fourth connection classification image, a fifth connection classification image, a sixth connection classification image, a seventh connection classification image, and an eighth connection classification image. Figure 5 As shown, for pixel PX0, the value of pixel PX0 in the first connected classification image represents the connection prediction probability from pixel PX0 to pixel PX1, and the value of pixel PX0 in the second connected classification image represents the disconnection prediction probability from pixel PX0 to pixel PX1; the value of pixel PX0 in the third connected classification image represents the connection prediction probability from pixel PX0 to pixel PX2, and the value of pixel PX0 in the fourth connected classification image represents the disconnection prediction probability from pixel PX0 to pixel PX2; the value of pixel PX0 in the fifth connected classification image represents the connection prediction probability from pixel PX0 to pixel PX3, and the value of pixel PX0 in the sixth connected classification image represents the disconnection prediction probability from pixel PX0 to pixel PX3; the value of pixel PX0 in the seventh connected classification image represents the connection prediction probability from pixel PX0 to pixel PX4, and the value of pixel PX0 in the eighth connected classification image represents the disconnection prediction probability from pixel PX0 to pixel PX4. It should be understood that the values ​​of the corresponding pixel points of the first connected classification image and the second connected classification image add up to 1, the values ​​of the corresponding pixel points of the third connected classification image and the fourth connected classification image add up to 1, the values ​​of the corresponding pixel points of the fifth connected classification image and the sixth connected classification image add up to 1, and the values ​​of the corresponding pixel points of the seventh connected classification image and the eighth connected classification image add up to 1.

[0074] For example, in some embodiments, a connection probability threshold can be set, for example, 0.7; when the connection prediction probability of two directly adjacent pixels is greater than or equal to the connection probability threshold, it indicates that the two adjacent pixels can be connected to each other; when the connection prediction probability of two directly adjacent pixels is less than the connection probability threshold, it indicates that the two directly adjacent pixels cannot be connected to each other.

[0075] It should be noted that the above-mentioned type probability threshold and connection probability threshold are merely illustrative, and the type probability threshold and connection probability threshold can be set according to actual application requirements.

[0076] For example, in some embodiments, the text feature image is a binary image, but is not limited thereto. For example, in some embodiments, based on the text classification prediction image and the connection probability prediction image, a text feature image is obtained, and the connection probability between each pixel in the text feature image and its directly adjacent pixels is obtained. This can include: binarizing each pixel in the text probability image in the text classification prediction image based on a comparison between its pixel value (the predicted probability of belonging to text, i.e., the type probability) and the type probability threshold to obtain the text feature image, and the connection probability between each pixel in the text feature image and its directly adjacent pixels can be correspondingly obtained by querying the connection probability prediction image. For example, in the text probability image, if the predicted probability of a pixel belonging to text is greater than or equal to the type probability threshold, the pixel is regarded as a positive pixel, that is, the text prediction probability of the positive pixel is greater than or equal to the type probability threshold; if the predicted probability of a pixel belonging to text is less than the type probability threshold, the pixel is regarded as a negative pixel, that is, the text prediction probability of the negative pixel is less than the type probability threshold; thereby, a text feature image including positive and negative pixels can be obtained.

[0077] Figure 6 A schematic diagram of a text feature image provided by at least one embodiment of the present disclosure. Figure 6 As shown, the text feature image includes positive pixels (such as Figure 6 Each gray square in the Figure 6 ).

[0078] It should be understood that the size of the text feature image is the same as the size of each feature image in the text classification prediction image and the connection probability prediction image.

[0079] It should be noted that Figure 4 The text detection neural network shown is schematic. In practical applications, neural networks with other structural forms can also be used to perform the operation of step S100; of course, Figure 4 The text detection neural network shown in FIG. 1 is partially modified to obtain a new text detection neural network that can also perform the operation of step S100. For example, in some examples, the text detection neural network can be omitted. Figure 4 The fourth upsampling module and the fifth dimensionality reduction module in the text detection neural network shown, as well as the corresponding fusion processing, are shown. At the same time, a classifier is used to classify the third fused feature map group to obtain a text classification prediction image and a connection probability prediction image. It should be noted that the embodiments of the present disclosure are not limited to this.

[0080] It should be understood that in the text detection methods provided in some examples, it can also be set that each pixel in the text feature image is directly adjacent to the 8 pixels above, below, left, right, upper left, lower left, upper right, and lower right thereof; in this case, the connection probability prediction image can correspond to 16 feature images. The embodiments of the present disclosure include but are not limited to this. For example, compared with a text detection method in which each pixel has 8 directly adjacent pixels, a text detection method in which each pixel has 4 directly adjacent pixels can reduce the amount of computation, increase processing speed, and at the same time improve the problem of text adhesion that may occur in the subsequent text box.

[0081] Step S200: taking a partial area of ​​the text feature image close to a first edge of the text feature image as a basic area, wherein at least part of the pixels in the basic area are positive pixels.

[0082] For example, the first edge of the text feature image corresponds to the first edge of the text image, and the text to be detected in the text image is close to the first edge of the text image (refer to Figure 3 ).

[0083] For example, in some embodiments, Figure 6 As shown, the lower part of the text feature image (ie, the part close to the first edge of the text feature image, as shown in FIG. Figure 6 As shown in the dotted box in the middle) as the basic area, at least part of the pixels in the basic area are positive pixels (such as Figure 6 (as shown by the grey squares in the dashed box).

[0084] For example, in some embodiments, assuming that the size of the text feature image is h*w (i.e., including h rows and w columns of pixels), the size of the basic area can be set to h base *w(including h base row w column pixels), where h, w, h base are all positive integers, and h base / h≤1. For example, in some examples, h base / h≤1 / 2; for example, in some examples, hbase The value range of / h is, for example, 1 / 10 to 1 / 2, for example, 1 / 5 to 2 / 5, for example, 1 / 4 to 1 / 3, etc. For example, h base The value range of / h can be set according to actual application requirements, for example, it can be set according to the range of font sizes to be recognized and the size of the coverage of the text image. It should be noted that if h base If the value of h is too small, the basic area may not include positive pixels, which may cause the text detection method provided by the embodiment of the present disclosure to be unable to be effectively implemented; if h is too small, the basic area may not include positive pixels, which may cause the text detection method provided by the embodiment of the present disclosure to be unable to be effectively implemented; base If the value of / h is too large, the reduction in the computational complexity of text detection may not be obvious, thereby reducing the beneficial effects of the embodiments of the present disclosure. Therefore, h base The value of / h should be set reasonably according to actual application requirements.

[0085] For example, since the length of the text to be detected may not be fixed, for example, English words are usually of different lengths, in an embodiment of the present disclosure, the width of the basic area may be set to be the same as the width of the text feature image, that is, both are w.

[0086] Step S300: Group at least part of the positive pixels in the basic area to obtain at least one connected region.

[0087] For example, in step S300, based on a union-find algorithm, at least some of the positive pixels in the basic area may be grouped according to the connection probability between each positive pixel in the basic area and its directly adjacent pixels to obtain at least one connected component.

[0088] For example, in some embodiments, a union-find algorithm may include: first, constructing an index set based on at least some positive pixels in a base region, for example, the index set includes at least some positive pixels in the base region, and in the index set, each positive pixel corresponds to a root node, and the initial value of the root node of each positive pixel is itself; then, in response to any directly adjacent pixel of each positive pixel in the index set being a positive pixel and each positive pixel having a positive connection relationship with the directly adjacent pixel, setting the value of the root node of the directly adjacent pixel to the value of the root node of the positive pixel; finally, treating each group of positive pixels with the same root node value as a connected domain to obtain at least one connected domain. It should be noted that the specific process of the above-mentioned union-find algorithm is illustrative and the embodiments of the present disclosure are not limited thereto. For example, in some examples, at least some positive pixels in the base region used to construct the index set include all positive pixels in the base region; for example, in other examples, at least some positive pixels in the base region used to construct the index set include positive pixels in one or more rows (which can be set according to actual needs) in the base region, for example, closest to the first edge of the text feature image, thereby reducing the amount of computation and improving processing speed. The embodiments of the present disclosure are not limited to this.

[0089] For example, the directly adjacent pixels of each positive pixel include pixels directly adjacent to each positive pixel in a first direction perpendicular to the first edge of the text feature image and pixels directly adjacent to each positive pixel in a second direction parallel to the first edge of the text feature image. For example, each positive pixel has four directly adjacent pixels.

[0090] For example, in an embodiment of the present disclosure, when the connection probability between two directly adjacent pixels is greater than a connection probability threshold, there is a positive connection relationship between the two pixels.

[0091] For example, in Figure 6 In the text feature image shown, all positive pixels in the base area are grouped to obtain four connected domains.

[0092] For example, in some embodiments, in order to prevent the influence of noise, the at least one connected domain mentioned above can be denoised. For example, in some examples, the connected domain with an area less than T1 pixels or the connected domain with a width (or height) less than T2 pixels in the at least one connected domain mentioned above can be removed, and the one or more connected domains remaining after the denoising process are used to determine the final connected domain corresponding to the text to be detected (refer to the relevant description in step S400 below). For example, in some examples, T1 can be, for example, 100 to 300, such as 200, but not limited thereto; for example, in some examples, T2 can be, for example, 5 to 15, such as 10, but not limited thereto. It should be understood that the values ​​of T1 and T2 can be set according to actual application requirements.

[0093] Step S400: Expanding at least one connected component in a direction away from a first edge of the text feature image to obtain at least one final connected component corresponding to the at least one connected component.

[0094] For example, in step S400 , the at least one final connected component includes a final connected component corresponding to the text to be detected.

[0095] Figure 7 A method corresponding to at least one embodiment of the present disclosure is provided Figure 2 The exemplary flow chart of step S400 shown in FIG. Figure 6 The text feature image pair shown Figure 7 Step S400 shown is described in detail.

[0096] For example, Figure 7 As shown, at least one connected domain is expanded in a direction away from the first edge of the text feature image to obtain at least one final connected domain corresponding to the at least one connected domain, that is, step S400, including steps S410 to S450.

[0097] Step S410: extracting a positive pixel in the current connected component that is farthest from the first edge of the text feature image in a first direction perpendicular to the first edge of the text feature image as a first positive pixel.

[0098] For example, in step S410, the current connected domain is at least one connected domain in the base region. Figure 6 As shown, the current connected domain is perpendicular to the first edge (i.e. Figure 6 The positive pixels farthest from the first edge of the text feature image in the first direction (i.e., the column direction from bottom to top) include pixels 1-5, so pixels 1-5 are all regarded as first positive pixels. Figure 6As shown, the first positive pixels (i.e., pixels 1-5) are located in the same row. Figure 6 As shown, pixel points 1-2 belong to the same connected domain, so pixel points 1-2 have the same root node; pixel points 3-5 belong to the same connected domain, so pixel points 3-5 have the same root node (different from the root node of pixel points 1-2).

[0099] Step S420: taking a pixel in the text feature image that is on a side of the first positive pixel away from a first edge of the text feature image and directly adjacent to the first positive pixel as a first neighboring pixel.

[0100] For example, Figure 6 As shown in FIG, the five pixels directly adjacent to the pixels 1-5 in the row above the pixels 1-5 are taken as the first neighboring pixels. Figure 6 As shown, the first neighboring pixels include pixel points 6-8, etc.; among them, pixel point 6 is directly adjacent to pixel point 1, pixel point 7 is directly adjacent to pixel point 2, pixel point 8 is directly adjacent to pixel point 4, and the first neighboring pixels of pixel points 3 and 5 are not given figure marks.

[0101] Step S430: In response to the first neighboring pixel being a positive pixel and having a positive connection relationship with the first positive pixel, modify the value of the root node of the first neighboring pixel to the value of the root node of the first positive pixel, and add the first neighboring pixel to the first neighboring pixel set.

[0102] For example, in some embodiments, when the connection probability between the first positive pixel and the first neighboring pixel is greater than a connection probability threshold, there is a positive connection relationship between the two.

[0103] For example, in some embodiments, the first set of neighboring pixels has a form similar to the aforementioned index set, that is, each pixel in the first set of neighboring pixels also has a corresponding root node. Figure 6 As shown, pixel 6 is a positive pixel and has a positive connection relationship with pixel 1, so pixel 6 can be added to the first set of neighboring pixels, and the value of the root node of pixel 6 is the same as the value of the root node of pixel 1. Similarly, pixel 7 can also be added to the first set of neighboring pixels, and the value of the root node of pixel 7 is the same as the value of the root node of pixel 2, that is, the same as the value of the root nodes of pixels 1 and 6; pixel 8 can also be added to the first set of neighboring pixels, and the value of the root node of pixel 8 is the same as the value of the root node of pixel 3.

[0104] Step S440: Expanding the first set of neighboring pixels in a second direction parallel to the first edge of the text feature image.

[0105] For example, in some embodiments, step S440 may include: adding positive pixels that are directly adjacent to any pixel in the first neighboring pixel set in a second direction parallel to the first edge of the text feature image and have a positive connection relationship to the first neighboring pixel set until the first neighboring pixel set can no longer be expanded in the direction parallel to the first edge of the text feature image.

[0106] For example, in some embodiments, the determination condition for the positive connection relationship in step S440 is the same as the determination condition in the aforementioned step S430.

[0107] For example, in some examples, such as Figure 6 As shown, pixel 9 is a positive pixel and has a positive connection relationship with pixel 6, so pixel 9 can be added to the first neighboring pixel set, and the value of the root node of pixel 9 is the same as the value of the root node of pixel 6; further, pixel 10 is a positive pixel and has a positive connection relationship with pixel 9, so pixel 10 can also be added to the first neighboring pixel set, and the value of the root node of pixel 10 is the same as the value of the root node of pixel 9. For example, Figure 6 As shown, the first neighboring pixel set includes only pixel points 6-8 before expansion, and includes pixel points 6-11 after expansion. Pixel points 6-7 and 9-11 have the same root node.

[0108] Step S450: Expand the current connected domain to include all pixels in the first set of neighboring pixels, and continue to expand the current connected domain in a direction away from the first edge of the text feature image until further expansion is impossible.

[0109] For example, Figure 6 As shown, the connected domain (first connected domain) including pixels 1-2 in the basic area also includes pixels 6-11 after the first expansion, and the connected domain (second connected domain) including pixels 3-5 also includes pixel 8 after the first expansion.

[0110] For example, based on the connected domain after the first expansion, the operations of steps S410 to S450 can be repeated to complete the second expansion of the connected domain. For example, during the second expansion, the pixels in the first neighboring pixel set obtained during the first expansion (i.e., pixels 6-11) can be used as the first positive pixels. Figure 6 As shown, after the second expansion, the first connected domain further includes pixel points 12-14, and the second connected domain further includes pixel points 15-16.

[0111] And so on, after multiple expansions, such as Figure 6As shown, the first connected domain also includes pixels 6-14, 17, and 19-20 outside the basic region, and the second connected domain also includes pixels 8, 15-16, 18, and 21 outside the basic region. Thus, two final connected domains can be obtained respectively.

[0112] It should be noted that, in the embodiments of the present disclosure, Figure 6 The expansion of the connected domain in the text feature image shown is exemplary and not restrictive. For example, in some embodiments, the connected domain that can be expanded outward (outside the basic area) in the basic area can be one or more, not limited to Figure 6 For example, in some embodiments, two or more connected domains in the basic region may form a final connected domain after being expanded outward, and the process is not limited to one final connected domain for each connected domain. For example, in some embodiments, the basic region also includes connected domains whose areas do not change after being processed in step S400, such as connected domains that cannot be expanded outward (i.e., cannot be expanded in a direction away from the first edge of the text feature image). Such connected domains are also used as final connected domains after being processed in step S400.

[0113] Step S500: determining at least one feature box corresponding to at least one final connected component, and mapping the at least one feature box to a text image to obtain at least one text box, wherein the at least one text box includes a text box of the text to be detected.

[0114] For example, in some embodiments, determining at least one feature box corresponding to at least one final connected domain may include: performing contour detection on the at least one final connected domain using a contour detection algorithm to obtain a contour of the at least one final connected domain; and processing the contour of the at least one final connected domain using a minimum bounding rectangle algorithm to obtain the at least one feature box corresponding to the at least one final connected domain. For example, the contour detection algorithm may include, but is not limited to, the contour detection (findContours) function of OpenCV; for example, the minimum bounding rectangle algorithm may include, but is not limited to, the minimum bounding rectangle (minAreaRect) function of OpenCV.

[0115] For example, in the embodiment of the present disclosure, the feature box can be a rectangular box, and correspondingly, the text box can also be a rectangular box. It should be noted that the embodiment of the present disclosure includes but is not limited to this.

[0116] For example, in some embodiments, Figure 3 As shown, after at least one feature box in the text feature image is mapped to the text image, at least one text box can be obtained (such as Figure 3As shown in the solid box in ). For example, the mapping includes two processes: scale transformation and projection. For example, taking the size of the text feature image as 1 / (2×2) of the size of the text image, in the scale transformation process, the width and height of the feature box are respectively doubled; in the projection process, the relative position of the text box and the text image and the relative position of the feature box and the text feature image are kept consistent, so that the corresponding text box can be obtained. For example, as Figure 3 As shown, each text box contains a text.

[0117] For example, Figure 3 As shown, in the text detection method provided by the embodiment of the present disclosure, only the text to be detected in the text image (such as Figure 3 In contrast, Figure 1 Commonly used text detection methods for the text image shown require traversal detection of the entire area of ​​the text image to obtain text boxes for all the text in the text image. Therefore, the text detection method provided by the embodiments of the present disclosure can reduce the amount of text detection computation (i.e., reduce the number of traversals) and shorten the response time of text detection.

[0118] Step S600: determining a text box containing text to be detected from at least one text box.

[0119] For example, in some embodiments, the text image is captured by a camera mounted on the body of a point-to-point translation pen, while the text to be detected is selected by the pen tip. Since the relative position of the pen tip and the camera is fixed, the relative position of the pen tip (assuming a virtual pen tip is created on the plane where the text image is located, i.e., the virtual pen tip) and the text image captured by the camera is also fixed. Therefore, step S600 can be implemented based on the above principle.

[0120] Figure 8 A method corresponding to at least one embodiment of the present disclosure is provided Figure 2 The exemplary flow chart of step S600 shown in FIG. Figure 9 A method corresponding to at least one embodiment of the present disclosure is provided Figure 2 The operation diagram of step S600 is shown in FIG. Figure 9 right Figure 8 Step S600 is described in detail.

[0121] For example, Figure 8 As shown, a text box with text to be detected is determined from at least one text box, that is, step S600 includes steps S610 to S620.

[0122] Step S610: constructing a virtual detection frame in the text image;

[0123] Step S620: Calculate the overlapping area between the virtual detection frame and each text frame, and use the text frame with the largest overlapping area with the virtual detection frame as the text frame of the text to be detected.

[0124] For example, in some embodiments, Figure 9 As shown, you can first Figure 9 For example, in some examples, the virtual pen tip (such as Figure 9 The virtual pen tip (shown as a black dot in the middle) can be set on the first edge of the text image, but is not limited thereto; for example, in other examples, the virtual pen tip can be set outside the text image and close to the first edge. Figure 9 As shown, the virtual pen tip can generally be set on the perpendicular midline of the first edge of the text image, or can be set near the perpendicular midline of the first edge of the text image, and the embodiments of the present disclosure are not limited to this. It should be understood that the virtual pen tip can be set according to actual application requirements, and the embodiments of the present disclosure are not limited to this.

[0125] Then, a virtual detection frame with a height of H and a width of W is constructed with the virtual pen tip as the bottom midpoint of the virtual detection frame (e.g. Figure 9 (as shown in the dashed box in the figure). For example, in some embodiments, H=H1+H2, where H1 represents the minimum value of the distance between the virtual pen tip and the center of each text box in the text image in the first direction perpendicular to the first edge (i.e., the column direction), and H2 is a preset height value; for example, H2 can be set to a height value of, for example, 30 pixels, but is not limited thereto. For example, in some embodiments, the width W is a preset width value; for example, W can be set to a width value of, for example, 60 pixels, but is not limited thereto. It should be understood that H2 and W can be set according to actual application requirements, and the embodiments of the present disclosure are not limited thereto.

[0126] For example, in some embodiments, after determining the text box of the text to be detected, the text detection method provided by the embodiments of the present disclosure may further include: performing text recognition processing on the text to be detected based on the text box of the text to be detected. For example, a commonly used text processing method can be used for text recognition processing, and the embodiments of the present disclosure are not limited to this. For example, a commonly used text processing method may include but is not limited to using a neural network (such as a multi-objective corrective attention network (MORAN)) to perform text recognition processing.

[0127] For example, in practical applications, text translation can be performed based on the results of the text recognition process to obtain and output a translation result of the text to be detected. For example, the results of the text recognition process can be indexed using a dictionary database to retrieve the translation result. For example, the translation result of the text to be detected can be displayed on a display or output as a voice through a speaker.

[0128] It should be noted that, in the embodiments of the present disclosure, the process of the above-described text detection method may include more or fewer operations, and these operations may be performed sequentially or in parallel. Although the process of the text detection method described above includes multiple operations that appear in a specific order, it should be clearly understood that the order of the multiple operations is not limited. The text detection method described above can be performed once or multiple times according to predetermined conditions.

[0129] It should be noted that in the embodiments of the present disclosure, the text detection neural network and the various functional modules and functional layers in the text detection neural network can be implemented by software, hardware, firmware or any combination thereof to perform the corresponding processing process.

[0130] The text detection method provided by the embodiments of the present disclosure can perform text detection based on pre-defined basic regions and adopt the concept of connected domains. This can reduce the amount of text detection computation (i.e., reduce the number of traversals) and shorten the response time of text detection. This text detection method is applicable to point translation pens and can increase the processing speed of point translation pens, thereby improving the user experience.

[0131] At least one embodiment of the present disclosure further provides a text detection device. Figure 10 A schematic block diagram of a text detection device provided in at least one embodiment of the present disclosure.

[0132] For example, Figure 10 As shown, the text detection device 1000 includes a memory 1001 and a processor 1002. It should be understood that, Figure 10 The components of the text detection device 1000 shown are merely exemplary and non-limiting. The text detection device 1000 may further include other components according to actual application requirements.

[0133] For example, the memory 1001 is used to store text images and computer-readable instructions; the processor 1002 is used to read the text images and run the computer-readable instructions. When the computer-readable instructions are run by the processor 1002, one or more steps in the text detection method described in any of the above embodiments are executed.

[0134] For example, in some embodiments, Figure 10As shown, the text detection device may further include an image acquisition component 1003. For example, the image acquisition component 1003 is used to acquire text images. For example, the image acquisition component 1003 is the image acquisition device or component described in the embodiment of the text detection method above. For example, the image acquisition component 1003 may be various types of cameras.

[0135] For example, in some embodiments, the text detection device 1000 may be a point translation pen, but is not limited thereto. For example, the point translation pen is used to select text to be detected. For example, the image acquisition component 1003 may be provided on the point translation pen, such as a camera provided on the point translation pen.

[0136] It should be noted that the memory 1001 and the processor 1002 may also be integrated into the point translation pen, that is, the image acquisition element 1003, the memory 1001 and the processor 1002 may all be integrated into the point translation pen. The embodiments of the present disclosure include but are not limited to this.

[0137] For example, the text detection device 1000 may further include an output unit, which is used to output the recognition result and / or translation result of the text to be detected. For example, the output unit may include a display, a speaker, etc. The display may be used to display the recognition result and / or translation result of the text to be detected, and the speaker may be used to output the recognition result and / or translation result of the text to be detected in the form of speech. For example, the translation pen may further include a communication module, which is used to enable communication between the translation pen and the output unit, for example, to transmit the translation result to the output unit.

[0138] For example, the processor 1002 can control other components in the text detection device 1000 to perform desired functions. The processor 1002 can be a device with data processing capabilities and / or program execution capabilities, such as a central processing unit (CPU) or a tensor processing unit (TPU). The central processing unit (CPU) can be an X86 or ARM architecture. The GPU can be directly integrated into the motherboard separately, or built into the north bridge chip of the motherboard. The GPU can also be built into the central processing unit (CPU).

[0139] For example, the memory 1001 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), a hard disk, an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, a flash memory, etc. One or more computer-readable instructions may be stored on the computer-readable storage medium, and the processor 1002 may execute the computer-readable instructions to implement various functions of the text detection device 1000.

[0140] For example, components such as the image acquisition element 1003, the memory 1001, the memory 1230 and the output unit can communicate with each other through a network connection. The network may include a wireless network, a wired network, and / or any combination of a wireless network and a wired network. The network may include a local area network, the Internet, a telecommunications network, an Internet of Things based on the Internet and / or telecommunications network, and / or any combination of the above networks. For example, a wired network can communicate using twisted pair, coaxial cable or optical fiber transmission, and for example, a wireless network can communicate using 3G / 4G / 5G mobile communication networks, Bluetooth, Zigbee or WiFi. The present disclosure does not limit the type and function of the network.

[0141] For example, for a detailed description of the process of the text detection apparatus 1000 performing text detection processing, reference may be made to the relevant description in the embodiment of the text detection method, and the repeated parts will not be repeated here.

[0142] The technical effects of the text detection device provided by the embodiments of the present disclosure can be referred to the corresponding description of the text detection method in the above embodiments, which will not be repeated here.

[0143] At least one embodiment of the present disclosure further provides a storage medium. Figure 11 A schematic diagram of a storage medium provided in at least one embodiment of the present disclosure. Figure 11 As shown, one or more computer-readable instructions 1101 may be non-transitory stored on the storage medium 1100. For example, when the computer-readable instructions 1101 are executed by a computer, one or more steps in the text detection method described above can be performed.

[0144] For example, the storage medium 1100 can be applied to the above-mentioned text detection device 1000, for example, it can serve as the memory 1001 in the text detection device 1000. For the description of the storage medium 1100, reference can be made to the description of the memory in the embodiment of the text detection device 100, and the repeated parts will not be repeated.

[0145] The technical effects of the storage medium provided by the embodiments of the present disclosure can be referred to the corresponding description of the text detection method in the above embodiments, which will not be repeated here.

[0146] Regarding this disclosure, the following points need to be explained:

[0147] (1) The drawings of the embodiments of the present disclosure only relate to the structures related to the embodiments of the present disclosure. Other structures may refer to conventional designs.

[0148] (2) For the sake of clarity, the thickness of layers or the size of regions in the drawings used to describe the embodiments of the present disclosure are exaggerated or reduced, i.e., these drawings are not drawn to scale. It is understood that when an element such as a layer, film, region, or substrate is referred to as being "on" or "under" another element, the element may be "directly" on or "under" the other element, or intervening elements may be present.

[0149] (3) In the absence of conflict, the embodiments of the present disclosure and the features therein may be combined with each other to form new embodiments.

[0150] The foregoing description is merely an exemplary embodiment of the present disclosure and is not intended to limit the scope of protection of the present disclosure. Any modifications or substitutions that can be readily conceived by a person skilled in the art within the technical scope disclosed herein are intended to be encompassed by the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure is determined by the appended claims.

Claims

1. A text detection method, comprising: Based on the text image, obtaining a text feature image corresponding to the text image, wherein each pixel in the text feature image has a connection probability with a directly adjacent pixel; using a partial area of ​​the text feature image close to a first edge of the text feature image as a base area, wherein the first edge of the text feature image corresponds to a first edge of the text image, the text to be detected in the text image is close to the first edge of the text image, and at least some pixels in the base area are positive pixels; Grouping at least some of the positive pixels in the basic area to obtain at least one connected region; Expanding the at least one connected domain in a direction away from a first edge of the text feature image to obtain at least one final connected domain corresponding to the at least one connected domain; and Determining at least one feature box corresponding to the at least one final connected domain, and mapping the at least one feature box to the text image to obtain at least one text box, wherein the at least one text box includes the text box of the text to be detected; Determine a text box containing the text to be detected from the at least one text box; The step of mapping the at least one feature frame to the text image to obtain at least one text frame includes: The relative positions of the at least one feature frame and the text feature image and the relative positions of the at least one text frame and the text image are kept consistent to obtain the at least one text frame.

2. The text detection method according to claim 1, wherein: In the case where the text feature image includes h rows and w columns of pixels, the basic area includes h base Rows and columns of pixels, Among them, h, w, h base are all positive integers, and h base / h≤1 / 2.

3. The text detection method according to claim 1 or 2, wherein: Grouping at least some of the positive pixels in the basic region to obtain the at least one connected region includes: Based on a union-find algorithm, the at least some positive pixels in the basic area are grouped according to a connection probability between each positive pixel in the at least some positive pixels in the basic area and a directly adjacent pixel, so as to obtain the at least one connected domain.

4. The text detection method according to claim 3, wherein: The method further comprises: grouping the at least some positive pixels in the basic region based on the union-find algorithm according to a connection probability between each positive pixel in the at least some positive pixels in the basic region and a directly adjacent pixel to obtain the at least one connected region, comprising: constructing an index set based on the at least some positive pixels in the base area, wherein the index set includes the at least some positive pixels in the base area, and in the index set, each positive pixel corresponds to a root node, and an initial value of the root node of each positive pixel is itself; In response to any directly adjacent pixel of each positive pixel in the index set being a positive pixel and each positive pixel having a positive connection relationship with the directly adjacent pixel, setting the value of the root node of the directly adjacent pixel to the value of the root node of each positive pixel; and Each group of positive pixels having the same root node value is regarded as a connected component to obtain the at least one connected component.

5. The text detection method according to claim 4, wherein: In the case that the connection probability between each positive pixel in the basic area and the directly adjacent pixel is greater than a connection probability threshold, it is determined that each positive pixel has the positive connection relationship with the directly adjacent pixel.

6. The text detection method according to claim 4 or 5, wherein: The directly adjacent pixels of each positive pixel in the base region include: pixels directly adjacent to each positive pixel in a first direction perpendicular to the first edge of the text feature image, and pixels directly adjacent to each positive pixel in a second direction parallel to the first edge of the text feature image.

7. The text detection method according to claim 4 or 5, wherein: Each positive pixel in the basis area has four directly adjacent pixels.

8. The text detection method according to claim 4 or 5, wherein: Expanding the at least one connected domain in a direction away from a first edge of the text feature image to obtain the at least one final connected domain corresponding to the at least one connected domain comprises: Extracting a positive pixel in the current connected domain that is farthest from the first edge of the text feature image in a first direction perpendicular to the first edge of the text feature image as a first positive pixel; taking a pixel in the text feature image that is on a side of the first positive pixel away from a first edge of the text feature image and directly adjacent to the first positive pixel as a first neighboring pixel; In response to the first neighboring pixel being a positive pixel and having a positive connection relationship with the first positive pixel, modifying the value of the root node of the first neighboring pixel to the value of the root node of the first positive pixel, and adding the first neighboring pixel to a first neighboring pixel set; Expanding the first set of neighboring pixels in a second direction parallel to the first edge of the text feature image; and The current connected domain is expanded to include all pixels in the first set of neighboring pixels, and the current connected domain is continued to be expanded in a direction away from the first edge of the text feature image until further expansion is impossible.

9. The text detection method according to claim 8, wherein: Expanding the first set of neighboring pixels in a second direction parallel to a first edge of the text feature image includes: Positive pixels that are directly adjacent to any pixel in the first set of neighboring pixels in a second direction parallel to the first edge of the text feature image and have a positive connection relationship are added to the first set of neighboring pixels until the first set of neighboring pixels can no longer be expanded in the direction parallel to the first edge of the text feature image.

10. The text detection method according to claim 8, wherein: The at least one final connected component includes a connected component in the basic region that cannot be extended in a direction away from the first edge of the text feature image.

11. The text detection method according to claim 3, wherein: Acquiring the text feature image corresponding to the text image based on the text image includes: The text image is processed using a text detection neural network to obtain the text feature image, and the connection probability between each pixel in the text feature image and a directly adjacent pixel is obtained.

12. The text detection method according to claim 11, wherein: The text detection neural network includes first to sixth convolution modules, first to fifth downsampling modules, first to fourth upsampling modules, and a classifier; Processing the text image using the text detection neural network to obtain the text feature image, and obtaining a connection probability between each pixel in the text feature image and a directly adjacent pixel, including: Using a first convolution module to perform convolution processing on the text image to obtain a first convolution feature map group; Using a first downsampling module to downsample the first convolution feature map group to obtain a first downsampled feature map group; Using a second convolution module to perform convolution processing on the first down-sampled feature map group to obtain a second convolution feature map group; Performing downsampling processing on the second convolution feature map group using a second downsampling module to obtain a second downsampled feature map group, and performing dimensionality reduction processing on the second convolution feature map group using a fifth dimensionality reduction module to obtain a fifth reduced dimensionality feature map group; Using a third convolution module to perform convolution processing on the second down-sampled feature map group to obtain a third convolution feature map group; Performing downsampling processing on the third convolution feature map group using a third downsampling module to obtain a third downsampling feature map group, and performing dimensionality reduction processing on the third convolution feature map group using a fourth dimensionality reduction module to obtain a fourth reduced dimensionality feature map group; Using a fourth convolution module to perform convolution processing on the third down-sampled feature map group to obtain a fourth convolution feature map group; Downsampling the fourth convolution feature map group using a fourth downsampling module to obtain a fourth downsampling feature map group, and reducing the dimension of the fourth convolution feature map group using a third dimensionality reduction module to obtain a third reduced dimensionality feature map group; Performing a convolution process on the fourth down-sampled feature map group using a fifth convolution module to obtain a fifth convolution feature map group; Performing downsampling processing on the fifth convolutional feature map group using a fifth downsampling module to obtain a fifth downsampled feature map group, and performing dimensionality reduction processing on the fifth convolutional feature map group using a second dimensionality reduction module to obtain a second reduced dimensionality feature map group; Performing convolution processing on the fifth down-sampled feature map group using a sixth convolution module to obtain a sixth convolution feature map group; Performing upsampling processing on the sixth convolutional feature map group using a first upsampling module to obtain a first upsampling feature map group; Using a first dimensionality reduction module to perform dimensionality reduction processing on the first up-sampled feature map group to obtain a first dimensionality reduced feature map group; Fusing the first dimensionality reduction feature map group and the second dimensionality reduction feature map group to obtain a first fused feature map group; Using a second upsampling module to upsample the first fused feature map group to obtain a second upsampled feature map group; Fusing the second up-sampled feature map group and the third dimensionality reduction feature map group to obtain a second fused feature map group; Using a third upsampling module to upsample the second fused feature map group to obtain a third upsampled feature map group; Fusing the third up-sampled feature map group and the fourth dimensionality reduced feature map group to obtain a third fused feature map group; Using a fourth upsampling module to upsample the third fused feature map group to obtain a fourth upsampled feature map group; Fusing the fourth up-sampled feature map group and the fifth dimensionality reduced feature map group to obtain a fourth fused feature map group; Using a classifier to perform classification processing on the fourth fused feature map group to obtain a text classification prediction image and a connection probability prediction image; and The text feature image is obtained based on the text classification prediction image and the connection probability prediction image, and the connection probability between each pixel in the text feature image and a directly adjacent pixel is obtained.

13. The text detection method according to claim 12, wherein: Each pixel in the text classification prediction image has a type probability, and each pixel in the connection probability prediction image has a connection probability between the pixel and a directly adjacent pixel; Obtaining the text feature image based on the text classification prediction image and the connection probability prediction image, and obtaining the connection probability between each pixel in the text feature image and its adjacent pixels, including: Pixels in the text classification prediction image whose type probability is greater than or equal to the type probability threshold are taken as positive pixels, and pixels in the text classification prediction image whose type probability is less than the type probability threshold are taken as negative pixels to obtain the text feature image, and the connection probability between each pixel in the text feature image and the directly adjacent pixels is correspondingly queried from the connection probability prediction image.

14. The text detection method according to claim 1 or 2, wherein: Determining the at least one feature box corresponding to the at least one final connected domain includes: Performing contour detection on the at least one final connected domain using a contour detection algorithm to obtain a contour of the at least one final connected domain; and processing the contour of the at least one final connected domain using a minimum circumscribed rectangle algorithm to obtain the at least one feature box corresponding to the at least one final connected domain.

15. The text detection method according to claim 1, wherein: Determining the text box of the to-be-detected text from the at least one text box includes: Constructing a virtual detection frame in the text image; and The overlapping areas between the virtual detection frame and each text frame are calculated, and the text frame having the largest overlapping area with the virtual detection frame is used as the text frame of the text to be detected.

16. The text detection method according to claim 1, further comprising: Based on the text box of the text to be detected, the text to be detected is recognized.

17. The text detection method according to claim 1 or 2, further comprising: Using an image acquisition component of the point translation pen to acquire the text image; When collecting the text image, the tip of the point translation pen is placed on the side of the text to be detected close to the first edge of the text image. The text image includes the text to be detected.

18. A text detection device, comprising: a memory for storing text images and computer-readable instructions; A processor is configured to read the text image and execute the computer-readable instructions, wherein the computer-readable instructions, when executed by the processor, execute the text detection method according to any one of claims 1 to 17.

19. The text detection device according to claim 18, further comprising: The image acquisition component is used to acquire the text image.

20. The text detection device according to claim 19, wherein: The text detection device is a point translation pen, wherein: The image acquisition element is provided on the point translation pen, and the point translation pen is used to select the text to be detected.

21. A storage medium non-transitorily storing computer-readable instructions, wherein: When the computer-readable instructions are executed by a computer, the text detection method according to any one of claims 1 to 17 can be executed.

Images