A text detection method and device based on target detection, and storage medium

Abstract

The present application relates to a text detection method and device based on target detection, and a storage medium, wherein the text detection method includes: obtaining an image to be detected, constructing candidate frames corresponding to each character according to the image to be detected, extracting feature information of the candidate frames and generating candidate character components corresponding to each character; predicting the connected areas of the candidate character components corresponding to each character to obtain the connection relationship between the candidate character components; merging several interconnected candidate character components to obtain candidate text lines, and generating text detection results for the candidate text lines. In the technical solution, by constructing candidate frames to extract image features and generate candidate character components corresponding to characters, it is beneficial to use candidate character components to individually locate each character in a complex background image, which can not only be integrated to form candidate text lines, but also enhance the accuracy of text line positioning.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a text detection method and device based on target detection, and a storage medium. Background Art

[0002] Natural scene text detection has extremely important and widespread applications in real life. It is an important prerequisite for text recognition, such as text retrieval, road sign recognition, and intelligent test paper grading. It can also be applied to image retrieval, machine translation, and autonomous driving. Due to various uncontrollable interference factors in natural scenes, such as light and shadow occlusion, shooting angle, foreign object occlusion, and some inherent properties of the text itself, such as artistic characters, deformed characters, or incomplete characters, natural scene text detection is still a very difficult task. In order to improve the efficiency of automated office, automated recognition, and automated production and manufacturing, automatic document generation based on images is an important application. For example, automatically generating bill information based on bill images can greatly reduce the labor intensity of bank staff.

[0003] There are several factors that make text detection in natural scenes difficult: (1) Diversity and variability. Compared with the text content in standard documents, the text in natural scenes may be multi-scale and multi-language, and may vary in shape, direction, scale, color, etc. These changes make text detection difficult; (2) Complex background factors. Scene text may appear in any background, including signal signs, bricks, grass, fences, etc. These backgrounds may have features very similar to those of text, thus becoming noise that affects the judgment of text. Of course, the lack of text caused by occlusion of foreign objects can also lead to potential detection errors; (3) Uneven imaging quality. Text images may be distorted or out of focus due to different shooting angles or shooting distances, or may form noise or shadows due to different lighting during shooting, making it impossible to guarantee the imaging quality of text images.

[0004] The text detection methods in natural scenes under complex backgrounds can be roughly divided into three categories: sliding window-based methods, connected component-based methods, and deep learning methods. Among them, the sliding window-based method does not make full use of the inherent intrinsic characteristics of the text itself, which will lead to the extraction of many more non-text candidate regions than real text regions, so that the subsequent non-text region filtering process becomes extremely complicated. In addition, this method is also sensitive to some external factors, such as illumination changes and image blur. Among them, the connected component-based method uses a multi-stage method to locate language text information. First, the MSER regions of the three channels of the image R, G, and B are extracted, and then the classifier is trained to remove the repeated MSER regions and non-text MSER regions to obtain candidate MSER regions. After that, the candidate text regions are connected into text strips and the obtained text strips are deduplicated. Although this method can detect and locate language text regions, the process is relatively cumbersome and is divided into multiple stages. The detection effect depends on the quality of the candidate regions generated by MSERs and is also affected by the manually designed feature extraction method. Among them, the deep learning method is mainly based on regression methods to process rectangular text lines. When the shape of the text is curved, its predicted detection box cannot accurately cover all text areas. In addition, for long text lines, once the aspect ratio of the text line is greater than the preset prediction threshold, there will be problems of frame loss or incomplete prediction. Summary of the invention

[0005] The main technical problem solved by the present invention is how to accurately detect text lines from complex background images. To solve the above technical problem, the present application provides a text detection method and device based on target detection, and a storage medium.

[0006] In a first aspect, an embodiment provides a text detection method based on target detection, comprising: acquiring an image to be detected, wherein the image to be detected includes one or more characters; constructing candidate frames corresponding to each character according to the image to be detected, extracting feature information of the candidate frames and generating candidate character components corresponding to each character; predicting connected areas for the candidate character components corresponding to each character to obtain connection relationships between the candidate character components; merging several interconnected candidate character components to obtain candidate text lines; and generating text detection results for the candidate text lines.

[0007] The method of constructing candidate frames corresponding to each character according to the image to be detected, extracting feature information of the candidate frames and generating candidate character components corresponding to each character includes: performing multiple consecutive convolution and downsampling operations on the image to be detected, and obtaining a first feature map of a corresponding scale after each operation; performing multiple consecutive convolution and upsampling operations on the first feature map of the minimum scale, and obtaining a second feature map of a corresponding scale after each operation; performing feature summation operations on the first feature map and the second feature map of the same scale to obtain a third feature map at the corresponding scale; constructing candidate frames corresponding to each character on the third feature map of each scale, and obtaining feature information of the candidate frames through regression processing; mapping the feature information of the candidate frames corresponding to each character to the image to be detected, and performing non-maximum suppression on the mapping results to obtain candidate character components corresponding to each character; the candidate character components include the position information and confidence scores of the characters.

[0008] The method predicts the connected regions of the candidate character components corresponding to each character to obtain the connection relationship between the candidate character components, including: obtaining the position information and confidence score of the candidate character components corresponding to each character formed by the third feature map of each scale; the position information of the candidate character components is represented as (x, y, w, h, θ), and the confidence score of the candidate character components is represented as S char ; where x and y are the coordinates or offsets of the X-axis and Y-axis respectively, w and h are the width pixels and height pixels respectively, and θ is the rotation angle relative to the X-axis; based on the position information of the candidate character components corresponding to each character formed by the third feature map of each scale, the first connection score of the candidate character component corresponding to each character and other candidate character components in the surrounding neighborhood is predicted, and the first connection score is expressed as S neighbor According to the position information of the candidate character components corresponding to each character formed by the third feature map of each scale, the second connection score between the candidate character component corresponding to each character and the candidate character components formed by the third feature map of other scales is predicted, and the second connection score is represented by S level ; Using the first connection score and the second connection score to determine the connection relationship between the candidate character components corresponding to each character.

[0009] The method of merging a plurality of interconnected candidate character components to obtain a candidate text line includes: forming a candidate set C using the interconnected candidate character components, and C = {c i}, where c i is the i-th connected candidate character component and the corresponding rotation angle is θ i , i ranges from 1 to n; a linear regression is performed on a process parameter b according to the least squares method to obtain a regression line; wherein the regression line is expressed as The objective function of the least squares method is expressed as

[0010]

[0011] in a i is a preset weight coefficient; according to the position information of the candidate character components corresponding to each character, projection is performed on the regression line to obtain the projection points on both sides and are recorded as (x l ,y l ) and (x r ,y r ); determine the candidate text line using the projection points on both sides of the acquired text line; the position information of the candidate text line is expressed as w i 、h i are the width pixels and height pixels of the i-th candidate character component respectively.

[0012] The generating of the text detection result of the candidate text line includes: obtaining a text feature map corresponding to the candidate text line by feature transformation; establishing a second neural network, and configuring the loss function of the second neural network as

[0013] L=L q +L p ;

[0014] Among them, L q is the regression loss function of the candidate text confidence score and satisfies

[0015]

[0016] λ2 is the preset weight coefficient, y′ is the predicted candidate text confidence score, represents the confidence score of the labeled candidate text; L p is the regression loss function of the candidate text position, and satisfies L p =L d +L β , d represents the candidate text position and is represented by the coordinates or coordinate offsets of the four corner points d = (x1, y1, x2, y2, x3, y3, x4, y4), smoothed L1 () is the activation function about L1 norm, d j is the predicted j-th candidate text position, is the jth candidate text position marked, β j is the predicted rotation angle of the jth candidate text, The j-th candidate text is rotated by an angle; the text feature map is input into the second neural network, and the candidate text confidence score y′ and the candidate text position d are calculated when the loss function of the second neural network converges; the text detection result of the candidate text line is obtained using the candidate text confidence score y′ and the candidate text position d.

[0017] According to the second aspect, an embodiment provides a text detection device, including: a camera, used to capture a natural scene and form an image to be detected, wherein the image to be detected includes one or more characters; a processor, connected to the camera, used to process the image to be detected according to the text detection method described in the first aspect above to obtain a text detection result; a display, connected to the processor, used to display the image to be detected and / or the text detection result.

[0018] According to the third aspect, an embodiment provides a computer-readable storage medium, characterized in that a program is stored on the medium, and the program can be executed by a processor to implement the text detection method described in the first aspect above.

[0019] The beneficial effects of this application are:

[0020] According to the above-mentioned embodiment, a text detection method and device based on target detection, and a storage medium, wherein the text detection method includes: obtaining an image to be detected, the image to be detected includes one or more characters; constructing candidate frames corresponding to each character according to the image to be detected, extracting feature information of the candidate frames and generating candidate character components corresponding to each character; predicting the connected areas of the candidate character components corresponding to each character to obtain the connection relationship between the candidate character components; merging several interconnected candidate character components to obtain candidate text lines; and generating text detection results for the candidate text lines. Since the features are extracted and the candidate character components corresponding to the characters are generated by constructing candidate frames, it is beneficial to individually locate each character in a complex background image, providing technical conditions for connecting the candidate character components into candidate text lines, and can enhance the accuracy of text line positioning.

[0021] In addition, when predicting the connection relationship between candidate character components, the position information is changed from the previous (x, y, w, h) to (x, y, w, h, θ), which fully considers the comprehensive influence of coordinates, width and height pixels and rotation angle on the position of the character, and also considers the connection scores of candidate character components in the same layer feature map and between non-same layer feature maps, so as to more accurately characterize the connection relationship between candidate character components with the help of multi-dimensional output parameters. When connecting several candidate character components and merging them to obtain candidate text lines, the least squares method is used to perform linear regression processing on the process parameters, so that the boundary positions on both sides of the text line formed by the character connection can be obtained more accurately, thereby further improving the positioning accuracy of the candidate text line. When generating the text detection results of the candidate text line, the neural network is used to perform regression operation processing on the candidate text confidence score and the candidate text position, which not only realizes the fine adjustment of the position of the candidate text line, but also can use the four corner point positioning method to adapt to the positioning requirements of different text scales, further improving the ability of the text detection method to adapt to the text scale and accurately locate. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 Flow chart of the text detection method based on target detection in this application;

[0023] Figure 2 Schematic diagram of the principle of the text detection method;

[0024] Figure 3 A flowchart for generating candidate character components;

[0025] Figure 4 A flowchart for obtaining the connection relationship of candidate character components;

[0026] Figure 5 Flow chart for obtaining candidate text lines;

[0027] Figure 6 Flowchart for generating text detection results;

[0028] Figure 7 Schematic diagram of the network for extracting feature maps of multiple scales;

[0029] Figure 8 A schematic diagram of the principle of connecting candidate character components into text lines;

[0030] Fig. 9 It is a structural schematic diagram of the text detection device in this application;

[0031] Fig.10a is a schematic diagram of each candidate character component in a product label;

[0032] Fig.10bis a schematic diagram of each candidate text line in a product label;

[0033] Fig.10c is a schematic diagram of text detection results for candidate text lines in product labels;

[0034] Fig.11 is a schematic diagram of the structure of the processor;

[0035] Fig.12 Schematic diagram of the structure of a text detection device in another embodiment of the present application. DETAILED DESCRIPTION

[0036] The present application is further described in detail below by specific embodiments in conjunction with the accompanying drawings. Wherein similar elements in different embodiments adopt associated similar element numbers. In the following embodiments, many detailed descriptions are intended to enable the present application to be better understood. However, those skilled in the art can easily recognize that some of the features can be omitted in different situations, or can be replaced by other elements, materials, and methods. In some cases, some operations related to the present application are not shown or described in the specification, in order to avoid the core part of the present application being overwhelmed by too much description, and for those skilled in the art, it is not necessary to describe these related operations in detail, and they can fully understand the related operations based on the description in the specification and the general technical knowledge in the art.

[0037] In addition, the features, operations or characteristics described in the specification can be combined in any appropriate manner to form various implementations. At the same time, the steps or actions in the method description can also be interchanged or adjusted in a manner that is obvious to those skilled in the art. Therefore, the various sequences in the specification and the drawings are only for the purpose of clearly describing a certain embodiment and are not meant to be a required sequence, unless otherwise specified that a certain sequence must be followed.

[0038] The serial numbers of the components in this document, such as "first", "second", etc., are only used to distinguish the objects described and do not have any order or technical meaning. The "connection" and "coupling" mentioned in this application, unless otherwise specified, include direct and indirect connections (couplings).

[0039] The technical solution of the present application is described in detail below with reference to the embodiments.

[0040] Embodiment 1

[0041] Please refer to Figure 1 , this embodiment discloses a text detection method based on target detection, which mainly includes steps 110-150, which are described below respectively.

[0042] Step 110, obtaining an image to be detected. The image to be detected may be an optical image of the surface of an object to be detected in a natural scene. Since the object to be detected is a type of object with text information on the surface (such as billboards, signs, product labels, book pages, engravings, handwriting, etc.), the image to be detected may include one or more characters (such as Chinese characters, letters, numbers, symbols, etc.). One or more character strings together constitute a text and represent a specific meaning of the text.

[0043] It should be noted that a camera or video camera can be used to capture an object to be detected in a natural scene, thereby obtaining an image to be detected containing one or more characters, and then the processor obtains the image to be detected from the camera or video camera to perform further image processing. In addition to the text composed of individual characters, the image to be detected may also contain some simple or complex background information (such as color, lines, stains, defects, patterns, etc.). At this time, it is necessary to eliminate the interference of the background information and detect the position of the text from it, and even be able to recognize the text content.

[0044] Step 120 , construct candidate boxes corresponding to the characters respectively according to the image to be detected, extract feature information of the candidate boxes and generate candidate character components corresponding to the characters respectively.

[0045] See also Figure 2 In the technical solution of the present application, a neural network is mainly used to extract the backbone features of the image to be detected to obtain the corresponding feature map. This is mainly to take into account the different receptive fields of each character in the image to be detected, and to obtain feature maps of receptive fields of different scales through backbone feature extraction. Afterwards, feature maps of different scales can be used as input, and character positioning technology (such as sliding window technology) can be used to use candidate boxes on the feature map to determine the position of each character, thereby extracting candidate character components.

[0046] For the detection task of each character in the image to be detected, it is not only necessary to return which characters are contained in the image, but also to use candidate boxes to mark the position of each character in the image. How to extract such candidate boxes from the image to be detected quickly and well requires the selection of appropriate character positioning technology, among which the sliding window technology can directly extract the candidate area. For example, in the process of extracting candidate character components, the feature maps of different receptive fields extracted by the trunk are used as input, and different numbers of candidate boxes are used on feature maps of different scales to extract candidate character components. The number of candidate boxes here can be designed in a Gaussian distribution. Six candidate boxes are used on the feature map of the intermediate scale (such as three candidate boxes of different sizes and each candidate box is combined with two different aspect ratios), and four candidate boxes are used on the feature maps of small and large scales (such as two candidate boxes of different sizes and each candidate box is combined with two different aspect ratios). Furthermore, the central coordinates, width and height pixels and confidence scores of the candidate boxes can be obtained by regression operation using a neural network.

[0047] In a specific embodiment, the size of the candidate box can be adaptively matched according to the size of the character. For example, if the width and height of the character are both 100 pixels, the candidate box is set to 150 pixels in width and height by 1.5 times.

[0048] In this embodiment, the candidate character components corresponding to each character in the image to be detected are mainly used to characterize the position information and confidence score of each character, where the position information can be represented by coordinates, scales in width and height, and rotation angle, and the confidence score can be represented by the confidence level of whether it is a character.

[0049] Step 130, predicting the connected regions of the candidate character components corresponding to each character, and obtaining the connection relationship between the candidate character components. The purpose of predicting the connected regions of the candidate character components is to fuse the candidate character components according to the candidate character components and the connection relationship between the candidate character components, so as to obtain the candidate text line. Then, how to obtain the connection relationship between the candidate character components will be described below.

[0050] Step 140: merge several interconnected candidate character components to obtain candidate text lines. It can be understood that after all candidate character components are detected, the candidate frame information and connection relationship information of the candidate character components output by each feature map can be merged to obtain information of the candidate text line.

[0051] for example Figure 2Since the candidate character components can represent the position information of the characters, when the connection relationship between the candidate character components is known, the corresponding characters can be connected into text lines by linear projection based on the position information of several interconnected candidate character components, that is, the candidate text lines are fused. Since the position information coordinates, scales and angles of the candidate character components are represented by five parameters, namely (x, y, w, h, θ), where x and y are the coordinates or offsets of the X-axis and Y-axis respectively, w and h are the width pixels and height pixels respectively, and θ is the rotation angle relative to the X-axis; then, the obtained candidate text lines can also be represented by these parameters. For example, the position information of the candidate text line is represented as (x t ,y t ,w t ,h t ,θ t ). In addition, the confidence scores of several interconnected candidate character components can be averaged to obtain the confidence score of the candidate text line.

[0052] Step 150, generate text detection results for candidate text lines. The position information and confidence scores of the candidate text lines obtained in step S140 are only preliminary results. In order to adapt to text areas of different shapes and obtain more accurate detection results, the position information and confidence scores of the candidate text lines can also be regressed and fine-tuned. For example, after obtaining the candidate text lines, feature transformation is performed through RoIPooling or RoIAlign to obtain a text feature map of a specific scale corresponding to the candidate text lines. The text feature map is used to regress the position information and confidence scores of the candidate text lines. After regression processing, the candidate text confidence scores and candidate text positions are obtained to form text detection results.

[0053] It should be noted that RoIPooling is the Pooling operation on RoI, where RoI (Region of interest) refers to the region of interest where the target exists in the image, and Pooling refers to the pooling process of the feature vector of the image. In the convolutional neural network, the pooling layer is often behind the convolution layer, and the feature vector output by the convolution layer is reduced by pooling. The specific operation of ROIPooling can be described as: according to the input image, the ROI is mapped to the corresponding position of the feature map, the mapped area is divided into parts of the same size, and each part is subjected to the maximum pooling (max pooling) or average pooling (average pooling) operation.

[0054] It should be noted that ROIAlign is an improvement on ROIPooling. It cancels the quantization operation and uses bilinear interpolation to obtain the image values ​​at the pixel points with floating-point coordinates, thereby converting the entire feature aggregation process into a continuous operation, which can solve the problem of regional mismatch caused by two quantizations in the ROIPooling operation. The specific operation of ROIAlign can be described as: traversing each candidate region in the image, keeping the floating-point boundaries unquantized, dividing the candidate region into kx k units, and leaving the boundaries of each unit unquantized, calculating four fixed coordinate positions in each unit, using bilinear interpolation to calculate the values ​​of these four positions, and then performing the maximum pooling operation.

[0055] In this embodiment, for candidate text lines of large objects in the image, either RoIPooling or RoIAlign can be selected; however, for candidate text lines of smaller objects in the image, RoIAlign can be selected first, and the result of such processing will be more accurate.

[0056] It will be easily understood by those skilled in the art that, since the present embodiment extracts features by constructing candidate boxes and generates candidate character components corresponding to characters, it is beneficial to individually locate each character in a complex background image, provides technical conditions for connecting the candidate character components into candidate text lines, and can enhance the accuracy of text line positioning.

[0057] In this embodiment, see Figure 3 The above-mentioned step 120 mainly involves the process of constructing a candidate box of a character, extracting feature information of the candidate box and generating candidate character components, which may specifically include steps 121-125, which are described as follows.

[0058] Step 121, performing multiple consecutive convolution and downsampling operations on the image to be detected, and obtaining a first feature map of a corresponding scale after each operation.

[0059] See also Figure 7 , the continuous convolution and downsampling operations on the image to be detected can be realized with the help of the VGG16 network. Each layer of the network can adopt the structural design form of convolution + pooling, which not only realizes the convolution and downsampling (Pooling) operations, but also can obtain feature extraction operations of different receptive fields, and realize the forward process of the network from bottom to top; in the forward process, the size of the feature map will change after passing through some layers, but will not change when passing through other layers. The layers that do not change the size of the feature map are classified as a processing block (Block), so each extracted feature is the output of each Block, thus forming a feature pyramid. The feature pyramid can be used to obtain four first feature maps with different resolutions, and the first feature maps are sorted from large to small as C2, C3, C4, and C5.

[0060] Step 122, performing multiple consecutive convolution and upsampling operations on the first feature map of the smallest scale, and obtaining a second feature map of the corresponding scale after each operation.

[0061] See also Figure 7 , when the first feature map C5 with the smallest scale is obtained, a top-down convolution and upsampling operation can be implemented with the help of the VGG16 network. Specifically, the upsampling (Unpooling) operation can be implemented by using transposed convolution. The second feature map with the smallest scale is P5. After multiple consecutive convolution and upsampling operations, the second feature maps P4, P3, and P2 with gradually increasing scales can be obtained.

[0062] It should be noted that, in order to ensure the consistency of upsampling and downsampling operations, the scale sizes (ie, width and height pixel sizes) of the first feature maps C2, C3, C4, C5 and the second feature maps P2, P3, P4, P5 are respectively the same.

[0063] Step 123, perform feature summation operation on the first feature map and the second feature map of the same scale to obtain a third feature map of the corresponding scale. Figure 7 , the first feature map C2 and the second feature map P2 are summed to obtain the third feature map R2 of the same scale as the first feature map C2, the first feature map C3 and the second feature map P3 are summed to obtain the third feature map R3 of the same scale as the first feature map C3, the first feature map C4 and the second feature map P4 are summed to obtain the third feature map R4 of the same scale as the first feature map C4; since the first feature map C5 and the second feature map P5 are both feature maps of the minimum scale, they will not be downsampled or upsampled, so the second feature map P5 can be used as the third feature map R5. Of course, in other cases, the feature map after the summation of the two can also be used as the third feature map R5. It should be noted that the summation operation involved here refers to Figure 7 In

[0064] In another embodiment, after obtaining the third feature map of each scale and before constructing the candidate box on the third feature map, that is, after step 123 and before step 124, it also includes: performing convolution processing (such as a 3×3 convolution processing process) on the third feature map of each scale. The purpose of the convolution processing is to eliminate the feature aliasing effect caused by upsampling in the third feature map.

[0065] Step 124, construct candidate boxes corresponding to each character on the third feature map of each scale, and obtain feature information of the candidate boxes through regression processing. In order to construct the candidate box corresponding to each character, the candidate box can be slid on the third feature map, and different numbers of candidate boxes are selected on the third feature maps of different scales to extract character features, so as to obtain the candidate box corresponding to each character. For example, the number of candidate boxes is designed in a Gaussian distribution, and 6 candidate boxes (3 different candidate box sizes and 2 different aspect ratio combinations) are used in the third feature map of the middle scale (such as R3, R4), and 4 candidate boxes (2 different candidate box sizes and 2 different aspect ratio combinations) are used in the third feature map of the two side scales (such as R2, R5), and then the center offset and width and height offset of the candidate box, as well as the confidence score (i.e., whether the character is contained) are obtained by regression.

[0066] In a specific implementation, in order to determine the feature information of the candidate box (including position information and confidence score), a neural network can be used to implement regression processing of the third feature map.

[0067] First, we need to build the first neural network. For example, we use a 2-layer fully connected layer to form the structure of the first neural network, and configure the loss function of the first neural network as

[0068] L rpn =L cls +L reg .

[0069] In L rpn In the formula, L reg is the regression loss function of the candidate box's location information, and λ1 represents the preset weight coefficient, w * represents the weights of the neural network to be regressed, the superscript T represents the transposition operation, where i is the serial number of the third feature map and its value range is 2 to 5, φ represents the activation function for activating the feature vector of the third feature map, and u * Represents the transformation relationship between the annotation position and the predicted position and satisfies u * =(u x ,u y ,u w ,u h ,u θ ),u x 、u y 、u w 、u h 、u θ They represent the transformation relationship of the candidate box about the X-axis coordinate, the transformation relationship about the Y-axis coordinate, the transformation relationship about the width pixel, the transformation relationship about the height pixel, and the transformation relationship relative to the X-axis rotation angle, and u x =(Gx -P′ x ) / P′ w 、u y =(G y -P′ y ) / P′ h 、u w =log(G w / P′ w )、u h =log(G h / P′ h )、u θ =P′ θ -G θ , G=(G x ,G y ,G w ,G h ,G θ ) is the marked position information, P′=(P′ x ,P′ y ,P′ w ,P′ h ,P′ θ ) is the predicted location information. cls is the regression loss function of the confidence score of the character in the candidate box, and α is the preset weight coefficient, p is the predicted confidence score (generally expressed as a value between 0 and 1, the larger the value, the higher the credibility of the character), The confidence score of the annotation (usually represented by a value of 0 or 1, 0 means it is not a character, and 1 means it is a character).

[0070] After the first neural network is trained using the labeled image (each character in the image is labeled with position information and confidence score), the first neural network can be established. Since the network training process is already very mature, this prior art will not be described in detail here.

[0071] Then, the third feature map of each scale is input into the first neural network, and the candidate boxes corresponding to each character are determined when the loss function of the first neural network converges. In this way, the feature information of the candidate boxes including the position information P′ and the confidence score p can be calculated.

[0072] It can be understood that the candidate box refers to a rectangular box that encloses the character area. The confidence score p of the candidate box reflects the confidence of whether the image feature in the rectangular box is a character. The position information P′ of the candidate box reflects the center coordinate offset, width and height scale and rotation angle of the rectangular box in the image.

[0073] Step 125, mapping the feature information of the candidate boxes corresponding to each character to the image to be detected, performing non-maximum suppression on the mapping results to obtain candidate character components corresponding to each character, each candidate character component including the position information and confidence score of the corresponding character.

[0074] It can be understood that the candidate box reflects the position and confidence of the characters in the third feature map. In order to further understand the situation of each character in the image to be detected, each candidate box needs to be mapped to the image to be detected to determine the position of the corresponding character in the image to be detected. There is a resolution difference between the third feature map and the image to be detected. For example, the width and high resolution of the image to be detected is twice the width and high resolution of the third feature map. Then the coordinates of the candidate box are multiplied by 2 to obtain the coordinates of the corresponding pixel points in the image to be detected, and the width and height scale of the candidate box are multiplied by 2 to obtain the number of width and height pixels of the corresponding character in the image to be detected; the rotation angle of the candidate box can remain unchanged when mapped to the image to be detected, and the confidence score of the candidate box can also remain unchanged when mapped to the image to be detected. The mapping result of each candidate box is non-maximum suppressed to obtain the candidate character component of the corresponding character. The candidate character component is essentially a data set of the position information and confidence score of the character, which reflects the attributes of the position and confidence of the character. Then, the position information of each candidate character component can be expressed as (x, y, w, h, θ), where x and y are the coordinates or offsets of the X-axis and Y-axis respectively, w and h are the width pixels and height pixels respectively, and θ is the rotation angle relative to the X-axis; the confidence score of each candidate character component can be expressed as S char , S char The value range is 0 to 1.

[0075] It should be noted that non-maximum suppression (NMS) is an existing image processing algorithm, the idea of ​​which is to search for local maximum values ​​and suppress smaller values. For example, in the process of target detection, a large number of candidate boxes will be generated at the position of the same target, and these candidate boxes may overlap with each other. At this time, non-maximum suppression is needed to find the best target bounding box and eliminate redundant bounding boxes.

[0076] for example Figure 7, the candidate boxes corresponding to each character in the third feature map R2 are mapped to the image to be detected, and the obtained candidate character component is D2, which includes parameter 2, which is the number of channels of the confidence score (two channels represent yes and no respectively, forming a binary classification result), and parameter 5 is the number of channels of the position information (x, y, w, h, θ each occupy one channel). Similarly, the parameters of the candidate character components D3, D4, and D5 can be known. Since the candidate character components D2, D3, D4, and D5 are the results of mapping based on different third feature maps (P2, P3, P4, and P5), there are obvious same-layer and different-layer relationships between the candidate character components corresponding to each character formed by the third feature maps of various scales. The candidate character components formed by the same third feature map belong to the same-layer relationship, and the candidate character components formed by different third feature maps belong to different-layer relationships.

[0077] In this embodiment, see Figure 4 The above step 130 mainly involves the process of predicting connected areas and obtaining the connection relationship of candidate characters, which may specifically include steps 131-134, which are described as follows.

[0078] Step 131, obtain the position information and confidence score of the candidate character components corresponding to each character formed by the third feature map of each scale. It can be understood that for the third feature map of any scale, if N candidate frames are formed, then these candidate frames will also be mapped to the image to be detected. N candidate character components will also be obtained; due to the differences between the features of the third feature maps of various scales, the number of candidate frames that can be constructed by each third feature map is also different, which leads to differences in the number of candidate character components formed, that is, the number of candidate character components belonging to different layers is not necessarily the same.

[0079] For each candidate character component corresponding to a character, the position information of the candidate character component can be expressed as (x, y, w, h, θ), and the confidence score of the candidate character component can be expressed as S char ; where x and y are the coordinates or offsets of the X-axis and Y-axis respectively, w and h are the width pixels and height pixels respectively, and θ is the rotation angle relative to the X-axis.

[0080] Step 132, based on the position information of the candidate character components corresponding to each character formed by the third feature map of each scale, predict the first connection score between the candidate character component corresponding to each character and other candidate character components in the surrounding neighborhood, wherein the first connection score is represented by S neighbor .

[0081] For each candidate character component belonging to the same layer relationship, there may be a connection relationship between them belonging to the same text. Since each candidate character component has information such as the coordinates, scale, and angle of the corresponding character, the connection relationship between each candidate character component and the candidate character components in the surrounding 8 directions (such as up, down, left, right, upper left, lower left, upper right, and lower right) can be judged based on the size of such information. If the coordinate distance between two candidate character components in a certain direction is less than a preset threshold, the scale difference or the angle difference is less than a preset threshold, it is predicted that there is a connection between the two, otherwise there is no connection between the two; the results of connection / no connection in all directions are combined to obtain the first connection score S neighbor .

[0082] See also Figure 7 , parameter 16 of candidate character component D3 represents the first connection score, which has a total of 2×8=16 quantities, where 8 represents eight directions and 2 represents the number of channels in each direction (two channels are connected and unconnected, respectively, forming a binary classification result).

[0083] Step 133, based on the position information of the candidate character components corresponding to each character formed by the third feature graph of each scale, predict the second connection score between the candidate character component corresponding to each character and the candidate character components formed by the third feature graph of other scales, wherein the second connection score is represented by S level ;

[0084] For each candidate character component belonging to a different layer relationship, there may also be a connection relationship between them belonging to the same character. Since each candidate character component has information such as the coordinates, scale, and angle of the corresponding character, the connection relationship between any candidate character component in a layer and the candidate character component at the same position in each layer can be judged based on the size of such information. If the coordinate difference between two candidate character components in different layers is less than a preset threshold, the scale difference or the angle difference is less than a preset threshold, it is predicted that there is a connection between the two, otherwise there is no connection between the two; then the second connection score S can be obtained by combining the results of connection / no connection between each two layers. level .

[0085] See also Figure 7 , parameter 8 of candidate character component D3 represents the second connection score, which has a total of 2×4=8 quantities, where 4 represents four layers and 2 represents the number of channels with or without connection between layers (the two channels are connected and not connected, respectively, forming a binary classification result).

[0086] Step 134, using the first connection score and the second connection score to determine the connection relationship between the candidate character components corresponding to each character. It can be understood that the first connection score reflects the existence / non-connection relationship between the candidate character components of the same layer, and the second connection score reflects the existence / non-connection relationship between the candidate character components of different layers, so the connection relationship between the candidate character components corresponding to all characters can be determined by using the first connection score and the second connection score.

[0087] It is easy for those skilled in the art to understand that in this embodiment, when predicting the connection relationship between candidate character components, the position information is changed from the previous (x, y, w, h) to (x, y, w, h, θ), which fully considers the comprehensive influence of coordinates, width and height pixels and rotation angle on the position of the character, and also considers the connection scores of candidate character components within the same layer feature map and between non-same layer feature maps, thereby using multi-dimensional output parameters to more accurately characterize the connection relationship between each candidate character component.

[0088] In this embodiment, see Figure 5 The above-mentioned step 140 mainly involves the process of merging candidate character components and fusing them to obtain candidate text lines, which may specifically include steps 141-144, which are described as follows.

[0089] Step 141, using interconnected candidate character components to form a candidate set C, and C = {c i}, where c i is the i-th connected candidate character component and the corresponding rotation angle is θ i , the value range of i is 1 to n.

[0090] It can be understood that since there are multiple groups of interconnected candidate character components, there may be multiple candidate sets C. In this embodiment, in order to facilitate the explanation of the process of a single candidate set C forming a candidate text line, only a single candidate set C is taken as an example.

[0091] Step 142, a process parameter b is subjected to linear regression processing according to the least square method to obtain a regression line; wherein the regression line is expressed as The objective function of the least squares method is expressed as

[0092]

[0093] in a i It is a preset weight coefficient and can adopt Huber, Tukey, Gauss or Drop weight coefficient.

[0094] Step 143, projecting the position information of the candidate character components corresponding to each character onto the regression line, obtaining the projection points on both sides (i.e., the leftmost point and the rightmost point of the projection) and recording them as (x l ,y l ) and (x r ,y r ).

[0095] Step 144, determine the candidate text line using the obtained projection points on both sides, and the position information of the candidate text line can be expressed as w i 、h i are the width pixels and height pixels of the i-th candidate character component respectively. In addition, x t ,y t is the coordinate of the middle pixel of the candidate text line, w t 、h t is the width and height pixel dimensions of the candidate text line.

[0096] It can be understood that since the candidate set C = {c i} includes multiple candidate character components c i , and each candidate character component has its own confidence score S char , then for the candidate text line determined by these candidate character components, its confidence score can be represented by the average of the confidence scores of these candidate character components.

[0097] It is easy for technicians in this field to understand that when the present embodiment connects several candidate character components to each other and merges them to obtain candidate text lines, the least squares method is used to perform linear regression processing on the process parameters, so that the boundary positions on both sides of the text line formed by the character connection can be obtained more accurately, thereby further improving the positioning accuracy of the candidate text line.

[0098] for example Figure 8 For the image to be detected containing the characters C, R, A, O, and C, the outer frame of each character represents the corresponding candidate character component. The position of the corresponding character can be determined by the coordinates and scale of each candidate character component. It is known that the upper C and R belong to the same text, and CR is determined to constitute a candidate text line through the connection relationship. Similarly, the position of the corresponding character is determined by the coordinates and scales of other candidate character components. It is known that the lower A, C, R, O, and C belong to the same text, and ACROC is determined to constitute another candidate text line through the connection relationship.

[0099] In this embodiment, see Figure 6 The above-mentioned step 150 mainly involves the process of generating text detection results, which may specifically include steps 151-153, which are described as follows.

[0100] Step 151, obtaining a text feature map corresponding to the candidate text line through feature transformation. Specifically, the RoIPooling or RoIAlign described above can be used to perform feature transformation on each candidate text line to obtain the corresponding text feature map.

[0101] Step 152: In order to further accurately determine the position and confidence of the candidate text line, a neural network may be used to implement regression processing of the text feature map.

[0102] First, establish the second neural network and configure the loss function of the second neural network as

[0103] L=L q +L p .

[0104] In the above formula L, L q is the regression loss function of the candidate text confidence score and satisfies λ2 is the preset weight coefficient, y′ is the predicted candidate text confidence score (the value range is 0 to 1), Indicates the confidence score of the labeled candidate text (the value is 0 or 1, 0 means it is not text, 1 means it is text). p is the regression loss function of the candidate text position and satisfies L p =L d +L β , d represents the candidate text position and is represented by the coordinates or coordinate offsets of the four corner points d = (x1, y1, x2, y2, x3, y3, x4, y4), smoothed L1 () is the activation function about L1 norm, d j is the predicted j-th candidate text position, is the jth candidate text position marked, β j is the predicted rotation angle of the jth candidate text, Rotate the jth candidate text labeled.

[0105] In a specific embodiment, if x′ is used as the activation function The input parameters of the activation function It can be specifically expressed by the formula

[0106]

[0107] After the second neural network is trained using the labeled image (each text line in the image is labeled with position information and confidence score), the established second neural network can be obtained. Since the network training process is already very mature, this prior art will not be described in detail here.

[0108] The text feature map is then input into the second neural network, and the candidate text confidence score y′ and the candidate text position d are calculated when the loss function of the second neural network converges.

[0109] Step 153, using the candidate text confidence score y′ and the candidate text position d to obtain the text detection result of the candidate text line.

[0110] It can be understood that the candidate text confidence score y′ is the result of regressing and fine-tuning the confidence score of the candidate text line using the second neural network, and the candidate text position d is the result of regressing and fine-tuning the position information on the candidate text using the second neural network. Compared with the confidence score and position information initially determined for the candidate text line, the candidate text confidence score y′ and the candidate text position d can more accurately reflect the text detection status of the corresponding candidate text line.

[0111] It is easy for those skilled in the art to understand that in this embodiment, when generating the text detection results of the candidate text lines, the second neural network is used to perform regression operation processing on the candidate text confidence scores and the candidate text positions, which not only realizes the fine-tuning of the positions of the candidate text lines, but also can use the four corner point positioning method to adapt to the positioning requirements of different text scales, further improving the ability of the text detection method to adapt to the text scale and accurately locate.

[0112] Embodiment 2

[0113] Please refer to Fig. 9 Based on the text detection method disclosed in the first embodiment, a text detection device is disclosed in this embodiment. The text detection device 2 includes a camera 21, a processor 22 and a display 23, which are described below respectively.

[0114] The camera 21 is used to capture images of natural scenes and form images to be detected. In order to achieve the purpose of text detection, the image to be detected formed by the image should include one or more characters. One or more character strings together constitute text and represent the specific meaning of the text.

[0115] It should be noted that the image to be detected can be an optical image of the surface of the object to be detected in a natural scene, so the object to be detected can be a type of object with text information on the surface (such as billboards, signs, product labels, book pages, lettering, handwriting, etc.), so the image to be detected may include one or more characters (such as Chinese characters, letters, numbers, symbols, etc.). Of course, in addition to the text composed of individual characters, the image to be detected may also contain some simple or complex background information (such as color, lines, stains, flaws, patterns, etc.), at this time, the processor 22 needs to be able to eliminate the interference of the background information and detect the position of the text therefrom.

[0116] The processor 22 is connected to the camera 21 and is used to process the image to be detected according to the text detection method introduced in the first embodiment, so as to obtain the text detection result. It can be understood that the processor 22 can be a CPU, GPU, FPGA, microcontroller or digital integrated circuit with data processing function, as long as it can implement the text detection method implemented in the above steps 110-150 according to its own logic instructions.

[0117] The display 23 is connected to the processor 22 and is used to display the image to be detected and / or the text detection result. It is understood that the display 23 can be a screen with an image display function, which can display the image to be detected and the text detection result separately or together, and the specific screen type and display layout are not limited.

[0118] In this embodiment, see Fig.11 The processor 22 includes an image acquisition module 22-1, a component extraction module 22-2, a relationship prediction module 22-3, a character fusion module 22-4 and a result generation module 22-5, which are described as follows.

[0119] The image acquisition module 22 - 1 can communicate with the camera 21 , thereby acquiring the image to be detected from the camera 21 .

[0120] The component extraction module 22-2 is connected to the image acquisition module 22-1, and constructs candidate boxes corresponding to each character according to the image to be detected, extracts the feature information of the candidate boxes and generates candidate character components corresponding to each character. For example, the component extraction module 22-2 can perform multiple consecutive convolution and downsampling operations on the image to be detected, and obtain the first feature map of the corresponding scale after each operation; and perform multiple consecutive convolution and upsampling operations on the first feature map of the minimum scale, and obtain the second feature map of the corresponding scale after each operation; then perform feature summation operation on the first feature map and the second feature map of the same scale to obtain the third feature map under the corresponding scale; then construct the candidate boxes corresponding to each character on the third feature map of each scale, and obtain the feature information of the candidate boxes through regression processing; finally, map the feature information of the candidate boxes corresponding to each character to the image to be detected, and perform non-maximum suppression on the mapping result to obtain the candidate character components corresponding to each character, and the candidate character components include the position information and confidence score of the character. The function of the component extraction module 22-2 can be specifically referred to steps 121-125 in Example 1, which will not be repeated here.

[0121] The relationship prediction module 22-3 is connected to the component extraction module 22-2, and mainly predicts the connected areas of the candidate character components corresponding to each character to obtain the connection relationship between the candidate character components. For example, the relationship prediction module 22-3 obtains the position information and confidence score of the candidate character components corresponding to each character formed by the third feature map of each scale; based on the position information of the candidate character components corresponding to each character formed by the third feature map of each scale, predicts the first connection score S between the candidate character component corresponding to each character and other candidate character components in the surrounding neighborhood. neighbor ; and, according to the position information of the candidate character components corresponding to each character formed by the third feature graph of each scale, predict the second connection score S between the candidate character component corresponding to each character and the candidate character component formed by the third feature graph of other scales level Then, the connection relationship between the candidate character components corresponding to each character is determined using the first connection score and the second connection score. The function of the relationship prediction module 22-3 can be specifically referred to steps 131-134 in the first embodiment, which will not be described here.

[0122] The character fusion module 22-4 is connected to the relationship prediction module 22-3 and is mainly used to merge several interconnected candidate character components to obtain candidate text lines. For example, the character fusion module 22-4 uses the interconnected candidate character components to form a candidate set C = {c i}, where c i is the i-th connected candidate character component and the corresponding rotation angle is θ i , where the value range of i is 1 to n; according to the least squares method, a linear regression process is performed on a process parameter b to obtain a regression line, which is expressed as Then, according to the position information of the candidate character components corresponding to each character, projection is performed on the regression line to obtain the projection points on both sides and record them as (x l ,y l ) and (x r ,y r ); Next, the candidate text lines are determined using the obtained projection points on both sides. The functions of the character fusion module 22-4 may be specifically referred to steps 141-144 in the first embodiment, which will not be described in detail here.

[0123] The result generation module 22-5 is connected to the character fusion module 22-4 and is mainly used to generate text detection results for candidate text lines. For example, the result generation module 22-5 obtains the text feature map corresponding to the candidate text line through feature transformation; the second neural network is established and trained in advance, and the loss function of the second neural network is configured as L=L q +L p; Then, the text feature map is input into the second neural network, and the candidate text confidence score y′ and the candidate text position d are calculated when the loss function of the second neural network converges; Finally, the candidate text confidence score y′ and the candidate text position d are used to obtain the text detection result of the candidate text line. The function of the component extraction module 22-2 can be specifically referred to steps 151-153 in the first embodiment, which will not be repeated here.

[0124] for example Fig.10a , Fig.10b , Fig.10c The displayed product label has a plurality of text lines consisting of some character strings. The component extraction module 22-2 extracts each character (number, letter, symbol) in the product label respectively, and obtains the following: Fig.10a The relationship prediction module 22-3 predicts the connected areas of each candidate character component to obtain the connection relationship between each character. The character fusion module 22-4 fuses the candidate character components with connection relationships to obtain the following: Fig.10b The fusion result shown in FIG. 2 shows that each bar area represents a candidate text line. The result generation module 22-5 regresses and fine-tunes the candidate text lines, accurately calculates the four corner coordinates of each candidate text line, and marks each candidate text line in the form of a detection frame to form a Fig.10c At this point, the position detection of each text line in the product label is completed.

[0125] In this embodiment, the disclosed text detection device 2 can realize the functions of text image acquisition, text detection and result display. Since the processor adopts the text detection method designed in steps 110-150, it is conducive to efficient and accurate text detection in natural scenes, solves the problem of multi-directional and variable-scale detection of text in natural scenes, and greatly improves the positioning accuracy of text; in addition, since the text detection method predicts multiple parameters of the text, such as coordinates, scale, angle, etc., the processor can not only detect horizontal or rotated text, but also detect text with annular distribution, thereby effectively improving the applicability of the device in various text detection occasions.

[0126] Embodiment 3

[0127] Please refer to Fig.11 This embodiment discloses a text detection device, which mainly includes a memory 31 and a processor 32.

[0128] The main components of the text detection device 3 are a memory 31 and a processor 32. The memory 31 is a computer-readable storage medium, mainly used to store a program, which may be a program code corresponding to the text detection method in the first embodiment. The processor 32 is connected to the memory 31, and is used to execute the program stored in the memory 31 to implement the text detection method. The functions implemented by the processor 32 can refer to the processor 22 in the second embodiment, and will not be described in detail here.

[0129] Those skilled in the art will appreciate that all or part of the functions of the various methods in the above-mentioned embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, and the storage medium can include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to implement the above-mentioned functions. For example, the program is stored in the memory of the device, and when the program in the memory is executed by the processor, all or part of the above-mentioned functions can be implemented. In addition, when all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can also be stored in a storage medium such as a server, another computer, disk, optical disk, flash disk or mobile hard disk, and can be downloaded or copied and saved in the memory of the local device, or the system of the local device is updated, and when the program in the memory is executed by the processor, all or part of the functions in the above-mentioned embodiments can be implemented.

[0130] The above specific examples are used to illustrate the present invention, which is only used to help understand the present invention and is not intended to limit the present invention. For those skilled in the art, according to the concept of the present invention, some simple deductions, modifications or substitutions can be made.

Claims

1. A text detection method based on target detection, characterized in that: include: Step 1: Acquire an image to be detected, wherein the image to be detected includes one or more characters; Step 2: extracting features from the image to be detected to obtain third feature maps of different scales, constructing candidate boxes corresponding to each character on the third feature map of each scale, extracting feature information of the candidate boxes and generating candidate character components corresponding to each character; Step 3: predicting the connected areas of the candidate character components corresponding to each character to obtain the connection relationship between the candidate character components; wherein, using the position information of the candidate character components corresponding to each character formed according to the third feature map of each scale, predicting the first connection score between the candidate character component corresponding to each character and other candidate character components in the surrounding area, and predicting the second connection score between the candidate character component corresponding to each character and the candidate character components formed by the third feature map of other scales according to the position information of the candidate character components corresponding to each character formed according to the third feature map of each scale, the connection relationship between the candidate character components is determined; Step 4: According to the position information of several interconnected candidate character components, the corresponding characters are connected into candidate text lines by linear projection; Step 5: using the text feature map of a specific scale corresponding to the candidate text line to perform regression processing on the position information and confidence score of the candidate text line, and obtaining the candidate text confidence score and the candidate text position after the regression processing to form a text detection result; Wherein, the step 2 includes establishing a first neural network, and configuring the loss function of the first neural network according to the regression loss function of the position information of the candidate box and the regression loss function of the confidence score of the character in the candidate box; wherein the regression loss function of the confidence score of the character in the candidate box adopts a weighted cross entropy loss function; the regression loss function of the position information of the candidate box adopts a minimum mean square error loss function and an L2 regularization term, the minimum mean square error loss function is the square of the error between the predicted value and the true value of the X coordinate of the center point of the candidate box, the Y coordinate of the center point, the width, the height, and the rotation angle relative to the X axis, and the L2 regularization term is the weighted sum of squares of the weights to be regressed by the first neural network; The third feature map of each scale is input into the first neural network, and when the loss function of the first neural network converges, candidate boxes corresponding to each character are determined; the feature information of the candidate boxes includes position information and confidence scores.

2. The text detection method according to claim 1, characterized in that: The step of extracting features from the image to be detected to obtain third feature maps of different scales, constructing candidate frames corresponding to each character on the third feature map of each scale, extracting feature information of the candidate frames and generating candidate character components corresponding to each character, includes: Performing multiple consecutive convolution and downsampling operations on the image to be detected, and obtaining a first feature map of a corresponding scale after each operation; Perform multiple consecutive convolution and upsampling operations on the first feature map of the smallest scale, and obtain a second feature map of the corresponding scale after each operation; Perform feature summation operation on the first feature map and the second feature map of the same scale to obtain a third feature map of the corresponding scale; Constructing candidate boxes corresponding to each character on the third feature map of each scale, and obtaining feature information of the candidate boxes through regression processing; The feature information of the candidate box corresponding to each character is mapped to the image to be detected, and non-maximum suppression is performed on the mapping result to obtain the candidate character components corresponding to each character; the candidate character components include the position information and confidence score of the character.

3. The text detection method according to claim 2, characterized in that: After obtaining the third feature map of each scale and before constructing the candidate frame on the third feature map, the method further includes: performing convolution processing on the third feature map of each scale to eliminate feature aliasing effects caused by upsampling in the third feature map.

4. The text detection method according to claim 2, characterized in that: The step of constructing candidate boxes corresponding to the characters on the third feature map of each scale, and obtaining feature information of the candidate boxes through regression processing, includes: Establish a first neural network and configure the loss function of the first neural network as L rpn =L cls +L reg ; Among them, L reg is the regression loss function of the position information of the candidate box, and λ1 represents the preset weight coefficient, w * represents the weights to be regressed by the neural network, the superscript T represents the transposition operation, φ represents the activation function for activating the feature vector of the third feature map, and u * Represents the transformation relationship between the annotation position and the predicted position and satisfies u * =(u x ,u y ,u w ,u h ,u θ ),u x 、u y 、u w 、u h 、u θ respectively represent the transformation relationship of the candidate frame with respect to the X-axis coordinate, the transformation relationship with respect to the Y-axis coordinate, the transformation relationship with respect to the width pixel, the transformation relationship with respect to the height pixel, and the transformation relationship with respect to the rotation angle of the X-axis, and u x =(G x -P′ x ) / P′ w 、u y =(G y -P′ y ) / P′ h 、u w =log(G w / P′ w )、u h =log(G h / P′ h )、u θ =P′ θ -G θ , G=(G x ,G y ,G w ,G h ,G θ ) is the marked position information, P′=(P x ′,P y ′,P w ′,P h ′,P θ ′) is the predicted position information; L cls is the regression loss function of the confidence score of the character in the candidate box, and α is the preset weight coefficient, p is the predicted confidence score, is the confidence score of the annotation; The third feature map of each scale is input into the first neural network, and when the loss function of the first neural network converges, the candidate boxes corresponding to the characters are determined; the feature information of the candidate boxes includes position information P′ and confidence score p.

5. The text detection method according to claim 2, characterized in that: The predicting of connected regions of the candidate character components corresponding to each character to obtain connection relations between the candidate character components includes: Obtain the position information and confidence score of the candidate character components corresponding to each character formed by the third feature map of each scale; the position information of the candidate character component is represented by (x, y, w, h, θ), and the confidence score of the candidate character component is represented by S char ; Where x and y are the coordinates or offsets of the X-axis and Y-axis respectively, w and h are the width and height pixels respectively, and θ is the rotation angle relative to the X-axis; According to the position information of the candidate character components corresponding to each character formed by the third feature map of each scale, the first connection score between the candidate character component corresponding to each character and other candidate character components in the surrounding neighborhood is predicted. The first connection score is represented by S neighbor ; According to the position information of the candidate character components corresponding to each character formed by the third feature graph of each scale, the second connection score between the candidate character component corresponding to each character and the candidate character components formed by the third feature graph of other scales is predicted, and the second connection score is represented by S level ; The first connection score and the second connection score are used to determine the connection relationship between the candidate character components corresponding to each character.

6. The text detection method according to claim 5, characterized in that: The method of connecting the corresponding characters into candidate text lines by linear projection according to the position information of the interconnected candidate character components includes: The candidate character components connected to each other are used to form a candidate set C, and C = {c i }, where c i is the i-th connected candidate character component and the corresponding rotation angle is θ i , the value range of i is 1 to n; According to the least square method, a linear regression process is performed on a process parameter b to obtain a regression line; wherein the regression line is expressed as The objective function of the least squares method is expressed as in a i is the preset weight coefficient, y i is the Y-axis coordinate of the i-th connected candidate character component, x i is the X-axis coordinate of the i-th connected candidate character component; Project the position information of the candidate character components corresponding to each character onto the regression line, obtain the projection points on both sides and record them as (x l ,y l ) and (x r ,y r ); The candidate text line is determined by using the projection points on both sides of the acquired text line; the position information of the candidate text line is expressed as w i 、h i are the width and height pixels of the i-th candidate character component, respectively, x t is the x-coordinate of the middle pixel of the candidate text line, y t is the y coordinate of the middle pixel of the candidate text line, w t is the width of the candidate text line in pixels, h t is the height pixel of the candidate text line, θ t is the rotation angle of the candidate text line.

7. The text detection method according to claim 6, characterized in that: The method of using the text feature map of a specific scale corresponding to the candidate text line to perform regression processing on the position information and confidence score of the candidate text line, and obtaining the candidate text confidence score and the candidate text position after the regression processing to form a text detection result, includes: Obtaining a text feature map corresponding to the candidate text line through feature transformation; Establish a second neural network and configure the loss function of the second neural network as L=L q +L p ; Among them, L q is the regression loss function of the candidate text confidence score and satisfies λ2 is the preset weight coefficient, y′ is the predicted candidate text confidence score, represents the confidence score of the labeled candidate text; L p is the regression loss function of the candidate text position, and satisfies L p =L d +L β , d represents the candidate text position and is represented by the coordinates or coordinate offsets of the four corner points d = (x1, y1, x2, y2, x3, y3, x4, y4), smoothed L1 ( ) is the activation function with respect to the L1 norm, d j is the predicted j-th candidate text position, is the jth candidate text position marked, β j is the predicted rotation angle of the jth candidate text, Rotate the jth candidate text marked; Inputting the text feature map into the second neural network, and calculating the candidate text confidence score y′ and the candidate text position d when the loss function of the second neural network converges; The candidate text confidence score y′ and the candidate text position d are used to obtain the text detection result of the candidate text line.

8. A text detection device, characterized in that: include: A camera, used to capture a natural scene and form an image to be detected, wherein the image to be detected includes one or more characters; A processor, connected to the camera, and configured to process the image to be detected according to the text detection method according to any one of claims 1 to 7 to obtain a text detection result; A display is connected to the processor and is used to display the image to be detected and / or the text detection result.

9. The text detection device according to claim 8, characterized in that: The processor comprises: An image acquisition module, used for acquiring the image to be detected from the camera; A component extraction module, used to extract features from the image to be detected, obtain third feature maps of different scales, construct candidate boxes corresponding to each character on the third feature map of each scale, extract feature information of the candidate boxes and generate candidate character components corresponding to each character; wherein, constructing candidate boxes corresponding to each character on the third feature map of each scale, and extracting feature information of the candidate boxes include: establishing a first neural network, and configuring the loss function of the first neural network according to the regression loss function of the position information of the candidate box and the regression loss function of the confidence score of the character in the candidate box; wherein, the regression loss function of the confidence score of the character in the candidate box adopts a weighted cross entropy loss function; the regression loss function of the position information of the candidate box adopts a minimum mean square error loss function and an L2 regularization term, wherein the minimum mean square error loss function is the square of the error between the predicted value and the true value of the X coordinate of the center point of the candidate box, the Y coordinate of the center point, the width, the height, and the rotation angle relative to the X axis, and the L2 regularization term is the weighted square sum of the weights to be regressed by the first neural network; Inputting the third feature map of each scale into the first neural network, and determining candidate boxes corresponding to each character when the loss function of the first neural network converges; the feature information of the candidate boxes includes position information and confidence scores; A relationship prediction module is used to predict the connected areas of the candidate character components corresponding to each character to obtain the connection relationship between the candidate character components; wherein, the first connection score between the candidate character component corresponding to each character and other candidate character components in the surrounding area is predicted based on the position information of the candidate character components corresponding to each character formed according to the third feature map of each scale, and the second connection score between the candidate character component corresponding to each character and the candidate character components formed by the third feature map of other scales is predicted based on the position information of the candidate character components corresponding to each character formed according to the third feature map of each scale, so as to determine the connection relationship between the candidate character components; A character fusion module is used to connect the corresponding characters into a candidate text line by linear projection according to the position information of several interconnected candidate character components; The result generation module is used to perform regression processing on the position information and confidence score of the candidate text line using the text feature map of a specific scale corresponding to the candidate text line, and obtain the candidate text confidence score and candidate text position after regression processing to form a text detection result.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program, and the program can be executed by a processor to implement the text detection method according to any one of claims 1 to 7.

Images