Text detection method, electronic device, storage medium and computer program product

Abstract

Embodiments of the present application provide a character detection method, an electronic device, a storage medium and a computer program product. The method comprises: obtaining a to-be-processed image; inputting the to-be-processed image into a character detection model to obtain at least one character box and at least one center line in the to-be-processed image, the character box being a box surrounding at least part of a predicted character region, and the center line being a center line of the predicted character region; clustering the at least one character box and the at least one center line to obtain at least one box line combination, each box line combination comprising a single center line and one or more character boxes; and determining, for any box line combination in the at least one box line combination, position information of a predicted character region corresponding to the box line combination based on the center line and / or the character box in the box line combination. By combining the character box and the center line, curved characters and characters with a very large aspect ratio can be effectively detected.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and more particularly to a text detection method, an electronic device, a storage medium and a computer program product. BACKGROUND

[0002] Existing text detection technical solutions can be divided into three categories according to the model prediction method: a detection solution based on regression, a detection solution based on segmentation, and a detection solution combining the two. Each of the three has advantages and disadvantages. The detection solution based on regression has a higher recall rate and is more sensitive to the arrangement direction of the text, but the detection effect is not good for curved text and text with a large aspect ratio. The detection solution based on segmentation can effectively solve the shortcomings of the regression-based solution, but it is not sensitive to the arrangement direction of the text, so the detection effect is not good for grid layout text, and it also cannot detect overlapping text well. Therefore, the solution combining the above two can take the advantages of each other and achieve better results, but the model structure of most mixed solutions is complex, and if it is to be applied to curved text detection, most solutions are based on a serial structure of first regression and then segmentation (first detecting a rectangular frame through regression, and then detecting a curved text area through segmentation) or rely on more expensive single-character labeling, so the practicality is very limited. SUMMARY

[0003] The present application is proposed in consideration of the above problems. The present application provides a text detection method, a target detection method, an electronic device, a storage medium and a computer program product.

[0004] According to an aspect of the present application, a text detection method is provided, comprising: obtaining a to-be-processed image; inputting the to-be-processed image into a text detection model to obtain at least one text frame and at least one center line in the to-be-processed image, the text frame being a frame surrounding at least part of a predicted text region, and the center line being a center line of the predicted text region; clustering the at least one text frame and the at least one center line to obtain at least one frame-line combination, each frame-line combination comprising a single center line and one or more text frames; and determining, for any frame-line combination in the at least one frame-line combination, position information of a predicted text region corresponding to the frame-line combination based on the center line and / or the text frame in the frame-line combination.

[0005] Exemplarily, clustering the at least one text frame and the at least one center line to obtain the at least one frame-line combination comprises: calculating the association degree between any text frame in the at least one text frame and any center line in the at least one center line; and for any text frame, clustering the text frame and the relevant center line into the same combination to constitute a frame-line combination corresponding to the relevant center line, wherein the relevant center line is a center line with the largest association degree and the association degree exceeding a first association degree threshold with the text frame.

[0006] Exemplarily, the clustering of the at least one text box and the at least one center line to obtain the at least one frame line combination further comprises: if there is a scattered text box, calculating a correlation degree and a relative position relationship between the scattered text box and each text box in any particular frame line combination, the scattered text box being a text box not belonging to any frame line combination; and if the correlation degree between the scattered text box and any particular text box in the particular frame line combination is greater than a second correlation degree threshold and the relative position relationship between the scattered text box and the particular text box meets a target requirement, assigning the scattered text box to the particular frame line combination.

[0007] Exemplarily, the position information comprises a region contour, and the determining of the position information of the predicted text region corresponding to the frame line combination based on the text boxes in the frame line combination comprises: for a first part of text boxes in the frame line combination, arranging the first part of text boxes in a target extension direction to obtain an arranged text box set, wherein the target extension direction is an extension direction of a line connecting feature points contained in each of a second part of text boxes, the first part of text boxes being at least part of the text boxes in the frame line combination, and the second part of text boxes being at least part of the text boxes in the first part of text boxes; and connecting at least part of corner points in the arranged text box set together in a clockwise or counterclockwise direction to obtain the region contour.

[0008] Exemplarily, the connecting of the at least part of corner points in the arranged text box set together in the clockwise or counterclockwise direction to obtain the region contour comprises: cyclically performing the following corner point connecting operation until all of the at least part of corner points in the arranged text box set are connected: connecting at least part of corner points of the first two text boxes in a current text box set together in the clockwise or counterclockwise direction to obtain a new text box, and determining the new text box and the remaining text boxes in the current text box set as a new text box set, wherein the current text box set is the arranged text box set when the first corner point connecting operation is performed, and the new text box set determined by any corner point connecting operation is the current text box set corresponding to the next corner point connecting operation.

[0009] Exemplarily, the connecting of the at least part of corner points of the first two text boxes in the current text box set together in the clockwise or counterclockwise direction to obtain the new text box comprises: in a case where the first two text boxes in the current text box set do not have an intersection, connecting all the corner points in the first two text boxes together in the clockwise or counterclockwise direction to obtain the new text box; and in a case where the first two text boxes in the current text box set have an intersection, discarding a top-right corner point of a text box on the left side and a bottom-left corner point of a text box on the right side in the first two text boxes, and connecting the remaining corner points together in the clockwise or counterclockwise direction to obtain the new text box.

[0010] Illustratively, before arranging the first part of text boxes in the target extension direction, the position information of the predicted text region corresponding to the frame line combination is determined based on the text boxes in the frame line combination, which further includes: filtering out the text boxes in the frame line combination whose area is greater than a first target multiple of the area of a standard text box and / or whose area is less than a second target multiple of the area of a standard text box, to obtain the first part of text boxes, wherein the standard text box is a text box in the frame line combination whose area is at the median.

[0011] Illustratively, the position information includes a region contour, and the position information of the predicted text region corresponding to the frame line combination is determined based on the center line in the frame line combination, which includes: inflating the height of the center line in the frame line combination to a first target height, wherein the first target height is a fixed value, or the first target height is the average of a first height and a second height, or the first target height is equal to the height of the center line before inflation inflated by a target inflation coefficient, the first height is the height of the left side edge of the leftmost text box in the frame line combination, and the second height is the height of the right side edge of the rightmost text box in the frame line combination; and determining the contour line of the inflated center line as the region contour.

[0012] Illustratively, the step of determining the position information of the predicted text region corresponding to the frame line combination based on the center line in the frame line combination is performed when the center line in the frame line combination exceeds a target range, and the target range is the range of the region contour determined based on the text boxes in the frame line combination.

[0013] Illustratively, the text detection model is obtained by training in the following manner: obtaining a sample image and text box annotation data, the text box annotation data including position information of an initial text box, the initial text box being a text box enclosing a real text region in the sample image; dividing the initial text box into a target number of sample text boxes; determining a sample center line based on the target number of sample text boxes, the sample center line being a center line of the real text region; and training the text detection model based on the sample image, the target number of sample text boxes, and the sample center line.

[0014] Illustratively, determining the sample center line based on the target number of sample text boxes includes: for any corner point in the target number of sample text boxes on a first long side and a corresponding corner point in the target number of sample text boxes on an opposite second long side, determining a center point of the two corner points; connecting all the center points to obtain an initial center line; and determining the initial center line or a new center line as the sample center line, wherein the new center line is obtained by: contracting both ends of the initial center line by a target proportion; and / or inflating the height of the initial center line to a second target height.

[0015] Exemplarily, dividing the initial text box into the target number of sample text boxes comprises: dividing a first long side of the initial text box equally according to the target number to obtain a first set of corner points; dividing a second long side of the initial text box equally according to the target number to obtain a second set of corner points; and combining each two adjacent corner points in the first set of corner points with two corresponding adjacent corner points in the second set of corner points to form a sample text box.

[0016] According to another aspect of the present application, an electronic device is also provided, comprising a processor and a memory, wherein the memory stores computer program instructions, which, when executed by the processor, are used to perform the text detection method described above.

[0017] According to another aspect of the present application, a storage medium is provided, on which program instructions are stored, which, when executed, are used to perform the text detection method described above.

[0018] According to another aspect of the present application, a computer program product is provided, comprising a computer program, which, when executed, is used to perform the text detection method described above.

[0019] According to the text detection method, the electronic device, the storage medium and the computer program product of the embodiments of the present application, the center line and the text box are predicted by using the text detection model, the center line and the text box corresponding to each predicted text region are determined through clustering and grouping of the center line and the text box, and then the position of the predicted text region is determined. This scheme can represent the text region by the combination of text boxes instead of a single text box, and the positioning of the text region and the grouping of the text boxes are assisted by the center line. In this way, the curved text and the text with a large aspect ratio can be effectively detected by the combination of the text box and the center line. In addition, in terms of recall rate, the text box and the center line have relatively obvious recall effects on large and small scales respectively, and the combination of the two can effectively improve the overall detection recall rate. In addition, the text detection model of the present application does not need to regress first and then segment, but can directly obtain the prediction results of the text box and the center line, and then determine the position of the text region by combining post-processing (clustering and position information determination operations). This network model is relatively simple and practical. BRIEF DESCRIPTION OF DRAWINGS

[0020] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which like reference characters refer to like parts throughout the figures. The accompanying drawings provide illustration of the embodiments of the present application and constitute a part of the specification, together with the description, to explain the embodiments of the present application and do not limit the present application. In the drawings, like reference numerals refer to like parts throughout the various figures.

[0021] Figure 1A schematic block diagram showing an example electronic device for implementing the text detection method and apparatus according to an embodiment of the present application is shown;

[0022] Figure 2 A schematic flow chart showing a text detection method according to an embodiment of the present application is shown;

[0023] Figure 3a A schematic diagram showing a text region and a corresponding text box according to an embodiment of the present application is shown;

[0024] Figure 3b A schematic diagram showing a text region and a corresponding center line according to an embodiment of the present application is shown;

[0025] Figure 4 A schematic diagram showing a box line combination of a text box and a center line according to an embodiment of the present application is shown;

[0026] Figure 5 A schematic diagram showing a region contour according to an embodiment of the present application is shown;

[0027] Figure 6a A schematic diagram showing connecting corner points of a text box according to an embodiment of the present application is shown;

[0028] Figure 6b A schematic diagram showing connecting corner points of a text box according to another embodiment of the present application is shown;

[0029] Figure 7a A schematic diagram showing obtaining corner points by equally dividing edges of an initial text box according to an embodiment of the present application is shown;

[0030] Figure 7b A schematic diagram showing combining corner points to form a sample text box is shown;

[0031] Figure 8 A schematic diagram showing obtaining an initial center line by a center point of a corner point according to an embodiment of the present application is shown;

[0032] Figure 9 A schematic diagram showing contracting the initial center line shown in Figure 8 according to an embodiment of the present application is shown;

[0033] Figure 10 A schematic diagram showing a training flow of a text detection model according to an embodiment of the present application is shown;

[0034] Figure 11 A schematic block diagram showing a text detection apparatus according to an embodiment of the present application is shown; and

[0035] Figure 12 A schematic block diagram showing an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0036] In recent years, important progress has been made in the research of computer vision, deep learning, machine learning, image processing, image recognition and other technologies based on artificial intelligence. Artificial intelligence (AI) is a new science and technology that studies and develops theories, methods, technologies and application systems for simulating and extending human intelligence. Artificial intelligence is a comprehensive discipline involving chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, neural networks and many other technology categories. Computer vision, as an important branch of artificial intelligence, specifically enables machines to recognize the world. Computer vision technology generally includes face recognition, image processing, fingerprint recognition and anti-fraud verification, biometric recognition, face detection, pedestrian detection, object detection, pedestrian recognition, image processing, image recognition, image semantic understanding, image retrieval, character recognition, video processing, video content recognition, three-dimensional reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, robot navigation and positioning, and other technologies. With the research and progress of artificial intelligence technology, this technology has been applied in many fields, such as city management, traffic management, building management, park management, face passage, face attendance, logistics management, warehouse management, robots, intelligent marketing, computational photography, mobile phone images, cloud services, smart home, wearable devices, driverless vehicles, autonomous driving, intelligent medical care, face payment, face unlocking, fingerprint unlocking, face and certificate verification, smart screens, smart televisions, cameras, mobile Internet, network live streaming, beautification, makeup, medical cosmetology, intelligent temperature measurement, and other fields.

[0037] In order to make the purpose, technical solutions and advantages of the present application more obvious, the example embodiments according to the present application will be described in detail below with reference to the drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all embodiments of the present application, and it should be understood that the present application is not limited to the example embodiments described herein. Based on the embodiments of the present application described in the present application, all other embodiments obtained by those skilled in the art without creative labor shall fall within the scope of protection of the present application.

[0038] The present application embodiment provides a character detection method, an electronic device, a storage medium and a computer program product. According to the character detection method of the present application embodiment, curved characters and characters with a very large aspect ratio can be effectively detected. The character detection technology according to the present application embodiment can be applied to any field involving character detection.

[0039] First, with reference to Figure 1 An example electronic device 100 for implementing the character detection method and device according to the present application embodiment will be described.

[0040] As shown in Figure 1 The electronic device 100 includes one or more processors 102, one or more storage devices 104. Optionally, the electronic device 100 can also include an input device 106, an output device 108, and an image capturing device 110, which are interconnected through a bus system 112 and / or other form of connection mechanism (not shown). It should be noted that Figure 1 The components and structure of the electronic device 100 shown are only exemplary and not limiting, and the electronic device can also have other components and structure as needed.

[0041] The processor 102 can be implemented in at least one hardware form of a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic array (PLA), a microprocessor, and can be a combination of one or several of a central processing unit (CPU), a graphics processor (GPU), an application-specific integrated circuit (ASIC), or other form of processing unit having data processing and / or instruction execution capabilities, and can control other components in the electronic device 100 to perform desired functions.

[0042] The storage device 104 can include one or more computer program products, which can include various forms of computer readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory can include, for example, random access memory (RAM), cache memory, and / or the like. The non-volatile memory can include, for example, read-only memory (ROM), hard disk, flash memory, and / or the like. One or more computer program instructions can be stored on the computer readable storage medium, and the processor 102 can execute the program instructions to implement the client functions (implemented by the processor) in the embodiments described below and / or other desired functions. Various application programs and various data, such as various data used and / or generated by the application programs, and the like, can also be stored in the computer readable storage medium.

[0043] The input device 106 can be a device used by a user to input instructions, and can include one or more of a keyboard, a mouse, a microphone, a touch screen, and the like.

[0044] The output device 108 can output various information (such as images and / or sounds) to the outside (such as a user), and can include one or more of a display, a speaker, and the like. Optionally, the input device 106 and the output device 108 can be integrated together, implemented with the same interactive device (such as a touch screen).

[0045] The image capturing device 110 can capture images and store the captured images in the storage device 104 for use by other components. The image capturing device 110 can be a separate camera or a camera in a mobile terminal, etc. It should be understood that the image capturing device 110 is merely an example, and the electronic device 100 can not include the image capturing device 110. In this case, images can be captured by other devices with image capturing capabilities and transmitted to the electronic device 100.

[0046] Exemplarily, an example electronic device for implementing the character detection method and device according to the embodiments of the present application can be implemented on a device such as a personal computer, a terminal device, an attendance machine, a panel machine, a camera, or a remote server, etc. Among them, the terminal device includes but is not limited to a tablet computer, a mobile phone, a PDA (Personal Digital Assistant), a touch screen all-in-one machine, a wearable device, etc.

[0047] Below, the character detection method according to the embodiments of the present application will be described with reference to the accompanying drawings. Figure 2 The character detection method according to the embodiments of the present application will be described below. Figure 2 A schematic flow chart of a character detection method 200 according to an embodiment of the present application is shown. As shown in the figure, the character detection method 200 includes steps S210, S220, S230, and S240. Figure 2

[0048] In step S210, a to-be-processed image is acquired.

[0049] Exemplarily, the to-be-processed image can be any type of image containing characters. The to-be-processed image can be a static image or any video frame in a dynamic video. The to-be-processed image can be an original image captured by an image capturing device (such as an image sensor in a camera), or an image obtained after pre-processing (such as digitization, normalization, smoothing, etc.) of the original image. It should be noted that the pre-processing of the original image can include the operation of extracting a sub-image containing characters from the original image captured by the image capturing device to obtain the to-be-processed image.

[0050] The to-be-processed image can be sent to the electronic device 100 by an external device (such as a cloud server) for processing by the processor 102 of the electronic device 100, or captured by an image capturing device 110 (such as a camera) included in the electronic device 100 and transmitted to the processor 102 for processing, or stored by a storage device 104 included in the electronic device 100 and transmitted to the processor 102 for processing.

[0051] ​At step S220, input the image to be processed into a text detection model to obtain at least one text box and at least one center line in the image to be processed, the text box being a box enclosing at least a part of a predicted text region, and the center line being a center line of the predicted text region.

[0052] The text detection model can be implemented by any suitable existing or future neural network model capable of realizing the function of text detection. Illustratively but not limitatively, the text detection model can include one or more of the following networks: Region-based Convolutional Neural Network (RCNN), Fully Convolutional One-Stage Object Detector (FCOS), connection text proposal network (CTPN), etc. In one example, the text detection model can include a text box detection module and a center line detection module. The text box detection module can be implemented by a structure similar to a conventional text detection network such as FCOS, mainly for detecting text boxes. The center line detection module can be implemented by a structure similar to a conventional image segmentation network such as Fully Convolutional Networks (FCN), mainly for detecting center lines. The text box detection module and the center line detection module can be independent of each other, or can share a feature extraction module.

[0053] Through the text detection model, a plurality of text boxes and a plurality of center lines can be detected and obtained. Each text box is a box enclosing at least a part of a predicted text region. Each text box is a quadrilateral box. The predicted text region is a text region predicted by the text detection model, and its specific position (e.g. contour size) can be determined through subsequent steps S230 and S240. The text region described herein (including the predicted text region and the real text region described below) can include text of any length, and the text region can have any shape, e.g. can be rectangular or curved, etc. The text region can be further divided into a plurality of smaller text boxes. The center line predicted by the text detection model is a center line of the predicted text region. The text region described herein (including the predicted text region and the real text region described below) can have a center line, which can have a specific height. The specific height can be equal to or greater than 0. Illustratively, the value of height can be expressed in the number of pixels, e.g. the specific height can be 1 pixel, 3 pixels, 5 pixels, etc.

[0054] Figure 3a A schematic diagram showing a text region and a corresponding text box according to one embodiment of the present application. Figure 3bFig. 2 shows a schematic diagram of a text region and a corresponding center line according to one embodiment of the present application. Figure 3a The text box is represented by a solid box. Figure 3b The center line is represented by a dashed line. From Figure 3a As can be seen, one text region can be divided into multiple text boxes, or said in another way, can be represented by multiple text boxes. From Figure 3b As can be seen, one text region corresponds to one center line.

[0055] In step S230, the at least one text box and the at least one center line are clustered to obtain at least one box line combination, each box line combination including a single center line and one or more text boxes. Each box line combination can include a single center line and one or more text boxes corresponding to the same predicted text region.

[0056] The text boxes and the center lines can be clustered and grouped such that each box line combination contains one center line and at least one text box. Each box line combination can correspond to one predicted text region.

[0057] In step S240, for any box line combination in the at least one box line combination, the position information of the predicted text region corresponding to the box line combination is determined based on the center line and / or the text boxes in the box line combination.

[0058] The position information of the predicted text region can include one or more of the following: the region contour of the predicted text region, the position information of multiple corner points of the predicted text region (the region surrounded by the multiple corner points connected to each other can be regarded as the predicted text region).

[0059] The position information of the predicted text region corresponding to any box line combination can be determined based on the center line and / or the text boxes in the box line combination, and there are multiple implementation schemes for determining the position information, which will be described below.

[0060] According to the text detection method, the center line and the text box are predicted by using the text detection model, the center line and the text box corresponding to each predicted text region are determined by clustering grouping of the center line and the text box, and then the position of the predicted text region is determined. This scheme can represent the text region by the combination of text boxes instead of a single text box, and the positioning of the text region and the grouping of the text boxes are assisted by the center line. In this way, the combination of the text box and the center line can effectively detect curved text and text with a large aspect ratio. In addition, in terms of recall rate, the text box and the center line have obvious recall effect on large and small scales respectively, and the combination of the two can effectively improve the overall detection recall rate. In addition, the text detection model of the present application does not need to regress first and then segment, but can directly obtain the prediction results of the text box and the center line, and then determine the position of the text region by combining post-processing (clustering and position information determination, etc.). This network model is relatively simple and practical.

[0061] The text detection method according to the embodiments of the present application can be deployed at a terminal, for example, can be deployed at a personal terminal or a server end.

[0062] Alternatively, the text detection method according to the embodiments of the present application can also be distributed and deployed at the server end (or cloud end) and the client end. For example, the image to be processed can be collected at the client end, the client end transmits the image to be processed to the server end (or cloud end), and the server end (or cloud end) performs text detection.

[0063] According to the embodiments of the present application, clustering the at least one text box and the at least one center line to obtain at least one combination of the box line includes: calculating the association degree between any text box in the at least one text box and any center line in the at least one center line; for any text box, clustering the text box and the related center line into the same combination to constitute the combination of the box line corresponding to the related center line, wherein the related center line is the center line with the maximum association degree and the association degree exceeding the first association degree threshold.

[0064] The first association degree threshold can be set to any suitable value as needed, which can be a value greater than or equal to 0, for example, 0.01, etc.

[0065] The same text region can be represented by a single center line and multiple text boxes. Thus, text boxes and center lines that are spatially associated relatively closely can be clustered and grouped, and text boxes and center lines corresponding to each text region can be found. Exemplarily, clustering can be performed by calculating the degree of association between text boxes and center lines. The degree of association can be represented by the intersection over union (10U) between text boxes and center lines. For example, the degree of association between text box A and center line C can be calculated as follows: the area of the region where text box A and center line C overlap / (the area of text box A + the area of center line C).

[0066] Exemplarily, an image coordinate system can be established on the image to be processed. For example, a planar image coordinate system can be established with the upper left corner of the image to be processed as the origin, the width direction of the image to be processed as the x-axis, and the height direction of the image to be processed as the y-axis. For the convenience of understanding, the various position information described below will be described along this coordinate system. Of course, the above-mentioned manner of establishing the image coordinate system is only an example, and the image coordinate system can also have other suitable establishment manners.

[0067] Exemplarily, in order to calculate the degree of association, the maximum and minimum values of the x-axis and the maximum and minimum values of the y-axis of each text box can be determined. In addition, the maximum and minimum values of the x-axis and the maximum and minimum values of the y-axis of each center line can also be determined. Subsequently, the degree of association (i.e., the above-mentioned 10U) between any text box and any center line can be calculated according to the maximum and minimum values of the x-axis and the maximum and minimum values of the y-axis of the text box, and according to the maximum and minimum values of the x-axis and the maximum and minimum values of the y-axis of the center line. Those skilled in the art can understand the specific calculation method of 10U, which will not be described herein.

[0068] In one example, the degree of association between each text box and each center line can be directly calculated. In another example, for any text box, center lines that have no intersection with the text box can be excluded first, and the degree of association is only calculated for the center lines that have an intersection with it. For example, whether any text box and any center line have an intersection can be determined by the maximum and minimum values of the x-axis and the maximum and minimum values of the y-axis of the text box, and according to the maximum and minimum values of the x-axis and the maximum and minimum values of the y-axis of the center line.

[0069] After the correlation degree is calculated, the correlation degree between the center line and the text box can be further grouped. For example, if the correlation degree between text box A1 and center line C2 is the largest and exceeds the first correlation degree threshold (for example, 0.01), it can be determined that text box A1 and center line C2 belong to the same combination. In this way, each text box can be clustered into a combination corresponding to each center line. After clustering is completed, a frame line combination corresponding to each center line can be obtained. It can be understood that as many center lines as there are, as many frame line combinations can be generated. Figure 4 A schematic diagram of a text box and a center line forming a frame line combination according to an embodiment of the present application is shown. Referring to Figure 4 The five text boxes in the upper half of the picture can be combined with the center line above to form a frame line combination, and the five text boxes in the lower half of the picture can be combined with the center line below to form another frame line combination. In addition, it can be seen that the frame line combination above corresponds to the text area where “EXCELSIOR” is located, and the frame line combination below corresponds to the text area where “CAFFE” is located.

[0070] The text box and the center line can be grouped quickly and conveniently by calculating the correlation degree of the text box and the center line, and this scheme has small calculation amount and simple implementation. Of course, other ways can also be used to cluster and group the text box and the center line, for example, the text box and the center line can be input into a trained clustering model for clustering and grouping. The clustering model can be a neural network model.

[0071] According to an embodiment of the present application, clustering at least one text box and at least one center line to obtain at least one frame line combination further includes: if there is a scattered text box, calculating the correlation degree and the relative position relationship between the scattered text box and each text box in any specific frame line combination, the scattered text box being a text box not belonging to any frame line combination; if the correlation degree between the scattered text box and any specific text box in the specific frame line combination is greater than a second correlation degree threshold and the relative position relationship between the scattered text box and the specific text box meets a target requirement, the scattered text box is assigned to the specific frame line combination.

[0072] The second correlation degree threshold can be set to any suitable value as needed, which can be a value greater than or equal to 0, for example, 0.01. The second correlation degree threshold can be equal to or different from the first correlation degree threshold described above.

[0073] If there is a text box not clustered into any frame line combination, the text box can be considered as a scattered text box. Exemplarily, the scattered text box can be a text box determined not to belong to any frame line combination based on the association degree with the center line. For the scattered text box, the association degree with the text boxes in each frame line combination can be further calculated to determine whether the scattered text box can be clustered into a frame line combination. For example, the association degree and the relative position relationship of the scattered text box and the text boxes in each frame line combination can be calculated. If there is any text box in a frame line combination and the association degree between the scattered text box and the text box is greater than a second association degree threshold (for example, 0) and the relative position relationship between the two text boxes meets the target requirement, the scattered text box can be classified into the frame line combination.

[0074] For the scattered text box, the association degree with the text boxes in the frame line combination is further calculated and the grouping is performed based on the association degree information. This scheme can improve the clustering accuracy of the text boxes and help improve the accuracy of text detection.

[0075] According to an embodiment of the present application, the relative position relationship is represented based on the distance between the center point coordinates of the two text boxes, and the target requirement is less than the distance threshold.

[0076] The relative position relationship between the two text boxes can be represented by the distance between the center point coordinates of the two text boxes, but this is only an example. The relative position relationship between the two text boxes can also be represented by other information. For example, the relative position relationship between the two text boxes can also be the distance between the two corresponding corner points of the two text boxes. The corresponding corner points refer to the corner points having the same relative position in the respective text boxes, for example, both are the top left corner points, or both are the bottom right corner points, and the like.

[0077] The distance threshold can be any suitable value, which can be set as needed, and the present application does not limit this. For example, the target requirement can be that the distance between the center point coordinates of the two text boxes is within 20 pixels.

[0078] According to an embodiment of the present application, the position information of the predicted text region corresponding to the frame line combination is determined based on the text boxes in the frame line combination, including: for a first part of the text boxes in the frame line combination, arranging the first part of the text boxes in a target extension direction to obtain an arranged text box set; determining at least part of the corner points from the arranged text box set in a clockwise or counterclockwise direction; and obtaining the position information of the predicted text region based on the at least part of the corner points.

[0079] Exemplarily, the position information of the predicted text region can include corner point position information of corner points of the predicted text region. In this case, in one example, at least part of the corner points can be determined from the arranged text box set for composing the predicted text region, without connecting the corner points. In another example, the region contour of the predicted text region can be determined through a corner point connecting operation, and then a plurality of contour points can be determined from the region contour as the corner points of the predicted text region. The position information of the contour points is taken as the corner point position information of the corner points of the predicted text region.

[0080] Exemplarily, the position information of the predicted text region can include a region contour of the predicted text region. In this case, the region contour of the predicted text region can be determined through a corner point connecting operation.

[0081] According to an embodiment of the present application, determining at least part of the corner points from the arranged text box set along the clockwise or counterclockwise direction can include: cyclically performing the following corner point combination operation until the combination of at least part of the corner points in the arranged text box set is completed: combining at least part of the corner points of the first two text boxes in the current text box set together along the clockwise or counterclockwise direction to obtain a new text box, and determining the new text box and the remaining text boxes in the current text box set as a new text box set, wherein the current text box set is the arranged text box set when the first corner point combination operation is performed, the new text box set determined by each corner point combination operation is the current text box set corresponding to the next corner point combination operation, and all the corner points included in the new text box set determined by the last corner point combination operation are the determined at least part of the corner points.

[0082] According to an embodiment of the present application, combining at least part of the corner points of the first two text boxes in the current text box set together along the clockwise or counterclockwise direction to obtain a new text box includes: in the case that the first two text boxes in the current text box set do not have an intersection, combining all the corner points in the first two text boxes together along the clockwise or counterclockwise direction to obtain a new text box; in the case that the first two text boxes in the current text box set have an intersection, discarding the top right corner point of the text box on the left side and the bottom left corner point of the text box on the right side in the first two text boxes, and combining the remaining corner points together along the clockwise or counterclockwise direction to obtain a new text box.

[0083] In the case that the position information of the predicted text region includes a region contour of the predicted text region, the above corner point combination operation can be implemented as a corner point connecting operation, and the following embodiment for determining the region contour can be understood according to the following embodiment.

[0084] According to an embodiment of the present application, the position information comprises a region contour, and determining the position information of the predicted text region corresponding to the frame line combination based on the text frame in the frame line combination comprises: for a first part of the text frames in the frame line combination, arranging the first part of the text frames in a target extension direction to obtain an arranged text frame set, wherein the target extension direction is an extension direction of a line connecting feature points contained in each of the second part of the text frames, the first part of the text frames is at least part of the text frames in the frame line combination, and the second part of the text frames is at least part of the text frames in the first part of the text frames; and connecting at least part of the corner points in the arranged text frame set together in a clockwise or counterclockwise direction to obtain the region contour.

[0085] The target extension direction is a uniform extension direction of the first part of the text frames, and can also be understood as an extension direction of the text contained in the first part of the text frames. Specifically, the target extension direction can be an extension direction of a line connecting feature points contained in each of the second part of the text frames. The relative positions of the feature points in different text frames in the corresponding text frames can be consistent, for example, being the center points of the text frames. The relative positions of the feature points in different text frames in the corresponding text frames can also be inconsistent, which can be understood with reference to the examples described below.

[0086] Exemplarily, the second part of the text frames can be all the text frames in the first part of the text frames, and the target extension direction can be determined by the following manner: determining the center point of each of the text frames in the second part of the text frames; and determining an extension direction of a line connecting all the center points of the second part of the text frames as the target extension direction. If the line connecting the center points extends more in the x-axis direction than in the y-axis direction, it can be considered that the extension direction of the text frame is horizontal, and vice versa. The center point of any text frame is the intersection of two diagonals of the text frame. Whether the line connecting the center points extends more in the x-axis direction or in the y-axis direction can be determined by the following manner. For example, the difference between the maximum value of the x-axis of all the center points and the minimum value of the x-axis can be calculated, and the difference between the maximum value of the y-axis of all the center points and the minimum value of the y-axis can be calculated, and if the x-axis difference is larger, it is determined that the line connecting the center points extends more in the x-axis direction than in the y-axis direction, and if the y-axis difference is larger, it is determined that the line connecting the center points extends more in the y-axis direction than in the x-axis direction.

[0087] Of course, the x-axis maximum value and the x-axis minimum value among all the corner points in the first part of the text boxes in any frame line combination (for example, the x value of the top right corner point of the rightmost text box is the x-axis maximum value, and the x value of the top left corner point of the leftmost text box is the x-axis minimum value), and the y-axis maximum value and the y-axis minimum value among all the corner points (for example, the y value of the bottom right corner point of the rightmost text box is the y-axis maximum value, and the y value of the top left corner point of the leftmost text box is the y-axis minimum value) can be directly determined. The difference between the x-axis maximum value and the x-axis minimum value can be calculated, and the difference between the y-axis maximum value and the y-axis minimum value can be calculated. If the x-axis difference is greater, it is determined that the line connecting the center points extends more in the x-axis direction than in the y-axis direction. If the y-axis difference is greater, it is determined that the line connecting the center points extends more in the y-axis direction than in the x-axis direction. In this embodiment, the second part of the text boxes can be part of the first part of the text boxes, for example, the rightmost text box and the leftmost text box in the first part of the text boxes. In this embodiment, the relative positions of the feature points of different text boxes in the corresponding text boxes can be different.

[0088] For example, the first part of the text boxes can be arranged in order. If the target extension direction is horizontal, all the text boxes can be arranged in ascending order according to the x-axis coordinates, or vice versa.

[0089] After the arrangement, at least part of the corner points of the arranged text boxes (arranged text box set) can be connected together in a clockwise or counterclockwise direction to obtain a region contour. Figure 5 A schematic diagram of a region contour according to an embodiment of the present application is shown. As shown in Figure 5 Based on the two frame line combinations shown in Figure 4 , the region contours of two text regions can be obtained, respectively.

[0090] According to an embodiment of the present application, connecting at least part of the corner points in the arranged text box set together in a clockwise or counterclockwise direction to obtain a region contour includes: cyclically performing the following corner point connection operation until all the corner points in the arranged text box set are connected: connecting at least part of the corner points of the first two text boxes in the current text box set together in a clockwise or counterclockwise direction to obtain a new text box, and determining the new text box and the remaining text boxes in the current text box set as a new text box set, wherein the current text box set is the arranged text box set when the first corner point connection operation is performed, and the new text box set determined by any corner point connection operation is the current text box set corresponding to the next corner point connection operation.

[0091] Referring back to Figure 4For the five text boxes above, the corner points of the left two text boxes can be first connected together to form a new text box, and then the remaining three text boxes are added to obtain four text boxes. Subsequently, for the updated four text boxes, the corner points of the left two text boxes are again connected together to form a new text box, and then the remaining two text boxes are added to obtain three text boxes. Subsequently, for the updated three text boxes, the corner points of the left two text boxes are again connected together to form a new text box, and then the remaining one text box is added to obtain two text boxes. Finally, the corner points of the two text boxes are connected together to obtain the required region contour. That is, the above corner point connection operation can be performed on two adjacent text boxes each time, and can be executed in a loop until the corner points of all text boxes are connected. Of course, this loop execution manner is only an example, and all corner points in the arranged text box set can be optionally connected together at one time.

[0092] According to an embodiment of the present application, the at least part of the corner points of the first two text boxes in the current text box set are connected together in the clockwise or counterclockwise direction to obtain a new text box, including: in the case that the first two text boxes in the current text box set do not have an intersection, connecting all the corner points of the first two text boxes together in the clockwise or counterclockwise direction to obtain a new text box; in the case that the first two text boxes in the current text box set have an intersection, discarding the top right corner point of the text box on the left and the bottom left corner point of the text box on the right in the first two text boxes, and connecting the remaining corner points together in the clockwise or counterclockwise direction to obtain a new text box.

[0093] If there is no intersection between the two text boxes, all the corner points can be used to generate a region contour as the contour key points. Figure 6a A schematic diagram of connecting the corner points of a text box according to an embodiment of the present application is shown. As shown in Figure 6a The five text boxes are detected as a whole, and the first two adjacent text boxes in the five text boxes do not intersect with each other. At this time, for the first text box (text box 1) and the second text box (text box 2), the corner points can be selected as the key points of the region contour in the following order to form a region contour: "text box 1" top left corner point— "text box 1" top right corner point— "text box 2" top left corner point— "text box 2" top right corner point— "text box 2" bottom right corner point— "text box 2" bottom left corner point— "text box 1" bottom right corner point— "text box 1" bottom left corner point, as shown in Figure 6a The points 0—1—2—3—4—5—6—7.

[0094] If there is a certain intersection between two text boxes, part of the corner points can be discarded, and the remaining corner points are used as the contour key points to generate the region contour. In this way, the problem of contour folding can be effectively avoided. Figure 6b Fig. 4 shows a schematic diagram of connecting the corner points of a text box according to another embodiment of the present application. As shown in Fig. 4, the text region is detected as five text boxes, and the first two adjacent text boxes intersect with each other. In this case, for the first text box (text box 1) and the second text box (text box 2), the corner points can be selected as the key points of the region contour in the following order to form the region contour: the top-left corner point of the text box 1 — the top-left corner point of the text box 2 — the top-right corner point of the text box 2 — the bottom-right corner point of the text box 2 — the bottom-right corner point of the text box 1 — the bottom-left corner point of the text box 1, as shown in Fig. 4. Figure 6b Figure 6b

[0095] According to an embodiment of the present application, before arranging the first part of text boxes in the target extension direction for the first part of text boxes in the frame line combination, determining the position information of the predicted text region corresponding to the frame line combination based on the text boxes in the frame line combination further includes: filtering out the text boxes in the frame line combination whose areas are greater than a first target multiple of the area of a standard text box and / or whose areas are less than a second target multiple of the area of the standard text box to obtain the first part of text boxes, wherein the standard text box is a text box in the frame line combination whose area is at a median value.

[0096] The first target multiple and the second target multiple can be set to any suitable value as needed, which is not limited in the present application. The first target multiple is greater than the second target multiple.

[0097] Optionally, for each frame line combination, the text boxes with large shape difference can be removed. For example, the text boxes whose areas are greater than 5 times the area of a standard text box or less than 1 / 5 of the area of the standard text box can be considered as text boxes with large shape difference. Since such text boxes are often low-quality noise text boxes, removing such text boxes helps to improve the accuracy of text detection.

[0098] ​​According to an embodiment of the present application, the position information comprises a region contour, and determining the position information of the predicted text region corresponding to the frame line combination based on the center line in the frame line combination comprises: inflating the height of the center line in the frame line combination to a first target height, wherein the first target height is a fixed value, or the first target height is an average of the first height and the second height, or the first target height is equal to the height of the center line before inflation after inflation by a target inflation coefficient, the first height is the height of the left side of the leftmost text box in the frame line combination, and the second height is the height of the right side of the rightmost text box in the frame line combination; and determining the contour line of the inflated center line as the region contour.

[0099] The region contour can also be generated based on the center line, that is, the center line can be inflated as the region contour. The inflated height of the center line in the frame line combination (i.e., the first target height) or the target inflation coefficient of the center line in the frame line combination (the inflation coefficient is also the inflation multiple) can need to be set to any suitable value. This contour generation method is simple and fast in processing speed.

[0100] Exemplarily, in the case that the position information of the predicted text region comprises the corner point position information of the corner points of the predicted text region, a plurality of contour points can be determined from the above-mentioned region contour as the corner points of the corner points of the predicted text region. The position information of the contour points is as the corner point position information of the corner points of the predicted text region.

[0101] According to an embodiment of the present application, before determining the position information of the predicted text region corresponding to the frame line combination based on the center line in the frame line combination, the method further comprises: averaging the height of the left side of the leftmost text box in the frame line combination and the height of the right side of the rightmost text box in the frame line combination to obtain the first target height.

[0102] Exemplarily but not limitatively, the first target height or the target inflation coefficient can be determined according to the average height of the text boxes in the frame line combination. The average height of the text boxes can be calculated by averaging the height of the left side of the leftmost text box in the frame line combination and the height of the right side of the rightmost text box in the frame line combination. Of course, this is only an example, and the average height of the text boxes can also be calculated in other ways. For example, the height of each text box can also be calculated respectively, and the average height of all text boxes can be obtained by averaging the heights of all text boxes. The height of each text box can be obtained by averaging the height of the left side of the text box and the height of the right side of the text box.

[0103] Exemplarily, the first target height or the target inflation coefficient can also be a fixed value, for example, the value of the first target height or the target inflation coefficient can be pre-set.

[0104] According to the embodiment of the present application, the step of determining the position information of the predicted text region corresponding to the frame line combination based on the center line in the frame line combination is performed in the case that the center line in the frame line combination exceeds the target range, which is the range of the region contour determined based on the text frame in the frame line combination.

[0105] In the case that the region contour constructed based on the text frame is shorter than the center line due to the lack of text frame, the region contour can be obtained by inflating the center line. This way can make up for the problem of inaccurate positioning of the region contour caused by missing detection, false detection and the like of the text frame, and help to improve the accuracy of text detection.

[0106] Exemplarily, the x-axis minimum value, the x-axis maximum value, the y-axis minimum value and the y-axis maximum value of the center line in any frame line combination can be determined, and the x-axis minimum value, the x-axis maximum value, the y-axis minimum value and the y-axis maximum value of at least part of the text frames used to obtain the region contour in the frame line combination can be determined. Optionally, the x-axis minimum value, the x-axis maximum value, the y-axis minimum value and the y-axis maximum value of at least part of the text frames here can use the coordinate data of the text frame determined in the correlation degree calculation operation. And it should be noted that the x-axis minimum value, the x-axis maximum value, the y-axis minimum value and the y-axis maximum value of at least part of the text frames refer to the x-axis minimum value, the x-axis maximum value, the y-axis minimum value and the y-axis maximum value of all the corner points in at least part of the text frames, rather than the x-axis minimum value, the x-axis maximum value, the y-axis minimum value and the y-axis maximum value of each corner point of each text frame in at least part of the text frames. For example, if at least part of the text frames includes 10 text frames, the x-axis minimum value of at least part of the text frames can refer to the x value of the lower left corner point of the first text frame, the x-axis maximum value of at least part of the text frames can refer to the x value of the upper right corner point of the tenth text frame, the y-axis minimum value of at least part of the text frames can refer to the y value of the upper left corner point of the first text frame, and the y-axis maximum value of at least part of the text frames can refer to the y value of the lower right corner point of the tenth text frame.

[0107] The x-axis minimum value, the x-axis maximum value, the y-axis minimum value and the y-axis maximum value of the center line are compared with the x-axis minimum value, the x-axis maximum value, the y-axis minimum value and the y-axis maximum value of at least part of the text frames, so as to determine whether the center line exceeds the target range.

[0108] According to an embodiment of the present application, the text detection model is obtained by training in the following manner: obtaining a sample image and text box annotation data, the text box annotation data including position information of an initial text box, the initial text box being a text box enclosing a real text region in the sample image; dividing the initial text box into a target number of sample text boxes; determining a sample center line based on the target number of sample text boxes, the sample center line being a center line of the real text region; and training the text detection model based on the sample image, the target number of sample text boxes, and the sample center line, to obtain the text detection model.

[0109] The initial text box can have any shape, such as a quadrilateral or a curved polygon, etc. The initial text box can be divided into the target number of sample text boxes, which can be understood as reducing the dimension of a complex-shaped text box such as a curved text box into small sample text boxes, which are easier to accurately detect. The sample text box is a smaller text box than the initial text box, and the sample text box can be a quadrilateral text box. The target number can be any suitable number, which can be set as needed, and the present application does not limit this. Exemplarily, the target number can be, for example, 3, 5, 8, 10, etc.

[0110] Based on the target number of sample text boxes, a sample center line indicating the position of the text in the sample image can also be further determined. The sample center line is a center line of the real text region. An exemplary implementation of determining the sample center line will be described below. Of course, the sample center line can also be directly labeled on the sample image by manual labeling.

[0111] Subsequently, the text detection model is trained using the sample image, the target number of sample text boxes, and the sample center line. It can be understood that the initial text detection model and the text detection model finally obtained by training have the same network structure, except that the parameter values (such as the weights and biases of the network layers, etc.) are different.

[0112] The training method of the above text detection model is relatively simple.

[0113] According to an embodiment of the present application, dividing the initial text box into the target number of sample text boxes includes: dividing a first long side of the initial text box into the target number of equal parts to obtain a first set of corner points; dividing a second long side of the initial text box into the target number of equal parts to obtain a second set of corner points; and combining each two adjacent corner points in the first set of corner points with two corresponding adjacent corner points in the second set of corner points to form a sample text box. Two adjacent sample text boxes share the two middle corner points.

[0114] For an initial text box of any shape represented by 2N points, the upper and lower edges of the text box are represented by N points respectively. In this application, the points representing the upper and lower edges can be first adjusted so that each edge is represented by M+1 points with approximately equal or completely equal spacing. M is the target number. Assuming M is 5, the upper and lower edges of the initial text box can be divided into 5 equal parts respectively. Then the equal parts of points on the upper and lower edges are connected in order to form 5 almost equally divided small quadrilaterals, i.e. 5 sample text boxes. Each sample text box is composed of 4 corner points, and the text detection model needs to regress and predict these 4 corner points.

[0115] Figure 7a A diagram showing how to obtain corner points by dividing the edges of an initial text box according to an embodiment of the present application is shown. Figure 7b A diagram showing how to form sample text boxes by combining corner points is shown. Figure 7a An initial text box 710 is shown, and 6 points obtained by dividing the upper and lower edges of the initial text box 710 are shown. Figure 7b A sample text box obtained by connecting the corresponding corner points of Figure 7a is shown.

[0116] According to an embodiment of the present application, based on the sample text boxes of the target number, determining the sample center line comprises: for any corner point on the first long edge in the sample text boxes of the target number and the corresponding corner point on the opposite second long edge in the sample text boxes of the target number, determining the center point of the two corner points; connecting all the center points to obtain an initial center line; determining the initial center line or a new center line as the sample center line, the new center line being obtained based on the initial center line.

[0117] For example, along with the above embodiment of obtaining a group of corner points by dividing and further obtaining sample text boxes, these text boxes have a total of 2(M+1) corner points (part of the corner points are shared by adjacent text boxes), the corresponding corner points on the upper and lower edges can be connected, the midpoint is taken as a node of the center line, then M+1 center line nodes are obtained, and the center line can be formed by connecting the nodes in order. Figure 8 A diagram showing how to obtain an initial center line by the center points of the corner points according to an embodiment of the present application is shown. Figure 8 The initial center line is shown by a dashed line in the diagram.

[0118] According to an embodiment of the present application, the new center line is obtained by: under the condition that the midpoint of the initial center line is fixed, respectively shrinking the parts of the initial center line on both sides of the midpoint by a target ratio; and / or expanding the height of the initial center line to a second target height.

[0119] The target ratio and the second target height can be set to any suitable value as needed, and the present application does not limit this.

[0120] The left and right halves of the initial center line can be respectively contracted by a certain ratio, so that the two end points thereof are close to the center. Figure 9 A schematic diagram of the initial center line is shown according to an embodiment of the present application. Figure 8 A schematic diagram of the initial center line after contraction is shown. Proper contraction of the center line can prevent the two text regions from being too close to each other so that the two center lines are pasted together, thereby affecting the training effect of the model.

[0121] In addition, the initial center line can also be inflated to a line segment with a certain height. If the initial center line is not inflated, the height is only one pixel, and if there is a missed detection, the center line may be broken. However, if the initial center line is inflated, such problems can be avoided, thereby improving the fault tolerance of the model prediction. The second target height can be any suitable height, such as 5 pixels, 10 pixels, etc.

[0122] The embodiments of the initial center line contraction and the initial center line height inflation can implement only one of them, or both. For example, the initial center line can be first contracted and then further inflated to a line segment with a certain height, thereby obtaining a new center line.

[0123] According to an embodiment of the present application, based on a sample image, a target number of sample text boxes and a sample center line, an initial detection model is trained to obtain a text detection model, which includes: inputting the sample image into a feature extraction module in the text detection model to obtain extracted features; inputting the extracted features into a classification module, a regression module and a segmentation module in the text detection model respectively to obtain predicted class information of each anchor point output by the classification module, predicted regression information of each anchor point output by the regression module and predicted segmentation information of each anchor point output by the segmentation module, the predicted class information is used to indicate whether the anchor point belongs to a predicted text box or background, the predicted regression information is used to indicate the distance from the anchor point to the four corner points of the corresponding predicted text box, and the predicted segmentation information is used to indicate whether the anchor point belongs to a center line, each anchor point corresponds to at least part of the pixels on the sample image; calculating a first loss based on the predicted class information and the target number of sample text boxes, calculating a second loss based on the predicted regression information and the target number of sample text boxes, and calculating a third loss based on the predicted segmentation information and the sample center line; calculating a total loss based on the first loss, the second loss and the third loss; and training the text detection model based on the total loss.

[0124] Figure 10 A schematic diagram of the training process of the text detection model is shown according to an embodiment of the present application. As shown in FIG. 6, the training process of the text detection model includes the following steps. Figure 10As shown, the text detection model can include a feature extraction module and two branches, one being a text box regression branch and the other being a centerline segmentation branch. The text box regression branch can include a classification module (which can be referred to as a classification head) and a regression module (which can be referred to as a regression head). The centerline segmentation branch can include a segmentation module (which can be referred to as a segmentation head).

[0125] Exemplarily, a sample image can be input into the text detection model to obtain predicted class information, position regression information and centerline segmentation information predicted by the text detection model. According to the sample text box of the target number, the real class information and the real regression information of each anchor point can be determined, and then the first loss and the second loss can be calculated. In addition, according to the sample centerline, the real segmentation information of each anchor point can also be determined, and the corresponding third loss can be calculated. Exemplarily but not limitatively, the first loss can be a generalized focal loss, the second loss can be an average absolute error loss (L1 loss), and the third loss can be a dice loss. Compared with the conventional focal loss, the generalized focal loss can improve the inference accuracy of the model obtained by training.

[0126] In this application, the regression information is used to indicate the distance from the anchor point to the four corner points of the corresponding text box, so that the text detection model can be applied to the prediction of any quadrilateral text box, and is not limited to a rectangle.

[0127] The first loss can be calculated based on the predicted class information and the real class information, the second loss can be calculated based on the predicted regression information and the real regression information, and the third loss can be calculated based on the predicted segmentation information and the real segmentation information. The total loss can be calculated based on the above losses. For example, the first loss, the second loss and the third loss can be directly added to obtain the total loss. Subsequently, based on the total loss, the parameters of the text detection model are optimized using the back propagation and gradient descent algorithm, and after multiple iterations, a text detection model meeting the requirements is finally obtained.

[0128] In the above embodiment, the centerline segmentation branch is trained together with the text box regression branch, which helps the pre-feature extraction module to learn the visual features of the text, and further achieves the purpose of implicit collaborative optimization with the text box regression branch.

[0129] It should be noted that in the description herein, the terms "left", "right", "up" and "down" are not limited to specific orientations, which can be set as needed. For example, in the case of establishing the above image coordinate system, a point with a small x-coordinate can be considered to be located on the left side of a point with a large x-coordinate, and a point with a small y-coordinate can be considered to be located above a point with a large y-coordinate.

[0130] According to another aspect of the present application, a text detection device is provided. Figure 11A schematic block diagram of the text detection apparatus 1100 according to an embodiment of the present application is shown.

[0131] As shown in Figure 11 the text detection apparatus 1100 according to an embodiment of the present application includes an obtaining module 1110, an inputting module 1120, a clustering module 1130 and a determining module 1140. Each of the modules can perform the respective steps of the text detection method described above. Figure 2 Hereinafter, only the main functions of the components of the text detection apparatus 1100 will be described, and the details already described above will be omitted.

[0132] The obtaining module 1110 is configured to obtain an image to be processed. The obtaining module 1110 can be implemented by the processor 102 in the electronic device shown in Figure 1 running program instructions stored in the storage 104.

[0133] The inputting module 1120 is configured to input the image to be processed into a text detection model to obtain at least one text box and at least one center line in the image to be processed, the text box being a box enclosing at least a part of a predicted text region, and the center line being a center line of the predicted text region. The inputting module 1120 can be implemented by the processor 102 in the electronic device shown in Figure 1 running program instructions stored in the storage 104.

[0134] The clustering module 1130 is configured to cluster the at least one text box and the at least one center line to obtain at least one box-line combination, each box-line combination including a single center line and one or more text boxes. The clustering module 1130 can be implemented by the processor 102 in the electronic device shown in Figure 1 running program instructions stored in the storage 104.

[0135] The determining module 1140 is configured to, for any of the at least one box-line combination, determine position information of a predicted text region corresponding to the box-line combination based on the center line and / or the text box in the box-line combination. The determining module 1140 can be implemented by the processor 102 in the electronic device shown in Figure 1 running program instructions stored in the storage 104.

[0136] Figure 12 A schematic block diagram of the electronic device 1200 according to an embodiment of the present application is shown. The electronic device 1200 includes a memory 1210 and a processor 1220.

[0137] The memory 1210 stores computer program instructions for implementing the respective steps in the text detection method according to an embodiment of the present application.

[0138] The processor 1220 is configured to execute computer program instructions stored in the memory 1210, so as to perform corresponding steps of the text detection method according to the embodiments of the present application.

[0139] In one embodiment, the computer program instructions, when executed by the processor 1220, are configured to perform the following steps: obtaining an image to be processed; inputting the image to be processed into a text detection model to obtain at least one text box and at least one center line in the image to be processed, the text box being a box surrounding at least part of a predicted text region, and the center line being a center line of the predicted text region; clustering the at least one text box and the at least one center line to obtain at least one combination of boxes and lines, each combination of boxes and lines including a single center line and one or more text boxes; and determining, for any combination of boxes and lines in the at least one combination of boxes and lines, position information of a predicted text region corresponding to the combination of boxes and lines based on the center line and / or the text box in the combination of boxes and lines.

[0140] For example, the electronic device 1200 can further include an image acquisition device 1230. The image acquisition device 1230 is configured to acquire the image to be processed. The image acquisition device 1230 is optional, and the electronic device 1200 can also not include the image acquisition device 1230. At this time, the processor 1220 can acquire the image to be processed in other manners, for example, from an external device or from the memory 1210.

[0141] In addition, according to the embodiments of the present application, a storage medium is also provided, and program instructions are stored on the storage medium. The program instructions, when executed by a computer or a processor, are configured to perform corresponding steps of the text detection method according to the embodiments of the present application, and are configured to implement corresponding modules in the text detection apparatus according to the embodiments of the present application. The storage medium can include, for example, a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, or any combination of the above storage media.

[0142] In one embodiment, the program instructions, when executed by the computer or the processor, can cause the computer or the processor to implement various functional modules of the text detection apparatus according to the embodiments of the present application, and / or can perform the text detection method according to the embodiments of the present application.

[0143] In one embodiment, the program instructions, when executed, cause the following steps to be performed: obtaining an image to be processed; inputting the image to be processed into a character detection model to obtain at least one character box and at least one center line in the image to be processed, the character box being a box enclosing at least part of a predicted character region, and the center line being a center line of the predicted character region; clustering the at least one character box and the at least one center line to obtain at least one combination of boxes and lines, each combination of boxes and lines including a single center line and one or more character boxes; and for any combination of boxes and lines in the at least one combination of boxes and lines, determining position information of a predicted character region corresponding to the combination of boxes and lines based on the center line and / or the character box in the combination of boxes and lines.

[0144] In addition, according to an embodiment of the present application, a computer program product is also provided, which includes a computer program. The computer program, when executed, is used to perform the character detection method 200.

[0145] The modules in the electronic device according to the embodiments of the present application can be realized by running computer program instructions stored in the memory by the processor of the electronic device for implementing character detection or character detection according to the embodiments of the present application, or can be realized when the computer instructions stored in the computer readable storage medium of the computer program product according to the embodiments of the present application are run by the computer.

[0146] In addition, according to an embodiment of the present application, a computer program is also provided. The computer program, when executed, is used to perform the character detection method 200.

[0147] Although the example embodiments have been described herein with reference to the accompanying drawings, it is to be understood that the example embodiments are merely exemplary and are not intended to limit the scope of the present application. Those of ordinary skill in the art can make various changes and modifications without departing from the scope and spirit of the present application. All such changes and modifications are intended to be included within the scope of the present application as claimed in the appended claims.

[0148] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in connection with the embodiments disclosed herein can be realized in electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those of ordinary skill in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0149] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is merely a logical function division. In actual implementation, another division manner can be used, for example, a plurality of units or components can be combined or integrated into another device, or some features can be omitted or not executed.

[0150] In the specification provided herein, a large number of specific details are described. However, it can be understood that the embodiments of the present application can be practiced without these specific details. In some examples, well-known methods, structures and techniques are not described in detail in order not to obscure the understanding of the specification.

[0151] Similarly, it should be understood that, in order to simplify the present application and help understand one or more of the various application aspects, in the description of the exemplary embodiments of the present application, various features of the present application are sometimes grouped together in a single embodiment, figure, or description thereof. However, the method of the present application should not be interpreted as reflecting the intention that the claimed present application requires more features than the features explicitly recorded in each claim. Rather, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with fewer features than all the features of a certain disclosed single embodiment. Therefore, the claims following the specific embodiments are hereby expressly incorporated into the specific embodiments, wherein each claim itself is a separate embodiment of the present application.

[0152] Those skilled in the art can understand that, except for the mutual exclusion between features, all features disclosed in the specification (including the accompanying claims, abstract and drawings) and all processes or units of any method or device disclosed in this way can be combined in any combination. Unless explicitly stated otherwise, each feature disclosed in the specification (including the accompanying claims, abstract and drawings) can be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0153] In addition, those skilled in the art can understand that although some embodiments herein include certain features rather than other features included in other embodiments, the combination of features of different embodiments means that it is within the scope of the present application and forms different embodiments. For example, in the claims, any one of the claimed embodiments can be used in any combination.

[0154] The various component embodiments of the present application can be implemented in hardware, or as software modules running in one or more processors, or in combinations thereof. As will be appreciated by one skilled in the art, a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functionality of some of the modules in the text detection apparatus according to the embodiments of the present application. The present application can also be implemented as a program for executing the part or all of the methods described herein on a computer (for example, a computer program and a computer program product). Such a program implementing the present application can be stored on a computer-readable medium, or can have one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0155] It should be noted that the above-mentioned embodiments illustrate rather than limit the application, and that one skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word 'comprising' does not exclude the presence of elements or steps other than those listed in a claim. The word 'a' or 'an' preceding an element does not exclude the presence of a plurality of such elements. The application can be implemented by means of both hardware and software, and any combination thereof. In a unit claim, several devices can be listed with a conjunction like 'or', and a single element can be substituted therefor. The use of the word 'at least' followed by a list of one or more items does not exclude additional such items. The use of the terms at the beginning of a claim is understood not to limit the scope of the claim, but rather - in conjunction with the wording of the claim - to emphasize that the features recited in the claim are required libe the claim is to be new, and are therefore essential to the claim as a whole.

[0156] The above merely illustrates the specific embodiments of the present application, or the description of the specific embodiments, and the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, and all such changes or replacements should be covered within the protection scope of the present application. The protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A text detection method, comprising: Obtain the image to be processed; The image to be processed is input into the text detection model to obtain at least one text box and at least one center line in the image to be processed. The text box is a box that surrounds at least a part of the predicted text region, and the center line is the center line of the predicted text region. Based on the correlation between any text box in the at least one text box and any center line in the at least one center line, the at least one text box and the at least one center line are clustered to obtain at least one frame line combination. Each frame line combination includes a single center line and one or more text boxes. Each text box in each frame line combination is distributed along the extension direction of the corresponding center line, and the center line crosses at least a portion of the area of ​​each text box in the frame line combination along the extension direction. The correlation is the intersection-union ratio between the text box and the center line. For any of the at least one frame combination, the position information of the predicted text region corresponding to the frame combination is determined based on the center line and / or text box in the frame combination, wherein the shape of the predicted text region includes a curved shape.

2. The method as described in claim 1, wherein, The step of clustering the at least one text box and the at least one center line based on the correlation between any text box and any center line of the at least one center line to obtain at least one box line combination includes: Calculate the correlation degree between any text box in the at least one text box and any center line in the at least one center line; For any text box, the text box and the relevant center line are clustered into the same group to form a frame line group corresponding to the relevant center line, wherein the relevant center line is the center line with the highest correlation with the text box and the correlation exceeds a first correlation threshold.

3. The method as described in claim 2, wherein, The step of clustering the at least one text box and the at least one center line based on the correlation between any text box and any center line of the at least one center line to obtain at least one box line combination further includes: If there are scattered text boxes, calculate the correlation and relative position relationship between the scattered text boxes and each text box in any specific frame combination. The scattered text boxes are text boxes that do not belong to any frame combination. If the correlation between the scattered text boxes and any specific text box in the specific frame combination is greater than the second correlation threshold and the relative positional relationship between the scattered text boxes and the specific text boxes meets the target requirements, then the scattered text boxes are assigned to the specific frame combination.

4. The method according to any one of claims 1-3, wherein, The location information includes the region outline. The location information for determining the predicted text region corresponding to the frame combination based on the text boxes within the frame combination includes: For the first part of the text boxes in the frame combination, the first part of the text boxes are arranged according to the target extension direction to obtain an arranged text box set, wherein the target extension direction is the extension direction of the line connecting the feature points contained in each of the second part of the text boxes, the first part of the text boxes is at least a part of the text boxes in the frame combination, and the second part of the text boxes is at least a part of the text boxes in the first part of the text boxes. Connect at least some of the corner points of the arranged text boxes together in a clockwise or counterclockwise direction to obtain the region outline.

5. The method of claim 4, wherein, Connecting at least some corner points of the set of arranged text boxes together in a clockwise or counterclockwise direction to obtain the region outline includes: Repeat the following corner connection operation until at least some of the corners in the set of arranged text boxes are connected: Connect at least some corner points of the first two text boxes in the current text box set in a clockwise or counterclockwise direction to obtain a new text box. Then, the new text box and the remaining text boxes in the current text box set are determined as a new text box set. When the first corner point connection operation is performed, the current text box set is the arranged text box set. The new text box set determined by any corner point connection operation is the current text box set corresponding to the next corner point connection operation.

6. The method of claim 5, wherein, The step of connecting at least some corner points of the first two text boxes in the current text box set together in a clockwise or counterclockwise direction to obtain a new text box includes: If the first two text boxes in the current text box set do not intersect, connect all the corner points of the first two text boxes together in a clockwise or counterclockwise direction to obtain the new text box; If the first two text boxes in the current text box set have an intersection, discard the upper right corner of the text box on the left and the lower left corner of the text box on the right, and connect the remaining corners together in a clockwise or counterclockwise direction to obtain the new text box.

7. The method of claim 4, wherein, Before arranging the first portion of text boxes in the frame combination according to the target extension direction, the step of determining the position information of the predicted text region corresponding to the frame combination based on the text boxes in the frame combination further includes: The first part of the text boxes is obtained by filtering out text boxes whose area is greater than a first target multiple of the area of ​​the standard text box and / or whose area is less than a second target multiple of the area of ​​the standard text box in the frame combination, wherein the standard text box is the text box whose area is the median in the frame combination.

8. The method as described in any one of claims 1-3, wherein, The location information includes the region outline. The location information for determining the predicted text region corresponding to the frame combination based on the center line in the frame combination includes: The height of the center line in the frame combination is expanded to a first target height, wherein the first target height is a fixed value, or the first target height is the average of the first height and the second height, or the first target height is equal to the height reached by the center line after expansion according to the target expansion coefficient, the first height is the height of the left side of the leftmost text box in the frame combination, and the second height is the height of the right side of the rightmost text box in the frame combination. The outline of the center line after expansion is defined as the outline of the region.

9. The method of claim 8, wherein, The step of determining the position information of the predicted text region corresponding to the frame combination based on the center line in the frame combination is performed when the center line in the frame combination exceeds the target range, wherein the target range is the range of the region outline determined based on the text box in the frame combination.

10. The method according to any one of claims 1-3, wherein, The text detection model was trained in the following way: Acquire sample images and text box annotation data, wherein the text box annotation data includes the position information of initial text boxes, and the initial text boxes are text boxes that surround the real text region in the sample image; Divide the initial text box into a target number of sample text boxes; Based on the target number of sample text boxes, a sample center line is determined, and the sample center line is the center line of the real text region. The text detection model is trained based on the sample image, the target number of sample text boxes, and the sample center line.

11. The method of claim 10, wherein, The determination of the sample centerline based on the target number of sample text boxes includes: For any corner point located on the first long side of the sample text box of the target number, and the corresponding corner point located on the opposite second long side of the sample text box of the target number, determine the center point of these two corner points; Connect all the center points to obtain the initial centerline; The initial centerline or the new centerline is determined as the centerline of the sample; The new centerline is obtained in the following way: With the midpoint of the initial centerline fixed, the portions of the initial centerline located on either side of the midpoint are contracted according to the target ratio; and / or, The height of the initial centerline is expanded to the second target height.

12. The method of claim 10, wherein, The step of dividing the initial text box into a target number of sample text boxes includes: Divide the first long side of the initial text box equally according to the target number to obtain the first group of corner points; Divide the second long side of the initial text box equally according to the target number to obtain the second set of corner points; Combine every two adjacent corner points in the first group with two corresponding adjacent corner points in the second group to form a sample text box.

13. An electronic device comprising a processor and a memory, wherein, The memory stores computer program instructions, which, when executed by the processor, are used to perform the text detection method as described in any one of claims 1 to 12.

14. A storage medium on which program instructions are stored, wherein, The program instructions are used to execute the text detection method as described in any one of claims 1 to 12 when the program is run.

15. A computer program product, the computer program product comprising a computer program, wherein, The computer program, when running, is used to perform the text detection method as described in any one of claims 1 to 12.

Images