Text detection method, device, equipment and storage medium

Abstract

The present invention discloses a text detection method, device, equipment and storage medium. The method includes: obtaining an image to be detected; inputting the image to be detected into a pre-constructed indentation offset text detection model to determine a target semantic segmentation feature map and a target offset feature map; determining a target expansion distance and a rectangular frame to be expanded according to the target semantic segmentation feature map and the target offset feature map; expanding the rectangular frame to be expanded by the target expansion distance to determine a target text detection frame; wherein the pre-constructed indentation offset text detection model includes a semantic segmentation sub-model and an offset regression sub-model. The technical solution of the embodiment of the present invention solves the problem that the existing text detection model obtained by training based on indentation text instances does not consider the offset of the text indentation process, resulting in slow detection speed and poor effect when detecting dense text areas, reduces the amount of calculation for determining the target text detection frame, and improves the efficiency of text detection.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a text detection method, apparatus, device and storage medium. Background Art

[0002] Text detection has a wide range of applications and is a prerequisite for many computer vision tasks, such as image search, text recognition, identity authentication, and visual navigation. The primary purpose of text detection is to locate text lines within an image. However, in natural scenes, text lines vary in size, font, color, shape, orientation, and background, often leading to overlap during the recognition process. With the rise of deep learning, research on text detection has become a hot topic, and a large number of text detection methods have emerged.

[0003] In natural text detection scenarios, semantic segmentation-based text detection algorithms typically use indented text instances to generate masks that separate text instances from each other as real samples. Currently, algorithms that use this idea for text detection include EAST and PSENet. They use a polygon indentation algorithm to process labeled real sample boxes into indented sample boxes, thereby separating densely packed text instances and making it easier for the neural network to learn the separated features.

[0004] However, after using the above algorithms to learn the indented samples, the EAST algorithm directly regresses the set distance of quadrilaterals on the mask position of the indented area to locate the text instance; while the PSENet algorithm expands from the inside to the outside on multiple masks with different indentation offsets to obtain an accurate mask, and then calculates the bounding box of the mask to locate the text instance. Both algorithms do not consider the offset that may be generated during the generation of the indented sample box, resulting in slow detection speed and poor detection effect when applied to dense text area detection, affecting the efficiency of text detection. Summary of the Invention

[0005] The present invention provides a text detection method, apparatus, device and storage medium for learning and detecting indented and offset scene text, thereby improving the efficiency and accuracy of text detection and balancing the precision and speed required for text detection.

[0006] In a first aspect, an embodiment of the present invention provides a text detection method, comprising:

[0007] Obtain the image to be detected;

[0008] Input the image to be detected into the pre-built indented offset text detection model to determine the target semantic segmentation feature map and the target offset feature map;

[0009] Determine the target expansion distance and the rectangular frame to be expanded according to the target semantic segmentation feature map and the target offset feature map;

[0010] Expand the rectangular frame to be expanded by the target expansion distance to determine the target text detection frame;

[0011] Among them, the pre-built indented offset text detection model includes a semantic segmentation sub-model and an offset regression sub-model.

[0012] Furthermore, the training steps of the indented text detection model include:

[0013] Extracting basic features from an image sample set in an indented offset text training sample set to determine a basic feature sample set; wherein the indented offset text training sample set includes an image sample set and a calibration sample set corresponding to the image sample set, and the calibration sample set includes an indented segmentation label and an offset label corresponding to each image sample;

[0014] Input the basic feature sample set into the initial semantic segmentation sub-model and extract the semantic segmentation intermediate results;

[0015] Input the basic feature sample set into the initial offset regression sub-model to extract the offset intermediate results;

[0016] Determine the corresponding first loss function according to the semantic segmentation intermediate result and the corresponding indentation segmentation label;

[0017] Determine the corresponding second loss function according to the offset intermediate result and the corresponding offset label;

[0018] The total loss function is determined according to the first loss function and the second loss function, and the initial semantic segmentation sub-model and the initial offset regression sub-model are trained based on the total loss function until a preset convergence condition is met to obtain an indented offset text detection model.

[0019] Furthermore, basic features are extracted from the image sample set in the indented offset text training sample set to determine a basic feature sample set, including:

[0020] Inputting the image sample set in the indented offset text training sample set into the feature extraction backbone network to determine a first feature atlas; wherein the first feature atlas includes a plurality of feature maps of different resolutions extracted from the image sample set;

[0021] Performing multi-scale feature extraction on the first feature atlas to determine a second feature atlas;

[0022] Multi-feature fusion is performed on the second feature atlas, and the set of fused feature maps is determined as the basic feature sample set.

[0023] Furthermore, the step of determining the indented segmentation label includes:

[0024] For each image sample, construct a first two-dimensional matrix corresponding to the image sample according to the size of the image sample, and determine the shortest side length of the image sample;

[0025] If the shortest side length is less than or equal to the preset minimum border length, each pixel corresponding to the position of the annotated text in the first two-dimensional matrix is ​​set to a first preset value;

[0026] If the shortest side length is greater than the preset minimum border length, a first indentation distance is determined according to the size of the image sample, the position of the annotation text is updated according to the first indentation distance, and each pixel corresponding to the updated position of the annotation text is set to a first preset value.

[0027] Furthermore, the step of determining the offset tag includes:

[0028] For each image sample, construct a second two-dimensional matrix corresponding to the image sample according to the size of the image sample, and determine the shortest side length of the image sample;

[0029] If the shortest side length is less than or equal to the preset minimum border length, each pixel corresponding to the position of the annotated text in the second two-dimensional matrix is ​​set to a first preset value;

[0030] If the shortest side length is greater than the preset minimum border length, a second indentation distance is determined according to the size of the image sample, and the second two-dimensional matrix is ​​updated and assigned according to the second indentation distance and the shortest side length.

[0031] Furthermore, the second two-dimensional matrix is ​​updated and assigned a value according to the second inward contraction distance and the shortest side length, including:

[0032] If the second indentation distance is less than the shortest side length, determine an offset strength value based on the second indentation distance and a preset reference value, update the position of the annotated text based on the second indentation distance, and set each pixel corresponding to the updated position of the annotated text as the offset strength value;

[0033] If the second inward distance is greater than or equal to the shortest side length, each pixel corresponding to the position of the annotated text in the second two-dimensional matrix is ​​set to a first preset value.

[0034] Furthermore, according to the semantic segmentation intermediate result and the corresponding indentation segmentation label, a corresponding first loss function is determined, including:

[0035] Compare the corresponding values ​​of each pixel in the semantic segmentation intermediate result with the corresponding values ​​of each pixel in the corresponding indented segmentation label;

[0036] A first loss function is determined according to the comparison result.

[0037] Furthermore, according to the intermediate offset result and the corresponding offset label, a corresponding second loss function is determined, including:

[0038] Compare the corresponding value of each pixel in the offset intermediate result with the corresponding value of each pixel in the corresponding offset label;

[0039] A second loss function is determined based on the comparison result.

[0040] Furthermore, a total loss function is determined according to the first loss function and the second loss function, and the initial semantic segmentation sub-model and the initial offset regression sub-model are trained based on the total loss function until a preset convergence condition is met to obtain an indented offset text detection model, including:

[0041] The first loss function and the second loss function are weighted and summed according to a preset weight value to determine a total loss function;

[0042] Based on the total loss function, the weight parameters in the initial semantic segmentation sub-model and the initial offset regression sub-model are adjusted until the preset convergence conditions are met to obtain the indented offset text detection model.

[0043] Furthermore, determining the target expansion distance and the rectangular frame to be expanded according to the target semantic segmentation feature map and the target offset feature map includes:

[0044] Threshold the target semantic segmentation feature map and determine the label map based on the connected component labeling algorithm;

[0045] Intersect the label map with the target offset feature map and update the target offset feature map;

[0046] After traversing the different marking values ​​in the marking graph after intersection, the circumscribed rectangular box of the marking object corresponding to the same marking value is determined as the rectangular box to be expanded;

[0047] Determine the average value of the pixel values ​​of each marked object in the updated target offset feature map as the predicted offset of the marked object;

[0048] The target expansion distance is determined based on the predicted offset and the preset benchmark value.

[0049] Furthermore, the text detection method further includes:

[0050] Determine the confidence level of the rectangular box to be expanded corresponding to the marked object according to the average value of the pixel values ​​of the marked object in the target semantic segmentation feature map;

[0051] If the confidence level is less than the preset confidence threshold, the rectangular box to be expanded is deleted.

[0052] Further, the rectangular frame to be expanded is expanded by the target expansion distance to determine the target text detection frame, including:

[0053] Sort the coordinates of the corner points of the rectangular frame to be expanded clockwise, and determine the coordinates of the center point of the rectangular frame to be expanded;

[0054] Shift the adjacent sides of the rectangular frame to be expanded outward by the target expansion distance relative to the center coordinates;

[0055] Extend both ends of each side of the rectangular frame to be expanded outward by the target expansion distance, and determine the new endpoint coordinates corresponding to each side;

[0056] Determine the intersection point after the expansion of each side of the rectangular frame to be expanded as the new intersection point coordinates;

[0057] If the coordinates of each new intersection point are consistent with the corresponding new endpoint coordinates, the expanded rectangular box to be expanded is determined as the target text detection box; otherwise, the original coordinates corresponding to the rectangular box to be expanded are returned.

[0058] In a second aspect, an embodiment of the present invention further provides a text detection device, comprising:

[0059] An image acquisition module, used to acquire an image to be detected;

[0060] A feature map determination module is used to input the image to be detected into a pre-built indentation offset text detection model to determine the target semantic segmentation feature map and the target offset feature map;

[0061] An expansion rectangle determination module is used to determine the target expansion distance and the rectangular frame to be expanded based on the target semantic segmentation feature map and the target offset feature map;

[0062] A detection frame determination module is used to expand the rectangular frame to be expanded by the target expansion distance to determine the target text detection frame;

[0063] Among them, the pre-built indented offset text detection model includes a semantic segmentation sub-model and an offset regression sub-model.

[0064] In a third aspect, an embodiment of the present invention further provides a text detection device, the text detection device comprising:

[0065] at least one processor; and

[0066] a memory communicatively connected to at least one processor; wherein,

[0067] The memory stores a computer program that can be executed by at least one processor. The computer program is executed by the at least one processor, so that the at least one processor can implement the text detection method of any embodiment of the present invention.

[0068] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the text detection method of any embodiment of the present invention when executed.

[0069] The embodiment of the present invention provides a text detection method, device, equipment and storage medium, which obtains an image to be detected; inputs the image to be detected into a pre-built indented offset text detection model to determine a target semantic segmentation feature map and a target offset feature map; determines a target expansion distance and a rectangular box to be expanded according to the target semantic segmentation feature map and the target offset feature map; expands the rectangular box to be expanded by the target expansion distance to determine a target text detection box; wherein the pre-built indented offset text detection model includes a semantic segmentation sub-model and an offset regression sub-model. By adopting the above technical solution, by inputting the image to be detected into the pre-built indented offset text detection model, the semantic segmentation sub-model and the offset regression sub-model in the indented offset text detection model are used to process the corresponding target semantic segmentation feature map and the target offset feature map, so as to simultaneously determine the rectangular box to be expanded of the indented text and the target expansion distance to be expanded according to the semantic segmentation result and the offset calculation result, and then expands the rectangular box to be expanded by the target expansion distance to obtain the final target text detection box. The method solves the problem that the existing text detection model trained based on indented text instances does not consider the offset of the text indentation process, resulting in slow detection speed and poor effect when detecting dense text areas. At the same time, semantic segmentation and offset regression are used to process the same image to be detected, fully considering the influence of the degree of adhesion and the offset generated by text of different sizes during the indentation process, so that the determined target text detection frame is more consistent with the text boundary of the position to be determined, thereby improving the accuracy of text detection. At the same time, after determining the rectangular frame to be expanded, it is only necessary to perform one expansion based on the determined target expansion distance, which reduces the calculation amount of determining the target text detection frame and improves the efficiency of text detection.

[0070] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0072] Figure 1is a flowchart of a text detection method in embodiment 1 of the present invention;

[0073] Figure 2 is a flowchart of a text detection method in embodiment 2 of the present invention;

[0074] Figure 3 This is an example flow chart of the training steps of an indented text detection model in the second embodiment of the present invention;

[0075] Figure 4 This is an example flow chart of extracting basic features from an image sample set in an indented offset text training sample set to determine a basic feature sample set in Embodiment 2 of the present invention;

[0076] Figure 5 This is an example flow chart of the steps for determining an indented segmentation label in the second embodiment of the present invention;

[0077] Figure 6 This is an example flow chart of the steps for determining an offset tag in the second embodiment of the present invention;

[0078] Figure 7 This is a structural diagram of a text detection device in Embodiment 3 of the present invention;

[0079] Figure 8 It is a structural diagram of a text detection device in embodiment 4 of the present invention. DETAILED DESCRIPTION

[0080] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0081] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0082] Example 1

[0083] Figure 1 A flowchart of a text detection method provided in Example 1 of the present invention is applicable to an indented offset text detection model obtained by training an indented offset text image to determine the indented text border and offset in the image to be detected, and then complete the expansion based on the determined indented text border and offset to determine the target text detection frame corresponding to the text position in the image to be detected. The method can be executed by a text detection device, which can be implemented by software and / or hardware, and can be configured on a computer device, which can be a notebook, desktop computer, smart tablet, etc.

[0084] like Figure 1 As shown, the text detection method provided in the first embodiment includes the following steps:

[0085] S101: Acquire an image to be detected.

[0086] In this embodiment, the image to be detected can be specifically understood as an image containing text information that needs to be recognized. For example, the image to be detected can be a static image captured by a surveillance camera, or an image frame captured in a video, which contains text information such as slogans and symbols that need to be recognized. The embodiment of the present invention does not limit this.

[0087] Specifically, in actual applications, when it is necessary to identify text information in the acquired image, or when text information needs to be extracted from a video or image, it is necessary to first determine the position of the text information in the image or video frame from which the text information extraction is required, and detect whether there is text information that needs to be extracted in the image. At this time, the image that needs to perform text information position detection is determined as the image to be detected.

[0088] S102: Input the image to be detected into a pre-built indented offset text detection model to determine a target semantic segmentation feature map and a target offset feature map.

[0089] Among them, the pre-built indented offset text detection model includes a semantic segmentation sub-model and an offset regression sub-model.

[0090] In this embodiment, the indented offset text detection model can be specifically understood as a neural network model composed of two sub-models, which is obtained by training text training samples that are indented after annotation and offset due to the indentation, and is used to extract features from the input image to be detected, and perform semantic segmentation and offset regression on the feature map after feature extraction. The semantic segmentation sub-model can be specifically understood as the neural network model used in the indented offset text detection model to perform semantic segmentation on the image to be detected after feature extraction. The offset regression sub-model can be specifically understood as the neural network model used in the indented offset text detection model to determine the offset of the indented text detected in the image to be detected after indentation relative to the text boundary. The target semantic segmentation feature map can be specifically understood as the feature image corresponding to the text position after indentation relative to the expected detected text boundary position, obtained after the image to be detected is input into the semantic segmentation sub-model for semantic segmentation after feature extraction. The target offset feature map can be specifically understood as the feature image used after the image to be detected is input into the offset regression sub-model for determining the relative offset of the indented text position.

[0091] Specifically, the image to be detected is input into a pre-built indented offset text detection model. After feature extraction of the image to be detected, it is respectively input into the semantic segmentation sub-model and the offset regression sub-model in the indented offset text detection model. The target semantic segmentation feature map and the target offset feature map are determined according to the output results of each sub-model.

[0092] S103 : Determine the target expansion distance and the rectangular frame to be expanded according to the target semantic segmentation feature map and the target offset feature map.

[0093] In this embodiment, the to-be-expanded rectangular box can be specifically understood as a rectangular box representing the target text's indented boundary, determined based on the target semantic segmentation feature map and the target offset feature map. The target expansion distance can be specifically understood as the distance the to-be-expanded rectangular box corresponding to the indented boundary of each target text must extend to reach the target text boundary, based on the different offsets of the target text.

[0094] Specifically, the processed target semantic segmentation feature map is marked and divided according to the connectivity of the target semantic segmentation feature map, and then a mark map with marks for characterizing the connectivity relationship is obtained. According to the mark map, the marked objects corresponding to each pixel point obtained after the intersection of the target semantic segmentation feature map and the target offset feature map are determined, and then the offset values ​​corresponding to the pixels corresponding to each marked object are determined. According to the connectivity, the rectangular box of the indented boundary corresponding to each marked object is determined and used as the rectangular box to be expanded. Then, the offset corresponding to each pixel in each marked object is determined according to the target offset feature map, so as to clarify the target expansion distance that needs to be expanded to the text boundary corresponding to each rectangular box to be expanded.

[0095] S104 , expanding the rectangular frame to be expanded by the target expansion distance to determine the target text detection frame.

[0096] Specifically, the rectangular box to be expanded is expanded outward by the target expansion distance through a predetermined expansion algorithm to achieve the expansion of the boundary of the target text corresponding to the rectangular box to be expanded, and the rectangular box finally obtained by expanding to the boundary of the target text is determined as the target text detection box.

[0097] The technical solution of this embodiment is as follows: obtaining an image to be detected; inputting the image to be detected into a pre-built indented offset text detection model to determine a target semantic segmentation feature map and a target offset feature map; determining a target expansion distance and a rectangular box to be expanded according to the target semantic segmentation feature map and the target offset feature map; expanding the rectangular box to be expanded by the target expansion distance to determine a target text detection frame; wherein the pre-built indented offset text detection model includes a semantic segmentation sub-model and an offset regression sub-model. By adopting the above technical solution, by inputting the image to be detected into a pre-built indented offset text detection model, the semantic segmentation sub-model and the offset regression sub-model in the indented offset text detection model are used to respectively process the corresponding target semantic segmentation feature map and target offset feature map, so as to simultaneously determine the rectangular box to be expanded of the indented text and the target expansion distance to be expanded according to the semantic segmentation result and the offset calculation result, and then expanding the rectangular box to be expanded by the target expansion distance to obtain the final target text detection frame. The method solves the problem that the existing text detection model trained based on indented text instances does not consider the offset of the text indentation process, resulting in slow detection speed and poor effect when detecting dense text areas. At the same time, semantic segmentation and offset regression are used to process the same image to be detected, fully considering the influence of the degree of adhesion and the offset generated by text of different sizes during the indentation process, so that the determined target text detection frame is more consistent with the text boundary of the position to be determined, thereby improving the accuracy of text detection. At the same time, after determining the rectangular frame to be expanded, it is only necessary to perform one expansion based on the determined target expansion distance, which reduces the calculation amount of determining the target text detection frame and improves the efficiency of text detection.

[0098] Example 2

[0099] Figure 2A flowchart of a text detection method provided for the second embodiment of the present invention. The technical solution of the embodiment of the present invention is further optimized on the basis of the above-mentioned optional technical solutions, and clarifies how to determine the indentation segmentation label and the offset label based on the indentation offset text training sample set, and further clarifies how to train the semantic segmentation sub-model and the offset regression sub-model respectively based on the indentation offset text training sample set, so as to finally obtain the constructed indentation offset text detection model, and further clarifies how to determine the label map by thresholding the target semantic segmentation feature map and the connected components, so as to determine the identified outward expansion matrix with the same label value according to the intersection of the label map and the target offset feature map. The target expansion distance is determined based on the predicted offset and the preset reference value, so as to complete the extension of each side of the rectangular box to be expanded according to the target expansion distance. At the same time, the rectangular boxes to be expanded that do not meet the preset confidence level are deleted according to the confidence level of the rectangular boxes to be expanded. Finally, the detected target text detection box is obtained, which fully considers the influence of the offset on the identified indented target text border. At the same time, the confidence level of the rectangular boxes to be expanded is calculated to delete the rectangular boxes to be expanded with lower confidence levels, thereby improving the determination accuracy of the target text detection box, reducing the calculation amount of the target text detection box, and improving the text detection efficiency.

[0100] like Figure 2 As shown, a text detection method provided in the second embodiment of the present invention specifically includes the following steps:

[0101] S201: Acquire an image to be detected.

[0102] S202: Input the image to be detected into a pre-built indented offset text detection model to determine a target semantic segmentation feature map and a target offset feature map.

[0103] Among them, the pre-built indented offset text detection model includes a semantic segmentation sub-model and an offset regression sub-model.

[0104] Specifically, the image to be detected is input into a pre-built indented offset text detection model, and the basic feature extraction of the image to be detected is completed through the backbone network in the indented offset text detection model and the corresponding network for completing multi-scale feature extraction and feature fusion. The extracted basic features are respectively input into the semantic segmentation sub-model and the offset regression sub-model in the indented offset text detection model, and the output result of the semantic segmentation sub-model is determined as the target semantic segmentation feature map, and the output result of the offset regression sub-model is determined as the target offset feature map.

[0105] Further, Figure 3 This is an example diagram of the training steps of a retracted text detection model provided in the second embodiment of the present invention, such as Figure 3 As shown, the specific steps include:

[0106] S301 , extracting basic features from an image sample set in an indented offset text training sample set to determine a basic feature sample set.

[0107] The indented offset text training sample set includes an image sample set and a calibration sample set corresponding to the image sample set, and the calibration sample set includes an indented segmentation label and an offset label corresponding to each image sample.

[0108] In this embodiment, the indented offset text training sample set can be specifically understood as a set of training objects composed of real images and calibration images containing text information to be detected, which are used to input into an untrained indented offset text detection model to train its backbone network, semantic segmentation sub-model and offset regression sub-model. Furthermore, since the indented offset text detection model in this application is a neural network model used to perform text semantic segmentation and indentation offset determination of the text after indentation based on the input image, the indented offset text training sample set input at this time should include an image sample set that is the same as the image input for detection later, and a calibration sample set corresponding to the image sample set, which contains calibrated information of the indented text range and indentation offset of the text to be detected, and is composed of calibration images after detection of each image sample in the image sample set. Each calibration sample in the calibration sample set has a one-to-one correspondence with the image samples in the image sample set, and is used to compare with the intermediate results of the semantic segmentation sub-model and the offset regression sub-model during the training process of the indented offset text detection model to generate a corresponding loss function. The calibration sample set includes the indentation segmentation labels and offset labels corresponding to each image sample. The indentation segmentation labels can be specifically understood as the calibration images or pixel matrices corresponding to the image samples, annotated with the indented boundary positions of the target text to be identified in the image samples; the offset labels can be specifically understood as the calibration images or pixel matrices corresponding to the image samples, annotated with the offsets generated by the indentation process of the target text to be identified in the image samples. The basic feature sample set can be specifically understood as the collection of feature maps obtained after feature extraction and multi-feature fusion of each image sample set.

[0109] Further, Figure 4 The second embodiment of the present invention provides a method for extracting basic features from an image sample set in an indented offset text training sample set to determine a process example of a basic feature sample set, such as Figure 4 As shown, the specific steps include:

[0110] S401: Input an image sample set in the indented offset text training sample set into a feature extraction backbone network to determine a first feature atlas.

[0111] The first feature map set includes multiple feature maps of different resolutions extracted from the image sample set.

[0112] Specifically, each image sample in the image sample set in the indented offset text training set is input into the backbone network for image feature extraction to obtain multiple feature maps of different resolutions corresponding to each image sample, and the set of feature maps is determined as the first feature map set.

[0113] Optionally, the backbone network can be resnet50 or other networks that can realize image feature extraction, which is not limited by the embodiment of the present invention. For example, assuming that the input image sample size is (b, 3, h, w), where h and w are the sizes of the image samples and b is the input batch, using resnet50 as the backbone network for image feature extraction, 5 feature atlases of different sizes and resolutions can be obtained, which can be expressed as F n {n∈(0,5)}.

[0114] S402: Perform multi-scale feature extraction on the first feature atlas to determine a second feature atlas.

[0115] Specifically, the first feature atlas is input into a pre-selected multi-scale feature extraction network or method to perform multi-scale feature extraction to obtain multiple feature maps of different sizes and resolutions, and the set of each feature map is determined as the second feature atlas.

[0116] Continuing with the above example, take an image sample as an example and take its corresponding first feature atlas F n {n∈(0,5)} is input into the standard Feature Pyramid Networks (FPN) to perform multi-scale feature extraction, and a feature atlas consisting of four feature maps of different sizes and resolutions is obtained, which can be expressed as F i {i∈(0,3)}.

[0117] S403: Perform multi-feature fusion on the second feature atlas, and determine the set of fused feature atlases as a basic feature sample set.

[0118] Specifically, the size of each feature map in the second feature map set is unified, and the feature maps after size unification are input into a pre-selected multi-feature fusion algorithm. After multi-feature fusion, basic feature samples corresponding to the image samples are obtained, and the set of technical feature samples corresponding to each image sample is determined as the basic feature sample set.

[0119] Continuing with the above example, the second feature atlas F iEach feature map in {i∈(0,3)} is resized to (b, C1, h / 4, w / 4) by bilinear interpolation and then input into a pre-selected multi-feature fusion algorithm or multi-feature fusion module to obtain a basic feature sample of size (b, C2, h / 4, w / 4), and the set of each basic feature sample is determined as the basic feature sample set. Furthermore, the multi-feature fusion module can be composed of three different convolution kernels and convolution blocks with a sampling interval of 1 in parallel. The feature maps with the same resolution output by each convolution block are added bit by bit to obtain the corresponding basic feature samples. Optionally, the three different convolution kernel sizes can be 3*3, 5*5 and 1*1 respectively, and the convolution block can be composed of a convolution layer, a batch normalization layer and an activation function layer.

[0120] Furthermore, Figure 5 This is an example flow chart of the steps for determining an indented segmentation label provided in the second embodiment of the present invention, as shown in FIG. Figure 5 As shown, the specific steps include:

[0121] S501 : For each image sample, construct a first two-dimensional matrix corresponding to the image sample according to the size of the image sample, and determine the shortest side length of the annotated text in the image sample.

[0122] Specifically, for each input image sample, a two-dimensional matrix of the same size as the image sample is generated, and the two-dimensional matrix is ​​determined as the first two-dimensional matrix, and the corresponding shortest side length is determined according to the number of pixels occupied by the annotated text in the image sample. Optionally, the size of the first two-dimensional matrix can be determined according to the number of pixels in the image sample, and the image sample can include an annotation of the outermost boundary of the text to be recognized. The annotated text information can be used as the annotated text, and then the pixel value corresponding to the shortest side length of the annotated text is determined according to the pixels corresponding to the position of the annotated text. Optionally, the constructed first two-dimensional matrix can be filled with all 0s or other preset initial values, and the embodiment of the present invention does not limit this.

[0123] S502: If the shortest side length is less than or equal to the preset minimum border length, set each pixel corresponding to the position of the annotated text in the first two-dimensional matrix to a first preset value.

[0124] In this embodiment, the preset minimum border length may be specifically understood as a pre-set border length value used to determine whether the annotated text to be recognized needs to be indented.

[0125] Specifically, when the shortest side length is less than or equal to the preset minimum border length, the text information to be recognized can be considered relatively small, and further indentation would make recognition more difficult. In this case, each pixel corresponding to the position of the annotated text in the first two-dimensional matrix is ​​directly set to the first preset value, without indenting the position of the annotated text. Optionally, the first preset value can be 1 or another preset value, and this is not limited in this embodiment of the present invention.

[0126] S503: If the shortest side length is greater than the preset minimum border length, determine a first indentation distance according to the size of the annotated text, update the position of the annotated text according to the first indentation distance, and set each pixel corresponding to the updated position of the annotated text to a first preset value.

[0127] Specifically, if the shortest side length is greater than the preset minimum border length, it can be considered that the annotation text to be identified is itself large and needs to be indented to avoid adhesion with other text to be identified. At this time, the first indentation distance that needs to be indented is determined according to the size of the annotation text, and then the position information corresponding to the annotation text is updated according to the first indentation distance, and the pixel corresponding to the position of the updated annotation text is set to the first preset value.

[0128] Optionally, the indentation distance to be indented can be determined based on the area, perimeter, and pre-set indentation parameters of the labeled text. The indentation distance can be calculated using the following formula:

[0129]

[0130] Wherein, A is the area of ​​the annotated text, L is the perimeter of the annotated text, and r is a preset indentation parameter. Optionally, in the embodiment of the present invention, r can be set to 0.4, or can be adjusted according to actual conditions, and the embodiment of the present invention does not limit this.

[0131] Furthermore, if the first indentation distance is less than the shortest side length, the position of the annotated text can be updated according to the first indentation distance, and each pixel corresponding to the position of the updated annotated text can be set to a first preset value; if the first indentation distance is greater than or equal to the shortest side length, the position of the annotated text is not updated according to the first indentation distance, and each pixel corresponding to the position of the annotated text in the first two-dimensional matrix is ​​directly set to the first preset value.

[0132] Furthermore, when updating the position of the annotation text according to the first indentation distance, the corner point coordinates of the circumscribed rectangular box of the annotation text area can be sorted clockwise, and the center point coordinates of the circumscribed rectangular box can be determined; the adjacent sides of the circumscribed rectangular box can be translated inward by the first indentation distance relative to the center point coordinates; the ends of each side in the circumscribed rectangular box can be retracted inward by the first indentation distance to determine the new endpoint coordinates corresponding to each side, as well as the new intersection coordinates of each adjacent side after the indentation is completed; if the new intersection coordinates are consistent with the corresponding new endpoint coordinates, the area contained in the indented circumscribed rectangular box is determined as the position of the updated annotation text, otherwise, the original coordinates corresponding to the circumscribed rectangular box are returned to re-indent the annotation text until the indentation of the annotation text relative to the first indentation distance is completed.

[0133] Further, Figure 6 This is an example flow chart of the steps for determining an offset tag provided in the second embodiment of the present invention, such as Figure 6 As shown, the specific steps include:

[0134] S601 : For each image sample, construct a second two-dimensional matrix corresponding to the image sample according to the size of the image sample, and determine the shortest side length of the annotated text in the image sample.

[0135] Specifically, for each input image sample, a two-dimensional matrix of the same size as the number of pixels in the image sample is generated. This two-dimensional matrix is ​​then determined as the second two-dimensional matrix. The corresponding shortest side length is then determined based on the number of pixels occupied by the annotated text in the image sample. Optionally, the constructed second two-dimensional matrix may be filled entirely with zeros or with other preset initial values, which is not limited in this embodiment of the present invention.

[0136] It is understandable that the method of determining the first two-dimensional matrix and the second two-dimensional matrix in S601 is the same as that in S501, and the method of determining the shortest side length of the annotated text is also the same, which is not described in detail in the embodiment of the present invention.

[0137] S602: If the shortest side length is less than or equal to the preset minimum border length, set each pixel corresponding to the position of the annotated text in the second two-dimensional matrix to a first preset value.

[0138] In this embodiment, the preset minimum border length may be specifically understood as a pre-set border length value used to determine whether the annotated text to be recognized needs to be indented.

[0139] Specifically, when the shortest side length is less than or equal to the preset minimum border length, the annotated text to be recognized is considered to be relatively small. Further indentation would result in the text occupying too small a space, making recognition more difficult. In this case, each pixel in the second two-dimensional matrix corresponding to the position of the annotated text is directly set to the first preset value, without updating the pixel range corresponding to the position of the annotated text. Optionally, the first preset value may be 1 or another preset value, and this is not limited in this embodiment of the present invention.

[0140] S603: If the shortest side length is greater than the preset minimum border length, determine a second indentation distance according to the size of the image sample, and update and assign values ​​to the second two-dimensional matrix according to the second indentation distance and the shortest side length.

[0141] Specifically, if the shortest side is greater than the preset minimum border length, it can be considered that the annotated text to be identified is itself large and needs to be retracted to avoid its adhesion with other texts to be identified. At this time, the second retraction distance for the annotated text to be retracted is determined based on the area, perimeter and pre-set retraction degree parameters of the annotated text, and then the position information corresponding to the annotated text is updated according to the second retraction distance. At the same time, the offset of the annotated text in the retraction process is determined based on the second retraction distance and the preset reference value. Whether to retract the annotated text is determined based on the calculated second retraction distance and the shortest side length, and then each pixel corresponding to the position of the retracted annotated text is assigned the determined offset.

[0142] It should be noted that the calculation method of the second retraction distance is consistent with the calculation method of the first retraction distance, and this embodiment of the present invention will not be described in detail.

[0143] Furthermore, if the second indentation distance is less than the shortest side length, the offset strength value is determined based on the second indentation distance and the preset reference value, the position of the annotation text is updated based on the second indentation distance, and each pixel corresponding to the position of the updated annotation text is set to the offset strength value; if the second indentation distance is greater than or equal to the shortest side length, each pixel corresponding to the position of the annotation text in the second two-dimensional matrix is ​​set to the first preset value.

[0144] In this embodiment, the preset reference value can be understood as a pre-set value that can be adjusted based on the processed image visualization result to verify the shrinkage effect. The offset strength value can be understood as a value used to represent the degree of deviation of the current shrinkage from the original size of the annotated text.

[0145] Specifically, when the second indentation distance is less than the shortest side length, it can be considered that the annotated text can still be recognized after indentation, and a certain offset will be generated due to the indentation. At this time, the offset strength value of this indentation is determined based on the second indentation distance and the preset reference value, and the position information corresponding to the annotated text is updated according to the second indentation distance. That is, the position information corresponding to each side of the original annotated text is offset inward by the second indentation distance as the updated position information corresponding to the annotated text, and each pixel corresponding to the position of the updated annotated text is set to the determined offset strength value. When the second indentation distance is greater than or equal to the shortest side length, it can be considered that if the annotated text is indented according to the second indentation distance, it will not exist or cannot reach the indentation distance. If it is indented, it cannot be recognized. At this time, the indentation of the annotated text is abandoned, and each pixel corresponding to the position of the annotated text in the second two-dimensional matrix is ​​set to the first preset threshold.

[0146] S302: Input the basic feature sample set into the initial semantic segmentation sub-model to extract the semantic segmentation intermediate results.

[0147] In this embodiment, the initial semantic segmentation submodel can be specifically understood as the untrained semantic segmentation submodel, in which the neural network layer composition architecture is exactly the same as that in the semantic segmentation submodel, and can use two 3*3 convolution blocks consisting of a convolution layer, a batch normalization layer, and an activation function layer, and a 1*1 convolution layer and a sigmoid activation function, but the weight parameters of each neural network layer have not yet been adjusted. The semantic segmentation intermediate result can be specifically understood as the intermediate result output by the untrained initial semantic segmentation submodel after semantic segmentation of the input basic feature sample set.

[0148] Specifically, the basic feature sample set is input into the initial semantic segmentation sub-model for training. During the training process, multiple different semantic segmentation intermediate results can be extracted after the initial semantic segmentation sub-model performs semantic segmentation on each basic feature sample in the basic feature sample set.

[0149] S303: Input the basic feature sample set into the initial offset regression sub-model to extract the offset intermediate result.

[0150] In this embodiment, the initial offset regression submodel can be specifically understood as the untrained offset regression submodel, in which the neural network layer composition architecture is exactly the same as that of the offset regression submodel, and can use two 3*3 convolution blocks consisting of a convolution layer, a batch normalization layer, and an activation function layer, and a 1*1 convolution layer and a sigmoid activation function, but the weight parameters of each neural network layer have not yet been adjusted. The intermediate offset result can be specifically understood as the intermediate result output after the offset regression is performed on the basic feature sample set input by the trained initial offset regression submodel.

[0151] Specifically, the basic feature sample set is input into the initial offset regression sub-model for training. During the training process, multiple different offset intermediate results of the offset obtained by the initial offset regression sub-model after indenting each basic feature sample in the basic feature sample set can be extracted.

[0152] S304: Determine a corresponding first loss function according to the semantic segmentation intermediate result and the corresponding indentation segmentation label.

[0153] Specifically, each pixel value in the semantic segmentation intermediate result is compared with each pixel value in its corresponding indented segmentation label, thereby determining a first loss function corresponding thereto. Optionally, the first loss function may be a smooth L1 loss function.

[0154] Furthermore, according to the semantic segmentation intermediate result and the corresponding indentation segmentation label, a corresponding first loss function is determined, which specifically includes the following steps:

[0155] S3041. Compare the corresponding value of each pixel in the semantic segmentation intermediate result with the corresponding value of each pixel in the corresponding indented segmentation label.

[0156] S3042. Determine a first loss function according to the comparison result.

[0157] S305. Determine a corresponding second loss function according to the offset intermediate result and the corresponding offset label.

[0158] Specifically, each pixel value in the offset intermediate result is compared with each pixel value in the corresponding offset label to determine a corresponding second loss function. Optionally, the second loss function may be a dice loss function.

[0159] Furthermore, according to the intermediate offset result and the corresponding offset label, a corresponding second loss function is determined, which specifically includes the following steps:

[0160] S3051. Compare the corresponding value of each pixel in the offset intermediate result with the corresponding value of each pixel in the corresponding offset label.

[0161] S3052. Determine a second loss function according to the comparison result.

[0162] S306. Determine a total loss function based on the first loss function and the second loss function, and train the initial semantic segmentation sub-model and the initial offset regression sub-model based on the total loss function until a preset convergence condition is met to obtain an indented offset text detection model.

[0163] In this embodiment, the preset convergence condition can be specifically understood as a condition for determining whether the trained initial network model has entered a converged state. Optionally, the preset convergence condition may include a weight parameter change between two model training iterations being less than a preset parameter change threshold, an iteration exceeding a set maximum number of iterations, or the completion of training for all indented offset text training samples, etc., which are not limited in this embodiment of the present invention.

[0164] Specifically, according to the different importance of the first loss function and the second loss function in the training process of the indented offset text detection model, different weight parameters are set for them, and then the first loss function and the second loss function are jointly constructed into a total loss function according to the weight parameters, and the initial semantic segmentation sub-model and the initial offset regression sub-model are back-propagated according to the total loss function, so that the weight parameters in each neural network layer used to constitute the initial semantic segmentation sub-model and the initial offset regression sub-model can be adjusted according to the total loss function, until the preset convergence condition is met, the trained initial semantic segmentation sub-model and the initial offset regression sub-model are jointly constructed into an indented offset text detection model.

[0165] Furthermore, a total loss function is determined according to the first loss function and the second loss function, and the initial semantic segmentation sub-model and the initial offset regression sub-model are trained based on the total loss function until a preset convergence condition is met to obtain an indented offset text detection model, which specifically includes the following steps:

[0166] S3061. Perform weighted summation of the first loss function and the second loss function according to a preset weight value to determine a total loss function.

[0167] For example, the total loss function L total It can be specifically expressed by the following formula:

[0168] L total =L seg +λ×L offset

[0169] Among them, L seg is the first loss function, L offset is the second loss function, λ is the weight of the weighted summation, optionally, in the embodiment of the present invention, λ can be set to 10, and the weight can be adjusted according to experimental results, and the embodiment of the present invention does not limit this.

[0170] S3062. Adjust the weight parameters in the initial semantic segmentation sub-model and the initial offset regression sub-model based on the total loss function until the preset convergence condition is met to obtain the indented offset text detection model.

[0171] S203: Thresholding the target semantic segmentation feature map, and determining a labeling map according to a connected component labeling algorithm.

[0172] In this embodiment, the connected component labeling algorithm can be specifically understood as an algorithm for scanning each pixel in a binary image, dividing pixels with the same pixel value and connected to each other into the same group, and finally obtaining all pixel connected components in the image.

[0173] Specifically, the corresponding values ​​of each point in the target semantic segmentation feature map are thresholded, and values ​​exceeding the preset threshold range are replaced with 1, and vice versa, with 0, to obtain a binary map with only 0 and 1 elements. The obtained binary map is then processed using a connected component labeling algorithm. By identifying components in the binary map with the same pixel values ​​and connectivity, the pixels belonging to the same component are marked with the same value, and different components are marked with different values. Pixels with the same value are treated as a labeled object, and finally a labeled map with one or more labeled objects is obtained.

[0174] S204: Intersect the label map with the target offset feature map, and update the target offset feature map.

[0175] Specifically, since there may be offset values ​​in the part where the non-labeled text is located in the target offset feature map, the target offset feature map is updated by intersecting the label map with the target offset feature map and eliminating the offset of the non-labeled object in the target offset feature map.

[0176] It should be clear that the target offset feature map can be updated by intersecting the label map and the target offset feature map, or by intersecting the thresholded binary map and the target offset feature map. In the embodiment of the present invention, only the intersection of the label map and the target offset feature map is taken as an example, and the specific implementation method is not limited in the embodiment of the present invention.

[0177] S205 , traverse different marking values ​​in the intersection marking graph, and determine the circumscribed rectangular box of the marking object corresponding to the same marking value as the rectangular box to be expanded.

[0178] Specifically, the different label values ​​in the resulting label map are traversed, and pixels with the same label value are identified as a label object. The bounding rectangle of each label object is then determined as the rectangle to be expanded to reach the outer boundary of the text to be recognized. This rectangle can be the indented bounding rectangle of the text to be recognized, or it can be the bounding rectangle of the text to be recognized without being indented, depending on the minimum bounding rectangle length determined during the training process.

[0179] Optionally, the marked object may be calculated using the minAreaRect function provided by OpenCV to obtain an outer rotation rectangular frame of the marked object, which is also the rectangular frame to be expanded in the embodiment of the present invention.

[0180] Furthermore, after traversing different marking values ​​in the intersection marking graph and determining the bounding rectangle of the marking object corresponding to the same marking value as the rectangular box to be expanded, the method further includes:

[0181] The confidence of the rectangular box to be expanded corresponding to the marked object is determined according to the average value of the pixel value of the marked object in the target semantic segmentation feature map; if the confidence is less than the preset confidence threshold, the rectangular box to be expanded is deleted.

[0182] Specifically, since the target semantic segmentation feature map output by the semantic segmentation sub-model may contain errors, the marked object determined by determining the connected area after thresholding the target semantic segmentation feature map may also contain errors. In an embodiment of the present invention, the confidence of the corresponding rectangular box to be expanded is determined by obtaining the pixel average value of each pixel point in the same marked object in the corresponding target semantic segmentation feature map. When the confidence score is less than the preset confidence threshold, the rectangular box to be expanded can be considered as a misidentified rectangular box, and the corresponding marked area does not include the target text to be identified. At this time, the rectangular box to be expanded is deleted to reduce the subsequent calculation amount of rectangular box expansion and improve the accuracy of the target text detection box determination.

[0183] S206 : Taking the average value of the pixel values ​​of each marked object in the updated target offset feature map as the predicted offset of the marked object.

[0184] Specifically, for each marked object, the average value of the pixels corresponding to the marked object in the updated target offset feature map is determined, and the average value is determined as the predicted offset corresponding to the marked object.

[0185] S207: Determine the target expansion distance according to the predicted offset and a preset reference value.

[0186] Specifically, the product of the predicted offset and the preset reference value is determined as the target expansion distance of the rectangular frame to be expanded corresponding to the predicted offset.

[0187] S208 , sorting the coordinates of the corner points of the rectangular frame to be expanded clockwise, and determining the coordinates of the center point of the rectangular frame to be expanded.

[0188] Specifically, for each rectangular box to be expanded, determine whether the four corner points of the rectangular box to be expanded are arranged in a clockwise order. If not, adjust the corner points to be arranged in a clockwise order to facilitate subsequent expansion operations. At the same time, connect the first corner point and the third corner point of the four clockwise corner points in the rectangular box to be expanded, and connect the second corner point and the fourth corner point. The intersection of the two lines is the center point of the rectangular box to be expanded, and the coordinates of the center point can be determined based on the coordinates of the four corner points of the rectangular box to be expanded.

[0189] S209 , translating two adjacent sides of the rectangular frame to be expanded outward relative to the center point coordinates by the target expansion distance.

[0190] Specifically, two adjacent sides in the rectangular frame to be expanded are sequentially translated relative to the center point coordinates in a direction opposite to the direction perpendicular to the center point coordinates by a target expansion distance.

[0191] For example, assuming that the center point of the rectangular frame to be expanded is C and the target expansion distance is D, one side of the rectangular frame to be expanded will be translated outward by a distance D relative to a direction perpendicular to C.

[0192] S210 , extending both ends of each side of the rectangular frame to be expanded outward by a target expansion distance, and determining new endpoint coordinates corresponding to each side.

[0193] Specifically, after all sides of the rectangular frame to be expanded have completed expansion relative to the center point, the end points of each side are extended toward both ends by the target expansion distance, and the coordinates of the two new end points corresponding to each side are determined.

[0194] S211 , determining the intersection points of the sides of the rectangular frame to be expanded after expansion as new intersection coordinates.

[0195] Specifically, after both ends of each side in the rectangular frame to be expanded are expanded, the two adjacent expanded sides should intersect. At this time, the intersection point of the two adjacent sides is determined as the new intersection point coordinates.

[0196] S212: Determine whether the coordinates of each new intersection point are consistent with the corresponding new endpoint coordinates. If so, execute step S213; if not, execute step S214.

[0197] Specifically, determine whether the coordinates of each new intersection point are consistent with the corresponding new endpoint coordinates. If so, it can be considered that the expanded edges can form a complete rectangular frame, and the expansion is successful. In this case, step S213 is executed. Otherwise, it can be considered that the expanded edges do not intersect, or the intersection points are not located at the endpoints of the edges, and the formed rectangular frame is not a complete rectangular frame. In this case, step S214 is executed.

[0198] S213: Determine the expanded rectangular frame as the target text detection frame.

[0199] S214: Return the original coordinates corresponding to the rectangular frame to be expanded.

[0200] Specifically, when it is confirmed that the expansion of the rectangular box to be expanded fails, the corresponding points of each side of the rectangular box to be expanded are returned to the original coordinates, and then an error may be reported, or the process may return to step S208 until the target text detection box corresponding to the rectangular box to be expanded is determined.

[0201] The technical solution of this embodiment is to determine the indentation segmentation label and the offset label according to each image sample in the indentation offset text training sample set and the shortest side length of the annotated text and the size of the annotated text, and then respectively train the semantic segmentation sub-model and the offset regression sub-model through the image sample set, the indentation segmentation label and the offset label in the indentation offset text training sample set, so as to finally obtain a constructed indentation offset text detection model, and determine the labeling map by thresholding and connecting components of the target semantic segmentation feature map, so as to determine the rectangular frame to be expanded with the same label value according to the intersection of the labeling map and the target offset feature map, and the rectangular frame to be expanded with the same label value according to the intersection of the labeling map and the target offset feature map. The predicted offset corresponding to each rectangular box to be expanded is used, and then the target expansion distance is determined based on the predicted offset and the preset benchmark value, so as to complete the extension and expansion of each side of the rectangular box to be expanded according to the target expansion distance. At the same time, the rectangular boxes to be expanded that do not meet the preset confidence level are deleted according to the confidence level of the rectangular box to be expanded, and finally the detected target text detection box is obtained, which fully considers the influence of the offset on the identified indented target text border, and at the same time, the confidence level of the rectangular box to be expanded is calculated to delete the rectangular boxes to be expanded with lower confidence level, thereby improving the determination accuracy of the target text detection box, reducing the calculation amount of the target text detection box, and improving the text detection efficiency.

[0202] Example 3

[0203] Figure 7 This is a structural diagram of a text detection device provided in Example 3 of the present invention. The text detection device includes: an image acquisition module 71, a feature map determination module 72, an outward expansion rectangle determination module 73 and a detection frame determination module 74.

[0204] Among them, the image acquisition module 71 is used to acquire the image to be detected; the feature map determination module 72 is used to input the image to be detected into a pre-built indented offset text detection model to determine the target semantic segmentation feature map and the target offset feature map; the outward expansion rectangle determination module 73 is used to determine the target outward expansion distance and the rectangular frame to be outward expanded based on the target semantic segmentation feature map and the target offset feature map; the detection frame determination module 74 is used to expand the rectangular frame to be outward expanded by the target outward expansion distance to determine the target text detection frame; wherein, the pre-built indented offset text detection model includes a semantic segmentation sub-model and an offset regression sub-model.

[0205] The technical solution of this embodiment solves the problem that the existing text detection model obtained by training based on indented text instances does not consider the offset of the text indentation process, resulting in slow detection speed and poor effect when detecting dense text areas. At the same time, semantic segmentation and offset regression are used to process the same image to be detected, fully considering the influence of the degree of adhesion and the offset generated by text of different sizes during the indentation process, so that the determined target text detection frame is more consistent with the text boundary of the position to be determined, thereby improving the accuracy of text detection. At the same time, after determining the rectangular frame to be expanded, it is only necessary to perform expansion once according to the determined target expansion distance, thereby reducing the calculation amount of determining the target text detection frame and improving the efficiency of text detection.

[0206] Furthermore, the training steps of the indented text detection model include:

[0207] Extracting basic features from an image sample set in an indented offset text training sample set to determine a basic feature sample set; wherein the indented offset text training sample set includes an image sample set and a calibration sample set corresponding to the image sample set, and the calibration sample set includes an indented segmentation label and an offset label corresponding to each image sample;

[0208] Input the basic feature sample set into the initial semantic segmentation sub-model and extract the semantic segmentation intermediate results;

[0209] Input the basic feature sample set into the initial offset regression sub-model to extract the offset intermediate results;

[0210] Determine the corresponding first loss function according to the semantic segmentation intermediate result and the corresponding indentation segmentation label;

[0211] Determine the corresponding second loss function according to the offset intermediate result and the corresponding offset label;

[0212] The total loss function is determined according to the first loss function and the second loss function, and the initial semantic segmentation sub-model and the initial offset regression sub-model are trained based on the total loss function until a preset convergence condition is met to obtain an indented offset text detection model.

[0213] Furthermore, basic features are extracted from the image sample set in the indented offset text training sample set to determine a basic feature sample set, including:

[0214] Inputting the image sample set in the indented offset text training sample set into the feature extraction backbone network to determine a first feature atlas; wherein the first feature atlas includes a plurality of feature maps of different resolutions extracted from the image sample set;

[0215] Performing multi-scale feature extraction on the first feature atlas to determine a second feature atlas;

[0216] Multi-feature fusion is performed on the second feature atlas, and the set of fused feature maps is determined as the basic feature sample set.

[0217] Furthermore, the step of determining the indented segmentation label includes:

[0218] For each image sample, construct a first two-dimensional matrix corresponding to the image sample according to the size of the image sample, and determine the shortest side length of the image sample;

[0219] If the shortest side length is less than or equal to the preset minimum border length, each pixel corresponding to the position of the annotated text in the first two-dimensional matrix is ​​set to a first preset value;

[0220] If the shortest side length is greater than the preset minimum border length, a first indentation distance is determined according to the size of the image sample, the position of the annotation text is updated according to the first indentation distance, and each pixel corresponding to the updated position of the annotation text is set to a first preset value.

[0221] Furthermore, the step of determining the offset tag includes:

[0222] For each image sample, construct a second two-dimensional matrix corresponding to the image sample according to the size of the image sample, and determine the shortest side length of the image sample;

[0223] If the shortest side length is less than or equal to the preset minimum border length, each pixel corresponding to the position of the annotated text in the second two-dimensional matrix is ​​set to a first preset value;

[0224] If the shortest side length is greater than the preset minimum border length, a second indentation distance is determined according to the size of the image sample, and the second two-dimensional matrix is ​​updated and assigned according to the second indentation distance and the shortest side length.

[0225] Furthermore, the second two-dimensional matrix is ​​updated and assigned a value according to the second inward contraction distance and the shortest side length, including:

[0226] If the second indentation distance is less than the shortest side length, determine an offset strength value based on the second indentation distance and a preset reference value, update the position of the annotated text based on the second indentation distance, and set each pixel corresponding to the updated position of the annotated text as the offset strength value;

[0227] If the second inward distance is greater than or equal to the shortest side length, each pixel corresponding to the position of the annotated text in the second two-dimensional matrix is ​​set to a first preset value.

[0228] Furthermore, according to the semantic segmentation intermediate result and the corresponding indentation segmentation label, a corresponding first loss function is determined, including:

[0229] Compare the corresponding values ​​of each pixel in the semantic segmentation intermediate result with the corresponding values ​​of each pixel in the corresponding indented segmentation label;

[0230] A first loss function is determined according to the comparison result.

[0231] Furthermore, according to the intermediate offset result and the corresponding offset label, a corresponding second loss function is determined, including:

[0232] Compare the corresponding value of each pixel in the offset intermediate result with the corresponding value of each pixel in the corresponding offset label;

[0233] A second loss function is determined based on the comparison result.

[0234] Furthermore, a total loss function is determined according to the first loss function and the second loss function, and the initial semantic segmentation sub-model and the initial offset regression sub-model are trained based on the total loss function until a preset convergence condition is met to obtain an indented offset text detection model, including:

[0235] The first loss function and the second loss function are weighted and summed according to a preset weight value to determine a total loss function;

[0236] Based on the total loss function, the weight parameters in the initial semantic segmentation sub-model and the initial offset regression sub-model are adjusted until the preset convergence conditions are met to obtain the indented offset text detection model.

[0237] Optionally, the outward-expanding rectangle determining module 73 includes:

[0238] a label map determination unit, configured to threshold the target semantic segmentation feature map and determine a label map according to a connected component labeling algorithm;

[0239] An offset map updating unit, configured to intersect the label map with the target offset feature map and update the target offset feature map;

[0240] A rectangular frame determining unit is used to traverse different marking values ​​in the marked graph after intersection, and determine the circumscribed rectangular frame of the marked object corresponding to the same marking value as the rectangular frame to be expanded;

[0241] an offset determination unit, configured to determine an average value of pixel values ​​of each marked object in the updated target offset feature map as a predicted offset of the marked object;

[0242] The outward expansion distance determination unit is used to determine the target outward expansion distance according to the predicted offset and a preset reference value.

[0243] Optionally, the text detection device further includes:

[0244] The confidence determination module is used to determine the confidence of the rectangular box to be expanded corresponding to the marked object based on the average value of the pixel values ​​of the marked object in the target semantic segmentation feature map; if the confidence is less than a preset confidence threshold, the rectangular box to be expanded is deleted.

[0245] Optionally, the detection frame determination module 74 includes:

[0246] A center point coordinate determining unit, configured to sort the coordinates of the corner points of the rectangular frame to be expanded clockwise and determine the coordinates of the center point of the rectangular frame to be expanded;

[0247] An edge translation unit is used to translate two adjacent edges of the rectangular frame to be expanded outward relative to the center point coordinates by the target expansion distance;

[0248] An endpoint coordinate determination unit, configured to extend both ends of each side of the rectangular frame to be expanded outward by a target expansion distance, and determine new endpoint coordinates corresponding to each side;

[0249] An intersection coordinate determining unit, configured to determine the intersection of each side of the rectangular frame to be expanded after expansion as a new intersection coordinate;

[0250] The detection frame determination unit is used to determine the expanded rectangular frame as the target text detection frame if the coordinates of each new intersection point are consistent with the corresponding new endpoint coordinates; otherwise, return the original coordinates corresponding to the rectangular frame to be expanded.

[0251] The text detection device provided by the embodiment of the present invention can execute the text detection method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0252] Example 4

[0253] Figure 8 A structural diagram of a text detection device provided for embodiment 4 of the present invention. The text detection device 80 may be an electronic device intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0254] like Figure 8As shown, the text detection device 80 includes at least one processor 81 and a memory, such as a read-only memory (ROM) 82 or a random access memory (RAM) 83, communicatively connected to the at least one processor 81. The memory stores a computer program executable by the at least one processor, and the processor 81 can perform various appropriate actions and processes based on the computer program stored in the read-only memory (ROM) 82 or loaded from the storage unit 88 into the random access memory (RAM) 83. The RAM 83 can also store various programs and data required for the operation of the text detection device 80. The processor 81, ROM 82, and RAM 83 are interconnected via a bus 84. An input / output (I / O) interface 85 is also connected to the bus 84.

[0255] Multiple components in the text detection device 80 are connected to the I / O interface 85, including an input unit 86, such as a keyboard, a mouse, etc.; an output unit 87, such as various types of displays, speakers, etc.; a storage unit 88, such as a magnetic disk, an optical disk, etc.; and a communication unit 89, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 89 allows the text detection device 80 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0256] The processor 81 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the processor 81 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 81 executes the various methods and processes described above, such as the text detection method.

[0257] In some embodiments, the text detection method can be implemented as a computer program that is tangibly contained in a computer-readable storage medium, such as storage unit 88. In some embodiments, part or all of the computer program can be loaded and / or installed on the text detection device 80 via ROM 82 and / or communication unit 89. When the computer program is loaded into RAM 83 and executed by processor 81, one or more steps of the text detection method described above can be performed. Alternatively, in other embodiments, processor 81 can be configured to perform the text detection method in any other suitable manner (e.g., by means of firmware).

[0258] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0259] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0260] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0261] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0262] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0263] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0264] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0265] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A text detection method, characterized in that: include: Obtain the image to be detected; Inputting the image to be detected into a pre-built indented offset text detection model to determine a target semantic segmentation feature map and a target offset feature map; Determine the target expansion distance and the rectangular frame to be expanded according to the target semantic segmentation feature map and the target offset feature map; Expand the rectangular frame to be expanded by the target expansion distance to determine the target text detection frame; The pre-built indented offset text detection model includes a semantic segmentation sub-model and an offset regression sub-model; The training steps of the indented text detection model include: Basic features are extracted from an image sample set in an indented offset text training sample set to determine a basic feature sample set; wherein the indented offset text training sample set includes an image sample set and a calibration sample set corresponding to the image sample set, and the calibration sample set includes an indentation segmentation label and an offset label corresponding to each image sample; the indentation segmentation label is a calibration image or pixel matrix corresponding to the image sample, which is labeled with the position of the indented boundary of the target text to be identified in the image sample; the offset label is a calibration image or pixel matrix corresponding to the image sample, which is labeled with the offset generated by the indentation process of the target text to be identified in the image sample; Inputting the basic feature sample set into the initial semantic segmentation sub-model to extract the semantic segmentation intermediate result; Inputting the basic feature sample set into the initial offset regression sub-model to extract the offset intermediate result; Determining a corresponding first loss function according to the semantic segmentation intermediate result and the corresponding indentation segmentation label; Determining a corresponding second loss function according to the offset intermediate result and the corresponding offset label; Determining a total loss function according to the first loss function and the second loss function, and training the initial semantic segmentation sub-model and the initial offset regression sub-model based on the total loss function until a preset convergence condition is met to obtain an indented offset text detection model; The step of determining the indented segmentation label includes: For each image sample, construct a first two-dimensional matrix corresponding to the image sample according to the size of the image sample, and determine the shortest side length of the annotated text in the image sample; If the shortest side length is less than or equal to a preset minimum border length, then setting each pixel corresponding to the position of the annotated text in the first two-dimensional matrix to a first preset value; If the shortest side length is greater than the preset minimum border length, a first indentation distance is determined according to the size of the annotated text, the position of the annotated text is updated according to the first indentation distance, and each pixel corresponding to the updated position of the annotated text is set to a first preset value.

2. The method according to claim 1, characterized in that The step of extracting basic features from the image sample set in the indented offset text training sample set to determine the basic feature sample set includes: Inputting the image sample set in the indented offset text training sample set into a feature extraction backbone network to determine a first feature atlas; wherein the first feature atlas includes a plurality of feature maps of different resolutions extracted from the image sample set; Performing multi-scale feature extraction on the first feature atlas to determine a second feature atlas; Multi-feature fusion is performed on the second feature graph set, and a set of fused feature graphs is determined as a basic feature sample set.

3. The method according to claim 1, characterized in that The step of determining the offset label includes: For each image sample, construct a second two-dimensional matrix corresponding to the image sample according to the size of the image sample, and determine the shortest side length of the annotated text in the image sample; If the shortest side length is less than or equal to the preset minimum border length, then setting each pixel corresponding to the position of the annotated text in the second two-dimensional matrix to a first preset value; If the shortest side length is greater than the preset minimum border length, a second indentation distance is determined according to the size of the annotated text, and the second two-dimensional matrix is ​​updated and assigned a value according to the second indentation distance and the shortest side length.

4. The method according to claim 3, characterized in that The updating and assigning of values ​​to the second two-dimensional matrix according to the second inward contraction distance and the shortest side length includes: If the second indented distance is less than the shortest side length, determining an offset strength value according to the second indented distance and a preset reference value, updating the position of the annotated text according to the second indented distance, and setting each pixel corresponding to the updated position of the annotated text to the offset strength value; If the second inward distance is greater than or equal to the shortest side length, each pixel corresponding to the position of the annotated text in the second two-dimensional matrix is ​​set to a first preset value.

5. The method according to claim 1, wherein The determining of a corresponding first loss function according to the semantic segmentation intermediate result and the corresponding indentation segmentation label includes: Compare the corresponding values ​​of each pixel in the semantic segmentation intermediate result with the corresponding values ​​of each pixel in the corresponding indented segmentation label; A first loss function is determined according to the comparison result.

6. The method according to claim 1, wherein Determining a corresponding second loss function according to the offset intermediate result and the corresponding offset label includes: Compare the corresponding value of each pixel in the offset intermediate result with the corresponding value of each pixel in the corresponding offset label; A second loss function is determined based on the comparison result.

7. The method according to claim 1, characterized in that The method further comprises: determining a total loss function according to the first loss function and the second loss function, and training the initial semantic segmentation sub-model and the initial offset regression sub-model based on the total loss function until a preset convergence condition is satisfied to obtain an indented offset text detection model, including: A total loss function is determined by weighting the first loss function and the second loss function according to a preset weight value; Based on the total loss function, the weight parameters in the initial semantic segmentation sub-model and the initial offset regression sub-model are adjusted until a preset convergence condition is met to obtain an indented offset text detection model.

8. The method according to claim 1, characterized in that The determining of the target expansion distance and the rectangular frame to be expanded according to the target semantic segmentation feature map and the target offset feature map includes: Thresholding the target semantic segmentation feature map and determining a labeling map according to a connected component labeling algorithm; Intersecting the label map with the target offset feature map, and updating the target offset feature map; After traversing the different marking values ​​in the marking graph after intersection, the circumscribed rectangular box of the marking object corresponding to the same marking value is determined as the rectangular box to be expanded; Determine an average value of pixel values ​​of each marked object in the updated target offset feature map as a predicted offset of the marked object; The target expansion distance is determined according to the predicted offset and a preset reference value.

9. The method according to claim 8, characterized in that Also includes: Determining the confidence of the rectangular box to be expanded corresponding to the marked object according to the average value of the pixel values ​​of the marked object in the target semantic segmentation feature map; If the confidence level is less than a preset confidence threshold, the rectangular frame to be expanded is deleted.

10. The method according to claim 1, characterized in that The step of expanding the rectangular frame to be expanded by the target expansion distance to determine the target text detection frame includes: Sort the coordinates of the corner points of the rectangular frame to be expanded clockwise, and determine the coordinates of the center point of the rectangular frame to be expanded; Shifting two adjacent sides of the rectangular frame to be expanded outward by the target expansion distance relative to the center point coordinates; Extending both ends of each side of the rectangular frame to be expanded outward by the target expansion distance to determine the new endpoint coordinates corresponding to each side; Determine the intersection points of the edges of the rectangular frame to be expanded after expansion as new intersection coordinates; If the coordinates of each of the new intersection points are consistent with the corresponding new endpoint coordinates, the expanded rectangular frame to be expanded is determined as the target text detection frame; otherwise, the original coordinates corresponding to the rectangular frame to be expanded are returned.

11. A text detection device, characterized in that: include: An image acquisition module, used for acquiring an image to be detected; A feature map determination module is used to input the image to be detected into a pre-built indented offset text detection model to determine a target semantic segmentation feature map and a target offset feature map; An expansion rectangle determination module, configured to determine a target expansion distance and a rectangular frame to be expanded based on the target semantic segmentation feature map and the target offset feature map; A detection frame determination module, configured to expand the rectangular frame to be expanded by the target expansion distance to determine a target text detection frame; The pre-built indented offset text detection model includes a semantic segmentation sub-model and an offset regression sub-model; The training steps of the indented text detection model include: Basic features are extracted from an image sample set in an indented offset text training sample set to determine a basic feature sample set; wherein the indented offset text training sample set includes an image sample set and a calibration sample set corresponding to the image sample set, and the calibration sample set includes an indentation segmentation label and an offset label corresponding to each image sample; the indentation segmentation label is a calibration image or pixel matrix corresponding to the image sample, which is labeled with the position of the indented boundary of the target text to be identified in the image sample; the offset label is a calibration image or pixel matrix corresponding to the image sample, which is labeled with the offset generated by the indentation process of the target text to be identified in the image sample; Inputting the basic feature sample set into the initial semantic segmentation sub-model to extract the semantic segmentation intermediate result; Inputting the basic feature sample set into the initial offset regression sub-model to extract the offset intermediate result; Determining a corresponding first loss function according to the semantic segmentation intermediate result and the corresponding indentation segmentation label; Determining a corresponding second loss function according to the offset intermediate result and the corresponding offset label; Determining a total loss function according to the first loss function and the second loss function, and training the initial semantic segmentation sub-model and the initial offset regression sub-model based on the total loss function until a preset convergence condition is met to obtain an indented offset text detection model; The step of determining the indented segmentation label includes: For each image sample, construct a first two-dimensional matrix corresponding to the image sample according to the size of the image sample, and determine the shortest side length of the annotated text in the image sample; If the shortest side length is less than or equal to a preset minimum border length, then setting each pixel corresponding to the position of the annotated text in the first two-dimensional matrix to a first preset value; If the shortest side length is greater than the preset minimum border length, a first indentation distance is determined according to the size of the annotated text, the position of the annotated text is updated according to the first indentation distance, and each pixel corresponding to the updated position of the annotated text is set to a first preset value.

12. A text detection device, characterized in that: The text detection device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the text detection method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the text detection method according to any one of claims 1 to 10 when executed.

Images