Text detection method and apparatus for images

Abstract

This application provides a method, apparatus, electronic device, and computer-readable storage medium for text detection in images; relating to the fields of artificial intelligence and transportation, the method includes: identifying multiple detection boxes from an image containing text, and obtaining a feature map of each detection box; performing mask segmentation processing based on the feature map of each detection box to obtain a text mask segmentation result; performing text box prediction processing in at least one manner based on the feature map of each detection box to obtain a text box recognition result; and determining the actual text box in the image based on the overlap rate of the text mask segmentation result and the text box recognition result. This application can improve the accuracy of text box recognition in images.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence technology and transportation, and in particular to a method and apparatus for text detection in images. Background Technology

[0002] Artificial intelligence (AI) is the theory, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0003] Text in images often suffers from tilting, stretching, and size variations. Current text detection techniques typically rely on foreground-background separation or pixel-level feature clustering, but these methods are ineffective for detecting text boxes. There are currently no effective solutions to improve the accuracy of text box detection. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for text detection in images, which can improve the recognition accuracy of text boxes in images.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a method comprising:

[0007] Multiple detection boxes are identified from an image including text, and a feature map of each detection box is obtained;

[0008] Based on the feature map of each detection box, a mask segmentation process is performed to obtain the text mask segmentation result;

[0009] Based on the feature map of each detection box, at least one method of text box prediction processing is performed to obtain the text box recognition result;

[0010] Based on the overlap rate between the text mask segmentation result and the text box recognition result, the actual text box in the image is determined.

[0011] This application provides a text detection device for an image, comprising:

[0012] The feature extraction module is configured to identify multiple detection boxes from an image including text and obtain a feature map of each detection box;

[0013] The mask segmentation module is configured to perform mask segmentation processing based on the feature map of each detection box to obtain text mask segmentation results;

[0014] The text box prediction module is configured to perform text box prediction processing in at least one manner based on the feature map of each detection box to obtain the text box recognition result;

[0015] The text box determination module is configured to determine the actual text boxes in the image based on the overlap rate of the text mask segmentation result and the text box recognition result.

[0016] This application provides a terminal device for text detection in images, the terminal device comprising:

[0017] Memory, used to store executable instructions;

[0018] The processor, when executing executable instructions stored in the memory, implements any of the image text detection methods provided in the embodiments of this application.

[0019] This application provides a computationally readable storage medium storing executable instructions for implementing the image text detection method provided in this application when executed by a processor.

[0020] This application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the image text detection method provided in this application.

[0021] The embodiments of this application have the following beneficial effects:

[0022] This application sets up multiple text box prediction methods. Based on the overlap rate of text mask segmentation results and multiple text box recognition results, the actual text boxes in the image are determined, which effectively improves the accuracy and efficiency of text box recognition in the image. Attached Figure Description

[0023] Figure 1 This is a schematic diagram illustrating an application scenario of the image text detection method provided in the embodiments of this application;

[0024] Figure 2A This is a schematic diagram of the structure of a terminal device 400 for text detection in an image provided in an embodiment of this application;

[0025] Figure 2B This is a schematic diagram of the structure of the image text detection device 455 provided in the embodiments of this application;

[0026] Figure 3AThis is a schematic flowchart of the text detection method for images provided in the embodiments of this application;

[0027] Figure 3B This is a schematic flowchart of the text detection method for images provided in the embodiments of this application;

[0028] Figure 3C This is a schematic flowchart of the text detection method for images provided in the embodiments of this application;

[0029] Figure 4A This is a schematic diagram of image processing for the text detection method provided in the embodiments of this application;

[0030] Figure 4B This is a schematic diagram of image processing for the text detection method provided in the embodiments of this application;

[0031] Figure 4C This is a schematic diagram of image processing for the text detection method provided in the embodiments of this application;

[0032] Figure 5 This is a schematic flowchart of the text detection method for images provided in the embodiments of this application;

[0033] Figure 6A This is a schematic diagram of the structure of the neural network model provided in the embodiments of this application;

[0034] Figure 6B This is a schematic diagram of the key edge prediction network model in the neural network model provided in the embodiments of this application;

[0035] Figure 6C This is a schematic diagram of image processing for the critical edge prediction network model provided in the embodiments of this application. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0037] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0038] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0039] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0040] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0041] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0042] 1) Image segmentation (OS, Object Segmentation), in the field of computer vision, refers to the process of subdividing a digital image into multiple image sub-regions (sets of pixels), where features within the same sub-region have a certain degree of similarity, while features between different sub-regions exhibit significant differences.

[0043] 2) Feature Map: Multiple features extracted directly or indirectly from the image to be processed. The resolution of the feature map can be less than or equal to the resolution of the image to be processed. Feature maps can be extracted directly from the image to be processed, or they can be further extracted (i.e., indirectly) from the feature maps of the image to be processed.

[0044] 3) Convolutional Neural Networks (CNNs): These are a class of feedforward neural networks (FNNs) that include convolutional computations and have a deep structure. They are one of the representative algorithms of deep learning. CNNs have representation learning capabilities and can perform shift-invariant classification of input images according to their hierarchical structure.

[0045] 4) Masking: Using a selected image, graphic, or object to occlude (all or part) the image to be processed, thereby controlling the area or process of image processing. In digital image processing, a pre-made region of interest mask can be multiplied with the image to be processed to obtain the region of interest image. The image values ​​within the region of interest remain unchanged, while the image values ​​outside the region are all 0.

[0046] 5) Regression processing, also known as data processing using regression algorithms, is a type of supervised algorithm. It involves building a model using test set data and then processing the model using data from the training set. Linear regression aims to find a line that minimizes the sum of distances to all sample points, and is commonly used in prediction and classification. Common regression algorithms include: Ordinary Least Squares, Stepwise Regression, Multivariate Adaptive Regression Splines, and Locally Estimated Scatterplot Smoothing.

[0047] For text detection in natural scene images such as road rectangular signs, ground text, and points of interest (POIs), the image quality of vehicle-mounted images in the road rectangular sign application is poor, requiring robust text detection algorithms. Text detection and recognition in natural scenes is challenging due to the rich backgrounds, brightness differences, contrast variations, occlusion, lighting conditions, perspective distortion, and incomplete occlusion. Inaccurate detection leads to a significant drop in recognition accuracy. Related technologies employ segmentation-based and detection-based approaches for text detection in these images, but the accuracy of text box recognition remains low.

[0048] This application provides a text detection method for images, a text detection device for images, an electronic device for text detection in images, a computer-readable storage medium, and a computer program product, which can improve the recognition accuracy of text boxes. The following describes exemplary applications of the electronic device provided in this application. The electronic device for text detection in images provided in this application can be implemented as various types of user terminals such as laptops, tablets, desktop computers, set-top boxes, and mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), or it can be implemented as a server.

[0049] See Figure 1 , Figure 1This is a schematic diagram of an application scenario for the text detection method for images provided in this application embodiment. To realize the application scenario of text detection for images (for example, the application scenario may be to detect points of interest, ground text, and text on road signs in natural scenes), the terminal device 400 connects to the server 200 through the network 300. The network 300 may be a wide area network or a local area network, or a combination of both.

[0050] In some embodiments, server 200 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Terminal device 400 may be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, smart voice interaction device, smart home appliance, vehicle terminal, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0051] In some embodiments, the terminal device 400 executes the image text detection method provided in this application to recognize text boxes in the image, and sends the recognized text classification, text boxes, and feature maps to the server 200. The server 200 generates the final text recognition result and sends the text recognition result to the terminal device 400. The terminal device 400 annotates the text recognition result in the image and displays the annotated image to the user.

[0052] In some embodiments, the terminal device 400 can implement the text detection method of this application embodiment independently without the aid of the server 200.

[0053] The embodiments of this application can be implemented with the help of cloud technology, which refers to a hosting technology that unifies a series of resources such as hardware, software, and network within a wide area network or local area network to realize the computation, storage, processing, and sharing of data.

[0054] The embodiments of this application can be implemented using computer vision (CV) technology. Computer vision is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes for target recognition, tracking, and measurement, and further performs image processing to make the computer-processed images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision researches related theories and technologies, attempting to establish artificial intelligence systems capable of extracting information from images or multidimensional data.

[0055] The embodiments of this application can also be implemented through Intelligent Vehicle Infrastructure Cooperative Systems (IVICS), which is a development direction of Intelligent Transportation Systems (ITS). IVICS utilizes advanced wireless communication and next-generation Internet technologies to implement dynamic real-time information interaction between vehicles and roads in all aspects. Based on the collection and fusion of dynamic traffic information across all times and spaces, it carries out active safety control of vehicles and cooperative road management, fully realizing effective coordination between people, vehicles, and roads, ensuring traffic safety, improving traffic efficiency, and thus forming a safe, efficient, and environmentally friendly road traffic system.

[0056] In some embodiments, server 200 can generate high-precision map data and send the map data to mobile terminal 400, allowing users to view the map through mobile terminal 400. Server 200 can perform text recognition on text areas in traffic elements (such as signs, billboards, landmarks, and text on iconic buildings) in map data collected by road testing equipment, and label the traffic elements based on the recognized text, facilitating the updating of names of relevant locations in the electronic map and facilitating the querying of the location of relevant locations in the electronic map.

[0057] Taking electronic devices as terminal devices as an example, see [link to relevant documentation]. Figure 2A , Figure 2A This is a schematic diagram of the structure of the terminal device 400 for the image text detection method provided in this application embodiment. Figure 2A The terminal device 400 shown includes at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components in the terminal device 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2A The general labeled all buses as Bus System 440.

[0058] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0059] User interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0060] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0061] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0062] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0063] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, for implementing various basic business functions and handling hardware-based tasks.

[0064] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including Bluetooth, WiFi, and Universal Serial Bus (USB).

[0065] Presentation module 453 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 associated with user interface 430 (e.g., a display screen, a speaker, etc.).

[0066] The input processing module 454 is used to detect and translate one or more user inputs or interactions from one or more input devices 432.

[0067] In some embodiments, the image text detection device provided in this application can be implemented in software. Figure 2A A device 455 stored in memory 450 is shown, which can be software in the form of programs and plug-ins, including the following software modules: feature extraction module 4551, mask segmentation module 4552, text box prediction module 4553, and text box determination module 4554. These modules are logically linked and can therefore be arbitrarily combined or further split according to the functions they implement. The functions of each module will be described below.

[0068] In other embodiments, the image text detection device provided in this application can be implemented in hardware. As an example, the image text detection device provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the image text detection method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0069] In some embodiments, the text detection method for images provided in this application can be implemented by running various computer programs. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), i.e., a program that needs to be installed in the operating system to run; it can also be a small program, i.e., a program that only needs to be downloaded to a browser environment to run; or it can be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module, or plugin.

[0070] See Figure 2B , Figure 2B This is a schematic diagram of the structure of the text detection device 455 for images provided in this application embodiment. The text detection device 455 includes a neural network model and a text box determination module 4554. The neural network model includes a feature extraction module 4551, a mask segmentation module 4552, and a text prediction module 4553. After inputting an image containing text into the neural network model, the feature extraction module 4551 first performs operations such as downsampling feature extraction, multi-scale feature fusion, detection box generation, and detection box size unification to obtain a feature map of the detection box. The mask segmentation module 4552 performs mask segmentation based on the feature map of the detection box to obtain a text mask, and sends the text mask to the text box determination module 4554. The text box prediction module 4553 performs at least one text box prediction based on the feature map of the detection box to obtain at least one text box recognition result, and sends the at least one text box recognition result to the text box determination module 4554. The text box determination module 4554 determines the actual text box based on the text mask and the at least one text box recognition result.

[0071] Figure 3A , Figure 3A This is a flowchart illustrating the text detection method for images provided in this application, with a terminal device as the execution entity. Figure 3A The steps shown are explained.

[0072] In step 101, multiple detection boxes are identified from the image including text, and the feature map of each detection box is obtained.

[0073] In some embodiments, step 101 can be implemented as follows: downsampling the image including text at multiple levels, wherein the downsampled feature maps obtained from downsampling at each level except the last level are used as input for downsampling at the next level; upsampling at multiple levels based on the downsampled feature map of the last level, wherein the upsampled feature maps obtained from upsampling at each level except the last level are fused with downsampled feature maps of the same size as input for upsampling at the next level; binary classification and bounding box regression based on the upsampled feature maps of each level to determine whether text is included, resulting in multiple detection boxes including text; and size unification processing of the upsampled feature maps within each detection box to obtain the processed feature map of each detection box.

[0074] As an example, the size of the detection boxes can be different for images of different sizes and for different text. The feature map of each detection box is the feature map of the region corresponding to each detection box in the overall feature map of the image.

[0075] As an example, step 101 can be based on Figure 2BThe feature extraction module 4551 is implemented in the image. This module includes a convolutional neural network (CNN) and a feature fusion pyramid (FPN) for feature extraction. The CNN can be a Visual Geometry Group (VGG) network model, a GoogLeNet deep neural network, a 101-layer residual network (ResNet101), or a 50-layer residual network (ResNet50), etc., used to extract feature maps from the image. Multiple layers in the feature fusion pyramid fuse the feature maps extracted by the CNN, with each layer outputting a fused feature map.

[0076] Let's take a 101-layer residual network as an example. A 101-layer residual network can downsample images. It consists of five layers: C1, C2, C3, C4, and C5. Each layer includes multiple identity residual blocks and multiple convolutional blocks. Each convolutional block comprises multiple convolutional layers, batch normalization layers, and ReLU (Rectified Linear Unit) activation layers. These blocks are used to extract features. The lower-level convolutional layers extract basic features such as image edges and textures, while higher-level convolutional layers combine and abstract these texture features. The normalization layers normalize the features to a normal distribution. The activation layers perform non-linear mapping on the extracted features, enhancing the model's generalization ability. The identity residual blocks incorporate identity mapping and shortcut mechanisms to ensure effective gradient return and prevent gradient vanishing during training. Identifier residual blocks are used to implement cross-layer connections within a 101-layer residual network, making the training of the 101-layer residual network easier.

[0077] As an example, the first layer C1 and the second layer C2 of the 101-layer residual network do not perform parameter updates or feature fusion. The third layer C3, the fourth layer C4, and the fifth layer C5 of the 101-layer residual network output feature maps. The third layer C3 is the first layer to output downsampled feature maps. Each of the third, fourth, and fifth layers C5 downsamples the feature map output from the previous layer, and the size of the feature map output by each layer is different. The fifth layer C5 of the 101-layer residual network is the last layer to output downsampled feature maps.

[0078] As an example, multiple levels of upsampling or downsampling can be cascaded, and sampling and feature fusion can be performed based on a feature fusion pyramid model.

[0079] As an example, a feature fusion pyramid is a structure that inversely fuses feature maps from multiple levels, with each level of the pyramid performing an upsampling operation. In this embodiment, the feature fusion pyramid includes seven layers, where the first layer P1 and the second layer P2 do not output feature maps; the third layer P3 to the seventh layer P7 output feature maps. The starting layer for inverse feature map fusion is the fifth layer P5 of the feature fusion pyramid, using the downsampled feature map of the fifth layer C5 of the 101-layer residual network as input to the fifth layer P5 of the feature fusion pyramid. Here, the size of the downsampled feature map of the fourth layer C4 is the same as the size of the feature map output from the fourth layer P4 of the feature fusion pyramid and the size of the feature map corresponding to the fifth layer P5 of the feature fusion pyramid after the upsampling operation. The size of the downsampled feature map of the third layer C3 is the same as the size of the feature map output from the third layer P3 of the feature fusion pyramid and the size of the feature map corresponding to the fourth layer P4 of the feature fusion pyramid after the upsampling operation.

[0080] As an example, the fifth layer P5 of the feature fusion pyramid outputs the downsampled feature map of the fifth layer C5. Simultaneously, the fifth layer P5 performs a convolution operation on the downsampled feature map of the fifth layer C5, and inputs the generated feature map into the sixth layer P6 and the seventh layer P7 of the feature fusion pyramid. The sixth layer P6 and the seventh layer P7 output the feature maps generated by the convolution operation, respectively. The fifth layer P5 also performs an upsampling operation on the downsampled feature map of the fifth layer C5, and fuses the resulting upsampled feature map with the downsampled feature map of the fourth layer C4 to obtain a first fused feature map. This first fused feature map is then input into the fourth layer P4 of the feature fusion pyramid. The fourth layer P4 outputs the first fused feature map. The fourth layer P4 also performs an upsampling operation on the first fused feature map, and fuses the resulting upsampled feature map with the downsampled feature map of the third layer C3 to obtain a second fused feature map. This second fused feature map is then input into the third layer P3 of the feature fusion pyramid. The third layer P3 outputs the second fused feature map.

[0081] As an example, based on the feature maps output from multiple levels of the feature fusion pyramid, it is determined which parts of the feature maps contain text, and detection boxes are formed on the regions corresponding to the text. Region Proposal Networks (RPNs) can be used to determine detection boxes containing text. The RPN predicts multiple boxes based on the pixels of the feature maps output from layers 3 to 7 (P3-P7) of the feature fusion pyramid. The size of the boxes is determined by scale and ratio, and each pixel can correspond to multiple boxes of different sizes. Simultaneously, the RPN can perform binary classification (classifying between text and non-text) and bounding box position regression (the process of extracting the feature map corresponding to the target from the feature map corresponding to the entire image to be detected using bounding boxes based on the location region of the target). Based on the text in the binary classification results, all detection boxes containing text are selected, resulting in multiple detection boxes of different sizes that include text.

[0082] As an example, after obtaining the detection boxes including text, these detection boxes need to be sized uniformly to facilitate subsequent processing. This can be achieved by using bilinear interpolation to process the feature maps of detection boxes with different sizes, reducing errors and ensuring that the feature maps of each detection box output to the next processing stage have the same dimension.

[0083] In step 102, mask segmentation is performed based on the feature map of each detection box to obtain the text mask segmentation result.

[0084] As an example, masks can be used to extract regions corresponding to text; the result of text mask segmentation is a text image corresponding to the text. (Reference) Figure 4A , Figure 4A This is a schematic diagram of image processing for the text detection method provided in the embodiments of this application; Figure 4A The Chinese text is "First Text", and the text mask corresponding to this text is all the pixels corresponding to this text.

[0085] In some embodiments, step 102 can be implemented as follows: performing multiple levels of convolution operations based on the feature map of each detection box to obtain a text mask in each detection box; performing segmentation processing based on the feature map of each detection box and the corresponding text mask to obtain a text image in each detection box; and using each text image as a text mask segmentation result, wherein each text image includes the pixels of the corresponding text.

[0086] As an example, in a text mask, all image values ​​except for the regions corresponding to the text are 0. Segmentation can be achieved by multiplying the text mask with the feature map of the detection box. After multiplication, the image values ​​of all regions except the text region in the feature map become 0, and only the image values ​​of the regions corresponding to the text are retained. The resulting feature map is the text map.

[0087] In step 103, at least one type of text box prediction processing is performed based on the feature map of each detection box to obtain the text box recognition result.

[0088] In some embodiments, at least one text box prediction method in step 103 can be implemented by at least one step in steps 1031 to 1034, with each step corresponding to a text box prediction method.

[0089] As an example, when predicting text boxes, you can choose one or more of the following four prediction methods. The number of prediction methods used can be determined based on the degree of text tilt; the degree of text tilt is positively correlated with the number of prediction methods used. For example, if the text in the image has a low degree of tilt (e.g., tilt angle less than or equal to 15 degrees), then two prediction methods can be used; if the text in the image has a high degree of tilt (e.g., tilt angle greater than 15 degrees), then four prediction methods can be used.

[0090] In step 1031, the text image corresponding to the feature map of each detection box is obtained from the text mask segmentation result. The bounding rectangle fitting process is performed on each text image to obtain the rectangular text box in each detection box. Each rectangular text box is used as the first text box recognition result.

[0091] As an example, the sides of the circumscribed fitted rectangle can be parallel or not parallel to the sides of the detection box.

[0092] In some embodiments, step 1031 can be implemented as follows: determine the outline of the text image corresponding to the feature map of each detection box, and fit the outline to a bounding rectangle; select the smallest bounding rectangle corresponding to the feature map of each detection box as the rectangular text box corresponding to each detection box.

[0093] As an example, the requirement for fitting a rectangle is that, based on the outline of the text graph, multiple points on the outline lie on the boundary line of the circumscribed rectangle, while the outline cannot extend beyond the boundary line of the rectangle.

[0094] refer to Figure 4A , Figure 4A This is a schematic diagram of image processing for the text detection method provided in the embodiments of this application; Figure 4A It includes a detection box 401A and a rectangular text box 402A. The text content within the detection box 401A is "first text". Based on the principle of masking, the text image of "first text" is all the pixels corresponding to the first text. The rectangle corresponding to the rectangular text box 402A is the smallest bounding rectangle of the text image of "first text".

[0095] In step 1032, the corresponding offset point is determined based on the endpoints of the feature map of each detection box, and the offset text box in each detection box is determined based on the multiple offset points corresponding to each detection box. Each offset text box is used as the second text box recognition result.

[0096] Since the endpoints of the actual text box must be near the endpoints of the detection box, the position of the endpoints of the actual text box can be determined by predicting the offset between the endpoints of the detection box and the endpoints of the actual text box.

[0097] In some embodiments, step 1032 can be implemented by: determining the offset point corresponding to each endpoint of the feature map of each detection box, wherein the feature map of each detection box includes multiple endpoints; sequentially connecting all the offset points corresponding to each detection box to obtain the offset text box in each detection box.

[0098] As an example, a deep learning model can predict the offsets between the endpoints and offset points of the feature map of each detection box. Using the parameters corresponding to the feature map of each detection box, the positions of the endpoints of each detection box, and the offsets between the endpoints and offset points as training data, the deep learning model can learn the relationship between the endpoints and offsets of each detection box. The endpoints of the feature map of each detection box are input into the deep learning model, which outputs the offsets between the endpoints and offset points of each detection box. Adding the position of each endpoint of each detection box to its corresponding offset gives the position of the offset point. Connecting the offset points of each detection box sequentially yields the offset text box corresponding to that detection box.

[0099] Let's take a quadrilateral text box as an example for illustration. (Refer to...) Figure 4B , Figure 4B This is a schematic diagram of image processing for the text detection method provided in the embodiments of this application. Figure 4B The text box includes a detection box 401B, points B1, B2, B3, and B4, an offset text box 402B, points C1, C2, C3, and C4. The text within the detection box 401B is "Second Text." Points B1, B2, B3, and B4 are the four endpoints of the detection box 401B. Points C1, C2, C3, and C4 are offset points. The text box formed by connecting points C1, C2, C3, and C4 sequentially (each point is connected to the two points closest to it, not cross-connected) is the offset text box 402B. The offset points can be inside the detection box, at the edge of the detection box, outside the detection box, etc., for example: [reference needed]. Figure 4B Point C2 is located on the boundary of detection box 401B, while point C3 is located inside the boundary of detection box 401B.

[0100] As an example, the shape of the text arrangement in the image is irregular, such as the text arrangement shape being arc-shaped, circular, etc. In this case, the shape of the text box is not limited to a quadrilateral, and the number of offset points can be multiple. For example, two offset points are predicted based on each endpoint of the detection box, and a total of eight offset points are obtained from the four endpoints of the detection box. Connecting the eight offset points yields the text box.

[0101] In step 1033, multiple candidate coordinate component values ​​of the endpoints of the text box are determined based on the feature map of each detection box. The multiple candidate coordinate component values ​​are combined to obtain multiple candidate text boxes for each detection box. Regression processing is performed based on the multiple candidate text boxes to obtain candidate text boxes in each detection box. Each candidate text box is used as the third text box recognition result.

[0102] Here, the types of candidate coordinate component values ​​include: candidate x-axis component values ​​and candidate y-axis component values. The number of candidate x-axis component values ​​and candidate y-axis component values ​​is the same.

[0103] As an example, convolution operations on the feature map of each detection box can yield multiple prediction components, each of which is used to predict a candidate coordinate component value.

[0104] As an example, combining a candidate x-axis component value with a candidate y-axis component value yields a coordinate point. Combining multiple candidate x-axis component values ​​with candidate y-axis component values ​​yields multiple coordinate points. Connecting multiple coordinate points corresponding to the same combination sequentially yields candidate text boxes. Regression processing is then performed based on the position of each candidate text box to obtain the candidate text boxes.

[0105] As an example, a sample set is generated based on multiple candidate text boxes. Each candidate text box is a sample, and each sample has a certain probability of becoming a final candidate text box. The probability of each candidate text box becoming a candidate text box is determined based on the feature map of each detection box and the normalization function (Softmax). The candidate text box with the highest probability is selected as the final candidate text box.

[0106] refer to Figure 3C , Figure 3C This is a schematic flowchart of the text detection method for images provided in the embodiments of this application; in some embodiments, step 1033 can be implemented by steps 301C to 304C.

[0107] In step 301C, multiple levels of convolution operations are performed based on the feature map of each detection box to obtain multiple prediction vectors.

[0108] As an example, multi-level convolution operations can be convolution operations that generate dual-channel feature maps, with each channel corresponding to a different type of prediction vector.

[0109] In some embodiments, step 301C can be implemented in the following manner: performing a convolution operation based on the feature map of each detection box to obtain a first channel feature map and a second channel feature map of each detection box; performing a convolution operation based on the first channel feature map of each detection box to obtain multiple prediction vectors corresponding to the first channel feature map; and performing a convolution operation based on the second channel feature map of each detection box to obtain multiple prediction vectors corresponding to the second channel feature map.

[0110] Here, the number of prediction vectors corresponding to the first channel feature map is the same as the number of prediction vectors corresponding to the second channel feature map.

[0111] As an example, the feature map of each detection box is multiplied twice by the same convolutional kernel to obtain the feature maps for the two channels. To generate candidate coordinate component values ​​corresponding to different coordinate axes for each channel, the feature map of each channel can be convolved with different convolutional kernels. For example, the feature map of each channel is N*N (N can be a common feature map size such as 28, 56, etc.). Convolving the feature map of one channel with a 1*N convolutional kernel generates multiple N-dimensional prediction vectors corresponding to the candidate horizontal coordinate component values; convolving the feature map of the other channel with an N*1 convolutional kernel generates multiple N-dimensional prediction vectors corresponding to the candidate vertical coordinate component values.

[0112] Taking a quadrilateral text box as an example, the number of prediction vectors can be 8, of which four prediction vectors correspond to candidate horizontal coordinate component values ​​and the other four prediction vectors correspond to candidate vertical coordinate component values.

[0113] In step 302C, regression processing is performed on each prediction vector to obtain the candidate coordinate component values ​​of multiple endpoints of the candidate text boxes in each detection box.

[0114] In some embodiments, step 302C can be implemented as follows: performing regression processing based on multiple prediction vectors corresponding to the first channel feature map of each detection box to obtain candidate horizontal coordinate component values ​​of multiple endpoints of candidate text boxes in each detection box; performing regression processing based on multiple prediction vectors corresponding to the second channel feature map of each detection box to obtain candidate vertical coordinate component values ​​of multiple endpoints of candidate text boxes in each detection box.

[0115] As an example, to facilitate regression processing of the predicted vectors, one-hot encoding can be used to represent multiple predicted vectors corresponding to the feature map of each channel as binary vectors (mapping classification values ​​to integer values, with each integer value represented in binary) for unsupervised learning. Regression processing can be implemented as follows: Regression processing is performed on each predicted vector corresponding to the first channel feature map (represented in binary) in the learning result using a normalized activation function to obtain candidate x-axis component values. Similarly, regression processing is performed on each predicted vector corresponding to the second channel feature map (represented in binary) in the learning result using a normalized activation function to obtain candidate y-axis component values.

[0116] As an example, each prediction vector is multidimensional, and since all prediction vectors have the same dimension, a regression process can be performed on the prediction vectors using a normalization function (Softmax) to obtain the candidate coordinate component values ​​corresponding to each prediction vector. The regression process is implemented as follows: Each prediction vector is mapped to a probability distribution of the same dimension using the normalization function (Softmax), with each dimension corresponding to a probability value ranging from (0, 1). The dimension with the highest probability value is selected as the target dimension, and the value corresponding to the target dimension is used as the candidate coordinate component value of the prediction vector.

[0117] refer to Figure 4C , Figure 4C This is a schematic diagram of image processing for the text detection method provided in the embodiments of this application; Figure 4C The detection box 401C contains the text "Third Text". Y=n1, Y=n2, Y=n3, and Y=n4 are the lines corresponding to the four candidate ordinate component values. X=m1, X=m2, X=m3, and X=m4 are the lines corresponding to the four candidate abscissa component values.

[0118] In step 303C, based on the coordinate axis corresponding to each candidate coordinate component value, the multiple candidate coordinate component values ​​corresponding to each detection box are combined to obtain multiple candidate points corresponding to each detection box.

[0119] As an example, combining the x-coordinate and y-coordinate values ​​yields the coordinates of the point. For further explanation, please refer to [link / reference needed]. Figure 4C The eight straight lines corresponding to the candidate coordinate component values ​​form sixteen intersection points, and these sixteen intersection points are the candidate points.

[0120] In step 304C, multiple candidate points corresponding to each detection box are connected and combined to obtain multiple connection combinations for each detection box, and the box corresponding to each connection combination is used as a text box.

[0121] Here, each connection combination includes four candidate points, and the coordinate values ​​of the four candidate points are different.

[0122] As an example, given four available x-coordinates and four available y-coordinates, to obtain four different coordinate points (each with a different numerical value), 24 (4*3*2*1=24) different combinations of four points can be generated. Connecting the four points in each combination sequentially will generate a candidate text box.

[0123] As an example, Figure 4C In the diagram, points D1, D2, D3, and D4 are candidate points. Connecting these four candidate points sequentially forms candidate text box 402C. The coordinates of the four endpoints of candidate text box 402C are different. Based on Figure 4C In addition to the candidate text box 402C, there are 23 other candidate text boxes among the eight alternative coordinate component values.

[0124] As an example, when there are four candidate ordinate components or four candidate abscissa components, at least two component values ​​can be identical. For example, in Y=n1, Y=n2, Y=n3, Y=n4, n1=n2, n3=n4. In this case, the resulting candidate text box can be a quadrilateral with two parallel sides. Another example: in Y=n1, Y=n2, Y=n3, Y=n4, n1=n2, n3=n4; and in X=m1, X=m2, X=m3, X=m4, m1=m2, m3=m4. In this case, the candidate text boxes for all 24 combinations are identical, and the candidate text box formed by connecting the candidate points can be a rectangle. Each candidate text box can serve as a candidate text box.

[0125] Continue to refer to Figure 3B In step 1034, the feature map of each detection box is classified to obtain detection boxes including text, and regression processing of the detection box position is performed based on the feature map of each detection box including text to obtain the corrected text box in each detection box. Each corrected text box is used as the fourth text box recognition result.

[0126] As an example, a convolutional neural network can be used to classify the feature map of each detection box. The feature map of each detection box is used as sample data to train the neural network, labeled as either including or excluding text, or including text (where the text category includes multiple text types) or excluding text. The feature map of each detection box is then input into the convolutional neural network model, causing the model to output the category corresponding to the feature map of each detection box.

[0127] As an example, the classification of the feature map of each detection box can be a binary classification of text and non-text, that is, distinguishing between detection boxes containing text and those without text. For example, for multiple detection boxes, some detection boxes do not contain text in their feature maps, while others do. Binary classification is used to distinguish the feature maps of detection boxes containing text from those not containing text. The detection boxes are then corrected in position using bounding box regression to obtain corrected detection boxes. These corrected detection boxes include text and can be considered as text boxes. In this embodiment, such text boxes are named "corrected text boxes" for easy distinction from text boxes predicted by other methods.

[0128] As an example, the classification of the feature map of each detection box can be fine-grained, that is, distinguishing between text and non-text, and between different types of text. This yields detection boxes that include text and the category labels of the text within those boxes. For instance, given multiple detection boxes, some may have text in their feature maps, while others may not. Fine-grained classification distinguishes between the feature maps of detection boxes containing text and those not containing text. Simultaneously, the detection boxes containing text are labeled according to their respective categories. Information fusion based on the corrected text boxes, the text category labels, and the text mask yields a single-instance text segmentation result.

[0129] As an example, regression processing of the detection box position based on the feature map of each detection box containing text can be implemented using a convolutional neural network model for predictive regression. The model is trained using the positions of the text in the feature map of the detection box and the positions of the detection boxes in the feature map of the text-containing detection boxes as training data, enabling the model to acquire the mapping relationship between the two. The feature map of each text-containing detection box is input into the convolutional neural network model, and the output is the value that each text-containing detection box's feature map needs to be translated or scaled relative to its current position. The feature map of each text-containing detection box is then adjusted for translation and scaling based on the corresponding values, resulting in the corrected detection box.

[0130] Continue to refer to Figure 3A In step 104, the actual text boxes in the image are determined based on the overlap rate between the text mask segmentation results and the text box recognition results.

[0131] In some embodiments, step 104 can be implemented by: determining the overlap rate between the text image recorded for each detection box in the text mask segmentation result and at least one text box recorded for each detection box in the text box recognition result; and selecting the text box with the maximum overlap rate as the actual text box in the image.

[0132] As an example, the overlap ratio, also known as the intersection-over-union (IOU), determines the overlapping portion between the text mask and the text box, and the union portion between the text mask and the text box. The overlap ratio is obtained by dividing the area of ​​the overlapping portion by the area of ​​the union portion. Since at least one prediction method is used in the embodiments of this application, multiple overlap ratios can be obtained, and the text box with the highest overlap ratio is taken as the actual text box.

[0133] As an example, a single image may contain multiple text segments. Different types of text have different degrees of tilt, magnification, or reduction, and different text types are suitable for different text box prediction methods. The text boxes corresponding to different text segments in the same image can be derived from different prediction methods.

[0134] In some embodiments, the text box prediction method and the corresponding text tilt can be recorded as samples. Training is then performed based on these samples to obtain the mapping relationship between the text box prediction method and the text tilt. When detecting text, the current tilt of the text is first determined. Based on the current tilt of the text and the mapping relationship between the text box prediction method and the text tilt, a suitable text box prediction method is determined, reducing unnecessary computation and improving the efficiency of text box prediction.

[0135] In some embodiments, based on the actual text box obtained from the embodiments of this application, the character box corresponding to each character in the text box can also be determined, for example: based on the actual text box... Figure 4C The text "third text" is segmented character by character to obtain character boxes, which improves the accuracy of text recognition.

[0136] This application sets up multiple text box prediction methods. Based on the overlap rate of text mask segmentation results and multiple text box recognition results, the actual text boxes in the image are determined. By using multiple methods to predict text boxes, the accuracy and efficiency of text box recognition in the image are effectively improved. This is beneficial for providing a good foundation for optical character recognition and for helping with point of interest recognition and road data recognition.

[0137] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0138] The embodiments of this application can have the following application scenarios, for example: for the collection of road data and identification of points of interest (POIs) in maps, in road data-related services, high-precision text detection of road name signs requires high text recognition accuracy; in POI-related services, text detection is required for aspects such as POI automation, POI misuse, POI expiration, and bus stop name recognition.

[0139] refer to Figure 5 , Figure 5 This is a schematic flowchart of the text detection method for images provided in the embodiments of this application; Figure 5 This includes steps 501 to 504. (See reference) Figure 6A , Figure 6A This is a schematic diagram of the structure of a neural network model for recognizing text boxes provided in an embodiment of this application; the following is based on... Figure 5 The steps in and combined Figure 6A The neural network model shown is explained below.

[0140] As an example, see reference Figure 6A , Figure 6AThe neural network models shown include a backbone network model 601, a Region Proposal Network (RPN) model 602, a single instance segmentation network 604, and a key edge prediction network model 603. The backbone network model 601 includes a residual network model and a Feature Pyramid Network (FPN) model, which is used to extract feature maps from the image to be detected. The Region Proposal Network model 602 uses 3x3 convolution kernels to convolve the feature maps of the image to be detected, and generates two-channel feature maps of the image to be detected using 1x1 convolution kernels. Binary classification (Softmax, normalized exponential function) and bounding box regression (BBox Reg) are performed on the feature maps of the two channels respectively. The outputs of the two channels are then fused to obtain the recommended region, i.e., the feature map of the detection box including text. The region recommendation network model 602 performs region-of-interest (ROI-Align) on the recommended regions and outputs the generated feature maps to the single-instance segmentation network 604. The single-instance segmentation network 604 includes multiple branches: First, a text mask is obtained by performing multi-level convolutional operations on the feature map through a head convolutional layer and fully convolutional layers. A bounding rectangle is fitted to the text mask to generate rectangular text boxes, and the rectangular text box corresponding to each detection box is output 1. Second, the feature map is processed through a fully connected layer to generate a dual-channel feature map. Bounding Box Regression is performed on one channel of the feature map to correct the detection box. A normalized exponential function (Softmax) is used to perform fine text classification (distinguishing between text categories), and the corrected detection box and the fine text classification result are output 4. Third, text box prediction is performed based on the endpoints of the feature map of the detection box using a four-keypoint regression method, and the predicted text box is output 2. Fourth, text box prediction is performed on the feature map of the detection box through a key-edge prediction network model 603, and the predicted text box is output 3. The optimal text box is determined by fusing these four outputs.

[0141] In step 501, feature extraction is performed on the image to be detected to obtain a feature map of the image to be detected.

[0142] As an example, step 501 is implemented through a backbone network model 601. After receiving the image to be detected, the backbone network model 601 performs a series of downsampling operations on the image to be detected to obtain the feature map of the image to be detected. In this embodiment, the backbone network model 601 adopts a 101-layer residual network (ResNet101) structure. The 101-layer residual network consists of multiple identity blocks and convolutional blocks. The convolutional blocks consist of multiple convolutional layers, batch normalization layers, and activation (RELU, Rectified Linear Unit) layers, used for feature extraction. Identity mapping and shortcut mechanisms are added to the identity residual blocks. These mechanisms do not generate additional parameters or increase computational complexity, and can ensure effective gradient backpropagation, so that gradients do not vanish during the training of the deep learning network.

[0143] The backbone network model 601 can be divided into five stages: C1, C2, C3, C4, and C5. In stages C1 and C2, no parameters are updated and no feature fusion is performed. In stages C1 and C2, the lower convolutional layers are responsible for extracting basic features such as image edges and textures, while the higher convolutional layers are responsible for combining and abstracting the texture features extracted from the lower layers. Normalization layers normalize the features to a normal distribution. Activation layers perform non-linear mapping on the extracted features, enhancing the model's generalization ability. Stages C3, C4, and C5 output the features, resulting in... Figure 6A The feature map output by the backbone network model 601.

[0144] As an example, the backbone network model 601 also incorporates a feature pyramid network model for multi-scale feature fusion. The feature pyramid network model is used to inversely fuse feature maps from high to low levels, fully utilizing the features extracted at each stage. Based on the feature pyramid mechanism, the feature map output from stage C5 is first extracted as the feature map of layer P5 (the fifth layer of the pyramid). A convolution operation with a stride of 2 is then performed on layer P5 to generate layers P6 and P7. Simultaneously, upsampling is performed from top to bottom: the feature map of layer P5 is upsampled by 2 and added to the feature map output from stage C4 to obtain the feature map of layer P4; the feature map of layer P4 is upsampled by 2 and added to the feature map of stage C3 to obtain the feature map of layer P3. Upsampling operations are performed on the feature maps of layers P3, P4, and P5 to achieve scale amplification, providing necessary feature information for higher-level upsampling fusion. The input to each upsampling module comes not only from the output of the previous upsampling module, but also from a feature map of the same size output by the backbone network model 601. To better integrate feature information, the feature pyramid network model sums the two types of features and performs convolution operations to achieve information fusion.

[0145] As an example, layers P3-P5 in the Feature Pyramid Network model all output feature maps. Layer P7 has the largest single-pixel receptive field (the size of the region mapped on the original image by a pixel in the feature map output by each layer of the convolutional neural network), while layer P3's single-pixel receptive field is 32 times smaller than that of layer P7. Based on the size of the single-pixel receptive field corresponding to the feature map output by each layer, the feature maps output by each layer in the Feature Pyramid model are used to predict instances of different sizes; a larger single-pixel receptive field allows for the prediction of larger instances.

[0146] In step 502, multiple detection boxes are generated based on the feature map of the image to be detected.

[0147] As an example, step 502 is implemented through a region recommendation network 602, which receives feature maps of various sizes output from layers P3-P7 of the pyramid network model in the backbone network model 601, and generates text detection boxes based on the feature maps.

[0148] As an example, the region recommendation network 602 in this embodiment recommends regions corresponding to text by generating a series of bounding boxes on the pixels of the feature map. The size of each box is determined by two parameters: the scale and the ratio of the feature map. Each pixel can correspond to multiple boxes of different sizes. The region recommendation network 602 is a binary classification network (used to classify text and non-text), and it can also perform preliminary determination of detection boxes. Based on the region recommendation network 602, multiple rectangular detection boxes including text can be generated.

[0149] As an example, multiple detection boxes containing text rectangles have different sizes. The size of the multiple detection boxes containing text rectangles is unified by the Interest Point Align (ROI Align) mechanism. ROI Align uses bilinear interpolation to obtain the pixel values ​​of four fixed point coordinates, thereby making discontinuous operations continuous. When returning to the original image, the error is smaller, while ensuring the consistency of the output feature dimensions.

[0150] In step 503, text mask segmentation and text box prediction in various ways are performed based on the feature map corresponding to each detection box.

[0151] As an example, step 503 can be implemented by a single-instance segmentation network 604, which includes four main branches: a box fine classification and location regression branch, a mask fitting branch, a four-keypoint regression branch, and a key edge prediction branch.

[0152] As an example, the detection box fine-classification and location regression branches correct the position of the detection boxes by performing bounding box regression, and then perform fine-classification of the text within the detection boxes using a normalization function to obtain the corresponding category label for the text. Because the feature pyramid network model can obtain a lot of useful information from features at different scales, fewer filters can be used during classification.

[0153] As an example, the mask fitting branch performs multiple convolution operations on the feature map of the cropped detection box output by the region recommendation network 602. For example, the feature map of the cropped detection box has a size of 14*14*256. The feature map is subjected to five identical convolution operations through a fully convolutional network model (FCN) to output a text mask corresponding to the text in the feature map with a size of 28*28. The minimum bounding rectangle is fitted based on the text mask, and the minimum bounding rectangle is used as a prediction result of the actual text box.

[0154] As an example, each detection box's feature map corresponds to a piece of text. Performing semantic segmentation on the feature maps of the detection boxes is essentially instance segmentation. Since most text is rectangular or approximately rectangular, the text mask output by the mask fitting branch more accurately reflects the text shape compared to other branches.

[0155] As an example, the four-keypoint regression branch uses keypoints as the endpoints of the actual text boxes. For a detection box containing text, the four keypoints must be located near the four endpoints of the detection box. This branch performs offset regression based on the four endpoints of each detection box to predict the offset values ​​of the actual text box endpoints relative to the endpoints of the detection boxes. That is, the x and y values ​​of the offset between each keypoint and the corresponding endpoint of the detection box. Adding the coordinate values ​​of the detection box endpoints to the offset values ​​yields the coordinate values ​​of the keypoints. Connecting the four keypoints sequentially forms a text box, which serves as one prediction result for the actual text box.

[0156] As an example, see reference Figure 6B , Figure 6B This is a schematic diagram of the key edge prediction network model in the neural network model provided in the embodiments of this application; Figure 6C This is a schematic diagram of image processing for the critical edge prediction network model provided in the embodiments of this application.

[0157] As an example, in this embodiment of the application, the feature map of the detection box output by the region recommendation network 602 is 28*28, and the key edge prediction network model 603 performs deconvolution on the feature map of the detection box to obtain a 56*56 feature map, which is used for illustration. (Reference) Figure 6BA 1x1 convolution kernel is used to transform a 56x56 feature map into a two-channel feature map. For one channel of the feature map, a 56x1 convolution kernel is used to generate four 56-dimensional prediction vectors (corresponding to y in the figure). min y2, y3, y max This is used to predict the four values ​​of y from the largest to the smallest. For the feature map of the other channel, a 1*56 convolutional kernel is used to generate four 56-dimensional prediction vectors (corresponding to x in the image). min x2, x3, x max This is used to predict four values ​​of x, from the largest to the smallest. One-hot encoding can be used to represent the prediction vector as a binary vector (mapping the classification value to integer values, with each integer value represented in binary) for unsupervised learning. Based on the learning results and the normalized activation function (Softmax), the most likely position value of each key edge is inferred, predicting four key edges corresponding to x and four key edges corresponding to y. The key edges are the lines containing the x or y coordinates of the text box corners. Here, learning only one x or y component is much easier than directly learning the position of a point (which includes both x and y components).

[0158] As an example, see reference Figure 6C , Figure 6C This is a schematic diagram of image processing for the critical edge prediction network model provided in the embodiments of this application. Figure 6C In the diagram, the dashed box represents detection box 601C, containing the text "noodle shop". The dashed lines corresponding to X5, X6, X7, and X8 are the four conditional key edges corresponding to the X values. The lines corresponding to Y5, Y6, Y7, and Y8 are the four conditional key edges corresponding to the Y values. These eight key edges form 16 intersection points, which are the candidate endpoints for the text box.

[0159] As an example, see reference Figure 6B Based on key edges, a feature map of the key edges is obtained as the basic feature. Then, a convolution operation is performed on this feature map using a 1*1 convolution kernel and a 56*56 convolution kernel to obtain 4 x values ​​and 4 y values ​​corresponding to 24 connection combinations (4*3*2*1=24). (Continue to refer to...) Figure 6C Text box 602C corresponds to one of the 24 connection combinations. The X and Y values ​​corresponding to the four endpoints of text box 602C include all values ​​from X5, X6, X7, X8, Y5, Y6, Y7, and Y8, but the X and Y values ​​corresponding to each endpoint are different. There are 23 other text boxes similar to text box 602C. The 24 connection combinations are normalized using a normalization function (Softmax) to obtain the final text box connection combination. The text corresponding to this combination is a prediction result of the actual text box, that is... Figure 6A Output 3 in the code.

[0160] In step 504, the actual text box is determined based on the text mask corresponding to each detection box and the prediction results of multiple text boxes.

[0161] As an example, step 504 can be implemented as follows: summarize the various text box prediction results output by the single instance segmentation network 604, calculate the intersection-over-union ratio (IOU) between the text box and the text mask corresponding to each result, and the text box corresponding to the text box prediction result with the highest IOU is the optimal text box, which is then used as the final text box.

[0162] As an example, in most cases, the output of the four-keypoint regression branch or the key-edge prediction branch is optimal. If the text in the image to be detected is too small, too long, or excessively skewed, the above two branches may fail, and the output of the mask fitting branch may be optimal.

[0163] As an example, the image text detection method based on embodiments of this application can provide a solid foundation for Optical Character Recognition (OCR). Optical character recognition technology is significant for the differentiation and updating of road signs in road data, and greatly assists in services such as linking and handling expiration in Points of Interest (POI) businesses. Efficient optical character recognition can significantly reduce manual workload and save operating costs. It also quickly maintains the freshness of map data, providing users with a better experience.

[0164] As an example, the neural network model provided in the embodiments of this application can simultaneously train multiple branches of the network, simultaneously optimize the loss function, fuse multiple results, complement each other's effects, improve the robustness and accuracy of text detection, and generate high-precision and compact text boxes.

[0165] The following continues to describe the exemplary structure of the image text detection device 455 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2AAs shown, the software modules in the text detection device 455 storing the image in the memory 450 may include: a feature extraction module 4551, configured to identify multiple detection boxes from an image including text and obtain a feature map of each detection box; a mask segmentation module 4552, configured to perform mask segmentation processing based on the feature map of each detection box to obtain a text mask segmentation result; a text box prediction module 4553, configured to perform text box prediction processing in at least one manner based on the feature map of each detection box to obtain a text box recognition result; and a text box determination module 4554, configured to determine the actual text boxes in the image based on the overlap rate of the text mask segmentation result and the text box recognition result.

[0166] In some embodiments, the feature extraction module 4551 is further configured to perform multi-level downsampling on an image including text, wherein the downsampled feature map obtained from each level of downsampling except the last level is used as input for the next level of downsampling; perform multi-level upsampling based on the downsampled feature map of the last level among the multiple levels, wherein the upsampled feature map obtained from each level of upsampling except the last level is used to fuse with a downsampled feature map of the same size as input for the next level of upsampling; perform binary classification and bounding box regression based on the upsampled feature map of each level to obtain multiple detection boxes including text; and perform size unification processing on the upsampled feature map within each detection box to obtain a processed feature map of each detection box.

[0167] In some embodiments, the mask segmentation module 4552 is further configured to perform multiple levels of convolution operations based on the feature map of each detection box to obtain a text mask in each detection box; perform segmentation processing based on the feature map of each detection box and the corresponding text mask to obtain a text image in each detection box; and use each text image as a text mask segmentation result, wherein each text image includes pixels of the corresponding text.

[0168] In some embodiments, the text box prediction module 4553 is further configured to perform at least one of the following text box prediction methods: obtaining the text map corresponding to the feature map of each detection box from the text mask segmentation result; performing bounding rectangle fitting processing on each text map to obtain a rectangular text box in each detection box; and using each rectangular text box as a first text box recognition result; determining the corresponding offset point based on the endpoints of the feature map of each detection box; determining the offset text box in each detection box based on multiple offset points corresponding to each detection box; and using each offset text box as a second text box recognition result; determining multiple candidate coordinate component values ​​of the endpoints of the text box based on the feature map of each detection box; combining the multiple candidate coordinate component values ​​to obtain multiple candidate text boxes for each detection box; performing regression processing on the multiple candidate text boxes to obtain candidate text boxes in each detection box; and using each candidate text box as a third text box recognition result; classifying the feature map of each detection box to obtain detection boxes including text; and performing detection box position regression processing on the feature map of each detection box including text to obtain corrected text boxes in each detection box; and using each corrected text box as a fourth text box recognition result.

[0169] In some embodiments, the text box prediction module 4553 is further configured to determine the outline of the text map corresponding to the feature map of each detection box, and fit a bounding rectangle to the outline; and select the smallest bounding rectangle corresponding to the feature map of each detection box as the rectangular text box corresponding to each detection box.

[0170] In some embodiments, the text box prediction module 4553 is further configured to determine the offset point corresponding to each endpoint of the feature map of each detection box, wherein the feature map of each detection box includes multiple endpoints; and sequentially connect all the offset points corresponding to each detection box to obtain the offset text box in each detection box.

[0171] In some embodiments, the text box prediction module 4553 is further configured to perform multiple levels of convolution operations based on the feature map of each detection box to obtain multiple prediction vectors; and to perform regression processing on each prediction vector to obtain candidate coordinate component values ​​of multiple endpoints of the candidate text box in each detection box.

[0172] In some embodiments, the text box prediction module 4553 is further configured to perform a convolution operation based on the feature map of each detection box to obtain a first channel feature map and a second channel feature map of each detection box; perform a convolution operation based on the first channel feature map of each detection box to generate multiple prediction vectors corresponding to the first channel feature map; and perform a convolution operation based on the second channel feature map of each detection box to generate multiple prediction vectors corresponding to the second channel feature map; wherein the number of prediction vectors corresponding to the first channel feature map is the same as the number of prediction vectors corresponding to the second channel feature map.

[0173] In some embodiments, the types of candidate coordinate component values ​​include: candidate horizontal coordinate component values ​​and candidate vertical coordinate component values. The text box prediction module 4553 is further configured to perform regression processing based on multiple prediction vectors corresponding to the first channel feature map of each detection box to obtain candidate horizontal coordinate component values ​​for multiple endpoints of candidate text boxes in each detection box; and to perform regression processing based on multiple prediction vectors corresponding to the second channel feature map of each detection box to obtain candidate vertical coordinate component values ​​for multiple endpoints of candidate text boxes in each detection box.

[0174] In some embodiments, the text box prediction module 4553 is further configured to combine multiple candidate coordinate component values ​​corresponding to each detection box according to the coordinate axis corresponding to each candidate coordinate component value to obtain multiple candidate points corresponding to each detection box; connect and combine the multiple candidate points corresponding to each detection box to obtain multiple connection combinations corresponding to each detection box, and use the box corresponding to each connection combination as a text box, wherein each connection combination includes four candidate points, and the coordinate values ​​of the four candidate points are different.

[0175] In some embodiments, the text box determination module 4554 is further configured to determine the overlap rate between the text image recorded for each detection box in the text mask segmentation result and at least one text box recorded for each detection box in the text box recognition result; and select the text box with the maximum overlap rate as the actual text box in the image.

[0176] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image text detection method described in this application embodiment.

[0177] This application provides a computer-readable storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to execute the image text detection method provided in this application. For example, ... Figure 3A The text detection method shown in the image.

[0178] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0179] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0180] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0181] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0182] In summary, by setting up multiple text box prediction methods through the embodiments of this application, and determining the actual text boxes in the image based on the overlap rate of the text mask segmentation results and multiple text box recognition results, the accuracy and efficiency of text box recognition in the image are effectively improved. This is beneficial for providing a good foundation for optical character recognition and for helping with point of interest recognition and road data recognition.

[0183] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A text detection method of an image, characterized by, The method comprises: identifying a plurality of detection boxes from an image comprising text, and obtaining a feature map of each detection box; performing mask segmentation processing based on the feature map of each detection box to obtain a text mask segmentation result; performing at least one of the following text box prediction modes to obtain a text box recognition result: determining a plurality of candidate coordinate component values of a text box endpoint based on the feature map of each detection box, combining the plurality of candidate coordinate component values to obtain a plurality of candidate text boxes of each detection box, performing regression processing based on the plurality of candidate text boxes to obtain a candidate text box in each detection box, and taking each candidate text box as a third text box recognition result; classifying the feature map of each detection box to obtain a detection box comprising text, performing detection box position regression processing based on the feature map of each detection box comprising text to obtain a corrected text box in each detection box, and taking each corrected text box as a fourth text box recognition result; determining an actual text box in the image based on an overlap rate of the text mask segmentation result and the text box recognition result.

2. The method of claim 1, wherein, The method comprises: performing multiple levels of down-sampling on the image comprising text, wherein the down-sampled feature map obtained by each level of down-sampling except the last level is used as the input of the next level of down-sampling; performing multiple levels of up-sampling based on the down-sampled feature map of the last level in the multiple levels, wherein the up-sampled feature map obtained by each level of up-sampling except the last level is used to fuse with a down-sampled feature map of the same size as the input of the next level of up-sampling; performing binary classification of whether the text is included and boundary box regression based on the up-sampled feature map of each level to obtain a plurality of detection boxes comprising text; performing size normalization processing on the up-sampled feature map in each detection box to obtain the feature map of each processed detection box.

3. The method of claim 1, wherein, The method comprises: performing multiple levels of convolution operation based on the feature map of each detection box to obtain a text mask in each detection box; performing segmentation processing based on the feature map of each detection box and the corresponding text mask to obtain a text image in each detection box, and taking each text image as a text mask segmentation result, wherein each text image comprises pixels of the corresponding text.

4. The method of claim 1, wherein, The method further comprises: obtaining a text image corresponding to the feature map of each detection box from the text mask segmentation result, and performing a bounding rectangle fitting process on each text image to obtain a rectangular text box in each detection box, and taking each rectangular text box as a first text box recognition result; determining a corresponding offset point based on the endpoint of the feature map of each detection box, determining an offset text box in each detection box based on a plurality of offset points corresponding to each detection box, and taking each offset text box as a second text box recognition result.

5. The method of claim 4, wherein, The fitting processing of the outer rectangle of each text image is performed to obtain a rectangular text box in each detection box, including: determining the contour line of the text image corresponding to the feature map of each detection box, and fitting an outer rectangle to the contour line; selecting the minimum outer rectangle box corresponding to the feature map of each detection box as the rectangular text box corresponding to each detection box.

6. The method of claim 4, wherein, The offset points corresponding to the end points of the feature map of each detection box are determined, and the offset text box in each detection box is determined based on the plurality of offset points corresponding to each detection box, including: determining the offset point corresponding to each end point of the feature map of each detection box, wherein the feature map of each detection box includes a plurality of end points; sequentially connecting all the offset points corresponding to each detection box to obtain the offset text box in each detection box.

7. The method of claim 4, wherein, The plurality of alternative coordinate component values of the text box end points are determined based on the feature map of each detection box, including: performing a plurality of levels of convolution operation based on the feature map of each detection box to obtain a plurality of prediction vectors; performing regression processing on each prediction vector to obtain the alternative coordinate component values of the plurality of end points of the candidate text box in each detection box.

8. The method of claim 7, wherein, The plurality of levels of convolution operation based on the feature map of each detection box to obtain a plurality of prediction vectors, including: performing convolution operation based on the feature map of each detection box to obtain a first channel feature map and a second channel feature map of each detection box; performing convolution operation based on the first channel feature map of each detection box to obtain a plurality of prediction vectors corresponding to the first channel feature map; performing convolution operation based on the second channel feature map of each detection box to obtain a plurality of prediction vectors corresponding to the second channel feature map; wherein the number of prediction vectors corresponding to the first channel feature map is the same as the number of prediction vectors corresponding to the second channel feature map; the type of the alternative coordinate component value includes an alternative horizontal coordinate component value and an alternative vertical coordinate component value; the regression processing on each prediction vector to obtain the alternative coordinate component values of the plurality of end points of the candidate text box in each detection box, including: performing regression processing on the plurality of prediction vectors corresponding to the first channel feature map of each detection box to obtain the alternative horizontal coordinate component values of the plurality of end points of the candidate text box in each detection box; performing regression processing on the plurality of prediction vectors corresponding to the second channel feature map of each detection box to obtain the alternative vertical coordinate component values of the plurality of end points of the candidate text box in each detection box.

9. The method of claim 4, wherein, The combination of the plurality of alternative coordinate component values to obtain a plurality of alternative text boxes of each detection box, including: combining the plurality of alternative coordinate component values corresponding to each detection box according to the coordinate axis corresponding to each alternative coordinate component value to obtain a plurality of alternative points corresponding to each detection box; connecting and combining the plurality of alternative points corresponding to each detection box to obtain a plurality of connection combinations corresponding to each detection box, and taking the box corresponding to each connection combination as a text box, wherein each connection combination includes four alternative points, and the coordinate values of the four alternative points are different.

10. The method according to any one of claims 1 to 9, characterized in that, The actual text box in the image is determined based on an overlapping rate of the text mask segmentation result and the text box recognition result, and the method comprises the following steps: determining an overlapping rate of the text graph recorded in the text mask segmentation result for each detection box and at least one text box recorded in the text box recognition result for each detection box; selecting a text box with the maximum overlapping rate as the actual text box in the image.

11. An apparatus for text detection of an image, characterized by comprising: The device comprises: a feature extraction module configured to identify a plurality of detection boxes from an image comprising text, and obtain a feature map of each detection box; a mask segmentation module configured to perform mask segmentation processing based on the feature map of each detection box to obtain a text mask segmentation result; a text box prediction module configured to perform at least one of the following text box prediction modes to obtain a text box recognition result: determining a plurality of candidate coordinate component values of text box endpoints based on the feature map of each detection box, combining the plurality of candidate coordinate component values to obtain a plurality of candidate text boxes for each detection box, performing regression processing based on the plurality of candidate text boxes to obtain a candidate text box in each detection box, and taking each candidate text box as a third text box recognition result; performing classification on the feature map of each detection box to obtain a detection box comprising text, and performing detection box position regression processing based on the feature map of each detection box comprising text to obtain a revised text box in each detection box, and taking each revised text box as a fourth text box recognition result; a text box determination module configured to determine an actual text box in the image based on an overlapping rate of the text mask segmentation result and the text box recognition result.

12. The device of claim 11, wherein: the feature extraction module is further configured to perform a plurality of levels of down-sampling on the image comprising text, wherein the down-sampled feature map obtained by each level of down-sampling, except for the last level, is used as the input for the next level of down-sampling; performing a plurality of levels of up-sampling based on the down-sampled feature map of the last level in the plurality of levels, wherein the up-sampled feature map obtained by each level of up-sampling, except for the last level, is used to fuse with the down-sampled feature map of the same size as the input for the next level of up-sampling; performing binary classification of whether to include text and boundary box regression based on the up-sampled feature map of each level to obtain a plurality of detection boxes comprising text; performing size normalization processing on the up-sampled feature map in each detection box to obtain the feature map of each processed detection box.

13. The device of claim 11, wherein: the mask segmentation module is further configured to perform a plurality of levels of convolution operation based on the feature map of each detection box to obtain a text mask in each detection box; performing segmentation processing based on the feature map of each detection box and the corresponding text mask to obtain a text graph in each detection box, and taking each text graph as a text mask segmentation result, wherein each text graph comprises the pixels of the corresponding text.

14. A terminal device, comprising: comprises: a memory for storing executable instructions; A processor for implementing the text detection method of the image according to any one of claims 1 to 10 when executing executable instructions stored in the memory.

15. A computer-readable storage medium, characterized in that, A memory storing executable instructions for implementing the text detection method of the image according to any one of claims 1 to 10 when executed by a processor.

16. A computer program product comprising computer-executable instructions or a computer program, characterized in that, The computer executable instructions or the computer program implement the text detection method of the image according to any one of claims 1 to 10 when executed by a processor.

Images