Font detection method, device, equipment and medium

Abstract

The application provides a font detection method, device and equipment and medium. The font detection method comprises: determining a region image containing characters and the characters in the region image on a to-be-detected image; performing image classification on the region image to obtain a candidate font corresponding to the region image; performing target segmentation on the region image to obtain a character image corresponding to the region image; and determining a target font to which the characters in the region image belong according to the characters in the region image, the candidate font and the character image. Thus, in the font detection process, the user only needs to provide the to-be-detected image, and the font of the characters in the to-be-detected image can be obtained, the font detection efficiency is improved, the font recognition of batch images can be realized, and the accuracy of the font detection is effectively improved through the image classification, target segmentation and multiple stages.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a font detection method, apparatus, device, and medium. Background Technology

[0002] On e-commerce platforms, merchants create promotional materials for new product launches and sales promotions, which often require the use of different fonts. To prevent merchants from unknowingly using copyrighted fonts commercially, and to protect the copyrights of font owners, it is necessary to provide font testing services for users (especially merchants).

[0003] In related technologies, the text in the input image is segmented to obtain segmented images, which are then output. The user then stitches the segmented images together according to the text structure and inputs the corresponding text into the stitched image. Font detection is then performed by combining the user-stitched image and the user-input text.

[0004] However, the above methods require users to perform cumbersome operations, resulting in low font detection efficiency. Summary of the Invention

[0005] This application provides a font detection method, apparatus, device, and medium to solve the problem of low font detection efficiency.

[0006] In a first aspect, embodiments of this application provide a font detection method applied to a server. The font detection method includes: determining a region image containing text and the text in the region image on an image to be detected; performing image classification on the region image to obtain candidate fonts corresponding to the region image; performing target segmentation on the region image to obtain text images corresponding to the region image; and determining the target font to which the text in the region image belongs based on the text in the region image, the candidate fonts, and the text images.

[0007] Secondly, embodiments of this application provide a font detection method applied to a client. The font detection method includes: responding to an interactive operation requesting the detection of text fonts in an image to be detected, sending a font detection request to a server, the font detection request being used to detect the font of the text in the image to be detected; receiving a font detection result returned by the server, the font detection result being related to the target font to which the text in the region image of the image to be detected belongs, the target font being determined by performing image classification and target segmentation on the region image and combining it with the text in the region image.

[0008] Thirdly, embodiments of this application provide a font detection device applied to a server. The font detection device includes: a first determining unit, used to determine a region image containing text and the text in the region image on an image to be detected; a font recognition unit, used to perform image classification on the region image to obtain candidate fonts corresponding to the region image; a target segmentation unit, used to perform target segmentation on the region image to obtain text images corresponding to the region image; and a second determining unit, used to determine the target font to which the text in the region image belongs based on the text in the region image, the candidate fonts, and the text images.

[0009] Fourthly, this application provides a font detection device applied to a client. The font detection device includes: a sending unit, configured to send a font detection request to a server in response to a user's interactive operation requesting the detection of the font of text in an image to be detected; and a receiving unit, configured to receive the font detection result returned by the server, wherein the font detection result is related to the target font to which the text in the region image of the image to be detected belongs, and the target font is determined by performing image classification and target segmentation on the region image and combining the text in the region image.

[0010] Fifthly, embodiments of this application provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the electronic device to perform the font detection method provided in the first and / or second aspects of this application.

[0011] In a sixth aspect, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the font detection method provided in the first and / or second aspects of this application.

[0012] In a seventh aspect, embodiments of this application provide a computer program product comprising: a computer program stored in a readable storage medium, at least one processor of an electronic device being able to read the computer program from the readable storage medium, and the at least one processor executing the computer program causing the electronic device to perform the font detection method provided in the first and / or second aspects of this application.

[0013] As can be seen from the above technical solutions, the embodiments of this application provide a font detection method, apparatus, device, and medium. On an image to be detected, a region image containing text and the text within that region image are determined. Based on the region image and the text within it, font detection is performed on the text in the region image. Compared to font detection on the entire image to be detected, font detection on a region image is more accurate and efficient. During the font detection process, the accuracy is improved by dividing font detection into the following stages: image classification of the region image to obtain candidate fonts corresponding to the region image; target segmentation of the region image to obtain text images corresponding to the region image; and combining the text in the region image, candidate fonts, and text images to determine the target font to which the text in the region image belongs. Therefore, in the font detection process, the user only needs to provide an image to achieve font detection of text in the image, especially for batch image text font detection, effectively improving both the efficiency and accuracy of font detection. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 A schematic diagram of a font detection method provided in an embodiment of this application;

[0016] Figure 2 Flowchart of the font detection method provided in the embodiments of this application Figure 1 ;

[0017] Figure 3 Schematic diagram of the target segmentation model provided in the embodiments of this application Figure 1 ;

[0018] Figure 4 Flowchart of the font detection method provided in the embodiments of this application Figure 2 ;

[0019] Figure 5(a) is a schematic diagram of the target segmentation model provided in the embodiment of this application. Figure 2 ;

[0020] Figure 5(b) is a schematic diagram of the target segmentation model provided in the embodiment of this application. Figure 3 ;

[0021] Figure 6 This is a structural block diagram of the font detection device 60 provided in an embodiment of this application;

[0022] Figure 7 This is a structural block diagram of the font detection device 70 provided in the embodiments of this application;

[0023] Figure 8 This is a schematic diagram of the structure of a cloud server provided as an exemplary embodiment of this application.

[0024] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in sequences other than those illustrated or described herein.

[0027] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, such that a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those steps or modules that are explicitly listed, but may include other steps or modules that are not explicitly listed or that are inherent to such process, method, product, or device.

[0028] In font detection technologies that do not support end-to-end image-to-text detection, user assistance is required, such as image stitching and text input. End-to-end image-to-text detection is not supported. In contrast, end-to-end image-to-text detection technologies have higher requirements for the input image, such as specific image size and text layout. Users need to select images that meet these requirements or manually process them before performing further font detection.

[0029] It is evident that the aforementioned font detection technologies all increase the user's operating costs and negatively impact the user experience. Furthermore, font detection efficiency is low, and batch image detection is not possible.

[0030] To address the aforementioned issues, this application provides a font detection method, apparatus, device, and medium. In an image to be detected, a region image containing text and the text within that region image are determined. The font of the text in the region image is detected by combining the region image and the text within it. Compared to performing font detection on the entire image to be detected, font detection on a region image is more efficient and accurate. The font detection process is improved by dividing font detection into several stages: image classification of the region image to obtain candidate fonts corresponding to the region image; target segmentation of the region image to obtain text images corresponding to the region image; and combining the text in the region image, candidate fonts, and text images to determine the target font to which the text in the region image belongs. Thus, end-to-end detection from image to font is achieved, without special requirements for the image, effectively improving the efficiency and accuracy of font detection and facilitating font detection of batch images.

[0031] Figure 1 This is a schematic diagram illustrating an application scenario of the font detection method provided in the embodiments of this application. For example... Figure 1 As shown, the device that performs the font detection method is a font detection device, which can be connected to a client.

[0032] The client can be any computing device with some data processing capabilities. In this case, the basic structure of the client can include at least one processor. The number of processors depends on the client's configuration and type. The client can also include memory, which can be volatile, such as RAM, or non-volatile, such as read-only memory (ROM), flash memory, etc., or both types. The memory typically stores the operating system (OS), one or more applications, and may also store program data. In addition to the processing unit and memory, the client also includes some basic configurations, such as a network interface card (NIC) chip, I / O bus, display components, and some peripheral devices. Optionally, some peripheral devices can include, for example, a keyboard, mouse, pen, printer, etc. Other peripheral devices are well known in the art and will not be described in detail here. Optionally, the client can be a PC (personal computer) terminal, a handheld terminal (e.g., a smartphone, tablet computer), etc.

[0033] A font detection device can be a device that provides font detection services to clients. Within the font detection device, a font detection model can be deployed. This model may include one or more algorithmic models related to font detection, based on which font detection of text on an image is achieved.

[0034] In terms of physical implementation, a font detection device can be any device capable of providing computing services, responding to service requests, and processing them. Examples include cluster servers, regular servers, cloud servers, cloud hosts, and virtual data centers. The main components of a font detection device include a processor, hard drive, memory, and system bus, similar to a general computer architecture.

[0035] The client can connect to the font detection device via a network, which can be wireless or wired. If the client and the font detection device are connected via a communication network, the mobile network standard can be any one of 2G (GSM), 2.5G (GPRS), 3G (WCDMA, TD-SCDMA, CDMA2000, UTMS), 4G (LTE), 4G+ (LTE+), WiMax, or 5G.

[0036] In font detection scenarios, a client can send a font detection request to a font detection device, requesting the device to perform font detection on the text in the image to be detected. The font detection device can respond to the client's request and perform font detection on the text in the image. Afterwards, the font detection device can return the font detection results to the client for the user to view. Furthermore, the font detection device can also be used with data from local or remote databases (…). Figure 1 Font detection can be performed on batch images (not shown), and the detected fonts can be written to a local or remote database.

[0037] In font detection scenarios, font detection devices are mainly used to: identify the regions and text in an image to be detected; recognize candidate fonts and segment the text images within the regions; and determine the target font to which the text in the regions belongs. This achieves font detection of text in an image.

[0038] The above is merely an exemplary application scenario. The embodiments of this application can be applied to any font detection scenario on the Internet, such as performing font detection on images locally on a client, or performing font detection on images locally on a server.

[0039] The technical solution of this application will now be described in detail with reference to the accompanying drawings and specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0040] Figure 2 Flowchart of the font detection method provided in the embodiments of this application Figure 1 This method can be applied to servers. For example... Figure 2 As shown, the font detection method provided in this application includes:

[0041] S201. On the image to be detected, determine the region image containing text and the text in the region image.

[0042] The image to be detected is an image containing text. It can be obtained from a local database or from other devices, such as a client. The text is distributed across one or more regions within the image; therefore, the number of text-containing regions in the image can be one or more, and the number of characters within each region can also be one or more. The text can include one or more types of characters, such as English letters, numbers, and / or Chinese characters.

[0043] In this embodiment, after obtaining the image to be detected, text within the image can be detected. Based on the image position distribution of the detected text in the image to be detected, a region image containing text is determined in the image to be detected, and the text in the region image can be obtained.

[0044] S202. Perform image classification on the region image to obtain the candidate fonts corresponding to the region image.

[0045] The font detection model includes an image classification model, which is used to classify images based on font type. In other words, the image classification model is used to determine the font category of an image, that is, the font category of the text in the image. For example, it determines that the font category of the text in the region image is KaiTi.

[0046] The number of candidate fonts can be one or more.

[0047] In this embodiment, an image classification model is used to classify the region images to initially predict the font type of the text in the region images. From the initially predicted font types, candidate fonts corresponding to the region images are determined. When there are multiple region images, each region image can be classified separately using the image classification model to obtain candidate fonts for each region image.

[0048] In the process of determining candidate fonts through image classification, an image classification model can be used to classify the region image and obtain the probability of the region image belonging to multiple font types. The probability of a region image belonging to a font type represents the probability that the text in the region image belongs to that font type. Based on the probability of the region image belonging to multiple font types, candidate fonts corresponding to the region image are selected from multiple font types. In one approach, if the probability of a region image belonging to a font type is greater than a probability threshold, then that font type is determined as the candidate font corresponding to the region image. In another approach, the candidate fonts corresponding to the region image are determined by selecting the font types ranked in descending order of probability, such as the top 10 font types ranked in descending order of probability.

[0049] As an example, an image classification model is used to classify the region image to obtain the probability of the region image under multiple font types. The top 10 (Top-10) possible font types are extracted as candidate fonts in descending order of probability. Among these 10 candidate fonts, there are usually fonts that are similar to the real font style of the text in the region image.

[0050] S203. Perform target segmentation on the region image to obtain the text image corresponding to the region image.

[0051] The font detection model also includes a target segmentation model, which is used to segment the image with text as the target to obtain a text image containing only the text.

[0052] In this embodiment, in addition to text, the region image may also include other content, such as people or products. The text can be regarded as the foreground and other content can be regarded as the background. The target segmentation model is used to perform target segmentation on the region image (that is, to distinguish the foreground and background in the region image to segment the foreground in the region image) to obtain a text image containing only text.

[0053] S204. Based on the text in the region image, the candidate fonts corresponding to the region image, and the text image corresponding to the region image, determine the target font to which the text in the region image belongs.

[0054] In this embodiment, based on the text in the regional image and the candidate fonts corresponding to the regional image, the text in the candidate fonts can be obtained. For example, if the text in the regional image is "Hello", and the candidate fonts include regular script and Song typeface, then based on the text and the candidate fonts, "Hello" in regular script and "Hello" in Song typeface can be generated. The text in the candidate fonts is matched with the text image corresponding to the regional image, and according to the matching result, the target font to which the text in the regional image belongs is determined among the candidate fonts. Thus, by combining the candidate fonts obtained through image classification, the text image obtained through target segmentation, and the text in the regional image, the accuracy of font detection for images is effectively improved.

[0055] In one way, among the candidate fonts, if the matching degree between the text in the candidate font and the text image corresponding to the regional image is greater than the matching threshold, then this candidate font is determined as the target font to which the text in the regional image belongs; in another way, among the texts in the candidate fonts, the text in the candidate font with the highest matching degree with the text image is determined, and this candidate font is determined as the target font; in yet another way, among the texts in the candidate fonts, the text in the candidate font with the highest matching degree and the matching degree greater than the matching threshold is determined, and this candidate font is determined as the target font.

[0056] In the embodiments of the present application, the font detection of the image to be detected is converted into the font detection of the regional image, and the font detection of the text on the regional image is divided into the following steps: performing image classification on the regional image to obtain the candidate fonts corresponding to the regional image; performing target segmentation on the regional image to obtain the text image corresponding to the regional image; and determining the target font to which the text in the regional image belongs among the candidate fonts according to the text in the regional image, the candidate fonts corresponding to the regional image, and the text image corresponding to the regional image. It can be seen that end-to-end detection from the image to the font is achieved, the font detection efficiency is improved, the font detection of batch images can be realized, and at the same time, the accuracy of font detection is also improved.

[0057] In some embodiments, the image processing of the image to be detected can be performed through Optical Character Recognition (OCR) to obtain the regional image containing text and the text in the regional image. Among them, OCR can recognize the text box and the text in the text box in the image to be detected. The text box is the regional image containing text, and the text in the text box is the text in the regional image. Thus, based on the character recognition ability of OCR, the accuracy of the regional image and the text in the regional image is improved.

[0058] In some embodiments, an image classification model can be obtained based on a pre-trained Vision Transformer (ViT) model. The pre-trained ViT model is trained on large-scale data and possesses high-accuracy image classification capabilities. After obtaining the pre-trained ViT model, it can be further trained on a font detection task, i.e., the model parameters of the ViT model are fine-tuned so that the ViT model can classify images based on font type, thus obtaining an image classification model. Therefore, the ViT model effectively improves the accuracy of image classification for region images and the accuracy of candidate fonts corresponding to the region images.

[0059] In some embodiments, Figure 3 Schematic diagram of the target segmentation model provided in the embodiments of this application Figure 1 ,like Figure 3 As shown, the target segmentation model includes a feature extraction network, a detection network, and a segmentation network. The feature extraction network extracts feature maps, while the detection and segmentation networks perform text detection and segmentation based on the feature maps extracted by the feature extraction network, respectively. After image feature extraction, text detection, and text segmentation, the region image of the image to be detected yields the corresponding text image. Thus, the accuracy of text detection and segmentation is improved through the collaboration of the feature extraction network, detection network, and segmentation network. Furthermore, data can be transferred between the detection network and the segmentation network; the detection results of the detection network can be passed to the segmentation network to assist it in target segmentation.

[0060] based on Figure 3 The target segmentation network shown is Figure 4 Flowchart of the font detection method provided in the embodiments of this application Figure 2 .like Figure 4 As shown, the font detection method provided in this application includes:

[0061] S401. On the image to be detected, determine the region image containing text and the text in the region image.

[0062] S402. Perform image classification on the region image to obtain the candidate fonts corresponding to the region image.

[0063] The implementation principles and technical effects of S401 to S402 can be referred to in the aforementioned embodiments, and will not be repeated here.

[0064] S403. In the target segmentation model, feature extraction networks are used to extract features from the region image to obtain the feature map of the region image.

[0065] The feature extraction network is a neural network, which includes multiple network layers.

[0066] In this embodiment, the region image can be input into the feature extraction network, or the region image can be preprocessed and then input into the feature extraction network. In the feature extraction network, image features are extracted from the region image through multiple network layers to obtain the feature map of the region network. When there are multiple region images, image features can be extracted from each region image separately through the feature extraction network to obtain the feature map of each region image.

[0067] In some embodiments, such as Figure 3 As shown, the target segmentation model also includes an edge detection network, which is used to detect more edge information in the image. The edge information detected by the edge detection network is used to assist the feature extraction network in feature extraction, in order to solve the problem that feature extraction, target detection, and target segmentation focus on the main body of the text while ignoring the edge parts of the text, which leads to inaccurate text images segmented from the region image. That is, it improves the accuracy of feature extraction from the region image, and improves the accuracy of subsequent target detection and target segmentation of the region image.

[0068] The target segmentation model also includes an edge detection network. One possible implementation of S403 includes: inputting the region image into the edge detection network, performing text edge detection on the region image in the edge detection network to obtain a text edge detection map of the region image; and extracting features from the region image based on the region image, the text edge detection map, and the feature extraction network to obtain a feature map of the region image.

[0069] In this implementation, either the region image or a pre-processed region image can be input into an edge detection network. The edge detection network detects the edge information of the text in the region image, obtaining a text edge detection map. This map displays the edges of the text while downplaying the background and the main body of the text. Next, either the region image or the pre-processed region image can be input into a feature extraction network. The feature extraction network extracts features from the region image. Simultaneously, the text edge detection map can assist the feature extraction network in extracting features from the region image, helping it acquire more text edge features and improving the accuracy of feature extraction. This addresses the problem in subsequent target detection and segmentation processes where attention is focused on the main body of the text while neglecting the edges, thus improving the accuracy of target detection and segmentation.

[0070] In the process of extracting features from a region image based on the region image, text edge detection map, and feature extraction network to obtain the feature map of the region image, the text edge detection map can be fused into the feature extraction network for the region image feature extraction process based on an attention mechanism.

[0071] Optionally, the edge detection network employs a holistically-nested edge detection (HED) network to improve the accuracy of text edge detection.

[0072] In some embodiments, such as Figure 3 As shown, the object detection network also includes an attention network, which processes the text edge detection map output by the edge detection network to obtain corresponding attention weights. These attention weights are used to assist the feature extraction network in extracting features from the region image, thereby improving the accuracy of feature extraction from the region image.

[0073] The object detection network also includes an attention network. In the process of extracting features from the region image based on the region image, text edge detection map, and feature extraction network to obtain the feature map of the region image, one possible implementation includes: inputting the text edge detection map into the attention network, performing feature extraction on the text edge detection map in the attention network to obtain the attention weights corresponding to the region image; inputting the region image into the feature extraction network, and performing feature extraction on the region image based on the attention weights corresponding to the region image to obtain the feature map of the region image.

[0074] The attention network learns attention weights on text edge detection images using an attention mechanism. The input data for the attention network is the text edge detection image, and the output data consists of multiple weight maps, each representing a different attention weight. These attention weights can be used to assist in the fusion of multiple network layers in the feature extraction network, and further, they can be used to fuse the output data of multiple network layers in the feature extraction network.

[0075] In this implementation, the text edge detection map is input into an attention network. The attention network uses an attention mechanism to extract features from the text edge detection map, obtaining multiple weight maps corresponding to the region image, i.e., multiple attention weights for the region image. Then, the region image is input into a feature extraction network. In the feature extraction network: features are extracted from the region image through multiple network layers, yielding output data for multiple network layers; the output data of different network layers can be weighted based on the correspondence between network layers and attention weights to achieve fusion between different network layers. In one approach, the output data of multiple network layers are weighted based on their corresponding attention weights to obtain the feature map of the region image; in another approach, for each network layer, the output data of that layer is weighted with its corresponding attention weight to obtain a weighted result, which is then input into the next network layer until the feature map output by the last network layer is obtained, i.e., the feature map of the region image.

[0076] Thus, based on the edge features of text on the regional image and the attention mechanism, the fusion of multiple network layers of the feature extraction network is guided, enabling the feature extraction network layers to extract more text edge features, thereby improving the accuracy and feature richness of the feature map.

[0077] The correspondence between the network layers in the feature extraction network and the attention weights output by the attention network can be preset. For example, the first network layer can be preset to correspond to the first attention weight, the second network layer to correspond to the second attention weight, and so on.

[0078] Besides using an attention mechanism to apply the text edge detection map to the feature extraction network, multiple input channels can be set for the feature extraction network. The text edge detection map and the region image can be output together to the feature extraction network for feature extraction, resulting in a feature map of the region image. Alternatively, the text edge detection map and the region image can be weighted, and the weighted image can be input to the feature extraction network for feature extraction, yielding a feature map of the region image. This provides the feature extraction network with richer text edge information, ensuring that the network extracts the text edge features.

[0079] In some embodiments, the feature extraction network may employ a Feature Pyramid Network (FPN). An FPN comprises bottom-up upsampling layers of varying scales and top-down downsampling layers of varying scales, enabling the extraction of a rich variety of image features. Therefore, in object segmentation models, using an FPN to extract features from region images yields feature maps of those regions, effectively improving the feature richness of the feature maps.

[0080] Based on the Feature Extraction Network (FPN), one possible implementation of S403 includes: In the target segmentation model, feature extraction is performed on the region image through the FPN to obtain the feature map output by the last layer from top to bottom in the FPN, which is the feature map of the region image. In the FPN, the resolution of the feature maps output by the top-down network layers decreases layer by layer, for example, by a factor of 2. Therefore, the feature map output by the last layer has the highest resolution. Obtaining the feature map output by the last layer means obtaining the highest resolution feature map, thus improving the accuracy of the feature map of the region image.

[0081] When using a Feature Processing Network (FPN) for feature extraction, combining it with the aforementioned edge detection network and attention network can improve the accuracy and richness of FPN's feature extraction of regional images. Specifically, attention networks can be used to guide the fusion of multiple network layers within the FPN. Furthermore, attention networks can be used to guide the fusion of feature maps output from different top-down network layers within the FPN, compensating for the scale and semantic differences in feature maps from different network layers within the FPN. This effectively alleviates the cue combination problem (also known as the hint combination problem) inherent in FPN. Clues refer to network layers within the FPN, and the cue combination problem refers to the difficulty in combining features with scale and semantic differences.

[0082] In one approach, the feature map output by the FPN layer can be weighted with the attention weights in the top-down network layer, and the weighted result can be input into the next network layer until the feature map output by the last network layer is obtained.

[0083] In another approach, the feature maps output by each top-down network layer in the FPN are obtained. Based on the correspondence between the top-down network layers and attention weights in the FPN, the attention weights are used as the weights in the weighted calculation to perform a weighted summation of the feature maps output by each top-down network layer in the FPN, thus obtaining the feature map of the region image. It can be seen that the attention weights are essentially parallel to the top-down network layers in the FPN, and the attention weights corresponding to the top-down network layers in the FPN determine the influence of the feature maps output by the network layers in the FPN on the feature map of the region image.

[0084] Optionally, the attention mechanism adopted by the feature extraction network is spatial attention. Spatial attention improves the fusion effect of different network layers in the FPN.

[0085] S404. In the target segmentation model, based on the feature map, the detection network and the segmentation network are used to perform text detection and text segmentation on the region image respectively to obtain the text image.

[0086] Both the detection network and the segmentation network are neural networks, each consisting of multiple network layers.

[0087] In this embodiment, after obtaining the feature map of the region image, the feature map can be input into the detection network. In the detection network, feature processing is performed on the feature map to detect the image position of the text in the region image. Based on the image position of the text in the region image and the feature map of the region image, a segmentation network can be used to segment the text in the region image, obtaining the segmented text image. Therefore, by combining the feature extraction network, detection network, and segmentation network, the accuracy of target segmentation is improved, that is, the accuracy of segmenting text images from the region image is improved.

[0088] In some embodiments, such as Figure 3 As shown, the detection network includes a first bounding box detection network and a second bounding box detection network. The first bounding box detection network is used for initial text detection on the region image, and the second bounding box detection network is used for further text detection on the region image. The detection results output by the first bounding box detection network can be used to assist the text detection of the second bounding box detection network. Thus, by combining the first and second bounding box detection networks, the accuracy of text detection is improved.

[0089] The detection network includes a first bounding box detection network and a second bounding box detection network. The image location of text in the region image includes the bounding box location of the text in the region image, such as... Figure 4 As shown, one possible implementation of S404 includes: S4041, in the first bounding box detection network, the text bounding box is initially detected based on the feature map of the region image to obtain the initial position of the bounding box of the text in the region image; S4042, based on the initial position of the bounding box of the text in the region image and the feature map of the region image, the text bounding box is detected again through the second bounding box detection network to obtain the target position of the bounding box of the text in the region image; S4043, according to the target position of the bounding box of the text in the region image and the feature map of the region image, the text is segmented through a segmentation network to obtain the mask image corresponding to the text in the region image; S4044, according to the mask image corresponding to the text in the region image and the region image, a text image is generated.

[0090] In S4041, the feature map of the region image can be input into the first bounding box detection network. In this network, feature processing is performed on the feature map to detect the class probabilities of multiple anchor boxes (i.e., the probability that the region belongs to text or background) and the bounding box positions of the text. For ease of distinction, the bounding box positions of the text in the region image detected by the first bounding box detection network are called the initial bounding box positions of the text in the region image. These initial bounding box positions can be used to assist the text detection in the second bounding box detection network. The class probabilities of the multiple anchor boxes in the region image mainly play a role in the training process of the font detection model, improving the accuracy of font detection.

[0091] Optionally, the first bounding box detection network may employ a Region Proposal Network (RPN) to improve the accuracy of text detection.

[0092] In S4042, based on the initial position of the bounding box of the text in the region image, initial image features related to the text are determined in the feature map of the region image. These initial image features are then input into the second bounding box detection network for further detection of the text bounding box, yielding the class probabilities of multiple anchor boxes in the region image and the bounding box position of the text. For ease of distinction, the bounding box position of the text in the region image detected by the second bounding box detection network is referred to as the target bounding box position of the text in the region image. Similarly, the class probabilities of multiple anchor boxes in the region image detected by the second bounding box detection network primarily play a role in the training process of the font detection model, improving the accuracy of font detection. The bounding box position of the text may include multiple positional parameters of the bounding box, such as the coordinates of the four vertices of the bounding box.

[0093] In step S4043, target image features related to the text can be determined from the feature map of the region image based on the bounding box target location of the text in the region image. These text-related target image features are then input into the segmentation network. The segmentation network performs text segmentation based on these text-related target image features, obtaining a mask image corresponding to the text in the region image. The mask image corresponding to the text is a binary mask image. On this mask image, a pixel value of 1 indicates that the pixel is located within the region containing the text, while a pixel value of 0 indicates that the pixel is not located within the region containing the text.

[0094] Optionally, one possible implementation of S4042 includes: determining a feature matrix corresponding to the initial position of the bounding box in the feature map of the region image; inputting the feature matrix corresponding to the initial position of the bounding box into a second bounding box detection network; and in the second bounding box detection network, performing a second detection of the text bounding box based on the feature matrix corresponding to the initial position of the bounding box to obtain the target position of the bounding box of the text in the region image. Here, the feature matrix corresponding to the initial position of the bounding box can be regarded as the feature matrix of the region of interest (i.e., the region where the text may appear) determined based on the initial position of the bounding box. Thus, based on the feature map and the initial position of the bounding box, the accuracy of the input data of the second bounding box detection network is improved, thereby improving the accuracy of text detection and text segmentation.

[0095] Optionally, considering that the number of characters in an image is finite, the number of bounding boxes detected by the first bounding box detection network can be preset as a first threshold, wherein the first threshold is less than the number of bounding boxes that the first bounding box detection network can detect (or, less than the standard number of bounding boxes detected by the first bounding box detection network). Thus, by reducing the number of bounding boxes detected by the first bounding box detection network, the computational resources occupied by text detection are reduced, reserving more computational resources for text segmentation, such as more GPU memory, thereby improving the efficiency and accuracy of text segmentation.

[0096] Optionally, one possible implementation of S4043 includes: determining the feature matrix corresponding to the bounding box target position in the feature map; inputting the feature matrix corresponding to the bounding box target position into the segmentation network; and performing text segmentation in the segmentation network based on the feature matrix corresponding to the bounding box target position to obtain a mask image corresponding to the text in the region image. Here, the feature matrix corresponding to the bounding box target position can be considered as the feature matrix of the region of interest (i.e., the region where the text may appear) determined based on the bounding box target position. Thus, based on the feature map and the bounding box target position, the accuracy of the input data to the segmentation network is improved, and the serialization of the second edge detection network and the segmentation network is achieved, that is, the data processed by the second edge detection network enters the segmentation network, improving the accuracy of text segmentation.

[0097] Optionally, considering the limited number of characters in an image, the number of feature matrices entering the segmentation network can be reduced. Including all bounding boxes detected by the second edge detection network in the segmentation stage is somewhat wasteful of resources. Ideally, we want to detect many bounding boxes, but only a few for text segmentation. Therefore, only bounding boxes close to the number of actual characters can be used for text segmentation, improving efficiency. Furthermore, with a reduced number of bounding boxes entering the segmentation stage, a higher-resolution feature matrix can be extracted based on the bounding box target location and the feature map of the region image, improving accuracy.

[0098] Optionally, besides determining the input data for the segmentation network based on the bounding box target positions of the text in the region image and the feature map of the region image, the input data can also be determined based on the initial bounding box positions of the text in the region image and the feature map of the region image. In this case, the second edge detection network and the segmentation network operate in parallel, rather than serially. The second edge detection network is mainly used to improve the model accuracy during training. It should be noted that because the accuracy of the bounding box target positions output by the second edge detection network is higher than that of the initial bounding box positions output by the first edge detection network, operating the second edge detection network serially with the segmentation network improves the accuracy of text segmentation compared to operating it in parallel.

[0099] The input data for the segmentation network is determined based on the initial position of the bounding box of the text in the region image and the feature map of the region image. The process of determining the input data for the segmentation network based on the target position of the bounding box of the text in the region image and the feature map of the region image will not be repeated here.

[0100] In S4044, after obtaining the mask image corresponding to the text in the region image, the text image can be extracted from the region image based on the mask image.

[0101] In some embodiments, such as Figure 3 As shown, the target segmentation model also includes a deep matting network, which is used to refine the mask image output by the segmentation model, solve the problems of excessive smoothing of text edges and blurring of text outlines on the mask image, and restore the text edge structure.

[0102] The target segmentation model also includes Deep Image Matting (DIM) networks. One possible implementation of S4044 involves processing the region image and mask image using DIM to generate a text image. DIM refines the mask image output by the segmentation network, using alpha matting to address issues like excessively smoothed edges and blurred text outlines, restoring the text edge structure and improving the image quality of the text image.

[0103] In this implementation, morphological operations (such as erosion and dilation, which can be understood as distortion of the mask image) can be performed on the mask image corresponding to the text in the region image to obtain the morphologically operated mask image. The region image and the morphologically operated mask image are then output to the DIM. In the DIM, feature processing is performed on the region image and the mask image corresponding to the text in the region image to generate the text image.

[0104] When an image is overexposed, the edge pixels of the text and the background pixels are highly similar. Using only local information is insufficient to distinguish the text edges from the background. Furthermore, when the region image and mask image are used as input to the Discrete Image Processing (DIM), the DIM lacks sufficient semantic information for determining text edges. Considering that the detection and segmentation networks extract semantic information, image features from these networks can also be used as part of the DIM's input data to provide sufficient semantic information. Therefore, optionally, image processing can be performed in the DIM based on the region image, mask image, and the feature matrix corresponding to the bounding box target position obtained from the feature map based on the region image, to obtain the text image. This is equivalent to improving the DIM's input layer from two channels to three channels, using the region image, mask image, and the feature matrix corresponding to the bounding box target position obtained from the feature map based on the region image as the DIM's input data, providing sufficient semantic information and improving the quality of the generated text image.

[0105] Optionally, the image processing in DIM is a multi-iterative process. Each iteration includes: performing image processing in DIM based on the output image from the previous DIM iteration, the region image, and the feature matrix corresponding to the bounding box target position obtained from the feature map of the region image. Thus, through multiple iterations, the image quality of the text image is improved.

[0106] In this optional approach, during the first iteration, the mask image (or the mask image after morphological operations), the region image, and the feature matrix corresponding to the bounding box target position obtained from the feature map based on the region image are input into the DIM for image processing, resulting in the output image of the DIM. During the second iteration, the output image of the DIM from the first iteration, the region image, and the feature matrix corresponding to the bounding box target position obtained from the feature map based on the region image are input into the DIM for image processing, again resulting in the output image of the DIM. Thus, the text image is optimized through multiple iterations.

[0107] Based on the foregoing description of the target segmentation model, Figure 5(a) is a schematic diagram of the target segmentation model provided in the embodiment of this application. Figure 2 Figure 5(b) is a schematic diagram of the target segmentation model provided in the embodiment of this application. Figure 3 In Figures 5(a) and 5(b): FPN represents the Feature Pyramid Network, which acts as the feature extraction network in the target segmentation model; RPN represents the Region Proposal Network, which acts as the first bounding box detection network in the target segmentation model; C0 represents the class probabilities of multiple anchor boxes in the region image detected by the Region Proposal Network; B0 represents the location of the bounding box containing the text detected by the Region Proposal Network, i.e., the initial position of the bounding box; R0 represents the network that determines the feature matrix of the region of interest (i.e., the region containing the text, or the bounding box) based on the feature map of the region image and the initial bounding box (i.e., the initial position of the bounding box), which can be understood as a feature matrix extraction network or a feature matrix extraction network for the region of interest; BH (Box Head (a network with multiple layers for detecting bounding boxes) represents the second bounding box detection network; C1 represents the class probabilities of multiple anchor boxes in the region image detected by BH, and B1 represents the location of the bounding box where the text is detected by BH, i.e., the target location of the bounding box; Rol1 represents the network that determines the feature matrix of the region of interest based on the feature map of the region image and the target bounding box (i.e., the target location of the bounding box), which can be understood as a feature matrix extraction network or a feature matrix extraction network of the region of interest; MH (Mask Head, a network with multiple layers for object segmentation) represents the segmentation network, and M represents the mask map obtained by object segmentation by the segmentation network.

[0108] In Figure 5(a), after the region image of the image to be detected is input into the FPN, the feature map of the region image is obtained. The feature map is then input into the RPN for text detection, resulting in the corresponding classification result C0 (i.e., the class probabilities of multiple anchor boxes) and the initial bounding box position B0 of the text in the region image. Based on the initial bounding box reflected by B0, the corresponding feature matrix is ​​extracted from the feature map through Rot0, and then the feature matrix is ​​input into BH. After BH, text detection is performed again, resulting in the classification result C1 and the target bounding box position B1 of the text in the region image. At this point, C0, C1, and B1 all play a role in the model training process, while B0 can be used in the text segmentation stage during model application. Based on the initial bounding box reflected by B0, the corresponding feature matrix is ​​extracted from the feature map through Rot1, and then the feature matrix is ​​input into MH. After MH, text segmentation is performed, resulting in the mask map M. Then, based on the mask map M, the text image is further obtained, completing the text image segmentation.

[0109] Unlike the target segmentation model shown in Figure 5(a), the target segmentation model shown in Figure 5(b) adds HED, an attention network, and DIM. It also serializes the detection and segmentation networks, meaning that the bounding box target position B1 obtained based on BH detection can be used in the text segmentation process of the MH network. Since B1 has higher accuracy than B0, using B1 for MH text segmentation improves the accuracy of MH compared to using B0. Here, HED represents a nested edge detection network, acting as the edge detection network in the target segmentation model, and DIM represents a depth matting network, used to recover the text edge structure in the target segmentation model.

[0110] In Figure 5(b), the region image of the image to be detected is input into HED to obtain the edge information of the text in the region image, as shown in the text edge detection map in the previous embodiment. Based on the edge information of the text, the corresponding attention weights are obtained through an attention network. The region image of the image to be detected is input into FPN, where feature extraction is performed in combination with the attention weights to obtain the feature map of the region image. The feature map is input into RPN for text detection to obtain C0 and B0. Based on the initial bounding box reflected by B0, the corresponding feature matrix is ​​extracted on the feature map through Rol0. Then, the feature matrix is ​​input into BH, and text detection is performed again through BH to obtain C1 and B1. C0 and C1 both play a role in the model training process. Based on the target bounding box reflected by B1, the corresponding feature matrix is ​​extracted on the feature map through Rol1. The feature matrix is ​​input into MH, where text segmentation is performed to obtain the mask image M. Then, based on the mask image M, the feature matrix extracted by Rol1, and the region image, iterative image processing is performed in DIM to obtain the text image.

[0111] S405. Based on the text in the region image, the candidate fonts corresponding to the region image, and the text image corresponding to the region image, determine the target font to which the text in the region image belongs.

[0112] The implementation principle and technical effects of S405 can be referred to in the aforementioned embodiments, and will not be repeated here.

[0113] In some embodiments, a possible implementation of S405 includes: generating a text candidate image corresponding to a candidate font based on the text in the region image; comparing the similarity between the text candidate image and the text image; and determining the target font to which the text in the region image belongs based on the comparison result. Thus, by comparing image similarity, the accuracy of font detection is improved.

[0114] Among them, the font of the text in the text candidate image is the candidate font corresponding to the text candidate image.

[0115] In this implementation, when there are multiple candidate fonts, for each candidate font, a candidate text image can be generated that presents the text in the region image in the style of the candidate font. For example, if the candidate fonts include "Songti" and "Kaiti", and the text in the region image is "Font Detection Test", then candidate text images for "Font Detection Test" containing the Songti style and candidate text images for "Font Detection Test" containing the Kaiti style can be generated. In this way, candidate text images corresponding to each candidate font are generated. The candidate text images are compared with the text images to obtain comparison results, including the similarity between the candidate text images and the text images. Among the candidate text images, the candidate text image with the highest similarity to the text image can be determined, and the target font to which the text in the region image belongs is determined as the candidate font corresponding to the candidate text image with the highest similarity to the text image.

[0116] Therefore, by performing image classification, target segmentation, and similarity comparison, end-to-end font detection of text in images is achieved, improving the accuracy of font detection in images.

[0117] In this embodiment, the font detection model includes an image classification model and a target segmentation model. The target segmentation model includes a feature extraction network, a detection network, and a segmentation network. Based on these models, end-to-end detection of text fonts in an image is achieved, which improves the detection efficiency and accuracy of text fonts in an image.

[0118] In some embodiments, the client may send a font detection request to the server in response to a user's interactive operation requesting the detection of text fonts in an image to be detected. The server, in response to the font detection request from the client, determines the region image containing the text and the text within the region image on the image to be detected. Then, it may determine the target font of the text in the region image. Afterward, the server may return the font detection result to the client, which is related to the target font to which the text in the region image belongs. Thus, by triggering a font detection request on the client, the user can detect the text fonts in the image to be detected, improving the convenience and efficiency of text font detection in images and enhancing the user experience.

[0119] The font detection request is used to request the detection of the font of the text on the image to be detected.

[0120] The target font is determined by performing image classification and target segmentation on the region image and combining the text in the region image. For details, please refer to the aforementioned embodiments, which will not be repeated here.

[0121] In this embodiment, on the client side, a font detection request can be sent to the server upon detecting user input of an image to be detected and / or input of relevant information about the image (such as the image's identifier or storage address), or upon detecting user clicking a font detection control. The font detection request may contain the image to be detected and / or its relevant information. Upon receiving the font detection request, the server can perform font detection on the image to be detected based on the request, obtaining the target font to which the text in one or more regions of the image belongs. The server can then return the font detection results to the client so that the user can understand the font detection status.

[0122] The font detection results may include at least one of the following: the target font, whether the target font is a copyrighted font, the copyright holder information of the target font, and the price information of the target font. The copyright holder information may include the copyright holder's name, contact information, and font licensing information provided by the copyright holder (which may include font sales information). Therefore, after receiving the font detection results, the client user can learn about the font of the text in the image to be detected, whether the font of the text in the image to be detected has any infringement risk, and the cost required to install the font in the image to be detected. It is evident that, based on the embodiments of this application, the convenience of font infringement detection can be effectively improved, the risk of font infringement on images can be reduced, font copyright protection can be enhanced, and the user experience can be improved.

[0123] Optionally, if the font detection result indicates that the target font is a copyright-protected font (for example, the font detection result indicates that the target font is a copyright-protected font, or the client obtains the target font from the font detection result and finds that the target font is a copyright-protected font in the font library), the client may also perform at least one of the following operations:

[0124] Operation 1: Display a prompt message to inform the user that the target font is a copyrighted font, to purchase the target font, and / or to modify the font of the text on the image to be detected; Operation 2: Display the purchase page for the target font; Operation 3: Display the font modification page corresponding to the image to be detected.

[0125] In Operation 1, the client can display a prompt message on the font detection page, or send a prompt message to the user via SMS or instant messaging. The prompt message may indicate that the target font is a copyrighted font, that the target font should be purchased, and / or that the font of the text in the image to be detected should be modified. It may also indicate that the font of the text in the image to be detected is the target font. For example, the prompt message may be: "The font of the text in the image is a copyrighted font," "The font of the text in the image is font A, and font A is a copyrighted font," "The font of the text in the image needs to be purchased to use," or "The font of the text in the image is a copyrighted font; please purchase or modify the font."

[0126] Optionally, the notification message can also display the copyright holder information of the target font to provide users with more detailed information, saving them time by eliminating the need for them to learn about the copyright of the target font.

[0127] In operation two, if the client determines that the target font is a copyrighted font, it can display the target font's purchase page based on the font's copyright holder information (which can be provided by the server or obtained by the client). This provides convenience for users to purchase fonts.

[0128] In Operation 3, if the client determines that the target font is a copyright-protected font, it displays the font modification page corresponding to the image to be detected. This allows users to directly modify the font of the text on the image to be detected, thus avoiding font infringement and improving user experience.

[0129] Figure 6 This is a structural block diagram of the font detection device 60 provided in an embodiment of this application. The font detection device 60 is applied to a server. Figure 6 As shown, the font detection device 60 includes: a first determining unit 61, a font recognition unit 62, a target segmentation unit 63, and a second determining unit 64, wherein:

[0130] The first determining unit 61 is used to determine, on the image to be detected, a region image containing text and the text in the region image;

[0131] The font recognition unit 62 is used to identify the font type of the region image and obtain the candidate font corresponding to the region image;

[0132] The target segmentation unit 63 is used to segment the region image to obtain the text image corresponding to the region image;

[0133] The second determining unit 64 is used to determine the target font to which the text in the region image belongs based on the text, candidate fonts and text images in the region image.

[0134] In one embodiment of this application, the target segmentation model includes a feature extraction network, a detection network, and a segmentation network. In the process of segmenting a region image to obtain a text image corresponding to the region image, the target segmentation unit 63 is specifically used to: extract features from the region image through the feature extraction network in the target segmentation model to obtain a feature map of the region image; and, based on the feature map, perform text detection and text segmentation on the region image through the detection network and the segmentation network respectively to obtain a text image.

[0135] In one embodiment of this application, the target segmentation model further includes an edge detection network. In the target segmentation model, during the process of extracting features from the region image through a feature extraction network to obtain a feature map of the region image, the target segmentation unit 63 is specifically used to: input the region image into the edge detection network, perform text edge detection on the region image in the edge detection network to obtain a text edge detection map of the region image; and extract features from the region image based on the region image, the text edge detection map, and the feature extraction network to obtain a feature map.

[0136] In one embodiment of this application, the target segmentation model further includes an attention network. During the process of extracting features from the region image based on the region image, the text edge detection map, and the feature extraction network to obtain a feature map, the target segmentation unit 63 is specifically used to: input the text edge detection map into the attention network, extract features from the text edge detection map in the attention network to obtain the attention weights corresponding to the region image; input the region image into the feature extraction network, extract features from the region image in the feature extraction network based on the attention weights to obtain a feature map, wherein the attention weights are used to assist in the fusion of multiple network layers in the feature extraction network.

[0137] In one embodiment of this application, the feature extraction network is a feature pyramid network. In the target segmentation model, during the process of extracting features from the region image through the feature extraction network to obtain the feature map of the region image, the target segmentation unit 63 is specifically used to: extract features from the region image through the feature pyramid network in the target segmentation model to obtain the feature map output by the last layer from top to bottom in the feature pyramid network.

[0138] In one embodiment of this application, the detection network includes a first bounding box detection network and a second bounding box detection network. In the target segmentation model, based on the feature map, the detection network and the segmentation network respectively perform text detection and text segmentation on the region image to obtain a text image. Specifically, the target segmentation unit 63 is used to: perform initial detection of text bounding boxes based on the feature map in the first bounding box detection network to obtain the initial position of the bounding box of the text in the region image; perform further detection of text bounding boxes based on the initial position of the bounding box and the feature map through the second bounding box detection network to obtain the target position of the bounding box of the text in the region image; perform text segmentation through the segmentation network based on the target position of the bounding box and the feature map to obtain the mask image corresponding to the text in the region image; and generate a text image based on the mask image and the region image.

[0139] In one embodiment of this application, during the process of segmenting text through a segmentation network based on the bounding box target position and feature map to obtain a mask image corresponding to the text in the region image, the target segmentation unit 63 is specifically used to: determine the feature matrix corresponding to the bounding box target position in the feature map; input the feature matrix into the segmentation network, and perform text segmentation based on the feature matrix in the segmentation network to obtain a mask image.

[0140] In one embodiment of this application, the target segmentation model further includes a deep matting network. In the process of generating a text image based on the mask image and the region image, the target segmentation unit 63 is specifically used to: perform image processing through the deep matting network based on the region image and the mask image to generate a text image.

[0141] In one embodiment of this application, the image processing process of the deep matting network is a multi-iteration process. Each iteration includes: performing image processing in the deep matting network based on the output image after the previous iteration of the deep matting network, the region image, and the feature matrix corresponding to the bounding box target position obtained based on the feature map.

[0142] In one embodiment of this application, in the process of determining the target font to which the text in the region image belongs based on the text in the region image, the candidate font, and the text image, the second determining unit 64 is specifically used to: generate a text candidate image corresponding to the candidate font based on the text in the region image; compare the similarity between the text candidate image and the text image; and determine the target font to which the text in the region image belongs based on the comparison result.

[0143] In one embodiment of this application, in the process of determining the region image containing text and the text in the region image on the image to be detected, the first determining unit 61 is specifically used to: in response to a font detection request from the client, determine the region image containing text and the text in the region image on the image to be detected, wherein the font detection request is used to detect the font of the text on the image to be detected.

[0144] The technical solutions provided in this application can be implemented on a cloud server.

[0145] Figure 7 This is a structural block diagram of the font detection device 70 provided in an embodiment of this application. The font detection device 70 is applied to a client. Figure 7 As shown, the font detection device 70 includes: a transmitting unit 71 and a receiving unit 72, wherein:

[0146] The sending unit 71 is used to send a font detection request to the server in response to the user's interactive operation of requesting to detect the font of the text in the image to be detected. The font detection request is used to detect the font of the text in the image to be detected.

[0147] The receiving unit 72 is used to receive the font detection result returned by the server. The font detection result is related to the target font to which the text in the region image of the image to be detected belongs. The target font is determined by performing font type recognition and target segmentation on the region image and combining it with the text in the region image.

[0148] In one embodiment of this application, the font detection result reflects that the target font is a copyright-protected font. The font detection device 70 further includes a display unit 73, which is used to perform at least one of the following operations: displaying a prompt message to prompt the user that the target font is a copyright-protected font, to purchase the target font and / or to modify the font of the text on the image to be detected; displaying the purchase page of the target font; and displaying the font modification page corresponding to the image to be detected.

[0149] The technical solutions provided in this application can be implemented on a terminal.

[0150] Figure 8 This is a schematic diagram of a cloud server provided as an exemplary embodiment of this application. The cloud server is used to run the font detection method provided in any of the foregoing method embodiments. Figure 8 As shown, the cloud server includes: memory 83 and processor 84.

[0151] Memory 83 is used to store computer programs and can be configured to store various other data to support operations on a cloud server. This memory 83 may be object storage (OSS).

[0152] The memory 83 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0153] The processor 84, coupled to the memory 83, is used to execute the computer program in the memory 83 for performing the font detection method provided in any of the foregoing embodiments.

[0154] Furthermore, such as Figure 8 As shown, the cloud server also includes other components such as a firewall 81, a load balancer 82, a communication component 85, and a power supply component 86. Figure 8 The diagram only shows some components and does not mean that a cloud server includes only these components. Figure 8 The components shown.

[0155] The above Figure 8 The communication component 85 is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component 85 can access wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G / LTE, 5G, or combinations thereof. In one exemplary embodiment, the communication component 85 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component 85 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0156] The above Figure 8 The power supply component 86 provides power to various components of the device in which it is located. The power supply component 86 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which it is located.

[0157] Accordingly, embodiments of this application also provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the electronic device to perform the steps in the above method embodiments.

[0158] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to implement the steps in the above method embodiments.

[0159] Accordingly, this application also provides a computer program product, including a computer program / instructions, which, when executed by a processor, cause the processor to implement the steps in the above method embodiments.

[0160] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0161] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0162] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0163] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0164] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0165] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0166] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0167] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0168] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A font detection method, applied to a server, characterized in that, include: On the image to be detected, determine the region image containing text and the text in the region image; The region image is classified to obtain candidate fonts corresponding to the region image; The region image is segmented to obtain the corresponding text image. The target segmentation is based on a target segmentation model, which includes an edge detection network, an attention network, and a feature pyramid network. The edge detection network performs text edge detection on the region image to obtain a text edge detection map. The attention network extracts features from the text edge detection map to obtain attention weights. The feature pyramid network fuses features from multiple network layers based on the attention weights to extract a feature map of the region image. Based on the feature map, text detection and text segmentation are performed by a detection network and a segmentation network, respectively, to obtain the text image. Based on the text in the region image, the candidate fonts, and the text image, determine the target font to which the text in the region image belongs.

2. The font detection method according to claim 1, characterized in that, The extraction of feature maps from the region image includes: In the target segmentation model, the feature pyramid network is used to extract features from the region image to obtain the feature map output from the last layer of the feature pyramid network from top to bottom.

3. The font detection method according to any one of claims 1 to 2, characterized in that, The detection network includes a first bounding box detection network and a second bounding box detection network. The step of performing text detection and text segmentation based on the feature map through the detection network and segmentation network to obtain the text image includes: In the first bounding box detection network, the initial detection of text bounding boxes is performed based on the feature map to obtain the initial position of the bounding box of the text in the region image; Based on the initial position of the bounding box and the feature map, the second bounding box detection network is used to detect the text bounding box again, so as to obtain the target position of the bounding box of the text in the region image. Based on the bounding box target location and the feature map, text segmentation is performed through the segmentation network to obtain the mask map corresponding to the text in the region image; The text image is generated based on the mask image and the region image.

4. The font detection method according to claim 3, characterized in that, The step of segmenting text using the segmentation network based on the bounding box target location and the feature map to obtain a mask image corresponding to the text in the region image includes: In the feature map, a feature matrix corresponding to the target position of the bounding box is determined; The feature matrix is ​​input into the segmentation network, and text segmentation is performed in the segmentation network based on the feature matrix to obtain the mask image.

5. The font detection method according to claim 3, characterized in that, The target segmentation model further includes a deep matting network, and the step of generating the text image based on the mask image and the region image includes: The text image is generated by performing image processing through the depth matting network based on the region image and the mask image.

6. The font detection method according to claim 5, characterized in that, The image processing of the depth matting network is a multi-iterative process, wherein one iteration includes: Image processing is performed in the deep matting network based on the output image after the previous iteration of the deep matting network, the region image, and the feature matrix corresponding to the target position of the bounding box obtained based on the feature map.

7. The font detection method according to any one of claims 1 to 2, characterized in that, The step of determining the target font to which the text in the region image belongs based on the text in the region image, the candidate font, and the text image includes: Based on the text in the region image, generate a text candidate image corresponding to the candidate font; The similarity between the candidate text image and the text image is compared, and the target font to which the text in the region image belongs is determined based on the comparison result.

8. The font detection method according to any one of claims 1 to 2, characterized in that, The step of determining the region image containing text and the text in the region image on the image to be detected includes: In response to a font detection request from a client, a region image containing text and the text in the region image are determined on the image to be detected. The font detection request is used to request the detection of the font of the text on the image to be detected.

9. A font detection method, applied to a client, characterized in that, include: In response to a user's interactive operation requesting the detection of text fonts in an image to be detected, a font detection request is sent to the server, the font detection request being used to detect the font of the text in the image to be detected; The system receives font detection results returned by the server. These results are related to the target font of the text in the region image of the image to be detected. The target font is determined by performing image classification and target segmentation on the region image and combining this with the text in the region image. The target segmentation is based on a target segmentation model, which includes an edge detection network, an attention network, and a feature pyramid network. The edge detection network performs text edge detection on the region image to obtain a text edge detection map. The attention network extracts features from the text edge detection map to obtain attention weights. The feature pyramid network fuses features from multiple network layers based on the attention weights to extract a feature map of the region image. Based on the feature map, text detection and text segmentation are performed by a detection network and a segmentation network, respectively, to obtain the text image corresponding to the region image.

10. The font detection method according to claim 9, characterized in that, The font detection result indicates that the target font is a copyright-protected font. After receiving the font detection result returned by the server, at least one of the following operations is performed: Display a prompt message to inform the user that the target font is a copyright-protected font, to purchase the target font, and / or to modify the font of the text on the image to be detected; Display the purchase page for the target font; Display the font modification page corresponding to the image to be detected.

11. A font detection device, applied to a server, characterized in that, include: The first determining unit is used to determine, on the image to be detected, a region image containing text and the text in the region image; A font recognition unit is used to perform image classification on the region image to obtain candidate fonts corresponding to the region image; A target segmentation unit is used to segment the region image to obtain a text image corresponding to the region image. The target segmentation is implemented based on a target segmentation model, which includes an edge detection network, an attention network, and a feature pyramid network. The edge detection network performs text edge detection on the region image to obtain a text edge detection map. The attention network extracts features from the text edge detection map to obtain attention weights. The feature pyramid network fuses features from multiple network layers based on the attention weights to extract a feature map of the region image. Based on the feature map, text detection and text segmentation are performed by a detection network and a segmentation network, respectively, to obtain the text image. The second determining unit is used to determine the target font to which the text in the region image belongs based on the text in the region image, the candidate font, and the text image.

12. A font detection device, applied to a client, characterized in that, include: The sending unit is used to send a font detection request to the server in response to an interactive operation in which a user requests the detection of the font of the text in the image to be detected. The font detection request is used to detect the font of the text in the image to be detected. A receiving unit is configured to receive the font detection result returned by the server. The font detection result is related to the target font to which the text in the region image of the image to be detected belongs. The target font is determined by performing image classification and target segmentation on the region image and combining the text in the region image. The target segmentation is implemented based on a target segmentation model, which includes an edge detection network, an attention network, and a feature pyramid network. The edge detection network performs text edge detection on the region image to obtain a text edge detection map. The attention network extracts features from the text edge detection map to obtain attention weights. The feature pyramid network fuses features from multiple network layers based on the attention weights to extract a feature map of the region image. Based on the feature map, text detection and text segmentation are performed by a detection network and a segmentation network, respectively, to obtain the text image corresponding to the region image.

13. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the electronic device to perform the font detection method according to any one of claims 1 to 10.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the font detection method according to any one of claims 1 to 10.

Images