Image-based font recognition method and device, electronic equipment and storage medium

Abstract

The application provides an image-based font recognition method and device, electronic equipment and storage medium. According to the embodiment of the application, first, the interference text elements outside the target text element are determined from the to-be-detected image, then the text relevance recognition result of the target image region outside the interference text element is obtained, which is used to represent at least one text paragraph region corresponding to the division of the target image region, and finally the target font corresponding to the target text element is obtained according to the font recognition result corresponding to the at least one text paragraph region. The above scheme can exclude the interference text element region in the to-be-detected image, so that the font recognition focuses on the target text element region in the to-be-detected image, reduces the calculation amount of font recognition, and further improves the efficiency of font recognition, which can adapt to the demand of image batch detection.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image-based font recognition method, apparatus, electronic device, and storage medium. Background Technology

[0002] In e-commerce, images are typically used to promote products, increase page views, drive traffic to products or stores, and boost sales. Some merchants also attract visitors by incorporating unique text into their images.

[0003] However, even fonts that come pre-installed on computers cannot be used commercially without permission. To avoid font infringement and protect the intellectual property rights of font owners, it is necessary to provide font detection services to assist in self-checking for copyright issues and mitigating copyright risks.

[0004] In related technologies, manual assistance is usually required to determine the image region where the text is located, resulting in low detection efficiency and a poor user experience. Summary of the Invention

[0005] This application provides an image-based font recognition method, apparatus, electronic device, and storage medium to solve one or more of the aforementioned technical problems.

[0006] In a first aspect, embodiments of this application provide an image-based font recognition method, including:

[0007] Identify interfering text elements other than the target text elements in the image to be detected;

[0008] Obtain the text association recognition result of the target image region other than the interfering text elements, wherein the text association recognition result represents at least one text paragraph region corresponding to the target image region;

[0009] Based on the font recognition results corresponding to the at least one text paragraph region, the target font corresponding to the target text element is obtained.

[0010] Secondly, embodiments of this application provide an image-based font recognition method, including:

[0011] Submit the target image to be recognized on the text recognition page;

[0012] Obtain the target font corresponding to the target image to be identified; the target font is determined based on the font recognition result corresponding to at least one text paragraph region, and the text paragraph region is obtained by dividing the target image region (excluding interfering text elements) based on the text association recognition result of the target image region;

[0013] The target font, along with corresponding infringement analysis and warning information and / or infringement handling strategies, are displayed on the text recognition page.

[0014] Thirdly, embodiments of this application provide an image-based font recognition device, comprising:

[0015] The interference text determination module is used to determine interference text elements other than the target text elements from the image to be detected;

[0016] The text association recognition module is used to obtain the text association recognition result of the target image region other than the interfering text elements, and the text association recognition result represents at least one text paragraph region corresponding to the target image region.

[0017] The target font acquisition module is used to obtain the target font corresponding to the target text element based on the font recognition result corresponding to the at least one text paragraph region.

[0018] Fourthly, embodiments of this application provide an image-based font recognition device, comprising:

[0019] The image submission module is used to submit the image to be detected on the text recognition page;

[0020] The target font acquisition module is used to acquire the target font corresponding to the image to be detected; the target font is determined based on the font recognition result corresponding to at least one text paragraph region, and the text paragraph region is obtained by dividing the target image region (excluding interfering text elements) based on the text association recognition result of the target image region.

[0021] The infringement information disclosure module is used to display the target font, the corresponding infringement analysis and warning information, and / or infringement handling strategies on the text recognition page.

[0022] Fifthly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method described in any of the above-mentioned embodiments.

[0023] Sixthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any of the preceding claims.

[0024] Compared with related technologies, this application has the following advantages:

[0025] According to the embodiments of this application, firstly, interfering text elements other than the target text element are identified in the image to be detected. Then, the text association recognition results of at least one text paragraph region corresponding to the target image region are obtained from the target image region other than the interfering text elements. Finally, the target font corresponding to the target text element is obtained based on the font recognition results corresponding to the at least one text paragraph region. This scheme can eliminate interfering text element regions in the image to be detected, allowing font recognition to focus on the target image region in the image to be detected, reducing the computational load of font recognition, thereby improving the efficiency of font recognition and meeting the needs of batch image detection. There is no need to manually confirm the image region corresponding to the text to be recognized; only the image needs to be provided to obtain the font recognition results of the target text element in the image. There are no special requirements for the image to be detected, making the operation simple. Furthermore, it can reduce the impact of interfering text element fonts on the accuracy of target text element font recognition.

[0026] Among them, the text correlation recognition of font recognition results belonging to the same text paragraph region in the image to be detected is equivalent to combining the related text to make a comprehensive judgment on the font, which eliminates some inaccurate text recognition results and can improve the accuracy of font recognition.

[0027] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0028] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this application and should not be construed as limiting the scope of this application.

[0029] Figure 1 A schematic diagram illustrating an application example of a font recognition scheme according to an embodiment of this application is shown;

[0030] Figure 2 A flowchart illustrating an image-based font recognition method according to an embodiment of this application is shown;

[0031] Figure 3 A flowchart illustrating another embodiment of the image-based font recognition method of this application is shown;

[0032] Figure 4 A structural block diagram of an image-based font recognition device according to an embodiment of this application is shown;

[0033] Figure 5A structural block diagram of an image-based font recognition device according to another embodiment of this application is shown; and

[0034] Figure 6 A block diagram of an electronic device used to implement embodiments of this application is shown. Detailed Implementation

[0035] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0036] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and all of them fall within the protection scope of the embodiments of this application.

[0037] Currently, most fonts on the market are copyrighted by font companies. Commercial use of these fonts as online images requires authorization from the font company and payment of corresponding copyright fees. However, due to various reasons, the number of font infringement cases has increased nearly a hundredfold in the past decade. Numerous e-commerce sellers and content organizations have suffered losses due to various copyright infringement lawsuits from different copyright holders. Lawyer's letters have become a common meme among sellers and self-media, with annual compensation payments reaching hundreds of millions of yuan. Therefore, it is necessary to provide a font recognition solution to identify the font type corresponding to text in online images, assisting in font copyright self-checks, protecting the intellectual property rights of font owners, and preventing font infringement.

[0038] In the related technologies prior to this application, manual assistance was required to determine the image region where the text was located. The font recognition efficiency was low, which could not meet the needs of batch image detection and increased the cost of using font recognition, resulting in a poor user experience.

[0039] In view of this, embodiments of this application provide a method, apparatus, electronic device, and storage medium for image-based font recognition to solve all or part of the above-mentioned technical problems.

[0040] Figure 1This is a schematic diagram illustrating an application example of the font recognition scheme according to an embodiment of this application. The scheme of this application embodiment can be implemented as a font recognition device. Corresponding to the font recognition device of this application, a client, page, or program function plugin can be provided for submitting font recognition requests. Individuals, merchants, font copyright holders, or corresponding platforms carrying images can provide the image to be detected based on the client, page, or program function plugin. The client, page, or program function plugin submits a font recognition request to the detection end, and the font recognition device at the detection end completes the font recognition and then returns the font recognition result.

[0041] Taking the example of merchants checking the fonts in banner ads, since banners often link to product or merchant webpages, merchants may use various fonts in banners to increase click-through rates, potentially leading to the risk of font copyright infringement. See also... Figure 1 After the merchant uploads the banner to the client-side text recognition page, the client sends a font recognition request to the detection terminal. The detection terminal has a font recognition device, which can include deep learning algorithms such as semantic segmentation, object detection, and text box detection models to identify interfering text elements in the image to be detected. This further identifies the target image region outside the interfering text elements and performs text association recognition. Then, based on the layout detection model, at least one text paragraph region corresponding to the target image region is divided, serving as the text association recognition result for the target image region. The font recognition results of one or more fonts within the same paragraph region are comprehensively evaluated to obtain the font recognition result for the entire paragraph region, which is then used as the target font for the target text element. The detection terminal sends the obtained font recognition results to the client for display on the text recognition page. Among the banner images uploaded by merchants, only “mini70, camera battery, CR2 battery x2” are the target text elements to be detected, while the other text elements are all interference text elements, which include three categories: object identification elements (the “Double” logo part in the banner); background elements (the text elements on the camera in the banner); and product object elements (the text elements on the battery in the banner).

[0042] In the above process, from the time the client sends a font recognition request to the detection end until the font detection result is output, the font recognition device at the detection end completes the entire process of image font recognition without any additional operations. There is no need to manually confirm the image region corresponding to the text to be recognized; only the image needs to be provided to obtain the font recognition result of the target text element in the image. There are no special requirements for the image to be detected, making the operation simple.

[0043] Secondly, a font recognition device performs text recognition on the image to be detected, identifying interfering text elements such as products, people, backgrounds, and logos. These interfering text elements might simply be text or logos on product packaging; detecting them not only wastes resources and reduces efficiency but may also affect the accuracy of the final results. Identifying interfering text elements allows the detection of these areas within the image to focus on the target image region, reducing computational load and improving efficiency, thus meeting the needs of batch image detection. Furthermore, it minimizes the impact of interfering text elements on the accuracy of target text element recognition.

[0044] Furthermore, the font recognition device divides the image to be detected into paragraphs. Since text elements in the image may belong to different parts of the image, some text elements may be processed as the same paragraph when edited into an image. Text within the same paragraph is more likely to use the same font. Therefore, by dividing the image into paragraphs, text correlation identification can be performed on font recognition results that may belong to the same text paragraph region. This is equivalent to combining related text for a comprehensive font judgment, eliminating some inaccurate text recognition results and improving the accuracy of font recognition.

[0045] It should be noted that the above-described font recognition scheme is merely an exemplary application scenario of this application, and the embodiments of this application can also be applied to any font recognition scenario. Besides the example shown above where merchants detect fonts in banners through a client, it could also be that font copyright holders detect fonts in images containing infringing fonts, or that the corresponding platform hosting the image (such as a trading platform or image website) detects fonts in images uploaded by third parties. This application does not impose any restrictions on this. The image to be detected can be, in addition to banners, online posters, WeChat official account images, website design interfaces, such as dynamic web pages in HTML5 (Hyper Text Markup Language 5) format. This application does not impose any restrictions on this.

[0046] The execution entity in this application embodiment can be an application, service, instance, functional module in software form, virtual machine (VM), container, or cloud server, or hardware device with data processing function (such as server or terminal device) or hardware chip (such as CPU, GPU, FPGA, NPU, AI accelerator card, or DPU). The device for implementing font recognition can be deployed on the computing device of the application provider offering the corresponding service or on a cloud computing platform providing computing power, storage, and network resources. The cloud computing platform can provide services in the following modes: IaaS (Infrastructure as a Service), PaaS (Platform as a Service), SaaS (Software-as-a-Service), or DaaS (Data-as-a-Service). Taking the platform providing SaaS (Software-as-a-Service) as an example, the cloud computing platform can utilize its own computing resources to provide model training or deploy font recognition devices. The specific application architecture can be built according to service requirements. For example, the platform can provide building services based on the above model to application users or individuals using platform resources, and further invoke the above model and implement the recognition function based on font recognition requests submitted by relevant client or server devices.

[0047] The technical solution of this application and how it solves the aforementioned technical problems are described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0048] like Figure 2 The diagram shown is a flowchart of an image-based font recognition method according to an embodiment of this application, which may include:

[0049] In step S201, interfering text elements other than the target text element are identified from the image to be detected.

[0050] The images involved in this application's embodiments include text elements and image elements. Text elements refer to text objects in the image, which can be various text forms such as Chinese, English, numbers, and symbols. The text object to be recognized can be identified as the target text element. Other text objects, since they do not require font recognition and their recognition may affect the font recognition result of the target text element, can be partially or entirely identified as interfering text elements. For example, when font recognition is required for advertising text in a banner, the advertising text is the target text element, and other parts that do not belong to the advertising text are interfering text elements that would interfere with the recognition of the advertising text. In specific implementations, the target text elements to be recognized can be determined according to the recognition requirements; this application does not impose any limitations on this. Image elements can be other content excluding text, such as icons, main objects, and background parts in the image.

[0051] Both the target text element and the distracting text element can be distributed in one or more regions, and each can be composed of one or more text forms.

[0052] It is understandable that identifying interfering text elements in the image to be detected and removing their influence on the target text elements during subsequent font recognition can improve detection efficiency and the accuracy of font recognition results.

[0053] There can be one or more images to be detected. For multiple images, concurrent detection can improve detection efficiency. An entry point for submitting images to be detected can be provided through a client, page, or program plugin. Requesters of font detection can upload images through this entry point. Alternatively, the device implementing font recognition can actively acquire the images to be detected. In one example, a control to trigger detection can be provided. Once triggered, the device implementing font recognition acquires the images to be detected according to a set path. For example, a "one-click full store detection" function can be provided to merchants. After the merchant triggers the corresponding control, all images uploaded by the merchant to the online platform can be actively searched as images to be detected.

[0054] In some embodiments, the aforementioned interfering text elements include at least one or more of the following: product object elements, background elements, or object identifier elements. Among them, the product object element can be a text object inherent in the main part of the image to be detected. For example, in the scenario where a merchant performs font detection on a banner on its product page, the product object element can be a text object inherent in the product displayed in the banner image, such as the text inherent in clothing, food, or daily necessities. The background element can be a text object inherent in the auxiliary part of the image to be detected. For example, in an image with a keyboard as the background, the keyboard may have text objects such as letters A, B, and C. The letters inherent in the keyboard itself are not the target for font detection. However, since the keyboard as the background may occupy all or most of the area of ​​the image, the area where the target text element is located overlaps with or directly overlaps with the area where the background element is located (for example, the commercial content to be displayed in the image to be detected is a keyboard cleaning cloth, while the advertising text content about the size and material of the cleaning cloth is displayed above the image area where the keyboard is located). In this case, the embodiments of this application can regard the target text element as the foreground and the keyboard and its inherent text as the background, thereby determining that the text inherent in the keyboard is the interference text element. For example, since background elements and target elements often reside in different network layers within the image to be detected, the position of background elements can be determined by detecting their pixel spatial locations. Object identifiers can be visual symbols, such as trademarks or logos of goods or businesses, or unique images that have undergone artistic processing.

[0055] In one possible implementation, the aforementioned interfering text elements include at least product object elements and background elements. Since product object elements and background elements in the image may contain text elements, but these text elements are not target text elements, the text inherent in the product object elements and background elements does not belong to the target text for font recognition and needs to be removed as interfering text. When determining interfering text elements other than target text elements from the image to be detected, the attention detection module of the semantic segmentation model can be invoked to detect product object elements and background elements in the image to be detected; the subject segmentation module of the semantic segmentation model can be invoked to perform edge segmentation on the product object elements and background elements, and the image regions delineated after edge segmentation are determined as the object image regions corresponding to the product object elements and background elements.

[0056] The semantic segmentation model's attention detection module classifies pixels in the image to be detected, assigning each pixel a category label based on its features and applying different weights to determine which pixels belong to the product object elements or background elements in the image. The semantic segmentation model's subject segmentation module performs edge segmentation on the pixels within the areas marked by the attention detection module, obtaining edge-segmented image regions. These regions are then designated as the object image regions corresponding to the product object elements or background elements. This allows text within these object image regions to be identified as interference text, which is then excluded from subsequent font recognition, saving resources and improving efficiency.

[0057] The attention detection module in the aforementioned semantic segmentation model extracts features from the image to be detected, obtaining a feature map. For example, the image to be detected can be input into the attention detection module, which extracts image features. This extraction can be done through a single network layer or multiple network layers, followed by merging the extracted feature maps to obtain a single overall feature map. When the image to be detected is divided into multiple regions, features can be extracted separately for each region, resulting in a feature map for that region. Furthermore, the attention detection module can apply different attention weights to different features in the obtained feature maps. Alternatively, the process of applying attention weights to different features can be parallel to the feature extraction process, applying attention weights to the feature maps of multiple network layers to facilitate the fusion of different feature layers in different networks and determine the influence weight of each feature layer on the output.

[0058] Optionally, the attention detection module in the semantic segmentation model obtains a weight distribution through pre-training and applies this weight distribution to the features extracted from the image. This weighting method can be to retain all components for weighting (i.e., soft attention), or to select a subset of components from the distribution using a sampling strategy for weighting (i.e., hard attention); it can be applied at the spatial scale, weighting different spatial regions; it can be applied at the channel scale, weighting different channel features; or it can be applied to the feature map, weighting each element. For example, the above attention mechanism can adopt the CV (channel attention) attention mechanism, applying the weight distribution to the channels to weight different channel features.

[0059] Optionally, the attention detection module in the semantic segmentation model can use a Feature Pyramid Network (FPN) to extract features from the image to be detected. For example, by using an FPN to extract features from the image to be detected, the feature map output by the last layer in the FPN is obtained. Since the resolution of the feature maps output by the top-down network layers in the FPN decreases layer by layer, the feature map output by the last layer has the highest resolution, which can make the features of the detected product object elements in the final feature map more accurate.

[0060] Furthermore, when using FPN for feature extraction, the aforementioned attention mechanisms, such as SAM (spatial attention), can be combined. This can improve the fusion effect of different network layers in FPN, compensating for the scale and semantic differences in feature maps of different network layers in FPN. This effectively alleviates the cue combination problem in FPN (where cues refer to network layers in FPN, and the cue combination problem refers to the problem of combining features with scale and semantic differences), and helps the semantic segmentation model obtain more edge information, making it less prone to incomplete edge segmentation even when small areas of edge loss occur. SAM can be used to utilize the spatial relationships between different feature maps, enabling the model to pay attention to the spatial positions of different features in the feature maps. For example, SAM can first perform global operations on the pixel values ​​at the same position on different feature maps along the axis to obtain two spatial attention maps and fuse them. Then, convolution is performed on this feature map, followed by a sigmoid function (S-shaped growth curve function) to obtain a spatial matrix (which can be a binary mask map) with the same dimension as the aforementioned language feature map and spatial attention weights. Finally, the obtained spatial matrix is ​​multiplied onto the original feature map to obtain a new feature map.

[0061] The main segmentation module of the semantic segmentation model described above can perform edge segmentation on the area where the product object element or background element is located, obtain the image region after edge segmentation, and define the region as the object image region corresponding to the product object element or background element.

[0062] Optionally, the main segmentation module of the semantic segmentation model can be implemented through the HED (Holistically-nested Edge Detection) mechanism. Compared with other edge detection mechanisms, the edge map generated by HED does a better job of preserving the boundaries of target objects in the image. It can preserve all the boundaries of the detected target, thereby improving the accuracy of edge detection.

[0063] For example, combining the aforementioned attention detection module, the feature map extracted by the attention detection module can be input into the HED mechanism. After edge detection by HED, the HED-processed feature map is output to SAM, which generates a weight map to assist in the fusion of different feature layers in the aforementioned FPN network. The weight map is equivalent to an attention weight layer running parallel to the feature extraction network, determining the influence weight of each feature layer on the output. In the top-down network layers of FPN, the feature map output by each network layer can be weighted with the attention weights, and the weighted result can be input into the HED mechanism until the final output feature map is obtained after SAM labeling. Alternatively, based on the correspondence between the top-down network layers and attention weights in FPN, the attention weights can be used as the weights in the weighting operation to perform a weighted sum of the feature maps output by each top-down network layer in FPN to obtain the final feature map.

[0064] In one possible implementation, the aforementioned interfering text elements include at least object identifier elements. Since object identifier elements are typically artistically rendered, unique images primarily serving to identify the source of goods, such as logos, they are not text elements that the client is concerned with that might constitute font infringement, and font detection is often unnecessary for the text within them. Object identifier elements are characterized by their generally unrotated or unflipped position within the image, their limited number, and their small size, requiring a more precise model for identification. Therefore, when determining interfering text elements other than the target text elements from the image to be detected, an object detection model can be invoked to detect the object identifier elements and the corresponding identifier image regions from the image to be detected. This object detection model is used for parallel detection of object identifier elements and their corresponding identifier image regions.

[0065] The object detection model involved samples a large number of regions in the image to be detected, then determines whether these regions contain the object of interest, and adjusts the region edges to more accurately predict the ground-truth bounding box of the object. The object detection model can be a deep learning model, such as the YOLO model (You Only Look Once: Unified, Real-Time Object Detection, an object detection system based on a single neural network). Compared to traditional object detection methods, YOLO offers an alternative approach: transforming the object detection problem into a regression problem, that is, given an input image, directly regressing the bounding box of the object and its classification category at multiple locations in the image. YOLO models include, but are not limited to, YOLOv3, YOLOv4, and YOLOv5 (all different versions of YOLO), etc., with different weights, network structures, algorithms, and region sampling methods.

[0066] The labeled image region involved can be the image region corresponding to the object label element, which is determined by the object detection model performing edge segmentation on the object label element within the region where the object label element is located.

[0067] Optionally, the object detection model also includes an FPN feature extraction module and a PAN (Pyramid Attention Network for Semantic Segmentation) feature extraction module, that is, it adopts the FPN+PAN mechanism. Traditional image feature extraction methods can perform upsampling on the image. This method is generally bottom-up sampling, so the highest layer of the image is the last layer. The higher the number of sampling layers, the larger the receptive field of the feature map, the more abstract the features, and the more semantic information, but the less positional information, and the lower the detection accuracy for small objects. If downsampling is performed on the image, it is generally top-down sampling. The lower the number of sampling layers, the less semantic information the feature map has, but more positional information. Although it can detect small objects, small objects are prone to misclassification. The advantage of using FPN and PAN modules for image feature extraction lies in their approach. Compared to traditional feature extraction networks, FPN is a top-down feature pyramid that passes down strong semantic features from higher levels to enhance the entire pyramid. However, it only enhances semantic information without passing on localization information (or rather, because the upward path is too long, the transmission effect is poor). PAN addresses this by adding a bottom-up pyramid after FPN, supplementing it and passing up low-level localization features. This results in a pyramid that combines both semantic and localization information, achieving a "double kill" effect. Therefore, FPN+PAN includes both bottom-up upsampling layers of different scales found in traditional methods and top-down downsampling layers of different scales, enabling the extraction of rich and diverse image features. It can not only detect small targets but also maintain the accuracy of classifying small targets, resulting in more detailed and accurate text boxes segmented based on image features.

[0068] For example, the object detection model YOLOv5l (a version of YOLO) can be used to detect logos in an image. Since logos are unique, artistically designed images, and the text within them is often specially designed and not based on any existing font, logos can be considered a special image element. By recognizing the features of pixels in the image, logos can be detected. The reason for using the YOLOv5l model is that, compared to other object detection models, it has a deeper network depth and a wider network width, resulting in a thicker feature map and stronger network feature extraction, feature fusion, and learning capabilities. Specifically, the pre-trained YOLOv5l object detection model downsamples the image to be detected five times. The feature maps obtained from these five downsamplings are then fused to obtain the feature map of the image to be detected. Simultaneously, the edges of the region containing the logo in the image are determined based on the feature map, and the edges are adjusted to finally determine the true bounding box of the logo region. The method for adjusting the region edges can be either to determine the region edges based on the results of each of the five downsampling processes and then merge the five region edges; or to directly determine the region edges based on the feature map after feature fusion. By performing multiple downsampling processes, the size of the image to be detected can be reduced, resulting in a smaller feature map. This allows for the rapid detection of pixels representing the logo with less computation, improving detection efficiency and accuracy.

[0069] Before using the Yolov 5l model to detect logo targets in an image, the model can be trained in advance. For example, input an image and tell the model the location of the logo in the image (if it exists). After multiple training sessions, the model can predict the location of the logo in a newly input image.

[0070] The way Yolov5l models detect logo targets can be divided into the following four parts:

[0071] First, at the input end, Mosaic data augmentation (a data augmentation mechanism), adaptive anchor box calculation, and / or adaptive image scaling can be used to perform preliminary processing on the image to be detected, so that the object detection model can better distinguish between object identifier elements and background elements, making the information in the feature map of the final output image to be detected more accurate. Mosaic data augmentation, through pre-training (a training method that stitches and overlays multiple images before identifying the target), can better distinguish between the target and the background in an image, enhancing the YOLOv 5l model's ability to differentiate between background elements and object identifiers. Adaptive anchor box computation is used to detect objects in network images and generates anchors of different sizes and aspect ratios on the unit elements of the network image feature map to determine candidate boxes. Compared to traditional object detection methods, which use a fixed-size window that slides step by step from left to right and from top to bottom according to a set schedule, adaptive anchor box computation is a sliding window that can be set to different sizes. It is more suitable for objects with large deformations, and the computational load is small with fewer windows. Adaptive anchor box computation can improve the accuracy of object detection. Furthermore, it can obtain the class probability (i.e., the probability that the region belongs to the object identifier or the background element) of multiple anchor boxes in the image to be detected and the bounding box position of the object identifier. The adaptive image scaling mechanism (letter box) improves detection performance by uniformly scaling input images of different sizes to the same size. However, this scaling process can lead to the loss of image information. Adaptive image scaling, on the other hand, can fully utilize the receptive field information in the network image. For example, in the feature map of the last layer of the YOLOv 5L model, each pixel corresponds to a 32x32 region in the original image. Therefore, as long as the overall image transformation ratio remains consistent, and both the length and width are divisible by 32, the information from the receptive field can be effectively utilized, improving the accuracy of feature extraction from the image to be detected.

[0072] Secondly, in the backbone structure, the Focus structure (a downsampling network structure) and the CSP structure (CSPNet, a cross-stage local fusion network structure that respects gradient variability by integrating feature maps at the beginning and end of network stages) can be used to ensure that the final output feature map of the target image retains more image features. Specifically, the Focus structure slices the image before it enters the backbone structure of the target detection model. This involves taking a value from every pixel in an image, similar to neighbor-to-neighbor downsampling, resulting in four complementary images without information loss. This concentrates image feature information into the channel space, expanding the input channels by four times. The resulting new image is then convolved to obtain a double-downsampled feature map without information loss, thus preserving more complete downsampled information. The CSP structure divides the original input into two branches, performs convolution operations on each branch to halve the number of channels, and then merges the two branches so that their input and output sizes are the same, allowing the detection model to detect more features.

[0073] Furthermore, in the feature fusion structure (i.e., the Neck side of the Yolov 5l model), the aforementioned FPN+PAN feature extraction module can be used to obtain downsampling layers of different scales from top to bottom, which can extract a rich variety of image features, taking into account both the ability to recognize small targets such as object identifiers and the accuracy of classification.

[0074] Finally, in the prediction structure, the GIOU Loss (Ground Truth Distance Loss Function) can be used. IoU is a distance metric that evaluates the relationship between two bounding boxes. This metric possesses all the characteristics of distance, including symmetry, nonnegativity, identity, and triangle inequality. The IoU loss function reflects the ratio between the intersection and union of two detection boxes, therefore it is independent of the size of the detection boxes but scale-dependent. For large and small targets with the same loss value, the detection performance of the large target is better than that of the small target, and it is less flexible in detecting small targets. The goal of GIoU is equivalent to adding a penalty to the IoU loss function based on the closure formed by the ground truth and the predicted bounding boxes. Its penalty term is the closure minus the area of ​​the union of the two boxes, and the smaller the proportion of the area in the closure, the better. When the two bounding boxes have no intersection, the IoU value is 0, while GIoU has different values ​​and is positively correlated with the detection performance. Therefore, GIoU loss optimizes the case where two bounding boxes do not overlap. When the two bounding boxes are very close, the values ​​of GIoU loss and IoU loss are very similar. In some scenarios, models using the two losses may perform similarly, but GIoU should have a faster convergence speed. Therefore, using GIoU Loss can better predict the location of small objects, thus enabling object detection models to more accurately predict the location of object identifiers in the image being detected.

[0075] In step S202, the text association recognition result of the target image region other than the interfering text elements is obtained. This text association recognition result represents at least one text paragraph region corresponding to the target image region. This step mainly addresses the problem that font recognition of the target image region alone may produce inaccurate recognition results. During image creation, text elements in the image may belong to different parts of the image, and some text elements may be treated as the same paragraph when edited into an image. Text within the same paragraph is more likely to use the same font. Therefore, by dividing into paragraphs, text association recognition can be performed on the font recognition results of possible text paragraph regions in the image to be detected. This is equivalent to combining the associated text for comprehensive font judgment, eliminating some inaccurate text recognition results, and improving the accuracy of font recognition.

[0076] The target image region involved in this application embodiment may contain the target text element to be detected, but does not contain interfering text elements. The number of target image regions may be one or more, and each target image region contains at least one target text element.

[0077] The text association recognition result is used to characterize at least one text paragraph region corresponding to the target image region, and then determine the text paragraph where the target text element is located. Specifically, the position, format, area, shape, image layer, character type, font recognition result, relative positional relationship between different target text elements, and differences in the number of characters of the target text element in the target image region can be extracted to perform association calculation and classify the associated text into the same text paragraph.

[0078] The text association recognition results mentioned above can be determined by a layout detection model. After determining the text paragraph region, font recognition can also be performed on target text elements belonging to the same paragraph, and the font recognition results of the entire target text can be output. The layout detection model can also be a pre-trained target detection model.

[0079] In this embodiment, the timing of text association recognition within the target image region can be determined to be before, after, or simultaneously with interfering text elements. The text within the target image region can be further divided into regions, and a text box detection model can be used to divide the text within the target image region into different text boxes, each of which can include one or more characters.

[0080] Optionally, the text box detection model includes an OCR (Optical Character Recognition) text recognition module. The character recognition capability based on OCR can improve the accuracy of the region image and the text in the region image.

[0081] Optionally, the text box detection model also includes an FPN feature extraction module. Further, a ResNet+FPN (i.e., residual neural network + feature pyramid network)-based text box detection model can be adopted. This model is suitable for object detection tasks and can set the detection target as a text box. This model can better adapt to possible deformations and angle transformations of text boxes in images (for example, the position of a text box in an image is not always horizontal; it may involve rotation or flipping, and this model can detect such text boxes). Specifically, ResNet is used to extract image feature maps from each layer of the image to be detected. The residual neural network can effectively solve the problems of gradient explosion and gradient vanishing that may occur as the network deepens during feature extraction. FPN is used to sample the image feature maps and obtain different features from each layer of the image feature maps. That is, the FPN feature extraction module in this model does not change the original backbone network but is more like an external device; the backbone network is still ResNet.

[0082] For example, the ResNet backbone outputs feature maps from multiple different network layers (possibly all layers from P0 to P4). The traditional approach is to extract features from these multiple output feature maps and then fuse them. However, since the resolution of feature maps from P0 to P4 decreases at twice the rate of each layer, the traditional method results in insufficiently high-resolution extracted features, impacting detection accuracy. Using the FPN pyramid network as a feature extractor allows for feature extraction only from layer P0, instead of adaptively selecting features from layers P0-P4, thus obtaining high-resolution features and improving detection accuracy. Furthermore, since the number of characters in the input image is limited, to allocate more GPU memory for subsequent high-resolution feature maps, the number of proposals (coordinates) in the extracted feature maps can be reduced from the standard 512 to 128. Experiments show that this reduction in the number of proposals does not affect the performance of the detection part. In short, ResNet+FPN extracts the feature maps from each layer of the image detected by ResNet and feeds them into the FPN for processing. Finally, the text box detection model outputs an image with the target text element after it has been marked by the text box.

[0083] In some embodiments, obtaining the text association recognition result of the target image region other than the interfering text elements can specifically include two methods: one is to obtain the font recognition result of the target image region other than the interfering text elements; the other is to perform text association recognition on the font recognition result of the target image region to obtain the text association recognition result of the target image region. Therefore, based on the font recognition result and the text association recognition result of the target image region, the target text elements can be further processed to finally obtain the corresponding target font.

[0084] For example, suppose there are three target image regions in the image to be detected, excluding interfering text elements. In each target image region, a text box detection model detects two text boxes containing the target text elements. Therefore, there are a total of six text boxes containing target text elements in the image to be detected. First, font recognition can be performed on the target text elements in these six text boxes. Then, text association recognition can be performed based on these six font recognition results. The weights of different types of text association recognition results can be pre-set to determine the text paragraph containing the target text element in each text box. For example, in these font recognition results, the font recognition result of the target text elements in two text boxes is "SimSun". Based on the association recognition of the positional relationship between these two target text elements, if they are found to be in the same target image region, they can be classified into the same text paragraph; if they are found to be in different target image regions, they can be classified into different text paragraphs. For example, even if the font recognition results of the target text elements in two text boxes are both "SimSun" and they are located in the same target image area, a comparison of the recognition results of other text associations of these two target text elements may reveal that they are located in different network layers or that the areas they occupy in the target image area are significantly different. Therefore, after comprehensive calculation based on the pre-set font recognition results, relative positional relationship, network layer, and area weights, these two target text elements may be classified into different text paragraphs.

[0085] One possible implementation is that obtaining the font recognition result of the target image region other than the interfering text element can include: determining the target image region other than the interfering text element based on the interfering image region where the interfering text element is located; performing font recognition on the target image region to obtain the font recognition result of the target image region.

[0086] In other words, based on the position of the interfering text elements in the image to be detected, the interfering image region where the interfering text elements are located can be defined. This allows us to determine that the regions in the image to be detected other than the interfering image region are the target image regions. Further, the text in the target image region is identified as the target text elements, and text recognition is performed on these target text elements to obtain the font recognition results of the target text elements in the target image region. There can be one or more interfering image regions, and there can also be one or more target image regions determined based on the interfering image regions; each target image region contains at least one target text element. Using this method, interfering text elements can be eliminated before the initial font recognition of the target text elements, narrowing down the area requiring font recognition and thus improving the efficiency of font recognition.

[0087] In another possible implementation, when obtaining the font recognition result of the target image region other than the interfering text element, the font recognition result obtained by performing font recognition on the image to be detected can be obtained first; the font recognition result corresponding to the interfering image region where the interfering text element is located can be removed from the font recognition result of the image to be detected to obtain the font recognition result of the target image region.

[0088] The process of obtaining font recognition results from the image to be detected can be achieved by performing font recognition on all text elements in the image, regardless of which region the text element belongs to. Alternatively, the obtained font recognition results can be obtained by using a text box detection model to identify text elements belonging to different text boxes. Other models, such as the aforementioned semantic segmentation model and object detection model, can be used to detect interfering text elements in the image. Based on the position of the interfering text elements in the image, the interfering image region where the interfering text elements are located is defined. Then, the font recognition results corresponding to the interfering image region where the interfering text elements are located are removed from the font recognition results of the image to obtain the font recognition results of the target image region. The order of font recognition and region segmentation is not particularly limited, and the method of performing font recognition on the target text elements is flexible. This can be done after obtaining the font recognition results of all text elements, before performing font recognition on the image to be detected, or simultaneously with font recognition.

[0089] In the embodiments of this application, font recognition can be achieved through at least the following steps: First, a hybrid recognition model is invoked to perform image recognition on the image region to be recognized, so as to detect the font of each character separately based on image features. This hybrid recognition model is trained to simultaneously perform font recognition and text recognition tasks. Second, a single-character recognition model is invoked to perform image recognition on the image region to be recognized, so as to detect the font of each character separately based on image features. Third, a holistic recognition model is invoked to perform image recognition on the image region to be recognized, so as to detect the font of all characters as a whole based on image features. The above three steps perform font recognition on the target text element in different ways. Therefore, if the above three steps are performed in parallel, it is equivalent to performing font recognition multiple times using different methods, which can enhance the accuracy of font detection and improve the accuracy rate of the font detection results.

[0090] The image region to be identified can be the target image region determined in the preceding steps based on the interfering image region where the interfering text elements are located; or it can be the entire or part of the image to be detected, which may include the interfering image region. Further, the image region to be identified can be segmented to obtain several smaller units of image region to be identified, and a text box detection model can be used to delineate one or more text elements belonging to the image region to be identified.

[0091] Existing technologies can only determine the font of the text content by comparing the text image in the image to be detected with font images in a pre-set database, i.e., by comparing features between individual images. This font recognition method not only has high storage requirements for the database, but also cannot recognize fonts in new images not stored in the database. In this embodiment, the font recognition of text in the image to be detected can be achieved not only by comparing individual images, but also by a hybrid "image + text" recognition method. Furthermore, constraints are placed on the accuracy of both image and text recognition tasks, ensuring that the results of the two tasks influence each other, thereby improving the accuracy of font recognition. Moreover, this embodiment does not have special requirements for the database or the image to be detected itself.

[0092] The font recognition method involved can be implemented through at least one of the three steps mentioned above, ultimately yielding at least one font recognition result. Different recognition models can be used in these different steps to perform font recognition on the text elements to be recognized in the target image region. These different font recognition models employ different technical approaches, and the font results for the same text element may be the same or different.

[0093] Furthermore, in some embodiments, the above-described font recognition method may further include: performing a weighted calculation on the various font recognition results using the recognition confidence scores corresponding to the various font detection results as weights, to obtain a weighted font recognition result. This can reduce the probability of inaccurate font recognition results for some texts and improve the accuracy of font recognition.

[0094] For example, three different recognition models based on CRNN (Convolutional Recurrent Neural Network) can be used to perform font recognition on the text elements to be recognized, and then the font results recognized by these different recognition models are weighted to output the weighted font corresponding to the text element to be recognized. In one example, CRNN includes a CNN (Convolutional Neural Network) feature extraction layer and an LSTM (Long Short Term Memory) sequence feature extraction layer, which can realize image-based sequence recognition. It is mainly used for end-to-end recognition of text sequences of variable length, without first segmenting individual characters, but transforming text recognition into a time-dependent sequence learning problem.

[0095] Optionally, the three different CRNN-based recognition models mentioned above include a text decoding module, used to decode the text in the text box after it has been delineated by the font detection model or recognized by the OCR module, and to determine the font corresponding to the text element to be recognized in the text box. The text decoding module can be one or more of the following: a CTC module (Connectionist Temporal Classification, a neural network-based temporal classification), an attention module (Attention Decoder, a decoding algorithm based on an attention mechanism), or an ACE module (Aggregation Cross-Entropy, a weakly supervised algorithm for solving sequence problems). In the model combining CRNN and the CTC module, firstly, the CNN in the CRNN extracts the image convolutional features, then the LSTM in the CRNN further extracts the sequence features from the image convolutional features, and finally, CTC is introduced to solve the problem of character misalignment during training.

[0096] One recognition model, named the CRNN-CTC-TEXT model, performs hybrid recognition of the image regions to be recognized. Hybrid recognition includes image recognition of the image regions to be recognized, performing font detection for each character based on image features; it also includes image recognition of the image regions to be recognized, and performing text detection for each character based on image features. Because the training process of the hybrid recognition model simultaneously constrains the recognition accuracy of both related tasks, the recognition results of the two tasks will influence each other, thus improving the accuracy of both font recognition and text recognition results.

[0097] Another recognition model can be named the CRNN-CTC-ACE model. This model performs image recognition on the image region to be identified, detecting the font of each character based on image features. This model does not concern itself with the content of the text; it only uses the model to output the font category corresponding to each character based on image features.

[0098] Another recognition model can be named the CRNN-attention model. It performs image recognition on the image region to be recognized, detecting the font of all text as a whole based on image features. This model doesn't care about the content of the text recognition; it only outputs the font category based on image features. Because this model does not contain a CTC module, it cannot handle variable-length tasks well. Therefore, this model does not adopt a character-by-character recognition approach; that is, it does not output the font category corresponding to each character in each text box, but only outputs the font category corresponding to the entire text box.

[0099] Furthermore, the outputs of different models in the above recognition task can be weighted and averaged. The recognition model will output the confidence score of the font category corresponding to each character or the entire text box. The confidence score is used as the weight for weighted calculation to output the font category of the text box.

[0100] Another possible approach is that, since text elements in an image may belong to different parts of the image, and some text elements may be processed as the same paragraph when edited into an image, the text within the same paragraph is more likely to use the same font. Therefore, by dividing into paragraphs, text correlation identification can be performed on the font recognition results of regions in the image to be detected that may belong to the same text paragraph. This is equivalent to combining the related text for a comprehensive font judgment, eliminating some inaccurate text recognition results and improving the accuracy of font recognition. For example, based on correlation identification results such as the location information of the target text element, it can be determined whether the target text element may belong to a paragraph. Then, by comparing the font recognition results of text elements within the paragraph, the possibility of some inaccurate text recognition results can be reduced. Correspondingly, when performing text correlation identification on the font recognition results of the target image region to obtain the text correlation identification results of the target image region, paragraph type detection can be performed on the target region image based on the font recognition results corresponding to at least one text paragraph region. This paragraph type can include tables, text paragraphs, or text titles; the image regions where font recognition results belonging to the same paragraph type are located are divided into the same text paragraph region.

[0101] In this embodiment of the application, the paragraph type detection method can be based on the object detection model of Faster RCNN (Faster Regions with CNN features). This model is suitable for object detection tasks and can set three detection targets, namely "table", "text paragraph" or "text title".

[0102] Faster-RCNN is a deep learning detection algorithm that integrates feature extraction, candidate box selection, bounding box regression, and classification into a single network by adding a Region Proposal Network (RPN) and an anchor mechanism, thereby effectively improving detection accuracy and efficiency. The anchor box is a multi-scale sliding window, also called a prior box, used to detect objects in the network image and generate anchors of different sizes and aspect ratios on the cell elements of the network image feature map to determine candidate boxes.

[0103] The detection steps of Faster R-CNN are as follows: First, input an image to be detected. Second, extract candidate regions: extract candidate regions from the input image and map these candidate regions to the final convolutional feature layer according to their spatial relationship. That is, after scaling the input image, it enters the convolutional layer to extract features to obtain a feature map. Then, the feature map is fed into the RPN network (Region Proposal Network, a type of fully convolutional network) to generate a series of possible candidate boxes. The task in the RPN network can be divided into two parts: one is classification: determining whether all preset anchors are positive or negative (i.e., whether there is a target within the anchor); the other is regression: using bounding box regression to correct the anchors to obtain more accurate proposals (coordinates). Therefore, the RPN network is equivalent to performing a part of the detection in advance, that is, determining whether there is a target and correcting the anchors to make the position of the candidate boxes more accurate. The third step is region normalization: Each candidate region on the original feature map and the convolutional layer feature map with candidate boxes output by the RPN is input into the RoI Pooling layer (Region of Interest Pooling) to obtain fixed-dimensional features. ROI Pooling processes proposals, pooling them completely into a fully connected layer without distortion, maintaining a fixed length, and preventing information loss. The fourth step is classification and regression: The extracted features are input into the fully connected layer for target classification and coordinate regression. For example, Softmax (normalized exponential function) can be used for classification, and the position of the candidate region can be regressed; alternatively, proposal feature maps (coordinate feature vectors) can be used to calculate the specific category of the candidate region within the candidate box, and then bounding box regression can be performed to obtain the final precise position of the candidate box.

[0104] For example, firstly, an image to be detected is input into the layout detection model. This image can be either a target detection region segmented by the aforementioned semantic segmentation model or an image detected by the aforementioned text box detection model. The Anchor mechanism in the layout detection model is then invoked to generate anchors of different sizes and aspect ratios on the input image. Secondly, the image feature points (fm) are fed into the RPN network. In the RPN, nine anchor boxes are predicted for each pixel in the input feature map. The center coordinates of the anchor boxes are the coordinates of the current pixel mapped to the input image. The width and height of the anchor boxes can be preset. The convolution in the RPN network does not change the resolution of the feature map, but the number of channels in the output feature map is related to the anchor boxes, indicating the probability that the anchor boxes are foreground or background. The RPN determines whether these anchor boxes (values ​​before coordinate adjustment) are positive or negative samples, thus identifying candidate boxes. If it is a negative sample, only the classification loss (classified as background) is calculated. If it is a positive sample, both the classification loss (classified as foreground) and the regression loss (correcting the coordinate values ​​of the anchor boxes) are calculated. Next, the original feature map and each candidate region on the convolutional layer feature map with candidate boxes output by the RPN are input into the RoI Pooling layer to obtain fixed-dimensional features. The RPN and Fast R-CNN share convolutional layer features. The Fast R-CNN model's task is to classify and regress the feature vector of each candidate box region input from the RPN. Then, based on the mapping of the candidate boxes, the ROI (Region of Interest) is obtained on the feature map, and ROI pooling is performed to obtain a fixed-length feature vector. Finally, the feature vector is fed into a fully connected layer to obtain the classification vector and location coordinates.

[0105] Optionally, the Faster-RCNN-based output model will have two outputs: Softmax (normalized exponential function) and bounding box regression. Softmax corresponds to the classification result, indicating the class of the current candidate box, while the bounding box regression outputs the position of the current candidate box on the image. In this embodiment, the output of softmax can be pre-defined to detect different targets; three detection targets can be set: "table," "text paragraph," or "text title."

[0106] In step S203, the target font corresponding to the target text element is obtained based on the font recognition result corresponding to the at least one text paragraph region.

[0107] The font recognition results involved can be obtained by segmenting the target text elements in different regions, resulting in the font recognition results for each region; or they can be obtained by performing multiple font recognition operations on the target text elements, resulting in all font recognition results; or they can be obtained by integrating and correcting (e.g., weighting) the font recognition results on the target text elements. The output font recognition results can be one type or multiple types; they can include the probability of each font recognition result simultaneously, or they can output only the font recognition results with the highest probability or a few types.

[0108] In this embodiment of the application, when obtaining the target font corresponding to the target text element based on the font recognition result corresponding to at least one text paragraph region, the target font corresponding to the target text element in the text paragraph region can be obtained by weighting the at least one font recognition result based on the area of ​​the text region corresponding to the at least one font recognition result in the text paragraph region and the recognition confidence.

[0109] In one possible implementation, regions that may belong to the same paragraph are segmented based on the features of text elements in the image to be detected. The segmentation can be based on the text box format within the target image region; text boxes conforming to table, paragraph, or heading formats are identified as belonging to the same paragraph region. Alternatively, a holistic character recognition (QR) can be performed on the text within the same paragraph region, outputting the text association recognition result within the same paragraph. Or, instead of performing a holistic QR, the text association recognition results of the text boxes within the same paragraph region are weighted based on the aforementioned text association recognition results in the target image region, and the weighted font detection results of the text boxes within the same paragraph are output. Since different text boxes correspond to different font confidence levels, font confidence can be set as the first weight, and text box area as the second weight. The ratio of the first and second weights is set, and the target font corresponding to the target text element within the paragraph is output through weighted calculation.

[0110] Optionally, font infringement analysis can be performed based on the identified target font, and corresponding infringement analysis warning information and / or infringement handling strategies can be provided.

[0111] The infringement analysis and early warning information involved in this application embodiment includes, but is not limited to: the target font, whether the target font is a copyrighted font, the copyright holder information of the target font, the price information of the target font, suggestions for changing the target font (e.g., which other fonts can be changed to without infringement risk), the method of obtaining font change services, the price of font change services, and the time, etc. The copyright holder information of the target font may include the copyright holder's name, contact information, font licensing information, font sales information, etc. Through this information, font detection results can be obtained, and further understanding can be achieved regarding whether the font on the image to be detected may have potential infringement risks, and the possible subsequent costs and time involved.

[0112] For example, when the identified target font belongs to a font that requires authorization from a font vendor, the client's display page can show which text in the image to be detected might belong to this type of font, the percentage of probability that the text in the image to be detected is the same font as the target font, and display the font with the highest similarity for reference; at the same time, suggestions for changing the font are provided, or image modification functions are provided, and professional designers will subsequently provide services to modify the font category in the image.

[0113] Figure 3 This is a flowchart of another embodiment of an image-based font recognition method according to this application, which may include:

[0114] In step S301, the image to be detected is submitted on the text recognition page.

[0115] The text recognition page can be deployed on a server or a client; it can be a static or dynamic page on various electronic media display screens such as mobile phones, desktop computers, laptops, tablets, and smartwatches; it can also be the display interface of program function plugins. This application does not impose any restrictions in this regard.

[0116] In step S302, the target font corresponding to the image to be detected is obtained; the target font is determined based on the font recognition result corresponding to at least one text paragraph region, and the text paragraph region is obtained by dividing the target image region from the image to be detected, excluding interfering text elements, based on the text correlation recognition result of the target image region.

[0117] The text paragraph regions, interfering text elements, target image regions, and text correlation recognition results involved can be referred to the concepts or process descriptions in other embodiments, and will not be repeated here.

[0118] Optionally, the font recognition result can be determined based on the matching degree between the recognized text in the image to be detected and the fonts in the font library. For example, candidate fonts corresponding to the target image region can be selected from multiple font types based on the probability of the target image region under multiple font types. If the probability of the region image under a certain font type in the font library is greater than the probability threshold, then that font type is determined as the candidate font corresponding to the target image region; or, the font types ranked first in descending order of probability can be selected to determine several candidate fonts corresponding to the target image region.

[0119] In one possible approach, if the matching degree between the text under the candidate font and the text image corresponding to the target image region is greater than the matching threshold, then the candidate font is determined as the target font to which the text in the target image region belongs. In another possible approach, among the text under the candidate font, the text under the candidate font with the highest matching degree with the text in the target image region is determined, and that candidate font is determined as the target font. In yet another possible approach, among the text under the candidate font, the text under the candidate font with the highest matching degree with the text in the target image region and a matching degree greater than the matching threshold is determined, and that candidate font is determined as the target font.

[0120] In step S303, the target font, along with corresponding infringement analysis and warning information and / or infringement handling strategies, are displayed on the text recognition page.

[0121] In one possible implementation, if the target font displayed on the text recognition page is a copyright-protected font, the client can also perform at least one of the following steps: First, display the target font and the probability that the target font may be a copyright-protected font, and provide the price for purchasing the target font or the method for modifying the target font, or provide suggestions for modifying the target font; second, display the purchase page for the target font, or display the purchase page for the target font modification service; finally, display the font modification page corresponding to the image to be detected.

[0122] Corresponding to the application scenarios and methods provided in the embodiments of this application, the embodiments of this application also provide an image-based font recognition device. For example... Figure 4 The diagram shown is a structural block diagram of an image-based font recognition device according to an embodiment of this application. This image-based font recognition device may include:

[0123] Interference text determination module 401 is used to determine interference text elements other than target text elements from the image to be detected.

[0124] The text association recognition module 402 is used to obtain the text association recognition result of the target image region other than the interfering text elements, and the text association recognition result represents at least one text paragraph region corresponding to the target image region.

[0125] The target font acquisition module 403 is used to obtain the target font corresponding to the target text element based on the font recognition result corresponding to the at least one text paragraph region.

[0126] In one possible implementation, the aforementioned interference text determination module may include:

[0127] The attention detection call submodule is used to call the attention detection module of the semantic segmentation model to detect product object elements and background elements from the image to be detected.

[0128] The main body segmentation call submodule is used to call the main body segmentation module of the semantic segmentation model to perform edge segmentation on the product object element and the background element, and to determine the image region defined after edge segmentation as the object image region corresponding to the product object element and the background element.

[0129] In another possible implementation, the aforementioned interference text determination module may include:

[0130] The object detection call submodule is used to call the object detection model to detect object identification elements and the identification image regions where the object identification elements are located in the image to be detected. The object detection model is used to detect object identification elements and corresponding identification image regions in parallel.

[0131] In some embodiments, the text association recognition module described above may include:

[0132] The recognition result acquisition submodule is used to acquire the font recognition results of the target image region other than the interfering text elements.

[0133] The text association recognition submodule is used to perform text association recognition on the font recognition results of the target image region to obtain the text association recognition results of the target image region.

[0134] In one possible implementation, the aforementioned recognition result acquisition submodule is specifically used to determine the target image region outside the interfering text element based on the interfering image region where the interfering text element is located; perform font recognition on the target image region to obtain the font recognition result of the target image region.

[0135] In another possible implementation, the above-mentioned recognition result acquisition submodule is specifically used to acquire the font recognition result obtained by performing font recognition on the image to be detected; and to remove the font recognition result corresponding to the interference image region where the interference text element is located from the font recognition result of the image to be detected, so as to obtain the font recognition result of the target image region.

[0136] In some embodiments, font recognition can be achieved through at least one of the following units:

[0137] The hybrid recognition unit is used to call the hybrid recognition model to perform image recognition on the image region to be recognized, so as to perform font detection on each character according to the image features. The hybrid recognition model is trained to perform font recognition and text recognition tasks simultaneously.

[0138] The single-character recognition unit is used to call the single-character recognition model to perform image recognition on the image region to be recognized, so as to perform font detection on each character according to the image features;

[0139] The overall recognition unit is used to call the overall recognition model to perform image recognition on the image region to be recognized, so as to perform font detection on all text as a whole based on image features.

[0140] In other embodiments, font recognition can also be achieved through at least one of the following units:

[0141] The recognition result weighting unit uses the recognition confidence scores corresponding to various font detection results as weights to perform weighted calculations on the various font recognition results, and obtains the weighted font recognition results.

[0142] In one possible implementation, the aforementioned text association recognition submodule may include:

[0143] A paragraph type detection unit is used to perform paragraph type detection on the target region image based on the font recognition result corresponding to the at least one text paragraph region. The paragraph type includes a table, a text paragraph, or a text title.

[0144] The paragraph region segmentation unit is used to divide the image regions where font recognition results of the same paragraph type are located into the same text paragraph region.

[0145] In some possible implementations, the target font acquisition module mentioned above may include:

[0146] The recognition result weighting submodule is used to perform a weighted calculation on the recognition result of at least one font based on the area of ​​the text region corresponding to the recognition result of at least one font in the text paragraph region and the recognition confidence, so as to obtain the target font corresponding to the target text element in the text paragraph region.

[0147] In some other possible implementations, the device may also include:

[0148] The font infringement analysis module performs font infringement analysis based on the identified target fonts and provides corresponding infringement analysis warning information and / or infringement handling strategies.

[0149] Corresponding to the application scenarios and methods provided in the embodiments of this application, the embodiments of this application also provide an image-based font recognition device. For example... Figure 5 The diagram shown is a structural block diagram of an image-based font recognition device according to an embodiment of this application, which may include:

[0150] The image submission module 501 is used to submit the image to be detected on the text recognition page.

[0151] The target font acquisition module 502 is used to acquire the target font corresponding to the image to be detected; the target font is determined based on the font recognition result corresponding to at least one text paragraph region, and the text paragraph region is obtained by dividing the target image region from the image to be detected, excluding interfering text elements, based on the text correlation recognition result of the target image region.

[0152] The infringement information disclosure module 503 is used to display the target font and the corresponding infringement analysis warning information and / or infringement handling strategy on the text recognition page.

[0153] The functions of each module in each device in the embodiments of this application can be found in the corresponding description in the above method, and they have corresponding beneficial effects, which will not be repeated here.

[0154] Figure 6 This is a block diagram of an electronic device used to implement embodiments of this application. For example... Figure 6 As shown, the electronic device includes a memory 601 and a processor 602. The memory 601 stores a computer program that can run on the processor 602. When the processor 602 executes the computer program, it implements the methods described in the above embodiments. The number of memories 601 and processors 602 can be one or more.

[0155] The electronic device also includes:

[0156] The communication interface 603 is used to communicate with external devices and perform data exchange and transmission.

[0157] If the memory 601, processor 602, and communication interface 603 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0158] Optionally, in a specific implementation, if the memory 601, processor 602, and communication interface 603 are integrated on a single chip, then the memory 601, processor 602, and communication interface 603 can communicate with each other through an internal interface.

[0159] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.

[0160] This application also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.

[0161] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0162] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0163] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0164] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0165] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0166] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0167] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0168] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0169] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0170] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0171] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image-based font recognition method, comprising: Identify interfering text elements other than the target text elements in the image to be detected; The text association recognition result of the target image region other than the interfering text element is obtained. The text association recognition result represents at least one text paragraph region corresponding to the target image region. The division of the text paragraph region is based on the text box format within the target image region, so as to determine the text boxes within the same text paragraph region according to the text box format. Based on the font recognition results corresponding to the at least one text paragraph region, the target font corresponding to the target text element is obtained; wherein, obtaining the target font corresponding to the target text element based on the font recognition results corresponding to the at least one text paragraph region includes: performing a weighted calculation on the at least one font recognition result based on the area of ​​the text region corresponding to the at least one font recognition result within the text paragraph region and the font recognition confidence level, to obtain the target font corresponding to the target text element within the text paragraph region; wherein, different text boxes correspond to different font recognition confidence levels, the font recognition confidence level is set as the first weight, the text box area is set as the second weight, and the font recognition results of the text boxes within the text paragraph region are weighted and calculated based on the pre-set ratio of the first weight and the second weight, so as to obtain the target font corresponding to the target text element within the text paragraph region.

2. The method according to claim 1, wherein, The interfering text elements include at least product object elements and background elements, and the step of determining interfering text elements other than the target text elements from the image to be detected includes: The attention detection module of the semantic segmentation model is invoked to detect product object elements and background elements from the image to be detected; The main segmentation module of the semantic segmentation model is invoked to perform edge segmentation on the product object element and the background element, and the image region defined after edge segmentation is determined as the object image region corresponding to the product object element and the background element.

3. The method according to claim 1, wherein, The interfering text elements include at least object identifier elements, and determining the interfering text elements other than the target text elements from the image to be detected includes: The object detection model is invoked to detect object identification elements and the identification image regions where the object identification elements are located in the image to be detected. The object detection model is used to detect object identification elements and corresponding identification image regions in parallel.

4. The method according to claim 1, wherein, The text association recognition result of the target image region other than the interfering text elements includes: Obtain the font recognition results for the target image region other than the interfering text elements; Text correlation recognition is performed on the font recognition results of the target image region to obtain the text correlation recognition results of the target image region.

5. The method according to claim 4, wherein, The step of obtaining the font recognition result of the target image region other than the interfering text elements includes: Based on the interfering image region where the interfering text element is located, determine the target image region outside the interfering text element; Font recognition is performed on the target image region to obtain the font recognition result of the target image region.

6. The method according to claim 4, wherein, The step of obtaining the font recognition result of the target image region other than the interfering text elements includes: Obtain the font recognition result obtained by performing font recognition on the image to be detected; Remove the font recognition results corresponding to the interfering image region where the interfering text element is located from the font recognition results of the image to be detected, and obtain the font recognition results of the target image region.

7. The method according to claim 5 or 6, wherein, The font recognition is achieved through at least one of the following steps: The hybrid recognition model is invoked to perform image recognition on the image region to be recognized, and to perform font detection on each character according to the image features. The hybrid recognition model is trained to perform font recognition and text recognition tasks simultaneously. The single-character recognition model is invoked to perform image recognition on the image region to be recognized, so as to detect the font of each character according to the image features; The overall recognition model is invoked to perform image recognition on the image region to be recognized, so as to perform font detection on all text as a whole based on image features.

8. The method according to claim 7, wherein, The font recognition also includes: The recognition results of various fonts are weighted by the recognition confidence scores corresponding to the detection results of various fonts, and a weighted calculation is performed to obtain the weighted font recognition results.

9. The method according to claim 4, wherein, The step of performing text association recognition on the font recognition results of the target image region to obtain the text association recognition results of the target image region includes: Based on the font recognition result corresponding to the at least one text paragraph region, paragraph type detection is performed on the target image region, and the paragraph type includes table, text paragraph or text title; Image regions containing font recognition results belonging to the same paragraph type are classified into the same text paragraph region.

10. The method according to claim 1, wherein, The method further includes: Font infringement analysis is performed based on the identified target fonts, and corresponding infringement analysis warning information and / or infringement handling strategies are provided.

11. An image-based font recognition method, comprising: Submit the image to be detected on the text recognition page; Obtain the target font corresponding to the image to be detected; The target font is determined based on the font recognition result corresponding to at least one text paragraph region. The text paragraph region is obtained by dividing the target image region from the image to be detected, excluding interfering text elements, based on the text correlation recognition result of the target image region. The division of the text paragraph region is based on the text box format within the target image region, so as to determine the text boxes within the same text paragraph region according to the text box format. The text recognition page displays the target font, along with corresponding infringement analysis and warning information and / or infringement handling strategies. The target font is the font corresponding to the target text element within the text paragraph region. The target font is obtained by weighting the at least one font recognition result based on the text region area and font recognition confidence level corresponding to the at least one font recognition result within the text paragraph region. Different text boxes correspond to different font recognition confidence levels. The font recognition confidence level is set as the first weight, and the text box area as the second weight. Based on the pre-set ratio of the first weight to the second weight, the font recognition results of the text boxes within the text paragraph region are weighted and calculated to obtain the target font corresponding to the target text element within the text paragraph region.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1-11.

13. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of any one of claims 1-11.

Images