Patent attached drawing number detection method and system

Abstract

The invention discloses a patent attached drawing number detection method and system, relates to the technical field of patents, and aims to solve the technical problem that the identification effect is not ideal when patent attached drawing number identification is performed by using a deep learning neural network model in the prior art. The method comprises the following steps: identifying and naming a number in each patent drawing in a training data set by using an identification frame; designing a target network structure, taking a text detection network as a text detection branch by the target network, and adding a text direction classification branch; training the target network by using the training data set; segmenting the patent drawing to be detected into a plurality of sub-graphs with overlapping areas; inputting the cut sub-graphs into a text detection model for prediction to obtain a number detection result and a corresponding number direction classification result; repeated numbers in the patent drawings to be detected are filtered out or incomplete numbers are spliced together, and the detected numbers are turned to be right according to the number direction classification result. The number detection rate can be improved, and the number identification effect is improved.

Description

Technical Field

[0001] The present application relates to the field of patent technology, and in particular to a method and system for detecting patent drawing numbers. Background Art

[0002] Patent drawings play an important role in patent application documents. They not only provide patent examiners with an intuitive display of technical solutions, but also help patent applicants express the specific implementation methods of technical solutions more clearly. Patent drawings usually contain elements such as diagrams, symbols, and numbers, where the numbers in the figures represent key technical features and their relationships. Accurately identifying the diagrams and numbers in patent drawings is of great significance to the review process of patent applications and the interpretation of technical solutions. Therefore, automated identification of numbers and graphics in patent drawings, especially in the fields of patent review, patent search, and technical analysis, has become a necessary technical means.

[0003] At present, the number recognition method in the patent drawings mainly relies on optical character recognition (OCR) technology, which is divided into OCR recognition based on traditional algorithms and OCR recognition based on deep learning. The traditional OCR method mainly relies on template matching and feature extraction of characters, and processes the image through a specific algorithm to recognize the character information therein. These methods can achieve good results in specific scenarios, especially for font rules and high image clarity. However, these methods often have low recognition accuracy when faced with complex situations (such as image blur, tilt, rotation or font differences), and poor adaptability to large-scale data sets. Compared with traditional algorithms, algorithms based on deep learning have significant advantages in feature extraction, processing complex situations (such as tilt, distortion, blur, font differences) and generalization capabilities.

[0004] The inventors realize that in patent drawings, the resolution of images often varies greatly, and there is no relatively fixed format for the numbering size. There are large images with small numbers (such as Figure 1 as shown) and small pictures with large numbers (as shown Figure 2 There are two extreme cases. Figure 1 In the patent drawings shown, the overall resolution of the drawings is 1000x543, but the width and height of the numbers are only 9x13; Figure 2 In the patent drawings shown, the overall resolution of the drawings is 224x209, but the width and height of the numbers are 68x23. Currently, when using a deep learning neural network model for number recognition, the entire image is scaled and then fed into the model, which results in unsatisfactory final recognition results. The reason for the poor recognition results is that the numbers are too small and are easily lost when the network extracts features.

[0005] Moreover, compared to natural scenes, in patent drawings, due to the shorter labels, lack of contextual information, and the existence of 90° or 270° rotation, the direction cannot be distinguished simply by the width and height of the detection box. After detection, it cannot be directly sent to the recognition network for recognition (the recognition effect is very poor). The text needs to be rotated (rotated 90° or 270°) first and then sent to the recognition network.

[0006] For a 90° rotation, see Figure 3 The number 110 in the image is cut out from the original image, and the small image obtained is as follows Figure 4 As shown in the figure, if the 110 small images captured according to the detection results are directly sent to the subsequent recognition network, they will be recognized incorrectly. They need to be rotated first before being sent to the subsequent recognition network; the 270° rotation can be seen in Figure 5 However, the existing deep learning-based image recognition method lacks a solution for correcting the rotated images. Summary of the invention

[0007] The present application provides a patent drawing number detection method and system, aiming to solve the technical problem that the existing patent drawing number recognition using deep learning neural network models has unsatisfactory recognition effect.

[0008] In a first aspect, a method for detecting patent drawing numbers comprises:

[0009] Label the training data set, mark the number in each patent drawing in the training data set with an identification box, and name the number as the encoding vector corresponding to the number direction;

[0010] Design a target network structure, wherein the target network uses a text detection network as a text detection branch and adds a text direction classification branch, wherein the text direction classification branch is used to obtain a text direction classification result;

[0011] Use the training data set to train the target network to obtain a trained text detection model;

[0012] The trained text detection model is deployed, the patent drawings to be detected are obtained, the patent drawings to be detected are preprocessed, and the patent drawings to be detected are divided into multiple sub-images with overlapping areas according to the aspect ratio and resolution of the patent drawings to be detected; and the cropped sub-images are scaled to the first target resolution size, and input into the trained text detection model for prediction, so as to obtain the number detection results and the corresponding number direction classification results;

[0013] Filter out duplicate numbers in the patent drawings to be detected or concatenate incomplete numbers, and convert the detected numbers to normal numbers based on the numbering direction classification results.

[0014] In the above scheme, optionally, when annotating the training data set, the PPOCRLabel annotation tool provided by PaddleOCR is used; the numbering directions include 0°, 90°, 180°, and 270°, the encoding vector corresponding to 0° is [1,0,0,0], the encoding vector corresponding to 90° is [0,1,0,0], the encoding vector corresponding to 180° is [0,0,1,0], and the encoding vector corresponding to 270° is [0,0,0,1].

[0015] In the above scheme, optionally, the text detection network is a DB detection network, and the text detection network includes MobileNetV3, DBFPN and DBHead; the text direction classification branch includes a first convolutional layer, a normalization layer, an activation layer and a second convolutional layer.

[0016] In the above solution, further optionally, using the training data set to train the target network includes:

[0017] Perform data augmentation on each patent drawing in the training dataset;

[0018] Each patent drawing in the training dataset is cropped to the first target resolution size and fed into the target network;

[0019] Each patent drawing is passed through the text detection network MobileNetV3 to obtain feature maps of four different resolutions: c2, c3, c4, and c5. The feature maps of c2, c3, c4, and c5 are passed through DBFPN to be fused into feature maps of the second target resolution, and then passed through DBHead to output probability maps and threshold maps.

[0020] The text direction classification branch uses the c5 feature map as input and outputs it after passing through the first convolutional layer, normalization layer, activation layer, and second convolutional layer;

[0021] Compute the detection branch loss based on the output of the text detection network , the classification branch loss is calculated based on the output of the text direction classification branch , and according to the detection branch loss and classification branch loss Calculate the model loss ;

[0022] Adjusting model loss using optimization algorithms Parameters.

[0023] In the above scheme, further optionally, the first target resolution is 960*960; the resolution of the c2 feature map is 240*240, the resolution of the c3 feature map is 120*120, the resolution of the c4 feature map is 60*60, and the resolution of the c5 feature map is 30*30; the second target resolution is 240*240.

[0024] In the above solution, further optionally, the branch loss is detected The calculation formula is:

[0025]

[0026] in, Used to measure the difference between the predicted text area and the true text area;

[0027]

[0028] in, represents the text region probability map predicted by the text detection branch, represents the true label map, is the number of all pixels;

[0029] Used to train the model to learn text boundaries;

[0030]

[0031] in, is the boundary threshold map of text detection branch prediction, is the true threshold label map, is the number of text border pixels;

[0032] Used to further increase the prediction accuracy of text areas;

[0033]

[0034] in, is the binary text region value predicted by the text detection branch, is the real binary text region label, is a constant, and is a hyperparameter;

[0035] Classification branch loss Using cross entropy loss, classification branch loss The calculation formula is:

[0036]

[0037] in, is the number of categories, is the encoding vector of the actual label, is the output of the text direction classification branch The probability distribution after the Softmax function is: The calculation formula is:

[0038]

[0039] Model Loss The calculation formula is:

[0040]

[0041] in, is the detection branch weight, is the classification branch weight.

[0042] In the above solution, further optionally, the text direction classification branch only calculates the classification loss for positive samples, and the selection of positive samples includes:

[0043] In the text detection branch, select the target whose real label IOU is greater than the first preset IOU threshold, calculate the center point position of the selected target, and proportionally correspond the center point position to a point in the feature map output by the text direction classification branch, and use the corresponding point as a positive sample of the text direction classification branch to participate in the classification branch loss. Calculation.

[0044] In the above scheme, optionally, the patent drawing to be detected is preprocessed, and the patent drawing to be detected is divided into multiple sub-images with overlapping areas according to the aspect ratio and resolution of the patent drawing to be detected, including:

[0045] Determine the interval range to which the aspect ratio of the patent drawing to be tested belongs;

[0046] When the aspect ratio of the patent drawing to be tested is less than or greater than , divide the patent drawing to be tested into three equal parts along the longest side, and extend the equal-division points in respective directions by a preset proportion, thereby obtaining three sub-images with overlapping areas;

[0047] When the aspect ratio of the patent drawing to be tested is In the range or located in In the interval, the patent drawing to be tested is divided into two equal parts along the longest side, and a preset proportion is extended in respective directions at the dividing point, so as to obtain two sub-images with overlapping areas;

[0048] When the aspect ratio of the patent drawing to be tested is Within the interval, determine whether the width and height resolutions of the patent drawing to be detected are greater than the preset resolution threshold; if the width or height resolution is greater than the preset resolution threshold, divide the patent drawing to be detected into two equal parts along the side of the width or height greater than the preset resolution threshold, and extend the equal-division point in each direction by a preset proportion, so as to obtain two sub-images with overlapping areas; if the width or height resolution is less than the preset resolution threshold, the patent drawing to be detected is not cropped;

[0049] in, , , , is the preset constant parameter, .

[0050] In the above scheme, optionally, the repeated numbers in the patent drawings to be detected are filtered out or the incomplete numbers are concatenated, including:

[0051] Calculate the IOU value of the two adjacent numbered identification boxes detected;

[0052] If the IOU value of two adjacent numbered identification boxes is greater than a second preset IOU threshold, filter out the identification box with a low detection score;

[0053] If the IOU value of two adjacent numbered identification boxes is greater than the third preset IOU threshold and less than the second preset IOU threshold, the outer identification box of the two identification boxes is taken as the merged number position.

[0054] In a second aspect, a patent drawing number detection system is provided, characterized in that it includes:

[0055] A training data set annotation module is used to annotate the training data set, mark the number in each patent drawing in the training data set with an identification box, and name the number as a coding vector corresponding to the number direction;

[0056] A target network structure design module is used to design a target network structure. The target network uses a text detection network as a text detection branch and adds a text direction classification branch. The text direction classification branch is used to obtain a text direction classification result.

[0057] The model training module is used to train the target network using the training data set to obtain a trained text detection model;

[0058] A detection module is used to deploy the trained text detection model, obtain the patent drawings to be detected, pre-process the patent drawings to be detected, and divide the patent drawings to be detected into multiple sub-images with overlapping areas according to the aspect ratio and resolution of the patent drawings to be detected; and scale the cropped sub-images to the first target resolution size, input them into the trained text detection model for prediction, and obtain the number detection results and the corresponding number direction classification results;

[0059] The post-processing module is used to filter out duplicate numbers in the patent drawings to be detected or to splice incomplete numbers together, and to convert the detected numbers into positive numbers according to the numbering direction classification results.

[0060] Compared with the prior art, this application has at least the following beneficial effects:

[0061] Based on further analysis and research on the problems of the prior art, the present application recognizes that the numbers in the patent drawings are relatively sparse and of different sizes. In the model training stage, the picture is sent to the model after being scaled and randomly cropped. However, in the test stage, if the whole picture is scaled and sent to the model, a large number of missed detections will occur; therefore, in the method provided in the embodiment of the present application, when the trained model is deployed for reasoning, a preprocessing link of the patent drawings to be detected is added. After the patent drawings to be detected are cropped according to the image resolution and aspect ratio, multiple overlapping sub-images are generated and then sent to the model for prediction. Finally, the prediction results of multiple sub-images are fused, and duplicates or integrated and truncated ones are removed, so as to avoid the situation where the number is too small and missed when performing number detection, greatly improve the detection rate of the number, and improve the number recognition effect; in addition, in response to the problem of the direction of the drawing, the target network constructed by the present application adds a text direction classification branch in the text detection network, which can make text direction classification predictions based on the text detection results, so as to finally realize the correction of the rotated numbers according to the direction prediction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 A schematic diagram of a patent drawing with a large figure and a small number;

[0063] Figure 2 A schematic diagram of a patent drawing with a small figure and a large number;

[0064] Figure 3 The patent drawings with 90-degree rotation in the numbers;

[0065] Figure 4 It is a small figure obtained by cutting out the number 110 from the patent drawing;

[0066] Figure 5 The patent drawings with the numbers rotated 270 degrees;

[0067] Figure 6 A specific flow chart of a method for detecting a patent drawing number provided for one embodiment of the present application;

[0068] Figure 7 This is a schematic diagram of the overall network structure of a target network model in one embodiment of the present application;

[0069] Figure 8 This is a schematic diagram of the network structure of the text direction classification branch CLSHead in one embodiment of the present application;

[0070] Fig. 9 A simplified overall flow chart of a patent figure number detection method provided for one embodiment of the present application;

[0071] Fig.10 This is a flow chart of the model deployment reasoning process in one embodiment of the present application;

[0072] Fig.11 A first sub-image with an overlapping area obtained by cropping an original image in an embodiment of the present application;

[0073] Fig.12 A second sub-image with an overlapping area obtained by cropping the original image in an embodiment of the present application;

[0074] Fig.13 In one embodiment of the present application, a numbered image is obtained by rotating a numbered thumbnail with an angle according to a direction prediction result;

[0075] Fig.14 A block diagram of the module architecture of a patent figure number detection system provided for one embodiment of the present application. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0077] In the description of this application: unless otherwise specified, "plurality" means two or more. The terms "first", "second", "third", etc. in this application are intended to distinguish the objects referred to, and do not have special meanings in terms of technical connotations (for example, they should not be understood as emphasizing the importance or order, etc.). Expressions such as "including", "comprising", "having", etc. also mean "not limited to" (certain units, components, materials, steps, etc.).

[0078] In one embodiment, Figure 6 As shown, a method for detecting patent figure numbers is provided, the method comprising the following steps:

[0079] S1, annotate the training data set, mark the number in each patent drawing in the training data set with an identification box, and name the number as the encoding vector corresponding to the number direction.

[0080] Among them, the identification box uses a rectangular box, and the encoding vector is a one-hot encoding vector.

[0081] Furthermore, the numbering directions include 0°, 90°, 180° and 270°, the coding vector corresponding to 0° is [1,0,0,0], the coding vector corresponding to 90° is [0,1,0,0], the coding vector corresponding to 180° is [0,0,1,0], and the coding vector corresponding to 270° is [0,0,0,1].

[0082] In other words, the first step is to label the training data set. Use the PPOCRLabel labeling tool provided by PaddleOCR to mark the numbers in the patent drawings with rectangular boxes. The names are the classification names of the number directions. There are 4 directions in total. 0 (one-hot is represented as [1,0,0,0]) represents 0°, 1 (one-hot is represented as [0,1,0,0]) represents 90°, 2 (one-hot is represented as [0,0,1,0]) represents 180°, and 3 (one-hot is represented as [0,0,0,1]) represents 270°. A total of 21,891 images were labeled.

[0083] S2, design the target network structure. The target network uses the text detection network as the text detection branch and adds a text direction classification branch. The text direction classification branch is used to obtain the text direction classification result.

[0084] Among them, the text detection network is a DB detection network, which includes MobileNetV3, DBFPN and DBHead; the text direction classification branch includes the first convolutional layer, normalization layer, activation layer and second convolutional layer.

[0085] For the OCR recognition method based on deep learning, end-to-end network and two-stage network can be used. The model with better effect is based on two-stage network, namely text detection network (based on CNN) and text recognition network (based on RNN). Compared with the end-to-end network, the two-stage network can be optimized separately in the two subtasks of detection and recognition. And because they do not share the backbone network, they can each choose a more suitable network structure for separate training. The update of a single model will not affect other models, and specific model tuning can be performed in complex scenarios.

[0086] The text detection network is a very important part in OCR recognition. Only after the text box position is detected can text recognition be performed later. The DB (Differentiable Binarization) text detection algorithm is currently a more commonly used algorithm. Compared with other algorithms, DB allows the model to directly output the precise boundaries of the text area from the image, and uses adaptive thresholds to generate text masks, which can increase the accuracy and consistency of the mask. Since only a small amount of post-processing is required, the DB algorithm maintains high accuracy and reduces overall computational overhead, and performs well in real-time applications.

[0087] In other words, in the second step, the network structure is modified, based on the DB detection network, which includes MobileNetV3 (backbone), DBFPN (neck) and DBHead (head), and CLSHead is added as the text direction classification branch to obtain the text direction classification result, which contains two layers of convolution (Conv+BN+ReLU+Conv).

[0088] S3, use the training data set to train the target network to obtain a trained text detection model.

[0089] Furthermore, using the training data set to train the target network specifically includes:

[0090] Perform data augmentation on each patent drawing in the training dataset;

[0091] Each patent drawing in the training dataset is cropped to the first target resolution size and fed into the target network;

[0092] Each patent drawing is passed through the text detection network MobileNetV3 to obtain feature maps of four different resolutions: c2, c3, c4, and c5. The feature maps of c2, c3, c4, and c5 are passed through DBFPN to be fused into feature maps of the second target resolution, and then passed through DBHead to output probability maps and threshold maps.

[0093] The text direction classification branch uses the c5 feature map as input and outputs it after passing through the first convolutional layer, normalization layer, activation layer, and second convolutional layer;

[0094] Compute the detection branch loss based on the output of the text detection network , the classification branch loss is calculated based on the output of the text direction classification branch , and according to the detection branch loss and classification branch loss Calculate the model loss ;

[0095] Adjusting model loss using optimization algorithms Parameters.

[0096] Among them, the first target resolution is 960*960; the resolution of the c2 feature map is 240*240, the resolution of the c3 feature map is 120*120, the resolution of the c4 feature map is 60*60, and the resolution of the c5 feature map is 30*30; the second target resolution is 240*240.

[0097] In other words, in the third step, the modified text detection network is used for training. The training images are finally cropped to a resolution of 960*960 after data enhancement and sent to the network. After MobileNetV3, four features of different resolutions, c2, c3, c4, and c5, are obtained, with resolutions of 240*240, 120*120, 60*60, and 30*30 respectively.

[0098] Then the text detection branch passes c2, c3, c4, and c5 through DBFPN and fuses them into a feature map with a resolution of 240*240. After passing through DBHead, the probability map and threshold map are obtained for subsequent loss calculation, back propagation, or as model prediction results. The text direction classification branch only uses the feature map of c5, which is output after passing through the first convolution layer, normalization layer, activation layer, and the second convolution layer. The number of output channels of the second convolution layer is 4 because the total number of classifications is 4.

[0099] The overall network structure of the target network model can also be found in Figure 7 The resolution of the image input is 960x960. After MobileNetV3 (backbone), four feature maps with different resolutions are output: c2 (16x240x240), c3 (16x120x120), c4 (24x60x60), and c5 (48x30x30). Among them, c5 is sent to CLSHead (head) for text direction classification, and c2, c3, c4, and c5 are sent to DBFPN (neck) together for feature fusion of different resolutions, and a feature map with a resolution of 96x240x240 is output, which is finally sent to DBHead (head) for text rectangular box prediction.

[0100] The network structure of the text direction classification branch CLSHead can also be seen Figure 8 In the CLSHead network structure, the first convolution layer is mainly used to extract the relevant features of the text direction, and the second convolution layer is mainly used to reduce the channel dimension to 4 for subsequent direction analysis. The resolution of the CLSHead output feature map is 30x30, that is, the receptive field of each point corresponding to the original image is 32x32.

[0101] Furthermore, the branch loss is detected The calculation formula is:

[0102]

[0103] in, Used to measure the difference between the predicted text area and the true text area; assuming is the text region probability map predicted by the text detection branch, is the true label map, then:

[0104]

[0105] in, is the number of all pixels;

[0106] It is used to train the model to learn text boundaries;

[0107]

[0108] in, is the boundary threshold map of text detection branch prediction, is the true threshold label map, is the number of text border pixels;

[0109] It is to further increase the prediction accuracy of the text area;

[0110]

[0111] in, is the binary text region value predicted by the text detection branch, is the real binary text region label, is a small constant, which is defined as 1e-6 in the embodiment of the present application; and are hyperparameters, which are defined as 5 and 10 respectively in the embodiment of the present application;

[0112] Classification branch loss (i.e. classification loss function) uses cross entropy loss, and only calculates classification loss for positive samples, classification branch loss The calculation formula is:

[0113]

[0114] in, is the number of categories, is the one-hot encoding vector of the actual label, such as [0,1,0,0], is the output of the text direction classification branch The probability distribution after the Softmax function is: The calculation formula is:

[0115]

[0116] Finally, the classification branch loss and the detection branch loss Putting it all together, the model loss The calculation formula is:

[0117]

[0118] Among them, is the detection branch loss, is the classification branch loss, is the detection branch weight, is the classification branch weight; in this application , , the hyperparameter is determined after parameter tuning. The parameter tuning method is: select ( , ), , ), , ) and other 9 combinations for training and verification.

[0119] Specifically, the text direction classification branch only calculates the classification loss for positive samples, and the selection of positive samples includes:

[0120] In the text detection branch, select the target whose real label IOU is greater than the first preset IOU threshold, calculate the center point position of the selected target, and proportionally correspond the center point position to a point in the feature map output by the text direction classification branch, and use the corresponding point as a positive sample of the text direction classification branch to participate in the classification branch loss. Calculation.

[0121] The first preset IOU threshold is 0.2. In other words, for the selection of positive samples for classification, firstly, in the detection branch, select the target that satisfies the true label IOU greater than 0.2, calculate the center point position of the target, and proportionally correspond the position to a point in the 30*30 feature map output by the classification branch, and use the point as the positive sample of the classification branch to participate in the loss calculation.

[0122] S4, deploy the trained text detection model, obtain the patent drawings to be detected, pre-process the patent drawings to be detected, and divide the patent drawings to be detected into multiple sub-images with overlapping areas according to the aspect ratio and resolution of the patent drawings to be detected; and scale the cropped sub-images to the first target resolution size, input them into the trained text detection model for prediction, and obtain the number detection results and the corresponding number direction classification results.

[0123] Furthermore, the patent drawings to be detected are preprocessed, and the patent drawings to be detected are divided into multiple sub-images with overlapping areas according to the aspect ratio and resolution of the patent drawings to be detected, including:

[0124] Determine the range of the aspect ratio of the patent drawing to be tested;

[0125] When the aspect ratio of the patent drawing to be tested is less than or greater than , divide the patent drawing to be tested into three equal parts along the longest side, and extend the equal-division points in respective directions by a preset ratio, thereby obtaining three sub-images with overlapping areas;

[0126] When the aspect ratio of the patent drawing to be tested is In the range or located in In the interval, the patent drawing to be tested is divided into two equal parts along the longest side, and the equal division points are extended in respective directions by a preset ratio, thereby obtaining two sub-images with overlapping areas;

[0127] When the aspect ratio of the patent drawing to be tested is Within the interval, determine whether the resolution of the width and height of the patent drawing to be detected is greater than the preset resolution threshold; if the resolution of the width or height is greater than the preset resolution threshold, divide the patent drawing to be detected into two equal parts along the side of the width or height greater than the preset resolution threshold, and extend 10% in each direction at the dividing point, so as to obtain two sub-images with overlapping areas; if the resolution of the width or height is less than the preset resolution threshold, the patent drawing to be detected is not cropped;

[0128] in, , , , is the preset constant parameter, . Optionally, It can be set to 0.35. It can be set to 0.5. Can be set to 2, It can be set to 2.8, and the preset ratio can be 10%.

[0129] In other words, in the fourth step, the trained text detection model is deployed on the server for application. In the testing phase, due to the large differences in the resolution and aspect ratio of the patent drawings, the numbers in the drawings are sparse and of different sizes, so it is necessary to add an image preprocessing step. This step divides the image into sub-images with overlapping areas according to the image aspect ratio and resolution.

[0130] The segmentation principle is: (1) For aspect ratios less than 0.35 or greater than 2.8, the longest side is divided into three equal parts. Considering the overlapping area, the points of division are extended by 10% in each direction, thus obtaining three sub-images with overlapping areas; (2) For aspect ratios between 0.35 and 0.5 and between 2 and 2.8, the longest side is divided into two equal parts. Considering the overlapping area, the points of division are extended by 10% in each direction, thus obtaining two sub-images with overlapping areas; (3) For aspect ratios between 0.5 and 2, if the resolution of the width (height) is greater than 800, the width (height) side needs to be divided into two equal parts to obtain the corresponding number of sub-images. If the resolution is less than 800, no cropping is required.

[0131] The cropped sub-image is scaled to 960*960 and fed into the network for prediction. By giving a threshold, the text box positions that meet the conditions are screened out, and their center points are mapped to the classification branch output to obtain the text direction classification result.

[0132] S5, filter out duplicate numbers in the patent drawings to be detected or concatenate incomplete numbers, and convert the detected numbers into positive numbers according to the numbering direction classification results.

[0133] Furthermore, duplicate numbers in the drawings of the patent to be tested are filtered out or incomplete numbers are concatenated, including:

[0134] Calculate the IOU value of the two adjacent numbered identification boxes detected;

[0135] If the IOU value of two adjacent numbered identification boxes is greater than a second preset IOU threshold, filter out the identification box with a low detection score;

[0136] If the IOU value of two adjacent numbered identification boxes is greater than the third preset IOU threshold and less than the second preset IOU threshold, the outer identification box of the two identification boxes is taken as the merged number position.

[0137] Among them, the second preset IOU threshold may be 0.8, and the third preset IOU threshold may be 0.1.

[0138] In other words, in the fifth step, due to the overlapping areas of the sub-images, duplicate number detection may occur, and post-processing is required to filter out duplicate numbers or splice incomplete numbers together.

[0139] Filtering or splicing principle: If the IOU of two numbered rectangular boxes is greater than 0.8, the rectangular box with a low detection score is filtered out. If the IOU is between 0.1 and 0.8, the circumscribed rectangular box of the two rectangular boxes is taken as the merged number position.

[0140] In the embodiments of the present application, Fig. 9 As shown in the figure, the overall steps in the above method can also be briefly summarized into three main steps: data labeling, model training, and model deployment reasoning. Fig.10 As shown, the model deployment reasoning link includes the entire reasoning process, which includes image preprocessing of the patent illustration to be detected, splitting the entire image into overlapping sub-images, and then sending them to the model for prediction, and finally fusing the prediction results of multiple sub-images to remove duplicates or integrate truncated ones.

[0141] In the patent drawings, the numbers are sparse and of different sizes. During the model training phase, the images are scaled and randomly cropped before being fed into the model. However, during the test phase, if the entire image is scaled and fed into the model, a large number of missed detections will occur. Therefore, it is necessary to increase the image preprocessing process, crop the image according to the image resolution and aspect ratio, generate multiple sub-images, and then feed them into the model, so as to improve the number detection rate (steps S4 and S5). The original patent drawing can be found in Figure 1 The two sub-images with overlapping areas obtained by cropping the original patent drawing can be found in Fig.11 and Fig.12 .

[0142] To solve the text direction problem, we add a text direction classification branch to the DB detection network, and send the backbone output c5 to the text direction classification branch. Based on the text detection results, we make a text direction classification prediction. The model detection uses the obtained number position to cut out a small image with an angle (see Figure 4 ), and then rotate the small image according to the direction prediction result (see Fig.13 ).

[0143] Using the method provided in the embodiment of the present application, in the test phase, 100 patent documents (including 824 drawings, a total of 6281 numbers) were used. Without image preprocessing (cropped into sub-images according to rules), the detection rate of the number was only 71.3%, while for image preprocessing, the detection rate can be increased to 98.5%. After adding the classification branch, the detection rate of the number was 98.4%, which was basically the same as before, and the accuracy of the direction classification was 99.2%. No distinction is made for horizontal or vertical inversion symmetric numbers, i.e. 6 and 9, 0 and 0, 8 and 8). Due to the shared backbone network, while ensuring that the detection effect is basically the same, higher-precision direction information can be given, thereby reducing computational overhead.

[0144] In one embodiment, Fig.14 As shown, a patent figure number detection system is provided, including the following modules:

[0145] The training data set annotation module 141 is used to annotate the training data set, mark the number in each patent drawing in the training data set with an identification box, and name the number as a coding vector corresponding to the number direction;

[0146] A target network structure design module 142 is used to design a target network structure. The target network uses a text detection network as a text detection branch and adds a text direction classification branch. The text direction classification branch is used to obtain a text direction classification result.

[0147] A model training module 143 is used to train the target network using the training data set to obtain a trained text detection model;

[0148] The detection module 144 is used to deploy the trained text detection model, obtain the patent drawings to be detected, pre-process the patent drawings to be detected, and divide the patent drawings to be detected into multiple sub-images with overlapping areas according to the aspect ratio and resolution of the patent drawings to be detected; and scale the cropped sub-images to the first target resolution size, input them into the trained text detection model for prediction, and obtain the number detection results and the corresponding number direction classification results;

[0149] The post-processing module 145 is used to filter out repeated numbers in the patent drawings to be detected or to splice incomplete numbers together, and to convert the detected numbers into positive numbers according to the numbering direction classification results.

[0150] The specific implementation content of each module can be found in the above definition of a patent figure number detection method, which will not be repeated here.

[0151] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which involves all or part of the processes in the above-mentioned embodiment method.

[0152] In one embodiment, a computer program product is also provided, including a computer program / instruction, which implements all or part of the process in the above-mentioned embodiment method when executed by a processor.

[0153] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A method for detecting patent drawing numbers, characterized in that: include: Label the training data set, mark the number in each patent drawing in the training data set with an identification box, and name the number as the encoding vector corresponding to the number direction; Design a target network structure, wherein the target network uses a text detection network as a text detection branch and adds a text direction classification branch, wherein the text direction classification branch is used to obtain a text direction classification result; Use the training data set to train the target network to obtain a trained text detection model; The trained text detection model is deployed, the patent drawings to be detected are obtained, the patent drawings to be detected are preprocessed, and the patent drawings to be detected are divided into multiple sub-images with overlapping areas according to the aspect ratio and resolution of the patent drawings to be detected; and the cropped sub-images are scaled to the first target resolution size, and input into the trained text detection model for prediction, so as to obtain the number detection results and the corresponding number direction classification results; Filter out duplicate numbers in the patent drawings to be detected or concatenate incomplete numbers, and convert the detected numbers to normal numbers based on the numbering direction classification results.

2. The patent drawing number detection method according to claim 1 is characterized in that: When annotating the training data set, the PPOCRLabel annotation tool provided by PaddleOCR was used; the numbering directions include 0°, 90°, 180°, and 270°. The encoding vector corresponding to 0° is [1,0,0,0], the encoding vector corresponding to 90° is [0,1,0,0], the encoding vector corresponding to 180° is [0,0,1,0], and the encoding vector corresponding to 270° is [0,0,0,1].

3. The patent drawing number detection method according to claim 1 is characterized in that: The text detection network is a DB detection network, which includes MobileNetV3, DBFPN and DBHead; the text direction classification branch includes a first convolutional layer, a normalization layer, an activation layer and a second convolutional layer.

4. The patent drawing number detection method according to claim 3 is characterized in that: Training the target network using the training dataset includes: Perform data augmentation on each patent drawing in the training dataset; Each patent drawing in the training dataset is cropped to the first target resolution size and fed into the target network; Each patent drawing is passed through the text detection network MobileNetV3 to obtain feature maps of four different resolutions: c2, c3, c4, and c5. The feature maps of c2, c3, c4, and c5 are passed through DBFPN to be fused into feature maps of the second target resolution, and then passed through DBHead to output probability maps and threshold maps. The text direction classification branch uses the c5 feature map as input and outputs it after passing through the first convolutional layer, normalization layer, activation layer, and second convolutional layer; Compute the detection branch loss based on the output of the text detection network , the classification branch loss is calculated based on the output of the text direction classification branch , and according to the detection branch loss and classification branch loss Calculate the model loss ; Adjusting model loss using optimization algorithms Parameters.

5. The method for detecting patent drawing numbers according to claim 4, characterized in that: The first target resolution is 960*960; the resolution of the c2 feature map is 240*240, the resolution of the c3 feature map is 120*120, the resolution of the c4 feature map is 60*60, and the resolution of the c5 feature map is 30*30; the second target resolution is 240*240.

6. The patent drawing number detection method according to claim 4 is characterized in that: Detection branch loss The calculation formula is: ； in, Used to measure the difference between the predicted text area and the true text area; ； in, represents the text region probability map predicted by the text detection branch, represents the true label map, is the number of all pixels; Used to train the model to learn text boundaries; ； in, is the boundary threshold map of text detection branch prediction, is the true threshold label map, is the number of text border pixels; Used to further increase the prediction accuracy of text areas; ； in, is the binary text region value predicted by the text detection branch, is the real binary text region label, is a constant, and is a hyperparameter; Classification branch loss Using cross entropy loss, classification branch loss The calculation formula is: ； in, is the number of categories, is the encoding vector of the actual label, is the output of the text direction classification branch The probability distribution after the Softmax function is: The calculation formula is: ； Model Loss The calculation formula is: ； in, is the detection branch weight, is the classification branch weight.

7. The method for detecting patent drawing numbers according to claim 6, characterized in that: The text direction classification branch only calculates the classification loss for positive samples. The selection of positive samples includes: In the text detection branch, select the target whose real label IOU is greater than the first preset IOU threshold, calculate the center point position of the selected target, and proportionally correspond the center point position to a point in the feature map output by the text direction classification branch, and use the corresponding point as a positive sample of the text direction classification branch to participate in the classification branch loss. Calculation.

8. The patent drawing number detection method according to claim 1 is characterized in that: The patent drawings to be detected are preprocessed, and the patent drawings to be detected are divided into multiple sub-images with overlapping areas according to the aspect ratio and resolution of the patent drawings to be detected, including: Determine the interval range to which the aspect ratio of the patent drawing to be tested belongs; When the aspect ratio of the patent drawing to be tested is less than or greater than , divide the patent drawing to be tested into three equal parts along the longest side, and extend the equal-division points in respective directions by a preset proportion, thereby obtaining three sub-images with overlapping areas; When the aspect ratio of the patent drawing to be tested is In the range or located in In the interval, the patent drawing to be tested is divided into two equal parts along the longest side, and a preset proportion is extended in respective directions at the dividing point, so as to obtain two sub-images with overlapping areas; When the aspect ratio of the patent drawing to be tested is Within the interval, determine whether the width and height resolutions of the patent drawing to be detected are greater than the preset resolution threshold; if the width or height resolution is greater than the preset resolution threshold, divide the patent drawing to be detected into two equal parts along the side of the width or height greater than the preset resolution threshold, and extend the equal-division point in each direction by a preset proportion, so as to obtain two sub-images with overlapping areas; if the width or height resolution is less than the preset resolution threshold, the patent drawing to be detected is not cropped; in, , , , is the preset constant parameter, .

9. The patent drawing number detection method according to claim 1 is characterized in that: Filter out duplicate numbers in the patent drawings to be tested or concatenate incomplete numbers, including: Calculate the IOU value of the two adjacent numbered identification boxes detected; If the IOU value of two adjacent numbered identification boxes is greater than a second preset IOU threshold, filter out the identification box with a low detection score; If the IOU value of two adjacent numbered identification boxes is greater than the third preset IOU threshold and less than the second preset IOU threshold, the outer identification box of the two identification boxes is taken as the merged number position.

10. A patent drawing number detection system, characterized in that: include: A training data set annotation module is used to annotate the training data set, mark the number in each patent drawing in the training data set with an identification box, and name the number as a coding vector corresponding to the number direction; A target network structure design module is used to design a target network structure. The target network uses a text detection network as a text detection branch and adds a text direction classification branch. The text direction classification branch is used to obtain a text direction classification result. The model training module is used to train the target network using the training data set to obtain a trained text detection model; A detection module is used to deploy the trained text detection model, obtain the patent drawings to be detected, pre-process the patent drawings to be detected, and divide the patent drawings to be detected into multiple sub-images with overlapping areas according to the aspect ratio and resolution of the patent drawings to be detected; and scale the cropped sub-images to the first target resolution size, input them into the trained text detection model for prediction, and obtain the number detection results and the corresponding number direction classification results; The post-processing module is used to filter out duplicate numbers in the patent drawings to be detected or to splice incomplete numbers together, and to convert the detected numbers into positive numbers according to the numbering direction classification results.

Images