Billet character recognition method, system and equipment based on multi-region collaboration and medium

Abstract

The invention provides a billet character recognition method, system and device based on multi-region collaboration and a medium, and belongs to the technical field of computer vision. The method comprises the steps that a to-be-detected billet image is acquired, the to-be-detected billet image comprises a plurality of redundant first character texts printed at different positions of the surface of a billet, and each first character text represents first production information of the billet; positioning and intercepting a plurality of first character texts contained in the to-be-detected billet image to obtain a plurality of corresponding first character text region images; performing feature extraction processing on the plurality of first character text region images to obtain a plurality of corresponding first character text features; and performing fusion identification on the plurality of first character text features to obtain a character identification result. According to the invention, the robustness of billet character recognition can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a billet character recognition method, system, device and medium based on multi-region cooperation. BACKGROUND

[0002] In the prior art, a deep learning framework can be used to realize basic character recognition functions, but at least the following defects exist when it is used in a billet industrial environment: In order to ensure the readability of information, redundant characters with the same content are usually sprayed on different positions on the surface of a billet as backups, but the existing deep learning framework often ignores this redundant annotation characteristic of billet characters in industrial printing. When uneven illumination, partial characters are corroded and blocked, etc. occur in a certain region, the existing deep learning framework lacks the ability to obtain information from other regions for supplementation, thereby limiting the improvement of the robustness of billet character recognition. SUMMARY

[0003] The main purpose of the present application is to propose a billet character recognition method, system, device and medium based on multi-region cooperation, aiming to improve the robustness of billet character recognition.

[0004] To achieve the above purpose, one aspect of the present application proposes a billet character recognition method based on multi-region cooperation, which comprises:

[0005] obtaining a to-be-tested billet image, the to-be-tested billet image containing a plurality of redundant first character texts sprayed on different positions on the surface of a billet, each first character text representing first production information of the billet;

[0006] locating and cutting the plurality of first character texts contained in the to-be-tested billet image to obtain a plurality of corresponding first character text region images;

[0007] performing feature extraction processing on the plurality of first character text region images to obtain a plurality of corresponding first character text features;

[0008] performing fusion recognition on the plurality of first character text features to obtain a character recognition result.

[0009] Further, the to-be-tested billet image further contains a single second character text sprayed on the surface of the billet, the second character text representing second production information of the billet; the locating and cutting of the plurality of first character texts contained in the to-be-tested billet image to obtain a plurality of corresponding first character text region images comprises:

[0010] performing coarse-grained detection on the to-be-tested billet image to locate a region of interest on the to-be-tested billet image, the region of interest containing the plurality of first character texts and the second character text;

[0011] performing fine-grained detection on the region of interest contained in the to-be-tested billet image to obtain a character text positioning result, the character text positioning result including boundary box coordinate information of M first character texts and boundary box coordinate information of the second character text detected;

[0012] When M < N, determining boundary box coordinate information of N-M first character texts that are not detected according to the boundary box coordinate information of the M first character texts, the boundary box coordinate information of the second character text, and a preset character text distribution constraint, wherein N is the number of the plurality of first character texts;

[0013] According to the boundary box coordinate information of the plurality of first character texts, a plurality of first character text region images corresponding to the plurality of first character texts are intercepted from the to-be-tested billet image.

[0014] Further, the character text distribution constraint is used to limit the geometric offset information of each first character text relative to the second character text; and the determination of the boundary box coordinate information of the N-M first character texts that are not detected according to the boundary box coordinate information of the M first character texts, the boundary box coordinate information of the second character text, and the preset character text distribution constraint includes:

[0015] According to the boundary box coordinate information of the M first character texts, the boundary box coordinate information of the second character text, and the character text distribution constraint, the boundary box size of the N-M first character texts and the geometric offset information of the N-M first character texts relative to the second character text are determined.

[0016] According to the boundary box coordinate information of the second character text, the boundary box size of the N-M first character texts, and the geometric offset information of the N-M first character texts relative to the second character text, the boundary box coordinate information of the N-M first character texts is determined.

[0017] Further, the feature extraction processing on the plurality of first character text region images to obtain a plurality of first character text features includes:

[0018] performing preliminary feature extraction processing on the plurality of first character text region images to obtain a plurality of initial first character text features;

[0019] performing multi-layer graph convolution processing on the plurality of initial first character text features to obtain the plurality of first character text features that are optimized.

[0020] Further, the fusing and recognizing the plurality of first character text features to obtain a character recognition result comprises:

[0021] obtaining lengths of the plurality of first character text features;

[0022] when the lengths of the plurality of first character text features are inconsistent, performing feature alignment and completion processing on the plurality of first character text features, and then performing fusing and recognizing the plurality of first character text features after processing to obtain the character recognition result.

[0023] Further, the feature alignment and completion processing on the plurality of first character text features comprises:

[0024] taking the first character text feature with the maximum length as a reference character text feature;

[0025] for each of the first character text features that is inconsistent with the length of the reference character text feature, taking the first character text feature as a missing character text feature, determining a character feature missing position according to a character matching condition between the missing character text feature and the reference character text feature, and then performing feature alignment and completion processing on the missing character text feature and the reference character text feature according to the character feature missing position.

[0026] Further, the determining the character feature missing position according to the character matching condition between the missing character text feature and the reference character text feature comprises:

[0027] calculating a similarity matrix between the missing character text feature and the reference character text feature, and then analyzing the similarity matrix by using a Hungarian matching algorithm to obtain the character feature missing position; wherein the similarity matrix records similarity values between character features at each position in the missing character text feature and character features at each position in the reference character text feature.

[0028] To achieve the above object, another aspect of the present application provides a billet character recognition system based on multi-region cooperation, which comprises:

[0029] a first module configured to obtain a to-be-tested billet image, wherein the to-be-tested billet image contains a plurality of redundant first character texts sprayed on different positions on a billet surface, and each of the first character texts represents first production information of the billet;

[0030] a second module configured to perform positioning and intercepting processing on the plurality of first character texts contained in the to-be-tested billet image to obtain a plurality of corresponding first character text region images;

[0031] a third module configured to perform feature extraction processing on the plurality of first character text region images to obtain a plurality of first character text features corresponding to the plurality of first character text region images;

[0032] a fourth module configured to perform fusion recognition on the plurality of first character text features to obtain a character recognition result.

[0033] To achieve the above object, another aspect of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the above steel billet character recognition method based on multi-region cooperation when executing the computer program.

[0034] To achieve the above object, another aspect of the present application provides a computer readable storage medium, which stores a computer program, and the computer program implements the above steel billet character recognition method based on multi-region cooperation when executed by a processor.

[0035] The present application has at least the following beneficial effects: for the multiple redundant first character texts printed on the steel billet surface at different positions of the steel billet image to be tested, by positioning and cutting out a plurality of first character text region images corresponding to the steel billet image to be tested and then performing feature extraction and fusion recognition, the information of different character text regions can be fully utilized for mutual complementation, and the precision and robustness of steel billet character recognition are improved. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 is a steel billet image schematic diagram collected under different working conditions provided by an embodiment of the present application;

[0037] Figure 2 is a flowchart of a steel billet character recognition method based on multi-region cooperation provided by an embodiment of the present application;

[0038] Figure 3 is a character distribution schematic diagram of a steel billet surface provided by an embodiment of the present application;

[0039] Figure 4 is a structural composition schematic diagram of a steel billet character recognition system based on multi-region cooperation provided by an embodiment of the present application;

[0040] Figure 5 is a hardware structure schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0041] For the purpose of making the purpose, technical scheme and advantages of the present application more clear, the present application is described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and are not intended to limit the present application. When the following description relates to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following example embodiments do not represent all implementations consistent with embodiments of the present application. They are only examples of systems and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0042] It can be understood that the terms "first", "second", and the like used in the present application can be used herein to describe various concepts, but unless specifically stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "when" or "in response to determining".

[0043] The terms "at least one", "multiple", "each", "any" and the like used in the present application include one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0044] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as understood by those skilled in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0045] In the field of industrial automation, existing billet character recognition technologies are mainly divided into two categories, as follows:

[0046] The first category is a method based on traditional image processing and artificial features. This method generally includes: first, image preprocessing is performed through grayscale, local adaptive binarization (such as Sauvola algorithm) and other means to enhance the readability of the characters; then, character regions are located using Canny edge detection, morphological operation and connected component analysis; finally, artificial designed features (such as SIFT features) are extracted, and a support vector machine (SVM) or other classifier is used for recognition. However, this kind of method has poor robustness, limited adaptability to character deformation, defects, blurring, and font and angle diversity, and the implementation effect is excessively dependent on artificial experience and environmental stability, which is difficult to meet the actual needs of complex and variable industrial sites for high-precision recognition;

[0047] The second category is end-to-end recognition methods based on deep learning. These methods, primarily driven by data, demonstrate enhanced recognition capabilities in complex scenarios. In character localization, models such as Fast R-CNN utilize a region proposal network (RPN) to generate candidate regions for effective character localization. In character recognition, the CRNN architecture fuses a convolutional neural network (CNN) with a recurrent neural network (RNN) to achieve end-to-end sequence recognition. Transformer-based models further enhance robustness against blurred and distorted characters through a global self-attention mechanism. While deep learning methods have achieved significant progress in character recognition performance, several key challenges remain. Research has shown that characters are still prone to missed detection in environments with uneven lighting. Furthermore, localized information loss due to occlusions such as rust can significantly interfere with recognition. More importantly, most current methods focus on recognizing characters in a single region, ignoring the common redundant annotation of characters in industrial printed steel billets. This inability to effectively exploit information across multiple regions limits further improvements in character recognition robustness.

[0048] To elaborate, methods based on traditional image processing and artificial features have the following defects: in the preprocessing stage, traditional threshold segmentation and morphological operations lack an adaptive mechanism, resulting in incomplete separation of characters and background in low-quality images; in the segmentation stage, a single strategy (such as edge detection or projection analysis) is difficult to take into account both the segmentation of adhered characters and the correction of tilted text; in the classification stage, a single feature descriptor and classifier cannot balance recognition accuracy and computational efficiency.

[0049] To elaborate, the end-to-end recognition method based on deep learning has made significant progress in the generalization ability and anti-interference ability of character recognition. However, it still has the following inherent disadvantages when dealing with the extreme steel billet industrial environment: First, the lighting conditions at the steel billet production site are often extremely uneven, with strong reflections or dim light. In this case, the steel billet images collected can be seen in the following examples: Figure 1 As shown in the figure, the feature extraction capability of existing deep learning frameworks for such images will be seriously affected, making it difficult to stably locate all character areas, prone to missed detection, and thus causing the loss of key information, which directly affects the integrity of subsequent character recognition. Secondly, when steel billets are stored or transported in high temperature and high humidity environments, their surfaces are prone to rust, corrosion or stains. These factors often block some characters, resulting in the loss of local information within the character area. In this case, the steel billet images collected can be seen in the figure. Figure 1As shown, the existing deep learning framework will interfere with the feature recognition and judgment ability of such images, leading to character misrecognition or inability to recognize; third, in order to ensure the readability of the information, redundant characters with the same content are usually sprayed on different positions on the surface of the billet as backup, but the existing deep learning framework often ignores this redundant labeling feature of the billet characters in industrial printing. When there is uneven illumination, some characters are corroded and blocked, etc., the existing deep learning framework lacks the ability to obtain information from other areas for supplementation, thereby limiting the improvement of the robustness of billet character recognition.

[0050] Therefore, the embodiments of the present application provide a billet character recognition method and system based on multi-region cooperation, device and medium. The scheme is aimed at the multiple redundant first character texts sprayed on different positions on the surface of the billet contained in the to-be-tested billet image. By positioning and intercepting the corresponding multiple first character text region images from the to-be-tested billet image and then performing feature extraction and fusion recognition, the information of different character text regions can be fully utilized for mutual supplementation, thereby improving the accuracy and robustness of billet character recognition.

[0051] The billet character recognition method based on multi-region cooperation provided by the embodiments of the present application is related to the technical field of computer vision, and can be applied to a terminal, a server, or software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, etc., but is not limited thereto. The server can be configured as a standalone physical server, a server cluster composed of multiple physical servers, or a distributed system. It can also be configured as a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and basic cloud computing services such as big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application that implements the above method, but is not limited to the above forms.

[0052] Please refer to Figure 2 , Figure 2 is an optional flowchart of a billet character recognition method based on multi-region cooperation provided by the embodiments of the present application. The method can include, but is not limited to, the following steps S101 to S104:

[0053] Step S101, obtaining a to-be-tested billet image containing multiple redundant first character texts sprayed on different positions on the surface of the billet, each first character text representing first production information of the billet;

[0054] Step S102, a plurality of first character texts contained in the image of the steel billet to be tested are positioned and intercepted, and a plurality of first character text region images corresponding thereto are obtained;

[0055] Step S103, a plurality of first character text region images are subjected to feature extraction processing, and a plurality of first character text features corresponding thereto are obtained;

[0056] Step S104, a plurality of first character text features are fused and recognized, and a character recognition result is obtained.

[0057] The steps S101 to S104 shown in the embodiments of the present application can fully utilize the information of different character text regions to complement each other, thereby improving the accuracy and robustness of steel billet character recognition.

[0058] In step S101 of some embodiments, the image of the steel billet to be tested can be obtained by an image acquisition device such as a camera, etc. pre-installed to image the character text region on the surface of the steel billet. The image of the steel billet to be tested also contains a single second character text sprayed on the surface of the steel billet, which represents the second production information of the steel billet. The second production information of the steel billet refers to the specific model of the steel billet, and the first production information of the steel billet refers to the production batch number of the steel billet. In general, the first production information and the second production information of the steel billet are related to each other, that is, the first production information of the steel billet can be obtained by looking up the table, etc. to obtain the second production information of the steel billet. In typical industrial spraying practice, a single second character text and four first character texts with the same content are usually sprayed on the surface of the steel billet, and the character distribution on the surface of the steel billet has a high degree of regularity, that is, the second character text is centered, and the four first character texts are distributed in different four directions, that is, the upper left, upper right, lower left and lower right of the second character text, as shown in Figure 3 The relative position relationship between the center point of the bounding box of each first character text and the center point of the bounding box of the second character text can be determined.

[0059] In some embodiments, the above step S102 can include but is not limited to the following steps S201 to S205:

[0060] Step S201, coarse-grained detection is performed on the image of the steel billet to be tested to locate a region of interest on the image of the steel billet to be tested, and the region of interest contains a plurality of first character texts and a single second character text;

[0061] In step S202, fine-grained detection is performed on the region of interest contained in the image of the steel billet to be tested to obtain a character text positioning result, which includes boundary box coordinate information of M first character texts and boundary box coordinate information of a single second character text detected.

[0062] In step S203, it is determined whether M < N is true, where N is the number of the first character texts, and N and M are integers; if true, step S204 is performed; if not true, i.e., M = N, step S205 is performed.

[0063] In step S204, boundary box coordinate information of N-M first character texts that are not detected is determined according to the boundary box coordinate information of the M first character texts, the boundary box coordinate information of the single second character text, and a preset character text distribution constraint.

[0064] In step S205, a plurality of first character text region images corresponding to the plurality of first character texts are cut out from the image of the steel billet to be tested according to the boundary box coordinate information of the plurality of first character texts.

[0065] In step S201, a YOLOv7 (You Only Look Once-version 7) model can be used to perform coarse-grained detection on the image of the steel billet to be tested to quickly locate a rough region containing all character texts, thereby narrowing the range of subsequent detection and excluding background interference.

[0066] In step S202, the YOLOv7 model can be reused to further perform fine-grained detection on the rough region containing all character texts, and the preliminary detection result obtained thereby includes boundary box coordinate information of each character text, a corresponding character category and a confidence thereof. The boundary box coordinate information of each character text includes a boundary box center point coordinate, a boundary box width and a boundary box height of the character text. The character category of each character text can be understood as the character text belonging to a first character text or a second character text. In general cases, after being sorted, the preliminary detection result actually includes boundary box coordinate information of at most N first character texts, a corresponding confidence thereof and boundary box coordinate information of a single second character text. Here, M first character text boundary box coordinate information with a suitable confidence is selected according to a preset confidence threshold, and the boundary box coordinate information of the single second character text is integrated to form a final character text positioning result output.

[0067] In the step S204, the preset character text distribution constraint is mainly used to limit the geometric offset information of each first character text relative to a single second character text, which includes an offset direction and a bound two-dimensional offset distance. The offset direction refers to which direction the center point of the bounding box of the first character text falls in the center point of the bounding box of the second character text. The two-dimensional offset distance refers to the X-axis offset distance and Y-axis offset distance between the center points of the bounding boxes of the first character text and the second character text. The two-dimensional offset distance bound by different offset directions is generally different.

[0068] On this basis, the determination manner of the bounding box coordinate information of the N-M first character texts which are not detected can include, but is not limited to, the following steps S301-S302:

[0069] In the step S301, the bounding box size of the N-M first character texts and the geometric offset information of the N-M first character texts relative to the single second character text are determined according to the bounding box coordinate information of the M first character texts, the bounding box coordinate information of the single second character text and the character text distribution constraint.

[0070] In the step S302, the bounding box coordinate information of the N-M first character texts is determined according to the bounding box coordinate information of the single second character text, the bounding box size of the N-M first character texts and the geometric offset information of the N-M first character texts relative to the single second character text.

[0071] In the step S301, the offset direction of the M first character texts relative to the single second character text is obtained by analyzing the center point coordinates of the bounding boxes of the M first character texts and the center point coordinates of the bounding box of the single second character text. Then, the offset direction of the N-M first character texts relative to the single second character text and the bound two-dimensional offset distance are obtained by deduplication analysis according to the offset direction of the M first character texts relative to the single second character text and the character text distribution constraint.

[0072] In the step S301, the bounding box width and the bounding box height of any one of the M first character texts can be directly used as the bounding box size of the N-M first character texts. Alternatively, the bounding box areas of the M first character texts are calculated and compared according to the bounding box width and the bounding box height of the M first character texts, and then the bounding box width and the bounding box height of the first character text with the largest bounding box area among the M first character texts are used as the bounding box size of the N-M first character texts. The present application does not make any limitation in this regard.

[0073] In step S302, the center point coordinates of the bounding box of the single second character text and the two-dimensional offset distance of the N-M first character texts relative to the single second character text are calculated to obtain the center point coordinates of the bounding box of the N-M first character texts, which are integrated with the bounding box size of the N-M first character texts to form the bounding box coordinate information of the N-M first character texts.

[0074] In the embodiments of the present application, the coarse and fine granularity cascade detection is performed on the image of the steel billet to be tested, and the spatial modeling strategy of geometric prior enhancement is introduced to cooperate with the positioning and interception of all first character text region images, which can effectively solve the problem of missed detection of character text regions caused by uneven illumination and other factors. When implementing the spatial modeling strategy of geometric prior enhancement, the other first character text regions that are not detected can be quickly inferred and completed according to the positioned first character text regions and the pre-defined character arrangement spatial topological relationship, thereby improving the integrity and stability of the character text region positioning in complex scenes.

[0075] In some embodiments, the above step S103 can include but is not limited to the following steps S401 to S402:

[0076] Step S401, performing preliminary feature extraction processing on the plurality of first character text region images to obtain a plurality of initial first character text features corresponding thereto;

[0077] Step S402, performing multi-layer graph convolution processing on the plurality of initial first character text features to obtain a plurality of first character text features after optimization corresponding thereto.

[0078] In the above step S401, the SVTR (Scene Text Recognition with a Single Visual Model) model can be used as a feature extraction backbone network to realize preliminary feature extraction of each first character text region image.

[0079] In the above step S402, the graph convolutional neural network is mainly used to realize information interaction and complementation of a plurality of different first character text regions at the feature level to repair the feature defects of part of the first character text regions, such as character feature blur caused by rust, shielding and other factors. The corresponding implementation process can include but is not limited to the following:

[0080] A graph structure is constructed and defined as G={V,E}, wherein the node set V includes a plurality of initial first character text features, and the edge set E includes edges formed by each two initial first character text features;

[0081] The weight between each two initial first character text features is calculated based on the cosine similarity principle, and the following expression can be used for calculation:

[0082]

[0083] In the formula, F i is the i-th initial first character text feature, F j is the j-th initial first character text feature, e ij is the weight between the i-th initial first character text feature and the j-th initial first character text feature, and the greater the weight, the more similar the content between the two initial first character text features, and the stronger the connectivity in the graph structure G.

[0084] An adjacency matrix is constructed according to the weight between each two initial first character text features, and the following expression can be used to represent the symmetric normalization processing of the adjacency matrix:

[0085]

[0086] In the formula, A is the adjacency matrix, is the symmetric normalized adjacency matrix, I is the unit matrix, and D is the diagonal matrix, also known as the degree matrix, which is determined according to the adjacency matrix A. It can be understood that the elements on the diagonal of the diagonal matrix D are the sum of the degrees of the corresponding rows (or columns) in the adjacency matrix A. The i-th diagonal element of the diagonal matrix D represents the degree of node i, i.e. the number of edges connected to node i.

[0087] The plurality of initial first character text features are input into the fully connected graph convolutional neural network. Each node in the graph convolutional neural network aggregates the information of its neighbor nodes. The graph convolutional neural network can include a first graph convolutional layer, a second graph convolutional layer, a third graph convolutional layer, and a random dropout layer connected in sequence. The first graph convolutional layer performs graph convolution and nonlinear activation processing on the plurality of initial first character text features to obtain a corresponding plurality of first correction features. The second graph convolutional layer continues to perform graph convolution and nonlinear activation processing on the plurality of first correction features to obtain a corresponding plurality of second correction features. The third graph convolutional layer again performs graph convolution and nonlinear activation processing on the plurality of second correction features to obtain a corresponding plurality of third correction features. The random dropout layer performs regularization processing on the plurality of third correction features to obtain a corresponding plurality of first character text features after optimization. Each graph convolutional layer introduces the graph structure G and the symmetric normalized adjacency matrix to assist in implementing graph convolution operations. The activation function used by each graph convolutional layer when performing nonlinear activation processing can be an exponential linear unit function.

[0088] In the embodiments of the present application, by introducing a self-loop strategy, that is, adding an identity matrix in the symmetric normalization process of the adjacency matrix, the node can also retain its own information when aggregating the information of its neighbor nodes, thereby ensuring the stability of the feature scale in subsequent graph convolution calculation; by continuously performing graph convolution and nonlinear activation processing on the input features, the input features can propagate and interact between the nodes of the graph structure, thereby enabling the effective strong features from the clear character text region to enhance and repair the weak features from the fuzzy character text region, realizing collaborative reasoning at the feature level, and solving the problem of serious interference with the recognition effect caused by the loss of local character information due to rust, shielding and other factors, and improving the robustness of the entire steel billet character recognition process.

[0089] In some embodiments, the above step S104 can include, but is not limited to, the following steps S501 to S502:

[0090] Step S501, obtaining the lengths of the plurality of first character text features;

[0091] Step S502, determining whether the lengths of the plurality of first character text features are consistent; if yes, that is, the lengths of at least two first character text features are inconsistent, it is determined that there is a character feature missing situation between the plurality of first character text features, at this time, the plurality of first character text features are processed for feature alignment and completion, and then the processed plurality of first character text features are recognized by alignment and fusion to obtain a character recognition result; if not, that is, the lengths of the plurality of first character text features are consistent, it is determined that there is no character feature missing situation between the plurality of first character text features, at this time, the plurality of first character text features are directly recognized by alignment and fusion to obtain a character recognition result.

[0092] In the above step S502, regarding the step of processing the plurality of first character text features for feature alignment and completion, the corresponding implementation process can include, but is not limited to, the following:

[0093] The first character text feature with the maximum length in the plurality of first character text features is taken as a reference character text feature, at this time, it is defaulted that there is no character feature missing situation in the reference character text feature; for each first character text feature with inconsistent length with the reference character text feature, the first character text feature is taken as a missing character text feature, and the character feature missing position is determined according to the character matching situation between the missing character text feature and the reference character text feature; then, the missing character text feature and the reference character text feature are processed for feature alignment and completion according to the character feature missing position.

[0094] In the step of determining the character feature missing position according to the character matching between the missing character text feature and the reference character text feature, the corresponding implementation process can include but is not limited to the following:

[0095] The similarity matrix between the missing character text feature and the reference character text feature is calculated, which mainly records the similarity values between the character features at each position in the missing character text feature and the character features at each position in the reference character text feature, and can be calculated by using the following expression:

[0096] S mn =Softmax(P j,m ·P k,n );

[0097] In the formula, S mn is the similarity value in the mth row and nth column of the similarity matrix, P j,m is the prediction probability distribution of the character feature at the mth position in the reference character text feature, and P k,n is the prediction probability distribution of the character feature at the nth position in the missing character text feature.

[0098] The similarity matrix is analyzed by using the Hungarian matching algorithm to obtain the character feature missing position in the missing character text feature. It can be understood that, according to the similarity matrix, the Hungarian matching algorithm is used to find the optimal matching relationship between the character features at each position in the missing character text feature and the character features at each position in the reference character text feature, so as to screen out the character feature missing position in the missing character text feature.

[0099] In the step of performing feature alignment and completion processing on the missing character text feature and the reference character text feature according to the character feature missing position, the corresponding implementation process can include but is not limited to the following:

[0100] The missing character text feature and the reference character text feature are aligned according to the character feature missing position. For example, it is assumed that the missing character text feature contains 4 character features and the reference character text feature contains 5 character features, and when it is deduced that the first character feature is missing in the missing character text feature, the 4 character features contained in the missing character text feature are aligned with the last 4 character features contained in the reference character text feature.

[0101] The default probability distribution is used or the character feature at the corresponding position and its prediction probability distribution are copied from the reference character text feature and filled into the character feature missing position in the missing character text feature.

[0102] In step S502, regarding the fusion recognition of the processed multiple first character text features, the corresponding embodiments can but not limited to include: averaging the prediction probability distribution of all character features at each same position in the processed multiple first character text features to construct a complete fusion probability matrix, and then analyzing the fusion probability matrix to obtain the character recognition result.

[0103] In the embodiments of the present application, considering the problem that the length of the recognized character sequence may not be consistent due to serious occlusion in some character text regions in actual industrial scenarios, the intelligent alignment and position supplement of each missing character text feature are performed by using the complete reference character text feature, so that the multi-region probability fusion is performed on the basis of complete structure, thereby obtaining a more accurate and reliable character recognition result, and the problem of insufficient utilization of multi-region redundant annotation can be solved.

[0104] Please refer to Figure 4 , Figure 4 is an optional structural composition schematic diagram of a billet character recognition system based on multi-region cooperation provided by the embodiments of the present application, which can implement the above-mentioned billet character recognition method based on multi-region cooperation. The system can but not limited to include the following:

[0105] The first module 601 is configured to obtain a to-be-tested billet image containing multiple redundant first character texts sprayed at different positions on the surface of the billet, and each first character text represents first production information of the billet.

[0106] The second module 602 is configured to perform positioning and cutting processing on the multiple first character texts contained in the to-be-tested billet image to obtain corresponding multiple first character text region images.

[0107] The third module 603 is configured to perform feature extraction processing on the multiple first character text region images to obtain corresponding multiple first character text features.

[0108] The fourth module 604 is configured to perform fusion recognition on the multiple first character text features to obtain a character recognition result.

[0109] It can be understood that the contents in the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those specifically implemented by the above method embodiments, and the beneficial effects achieved by the present system embodiments are also the same as those achieved by the above method embodiments.

[0110] The embodiment of the present application further provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned billet character recognition method based on multi-region cooperation when executing the computer program. The electronic device can include any smart terminal such as a tablet computer.

[0111] It can be understood that the contents in the above-mentioned method embodiments are applicable to the present device embodiment, the present device embodiment specifically implements the same functions as the above-mentioned method embodiments, and the present device embodiment achieves the same beneficial effects as the above-mentioned method embodiments.

[0112] Please refer to Figure 5 , Figure 5 The hardware structure schematic diagram of the electronic device of another embodiment is shown in FIG. 7. The electronic device comprises the following:

[0113] The processor 701 can be implemented in the form of a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is used to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0114] The memory 702 can be implemented in the form of a ROM (Read-Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory). The memory 702 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 702 and are called and executed by the processor 701 to implement the technical solutions provided by the embodiments of the present application.

[0115] The input / output interface 703 is used to realize information input and output.

[0116] The communication interface 704 is used to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0117] The bus 705 transmits information between various components (for example, the processor 701, the memory 702, the input / output interface 703, and the communication interface 704) of the device.

[0118] The processor 701, the memory 702, the input / output interface 703, and the communication interface 704 are communicatively connected with each other through a bus 705.

[0119] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the slab character recognition method based on multi-region cooperation.

[0120] It can be understood that the contents in the above method embodiments are all applicable to the present storage medium embodiment, the functions specifically implemented by the present storage medium embodiment are the same as the functions specifically implemented by the above method embodiments, and the beneficial effects achieved by the present storage medium embodiment are also the same as the beneficial effects achieved by the above method embodiments.

[0121] The memory, as a non-transitory computer readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0122] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0123] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than the figures shown, or combine certain steps, or different steps.

[0124] The system embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, can be located in one place, or can be distributed on multiple network units. According to actual needs, part or all of the modules can be selected to achieve the purpose of the present embodiment.

[0125] Those skilled in the art can understand that all or some steps in the above disclosed method, the functions of the modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0126] The terms "first", "second", "third", "fourth", and the like in the description of this application and in the claims hereof, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so termed herein is solely for descriptive purposes and not for pronouncing the limitations of the application described except as described in the claims. Moreover, the terms "comprise", "have", "contain" and "include" and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, product or apparatus that comprises, has, contains or includes a list of steps or elements, but not those not expressly listed or inherent to such process, method, system, product or apparatus, is not excluded from the scope of this application.

[0127] It should be understood that, in this application, "at least one" means one or more, "multiple" means two or more. "And / or", used to describe the relationship between associated objects, means that there can be three relationships, for example, "A and / or B" can mean: only A, only B, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including single or multiple combinations. For example, at least one of a, b or c, can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0128] In several embodiments provided in this application, it should be understood that the disclosed system and method can be implemented in other ways. For example, the above-described system embodiments are only illustrative, for example, the division of the above-mentioned units is only a logical function division, and actual implementation can have another division mode, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be through some interface, indirect coupling or communication connection between systems or units, which can be electrical, mechanical or other forms.

[0129] The units described above as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e. they can be located in one place or distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0130] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0131] When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in part, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions used to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various other media that can store programs.

[0132] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A steel billet character recognition method based on multi-region collaboration, characterized in that: The method comprises: Acquire an image of a steel billet to be tested, wherein the image of the steel billet to be tested includes a plurality of redundant first character texts printed at different positions on the surface of the steel billet, each of the first character texts representing first production information of the steel billet; Performing positioning and interception processing on the plurality of first character texts contained in the steel billet image to be tested to obtain corresponding plurality of first character text area images; Performing feature extraction processing on the plurality of first character text area images to obtain corresponding plurality of first character text features; The multiple first character text features are fused and recognized to obtain a character recognition result.

2. The steel billet character recognition method based on multi-region collaboration according to claim 1 is characterized in that: The steel billet image to be tested further includes a single second character text printed on the surface of the steel billet, wherein the second character text represents second production information of the steel billet; and the positioning and intercepting processing of the multiple first character texts included in the steel billet image to obtain the corresponding multiple first character text area images includes: Performing coarse-grained detection on the steel billet image to be tested to locate a region of interest on the steel billet image to be tested, where the region of interest includes the plurality of first character texts and the second character text; Performing fine-grained detection on the region of interest contained in the steel billet image to be tested to obtain a character text positioning result, wherein the character text positioning result includes the detected bounding box coordinate information of the M first character texts and the bounding box coordinate information of the second character text; When M<N, determining the bounding box coordinate information of the NM undetected first character texts based on the bounding box coordinate information of the M first character texts, the bounding box coordinate information of the second character texts, and a preset character text distribution constraint; wherein N is the number of the plurality of first character texts; According to the coordinate information of the bounding boxes of the plurality of first character texts, the corresponding plurality of first character text area images are cut out from the steel billet image to be measured.

3. The steel billet character recognition method based on multi-region collaboration according to claim 2 is characterized in that: The character text distribution constraint is used to limit the geometric offset information of each first character text relative to the second character text; determining the bounding box coordinate information of the NM undetected first character texts based on the bounding box coordinate information of the M first character texts, the bounding box coordinate information of the second character text, and the preset character text distribution constraint includes: Determining, based on the bounding box coordinate information of the M first character texts, the bounding box coordinate information of the second character texts, and the character text distribution constraint, bounding box sizes of the NM first character texts and geometric offset information of the NM first character texts relative to the second character texts; Determine the bounding box coordinate information of NM first character texts based on the bounding box coordinate information of the second character text, the bounding box sizes of the NM first character texts, and the geometric offset information of the NM first character texts relative to the second character text.

4. The steel billet character recognition method based on multi-region collaboration according to claim 1 is characterized in that: The performing feature extraction processing on the plurality of first character text area images to obtain corresponding plurality of first character text features comprises: Performing preliminary feature extraction processing on the plurality of first character text area images to obtain corresponding plurality of initial first character text features; Multi-layer graph convolution processing is performed on the multiple initial first character text features to obtain the corresponding optimized multiple first character text features.

5. The steel billet character recognition method based on multi-region collaboration according to claim 1 is characterized in that: The fusing and recognizing the plurality of first character text features to obtain a character recognition result includes: Obtaining the lengths of the plurality of first character text features; When the lengths of the multiple first character text features are inconsistent, feature alignment and completion processing are performed on the multiple first character text features, and then the processed multiple first character text features are aligned and fused for recognition to obtain the character recognition result.

6. The steel billet character recognition method based on multi-region collaboration according to claim 5 is characterized in that: The feature alignment and completion processing of the plurality of first character text features includes: Taking the first character text feature with the largest length as the reference character text feature; For each first character text feature whose length is inconsistent with that of the benchmark character text feature, the first character text feature is used as a missing character text feature, and the missing position of the character feature is determined based on the character matching between the missing character text feature and the benchmark character text feature; then, based on the missing position of the character feature, the missing character text feature and the benchmark character text feature are subjected to feature alignment and completion processing.

7. The steel billet character recognition method based on multi-region collaboration according to claim 6 is characterized in that: The determining of the missing position of the character feature according to the character matching between the missing character text feature and the reference character text feature includes: A similarity matrix between the missing character text feature and the benchmark character text feature is calculated, and the similarity matrix is ​​then analyzed using a Hungarian matching algorithm to obtain the missing position of the character feature; wherein the similarity matrix records the similarity values ​​between the character features at each position in the missing character text feature and the character features at each position in the benchmark character text feature.

8. A steel billet character recognition system based on multi-region collaboration, characterized in that: The system comprises: A first module is configured to obtain an image of a steel billet to be tested, wherein the image of the steel billet to be tested comprises a plurality of redundant first character texts printed at different positions on the surface of the steel billet, each of the first character texts representing first production information of the steel billet; The second module is used to locate and intercept the multiple first character texts contained in the steel billet image to obtain corresponding multiple first character text area images; A third module is configured to perform feature extraction processing on the plurality of first character text area images to obtain corresponding plurality of first character text features; The fourth module is used to perform fusion recognition on the multiple first character text features to obtain a character recognition result.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the steel billet character recognition method based on multi-region collaboration as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steel billet character recognition method based on multi-region collaboration according to any one of claims 1 to 7 is implemented.

Images