Box-type text recognition method, device and electronic equipment

Abstract

The present application provides a box-type structure text recognition method, device, electronic device and computer-readable storage medium; the method includes: performing feature extraction processing on the image of the box-type structure; performing field-level text region segmentation processing on the image based on the extracted features to obtain field-level segmentation results; performing single-character level detection processing on the image based on the extracted features to obtain character-level detection results; determining the character category corresponding to the character-level detection results based on the character-level detection results; determining the field recognition results of the box-type structure based on the character category and the field-level segmentation results; performing structured information extraction processing on the field recognition results to obtain text information on the box-type structure. Through the present application, the accuracy of text recognition of box-type structures can be improved.

Description

Technical Field

[0001] The present application relates to artificial intelligence technology, and in particular to a box-type text recognition method, device, electronic device and computer-readable storage medium. Background Art

[0002] Artificial Intelligence (AI) is a comprehensive technology in computer science. By studying the design principles and implementation methods of various intelligent machines, machines are given the functions of perception, reasoning and decision-making. Artificial intelligence technology is an interdisciplinary subject that covers a wide range of fields, such as natural language processing technology and machine learning / deep learning. With the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0003] Box-structured text recognition is one of the important applications in the field of artificial intelligence. It can use computers to process, analyze and understand images to identify text information in images.

[0004] Although the box-structured text recognition system in the related art can recognize the box-structured text included in the image, the box-structured text recognition in the related art is not accurate enough. Summary of the invention

[0005] The embodiments of the present application provide a box-type structure text recognition method, device, electronic device and computer-readable storage medium, which can improve the text recognition accuracy of the box-type structure.

[0006] The technical solution of the embodiment of the present application is implemented as follows:

[0007] The present application embodiment provides a box-type structure text recognition method, including:

[0008] Performing feature extraction processing on the image of the box-type structure;

[0009] Performing field-level text region segmentation processing on the image based on the extracted features to obtain a field-level segmentation result;

[0010] Performing single character level detection processing on the image based on the extracted features to obtain a character level detection result;

[0011] Based on the character-level detection result, determining a character category corresponding to the character-level detection result;

[0012] Determining a field recognition result of the box-type structure based on the character category and the field-level segmentation result;

[0013] The field recognition result is subjected to structured information extraction processing to obtain text information on the box structure.

[0014] An embodiment of the present application provides a text recognition device of a box-type structure, comprising: a feature module, used to perform feature extraction processing on an image of the box-type structure; a character detection and field segmentation module, used to perform field-level text area segmentation processing on the image based on the extracted features to obtain a field-level segmentation result; the character detection and field segmentation module is also used to perform single-character level detection processing on the image based on the extracted features to obtain a character-level detection result; a character category determination module, used to determine the character category corresponding to the character-level detection result based on the character-level detection result; a field recognition module, used to determine the field recognition result of the box-type structure based on the character category and the field-level segmentation result; a text extraction module, used to perform structured information extraction processing on the field recognition result to obtain text information on the box-type structure.

[0015] In the above scheme, the feature extraction process is implemented by calling a feature extraction network, and the feature extraction network includes a multi-scale feature extraction network and a multi-scale feature fusion network. The feature module is also used to: perform multi-scale feature extraction processing on the image of the box-type structure through the multi-scale feature extraction network to obtain sub-features of multiple scales of the box-type structure; and perform fusion processing on the sub-features of multiple scales through the multi-scale feature fusion network to obtain the features of the box-type structure.

[0016] In the above scheme, the multi-scale feature extraction network includes a convolution layer, a pooling layer and a residual layer corresponding to multiple scales. The feature module is also used to: perform convolution processing on the image through the convolution layer to obtain a convolution processing result; perform pooling processing on the convolution processing result through the pooling layer to obtain a pooling result; perform residual processing on the pooling result through cascaded residual layers corresponding to multiple scales to obtain sub-features of multiple scales of the box-type structure.

[0017] In the above scheme, the feature module is also used to: perform residual processing on the input of the nth residual layer through the nth residual layer in N cascaded residual layers, and transmit the nth residual processing result output by the nth residual layer to the n+1th residual layer to continue residual processing; wherein n is an integer whose value increases from 1, and the value range of n satisfies 1≤n≤N-1, and N is an integer greater than or equal to 2; when n is 1, the input of the nth residual layer is the pooling result, and when n is 2≤n≤N-1, the input of the nth residual layer is the residual processing result of the n-1th residual layer, and the residual processing result of the nth residual layer is the sub-feature corresponding to the scale.

[0018] In the above scheme, the multi-scale feature fusion network includes at least one fusion layer, and the feature module is further used to: when the number of the fusion layer is 1, the sub-features output by the Nth residual layer and the sub-features output by the N-1th residual layer are fused through the fusion layer to obtain the features of the box structure; when the number of the fusion layers is multiple, the input of the mth fusion layer is fused through the mth fusion layer in M ​​cascaded fusion layers, and the mth fusion processing result output by the mth fusion layer is transmitted to the m+1th fusion layer to continue Fusion processing; wherein, m is an integer whose value increases from 1, and the value range of m satisfies 1≤m≤M-1, and M is N-1; when m is 1, the input of the mth fusion layer is the sub-features output by the Nth residual layer and the sub-features output by the N-1th residual layer; when m is 2≤m≤M-1, the input of the mth fusion layer is the m-1th fusion processing result of the m-1th fusion layer and the sub-features output by the Nmth residual layer; when m is M-1, the fusion processing result of the m+1th fusion layer is the feature of the box structure.

[0019] In the above scheme, the single character detection process is implemented through a pixel category classification network and a pixel position regression network, and the character detection and field segmentation module is also used to: perform pixel detection processing based on the feature on the image through the pixel category classification network to obtain a first feature map; perform pixel position regression processing based on the feature on the image through the pixel position regression network to obtain a second feature map; and combine the first feature map and the second feature map into the character-level detection result.

[0020] In the above scheme, the text area segmentation processing is implemented by calling the text area segmentation network, and the character detection and field segmentation module is also used to: perform text area segmentation processing based on the features on multiple pixels of the image through the text area segmentation network to obtain a third feature map corresponding to the multiple pixels; and use the third feature map as the field-level segmentation result.

[0021] In the above scheme, the character category determination module is also used to: determine the first pixel belonging to a single character and the distance between the first pixel and the border of the character to which it belongs based on the character-level detection result; determine the character category of the single character based on the first pixel and the distance between the first pixel and the border of the character to which it belongs.

[0022] In the above scheme, the character category is determined by calling a single character classification network, and the character category determination module is also used to: determine the position of the character border based on the first pixel and the distance between the first pixel and the character border to which it belongs; crop the image according to the position of the character border to obtain the character area; perform character classification processing on the character area through the single character classification network to obtain the character category of the single character in the character area.

[0023] In the above scheme, the character-level detection result includes a first feature map of multiple pixels and a second feature map of the multiple pixels, and the character category determination module is also used to: perform maximum likelihood processing on the first feature map of the multiple pixels to obtain a first probability that each of the pixels belongs to the single character; determine the pixels whose first probability is greater than a first probability threshold as the first pixels belonging to the single character; obtain the value of each of the first pixels in the second feature map, wherein the value represents the distance between the first pixel and the character border to which it belongs.

[0024] In the above scheme, the field-level segmentation result includes a third feature map of multiple pixels, and the field recognition module is also used to: perform maximum likelihood processing on the third feature map of the multiple pixels to obtain a second probability that each of the pixels is in the field area; determine the pixel whose second probability is greater than a second probability threshold as a second pixel in the field area; perform connected domain analysis based on the position of the second pixel to obtain multiple connected domains; obtain edge information of each of the connected domains, and perform outward expansion processing based on the edge information to obtain the field area; determine the overlap between the pixels belonging to the character category and the field area; when the overlap is greater than the overlap threshold, determine the character category as the field recognition result.

[0025] In the above scheme, the text extraction module is also used for: when there is a field recognition result in the image that meets the numbering rule of the box structure, determining the field recognition result as the candidate number of the box structure; when there is no field recognition result that meets the numbering rule of the box structure in the image, combining multiple field recognition results in the image to obtain a combined recognition result, and determining the combined recognition result that meets the numbering rule of the box structure as the candidate number of the box structure; when the field recognition result is in the box type list of the box structure, determining the field recognition result as the candidate box type of the box structure; when the field recognition result meets the weight identification rule, determining the field recognition result as the candidate weight of the box structure; based on the candidate number, the candidate box type and the candidate weight, determining the text information on the box structure.

[0026] In the above scheme, when the image is a plurality of images collected from different surfaces of the box-type structure, the text extraction module is further used to: perform filling and combining processing on the candidate numbers obtained based on different images, and use the obtained filling and combining processing results as the number of the box-type structure; perform filling and combining processing on the candidate box types obtained based on different images, and use the obtained filling and combining processing results as the box type of the box-type structure; perform filling and combining processing on the candidate weights obtained based on different images, and use the obtained filling and combining processing results as the weight of the box-type structure; combine the number of the box structure, the box type of the box structure and the weight of the box structure into text information on the box structure.

[0027] An embodiment of the present application provides an electronic device, including:

[0028] A memory for storing executable instructions;

[0029] The processor is used to implement the box-type structure text recognition method provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0030] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for implementing the box-type structure text recognition method provided in the embodiment of the present application when executed by a processor.

[0031] The embodiments of the present application have the following beneficial effects:

[0032] Based on the extracted features, the image is simultaneously processed at the single character level detection and the field level text area segmentation to obtain the character level detection results and the field level segmentation results. Then, based on the character level detection results and the field level segmentation results, the box-structured field recognition results are determined. This not only makes the effects of the dual-branch processing promote each other, but also can adapt to the complex and diverse character arrangement situations, effectively improving the accuracy of box-structured text recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a structural schematic diagram of a box-type text recognition system provided in an embodiment of the present application;

[0034] Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0035] Figures 3A-3D It is a flowchart of a box-type structure text recognition method provided in an embodiment of the present application;

[0036] Figure 4 is a structural diagram of a text recognition model provided in an embodiment of the present application;

[0037] Figures 5A-5C It is an image of a box-type structure provided in an embodiment of the present application;

[0038] Figure 6 It is a logical schematic diagram of the box-type structure text recognition method provided in the embodiment of the present application;

[0039] Figures 7A-7C It is a partial image schematic diagram of the box-type structure text recognition method provided in the embodiment of the present application;

[0040] Figure 8 is a structural diagram of a text recognition model provided in an embodiment of the present application;

[0041] Figures 9A-9B It is a partial image schematic diagram of the box-type structure text recognition method provided in the embodiment of the present application;

[0042] Fig.10 is a schematic diagram of the result of the box-type structure text recognition method provided in the embodiment of the present application;

[0043] Fig.11 It is a schematic diagram of the output information of the box-type structure text recognition method provided in the embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0045] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0046] In the following description, the terms "first\second" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0048] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0049] 1) Deep learning technology: It is a branch of machine learning. It is an algorithm that uses artificial neural networks as its architecture to represent and learn data. This technology can simulate the human brain and perform different degrees of perception and analysis of the real world.

[0050] 2) Multi-scale: It is actually the sampling of signals at different scales. Usually, we can observe different features at different scales to complete different tasks. For example, when sampling images at different resolutions, the receptive field of large-scale features is sufficient for small-resolution information, so it is suitable for obtaining detailed image information in a small range. The resolution information of small-scale features is insufficient, so it is suitable for processing large-scale overall image information.

[0051] 3) Softmax function: also known as normalized exponential function, is a generalization of the logical function. It can compress a K-dimensional vector z containing any real number into another K-dimensional real vector σ(z), so that the range of each element is between (0, 1) and the sum of all elements is 1. This function is more than in multi-classification problems.

[0052] 4) Non-maximum suppression technology: All local maxima are selected by defining local methods. In the target detection task, it first generates a detection frame based on the object detection score. The detection frame M with the highest score is selected, and other detection frames that have obvious overlap with the selected detection frame are suppressed.

[0053] 5) Connected domain: There is a region G on the complex plane. If we make a simple closed curve in it and the interior of the closed curve always belongs to G, then G is called a connected region.

[0054] 6) Box-type structure, a three-dimensional structure with top, bottom and side surrounds, with a containment function. For example, containers, packaging boxes, modular houses, etc.

[0055] The related technology discloses a container number recognition method based on hierarchical clustering and deep learning. The system first obtains the candidate character area through a single character detection algorithm, and then obtains the real container number characters through character filtering and character merging. Then, the container number characters are obtained through a hierarchical clustering algorithm to obtain candidate text lines and filter the text lines. Finally, deep learning technology is used to recognize the single characters in the text lines to finally obtain the container number result.

[0056] The applicant found that the related technology adopts the maximum stable extreme area algorithm to obtain the detection results of a single character, which is greatly affected by changes in image brightness, scale, etc., and the effect is not robust. Subsequent character filtering and character merging operations are required, and a hierarchical clustering method is used to cluster adjacent characters into text lines. It is assumed that the letters and numbers in the box number are in a straight line, which is not in line with the actual situation. In actual scenarios, the box number text may be composed of one line, one column, two lines, three lines, or even four lines of text. In addition, it does not support the output of text information on the box structure such as box type, total weight, load, dead weight, and volume.

[0057] The related technology also discloses a container number positioning method based on edge detection technology. The method first converts the original container number image into a grayscale format image, performs median filtering on the obtained grayscale image to remove noise interference, and then selects the Sobel operator to perform vertical edge detection and horizontal edge detection respectively. Then, based on the edge detection results, column positioning attempts and row positioning attempts are performed respectively to obtain the vertical candidate container number character connected area and the horizontal candidate container number character connected area. Finally, the connected area is morphologically processed and analyzed to obtain the location information of the container number.

[0058] The applicant found that the related art uses edge detection technology to detect the area where the characters are located, which requires that the edge information of the detected container number area is complete. In fact, due to the interference of weather and lighting, the detected edge information is not ideal. Moreover, this method only locates the area where the container number is located, does not output the container number text, and does not support the output of text information on the container structure such as box type, gross weight, load, dead weight, volume, etc.

[0059] In response to the above-mentioned problems in the related art, the embodiments of the present application provide a text recognition method, device, electronic device and computer-readable storage medium of a box-type structure, which can improve the text recognition accuracy of the box-type structure.

[0060] The box-type text recognition provided in the embodiment of the present application can be implemented by various electronic devices, for example, it can be implemented by a terminal or a server alone, or it can be implemented by a terminal and a server in collaboration.

[0061] For example, the terminal independently undertakes the text recognition of the box-type structure described below, or the terminal sends a recognition request for the image to be recognized (including the image to be recognized) to the server, and the server performs text recognition of the box-type structure according to the received recognition request for the image to be recognized, and sends the recognition result of the box-type structure to the terminal, that is, the text information on the box-type structure such as box type, total weight, load, dead weight, and volume.

[0062] The server can be a server cluster deployed in the cloud, opening artificial intelligence cloud services (AI as a Service, AIaaS) to users. The AIaaS platform will split several common AI services and provide independent or packaged services in the cloud. This service model is similar to an AI theme mall. All users can access one or more artificial intelligence services provided by the AIaaS platform through an application programming interface.

[0063] For example, one of the artificial intelligence cloud services may be a box-type structure text recognition service, that is, the server in the cloud is encapsulated with the box-type structure text recognition program provided by the embodiment of the present application. The user calls the box-type structure text recognition service in the cloud service through a terminal (running a client, such as a box-type structure text recognition client), so that the server deployed in the cloud calls the encapsulated box-type structure text recognition program, performs feature extraction processing on the box-type structure image, performs single-character level detection processing on the image based on the extracted features, and obtains character-level detection results; based on the box-type structure features, performs field-level text area segmentation processing on the image to obtain field-level segmentation results; based on the character-level detection results, determines the character category corresponding to the character-level detection results; based on the character category and the field-level segmentation results, determines the field recognition results of the box-type structure; performs structured information extraction processing on the field recognition results to obtain text information on the box-type structure, in response to the text to be recognized. For example, for a text recognition application of a container structure, feature extraction is performed on the image of the container, and single character level detection is performed on the image based on the extracted features to obtain character level detection results; based on the features of the container, field level text region segmentation is performed on the image to obtain field level segmentation results; based on the character level detection results, the character category corresponding to the character level detection results is determined; based on the character category and the field level segmentation results, the field recognition results of the container are determined; structured information extraction is performed on the field recognition results to obtain text information on the container to respond to the text recognition request for the container, so that accurate information of the container can be entered according to the recognition results later, thereby improving the transfer efficiency and use efficiency of the container, see Fig.11 , Fig.11 It is a schematic diagram of the output information of the text recognition method for the box-type structure provided in the embodiment of the present application. When the box-type structure is a container, the output information includes the container type, container number, gross weight, dead weight, load and volume. By extracting the text of these fields, accurate information of the container is entered, thereby improving the transfer efficiency and utilization efficiency of the container.

[0064] See also Figure 1 , Figure 1It is a structural diagram of a box-type text recognition system provided in an embodiment of the present application. The terminal 400 is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0065] Terminal 400 (running a client, such as a box-type text recognition client) can be used to obtain a recognition request for an image to be recognized. For example, after a user inputs an image to be recognized including a box-type structure in the client, the terminal automatically obtains a recognition request for the image to be recognized.

[0066] In some embodiments, a text recognition plug-in of a box-type structure may be implanted in the client running in the terminal to implement text recognition of the box-type structure locally on the client. For example, after the terminal 400 obtains the recognition request for the image to be recognized, it calls the text recognition plug-in of the box-type structure to implement text recognition of the box-type structure, by performing feature extraction processing on the image of the box-type structure, performing single-character level detection processing on the image based on the extracted features, and obtaining character-level detection results; based on the features of the box-type structure, performing field-level text area segmentation processing on the image to obtain field-level segmentation results; based on the character-level detection results, determining the character category corresponding to the character-level detection results; based on the character category and the field-level segmentation results, determining the field recognition results of the box-type structure; performing structured information extraction processing on the field recognition results to obtain text information on the box-type structure in response to the text recognition request for the image to be recognized. For example, for a box-type structure detection application, the user input includes modular After obtaining the image to be recognized of the house, the recognition request for the image to be recognized is automatically obtained, and feature extraction processing is performed on the image of the modular house. Based on the extracted features, single character level detection processing is performed on the image to obtain character level detection results; based on the features of the modular house, field level text area segmentation processing is performed on the image to obtain field level segmentation results; based on the character level detection results, the character category corresponding to the character level detection results is determined; based on the character category and the field level segmentation results, the field recognition result of the modular house is determined; structured information extraction processing is performed on the field recognition result to obtain text information on the modular house, so as to respond to the text recognition request for the modular house, so as to subsequently enter the information of the modular house according to the recognition results, thereby improving the transportation efficiency and use efficiency of the modular house.

[0067] In some embodiments, after the terminal 400 obtains the recognition request for the image to be recognized, it calls the box-type structure text recognition interface of the server 200 (which can be provided in the form of a cloud service, i.e., a box-type structure text recognition service), and the server 200 performs feature extraction processing on the box-type structure image, and performs single-character level detection processing on the image based on the extracted features to obtain character-level detection results; based on the features of the box-type structure, the image is subjected to field-level text area segmentation processing to obtain field-level segmentation results; based on the character-level detection results, the character category corresponding to the character-level detection results is determined; based on the character category and the field-level segmentation results, the field recognition results of the box-type structure are determined; the field recognition results are subjected to structured information extraction processing on the field recognition results to obtain text information on the box-type structure, so as to obtain the text information on the box-type structure. In response to the recognition request for the image to be recognized, for example, for the box-type structure detection application, the image of the packaging box is subjected to feature extraction processing, and the image is subjected to single-character level detection processing based on the extracted features to obtain character-level detection results; based on the features of the packaging box, the image is subjected to field-level text area segmentation processing to obtain field-level segmentation results; based on the character-level detection results, the character category corresponding to the character-level detection results is determined; based on the character category and the field-level segmentation results, the field recognition results of the packaging box are determined; structured information extraction processing is performed on the field recognition results to obtain text information on the packaging box, in response to the recognition request for the image to be recognized, so that the information of the packaging box can be entered according to the recognition results later, thereby improving the transportation efficiency and utilization efficiency of modular houses.

[0068] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal 400 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0069] Next, the structure of the electronic device for implementing the box-type text recognition method provided by the embodiment of the present application is described. As mentioned above, the electronic device provided by the embodiment of the present application can be Figure 1 Server 200 in . Figure 2 , Figure 2 is a schematic diagram of the structure of the server 200 provided in an embodiment of the present application, Figure 2The server 200 shown includes: at least one processor 210, a memory 250, and at least one network interface 220. The various components in the server 200 are coupled together via a bus system 240. It is understood that the bus system 240 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 240 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 240 is not described in detail. Figure 2 Various buses are labeled as bus system 240 .

[0070] The processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0071] The memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 250 may optionally include one or more storage devices that are physically remote from the processor 210.

[0072] The memory 250 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 250 described in the embodiments of the present application is intended to include any suitable type of memory.

[0073] In some embodiments, memory 250 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0074] The operating system 251 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic businesses and processing hardware-based tasks; a network communication module 252 is used to reach other computing devices via one or more (wired or wireless) network interfaces 220. Exemplary network interfaces 220 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.

[0075] In some embodiments, the box-type text recognition device provided in the embodiments of the present application can be implemented in a software manner. Figure 2 A box-type text recognition device 255 stored in a memory 250 is shown, which may be software in the form of a program or plug-in, including the following software modules: a feature module 2551, a character detection and field segmentation module 2552, a character category determination module 2553, a field recognition module 2554, and a text extraction module 2555. These modules are logical, and therefore may be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0076] The box-type structure text recognition method provided in the embodiment of the present application will be described in combination with the exemplary application and implementation of the server 200 provided in the embodiment of the present application.

[0077] See also Figure 4 , Figure 4 It is a structural schematic diagram of a text recognition model provided by an embodiment of the present application, wherein the text recognition model includes a feature extraction network, and the text recognition model also includes a pixel category classification network, a pixel position regression network, a text region segmentation network, and a single character classification network. The feature extraction network performs feature extraction processing on the image of the box-type structure to obtain features, the pixel category classification network performs feature-based pixel detection processing on the image to obtain a first feature map, the pixel position regression network performs feature-based pixel position regression processing on the image to obtain a second feature map, the text region segmentation network performs feature-based text region segmentation processing on multiple pixels of the image to obtain a third feature map, based on the second feature map and the first feature map, the position of the character border is determined, the image is cropped according to the position of the character border to obtain a character region, the character region is subjected to character classification processing by the single character classification network to obtain the character category of the single character in the character region, and the field recognition result of the box-type structure is determined based on the third feature map and the character category of the single character.

[0078] See also Figure 3A , Figure 3A is a flowchart of a box-type text recognition method provided in an embodiment of the present application, which will be combined with Figure 3A Steps 101-105 are shown for illustration.

[0079] In step 101, feature extraction processing is performed on the image of the box-type structure.

[0080] As an example, feature extraction is achieved by calling a feature extraction network. The feature network includes a multi-scale feature extraction network and a multi-scale feature fusion network. The multi-scale feature extraction network serves as a backbone network and continuously fuses multi-scale features upward to output a high-resolution response map. The multi-scale feature extraction network is used to obtain multi-scale information of the input image to be identified. The high-resolution feature map has a stronger representation capability for small objects, and the low-resolution feature map has a stronger representation capability for large objects. Therefore, it is very necessary to use both high-resolution and low-resolution feature maps at the same time.

[0081] In some embodiments, see Figure 3B , Figure 3B It is a flow chart of the box-type structure text recognition method provided in an embodiment of the present application. The feature extraction process is realized by calling a feature extraction network. The feature extraction network includes a multi-scale feature extraction network and a multi-scale feature fusion network. In step 101, feature extraction process is performed on the image of the box-type structure, which can be realized by steps 1011-1012.

[0082] In step 1011, a multi-scale feature extraction network is used to perform multi-scale feature extraction processing on the image of the box-type structure to obtain sub-features of multiple scales of the box-type structure.

[0083] In some embodiments, the multi-scale feature extraction network includes a convolution layer, a pooling layer, and a residual layer corresponding to multiple scales. In step 1011, the multi-scale feature extraction network performs multi-scale feature extraction processing on the box-type structure image to obtain sub-features of multiple scales of the box-type structure. This can be achieved through the following technical solutions: performing convolution processing on the image through a convolution layer to obtain a convolution processing result; performing pooling processing on the convolution processing result through a pooling layer to obtain a pooling result; performing residual processing on the pooling result through cascaded residual layers corresponding to multiple scales one by one to obtain sub-features of multiple scales of the box-type structure.

[0084] In some embodiments, the above-mentioned residual processing of the pooling result by cascading residual layers corresponding to multiple scales one by one to obtain sub-features of multiple scales of the box-type structure can be achieved by the following technical solution: residual processing is performed on the input of the nth residual layer through the nth residual layer in N cascaded residual layers, and the nth residual processing result output by the nth residual layer is transmitted to the n+1th residual layer to continue the residual processing; wherein n is an integer whose value increases from 1, and the value range of n satisfies 1≤n≤N-1, and N is an integer greater than or equal to 2; when n is 1, the input of the nth residual layer is the pooling result, and when n is 2≤n≤N-1, the input of the nth residual layer is the residual processing result of the n-1th residual layer, and the residual processing result of the nth residual layer is the feature of the corresponding scale.

[0085] As an example, see Figure 8 , Figure 8 It is a structural schematic diagram of a text recognition model provided in an embodiment of the present application, wherein a convolution layer (conv) is used to convolve an image to obtain a convolution result, a pooling layer (pooling) is used to pool the convolution result to obtain a pooling result, and a residual layer corresponding to multiple scales is used to perform residual processing on the pooling result to obtain sub-features of multiple scales of a box-type structure. The cascaded residual layers corresponding to multiple scales can be ResBlock1-4, where N is 4. The output of each residual layer is transmitted to the next residual layer and is also output to a multi-scale feature fusion network. For example, ResBlock1 outputs features corresponding to the first scale, ResBlock2 outputs features corresponding to the second scale, ResBlock3 outputs features corresponding to the third scale, and ResBlock4 outputs features corresponding to the fourth scale.

[0086] In step 1012, sub-features of multiple scales are fused through a multi-scale feature fusion network to obtain features of a box structure.

[0087] In some embodiments, the multi-scale feature fusion network includes at least one fusion layer. In step 1012, the sub-features of multiple scales are fused through the multi-scale feature fusion network to obtain the features of the box structure, which can be achieved by the following technical solutions: when the number of fusion layers is 1, the sub-features output by the Nth residual layer and the sub-features output by the N-1th residual layer are fused through the fusion layer to obtain the features of the box structure; when the number of fusion layers is multiple, the input of the mth fusion layer is fused through the mth fusion layer in the M cascaded fusion layers, and the output of the mth fusion layer is The mth fusion processing result is transmitted to the m+1th fusion layer for further fusion processing; wherein, m is an integer increasing from 1, and the value range of m satisfies 1≤m≤M-1, and M is N-1; when m is 1, the input of the mth fusion layer is the sub-features output by the Nth residual layer and the sub-features output by the N-1th residual layer; when m is 2≤m≤M-1, the input of the mth fusion layer is the m-1th fusion processing result of the m-1th fusion layer and the sub-features output by the Nmth residual layer; when m is M-1, the fusion processing result of the m+1th fusion layer is the feature of the box structure.

[0088] As an example, see Figure 8When the number of fusion layers is 1, the number of residual layers is 2. The sub-features output by the second residual layer and the sub-features output by the first residual layer are fused through the fusion layer to obtain the features of the box structure. When the number of fusion layers is multiple, the number of fusion layers is the number of residual layers minus 1. For example, when the number of residual layers is 4, the number of fusion layers is 3, and the fusion layers are Merge1-3. The fusion layer Merge1 fuses the features of the fourth scale output by ResBlock4 and the features of the third scale output by ResBlock3. The first fusion processing result of the fusion layer Merge1 is transmitted to the second fusion layer Merge2. The fusion layer Merge2 fuses the first fusion processing result and the features of the second scale output by ResBlock2. The second fusion processing result of the fusion layer Merge2 is transmitted to the third fusion layer Merge3. The fusion layer Merge3 fuses the second fusion processing result and the features of the first scale output by ResBlock1 to obtain the features of the box structure.

[0089] In step 102, field-level text region segmentation processing is performed on the image based on the extracted features to obtain a field-level segmentation result.

[0090] In some embodiments, see Figure 3D , Figure 3D It is a flow chart of the box-type structure text recognition method provided in an embodiment of the present application. The text region segmentation process is implemented by calling a text region segmentation network. In step 102, field-level text region segmentation is performed on the image based on the extracted features to obtain a field-level segmentation result, which can be implemented through steps 1021-1022.

[0091] In step 1021, a text region segmentation network is used to perform feature-based text region segmentation processing on a plurality of pixels of the image to obtain a third feature map corresponding to the plurality of pixels;

[0092] In step 1022, the third feature map is used as a field-level segmentation result.

[0093] As an example, the output of the text region segmentation network is a third feature map with 2 channels. There is a corresponding value for each pixel on the third feature map. The value of the third feature map after maximum likelihood processing represents the probability that the corresponding pixel is in the field area. The field area is defined as an area surrounded by adjacent characters. It is a field-level concept. A polygonal mask is usually used to represent a field area, which covers all characters in the field. On the other hand, if the distance between two field areas is close, their masks will overlap, which is not conducive to the subsequent segmentation of the mask. See Figures 9A-9B , Figures 9A-9Bis a partial image schematic diagram of a box-type structure text recognition method provided in an embodiment of the present application, Fig. 9A The two field areas in the image are close to each other, and their masks will overlap. The masks can be shrunk in the direction of the shortest side, with each side shrinking by 0.2*r, where r is the length of the shortest side. Fig. 9B The shrunken mask is easier to distinguish different field areas and easier to converge during the training process. The training process adopts the binary cross entropy loss function shown in formula (1).

[0094] In step 103, single character level detection processing is performed on the image based on the extracted features to obtain a character level detection result.

[0095] As an example, the execution order between step 102 and step 103 can be executed simultaneously or in any order.

[0096] In some embodiments, see Figure 3C , Figure 3C It is a flow chart of the box-type structure text recognition method provided in an embodiment of the present application. Single character detection processing is achieved through a pixel category classification network and a pixel position regression network. In step 103, single character level detection processing is performed on the image based on the extracted features to obtain character level detection results, which can be achieved through steps 1031-1033.

[0097] In step 1031, feature-based pixel detection processing is performed on the image through a pixel category classification network to obtain a first feature map.

[0098] In step 1032, a pixel position regression network is used to perform feature-based pixel position regression processing on the image to obtain a second feature map.

[0099] In step 1033, the first feature map and the second feature map are combined into a character-level detection result.

[0100] As an example, the output of the pixel classification network is a first feature map with a channel number of 2. There is a corresponding value for each pixel on the first feature map. The value of the first feature map obtained after maximum likelihood processing represents the probability that the corresponding pixel is on or inside the character border. During the training process, the pixels on or inside the character annotation box are regarded as positive samples (class 1), otherwise they are regarded as negative samples (class 0). The loss function adopts the standard cross entropy loss, see formula (1):

[0101]

[0102] Where N is the total number of pixels, xyc means the pixel is located at the (x, y) position of the cth channel, and Yxyc Indicates the category of the pixel, with a value of 0 or 1. Indicates the probability of predicting that the pixel is a positive sample, L cls is the loss value of the pixel category classification network.

[0103] As an example, the output of the pixel position regression network is a second feature map with 8 channels. The value of each pixel point on the second feature map represents the vertical and horizontal distances of the pixel point from the four vertices of the character border, thereby determining the border position of the character to which the point belongs. Different from the segmentation principle-based algorithm that only uses the information of the center point of the border, the information of the four vertices of the border can be used at the same time, so that a more accurate edge detection result can be obtained. The training process of the pixel position regression network adopts the smooth_l1 loss function, which has high robustness to outliers and is defined as formula (2):

[0104]

[0105] Among them, N pos is the total number of pixels belonging to positive samples. Negative samples do not participate in the training of the position regression network. is the predicted distance, d is the actual marked distance, L reg is the loss value of the pixel position regression network.

[0106] In step 104, based on the character-level detection result, a character category corresponding to the character-level detection result is determined.

[0107] In some embodiments, in step 104, determining the character category corresponding to the character-level detection result based on the character-level detection result can be achieved through the following technical scheme: based on the character-level detection result, determining the first pixel belonging to the single character and the distance between the first pixel and the character border to which it belongs; based on the first pixel and the distance between the first pixel and the character border to which it belongs, determining the character category of the single character.

[0108] In some embodiments, the character-level detection result includes a first feature map of multiple pixels and a second feature map of multiple pixels. The above-mentioned determination of the first pixel belonging to a single character and the distance between the first pixel and the border of the character to which it belongs based on the character-level detection result can be achieved through the following technical scheme: performing maximum likelihood processing on the first feature map of multiple pixels to obtain a first probability that each pixel belongs to a single character; determining pixels whose first probability is greater than a first probability threshold as first pixels belonging to a single character; and obtaining the value of each first pixel in the second feature map, wherein the value represents the distance between the first pixel and the border of the character to which it belongs.

[0109] As an example, the output of the pixel category classification network is a first feature map with 2 channels. The value of each pixel point on the first feature map after being processed by the Softmax function represents the first probability that the pixel is classified as a character foreground, and the range of the first probability is between 0 and 1. The value of each pixel point on the second feature map represents the vertical and horizontal distances of the pixel point from the four vertices of the character border, thereby determining the border position of the character to which the point belongs.

[0110] In some embodiments, the character category is determined by calling a single character classification network. The above-mentioned determination of the character category of a single character based on the first pixel and the distance between the first pixel and the character border to which it belongs can be achieved by the following technical scheme: determining the position of the character border based on the first pixel and the distance between the first pixel and the character border to which it belongs; cropping the image according to the position of the character border to obtain the character area; performing character classification processing on the character area through the single character classification network to obtain the character category of the single character in the character area.

[0111] As an example, ResNet18 is used as the network structure of the single-character classification network. The classification categories include numbers 0 to 9, uppercase letters AZ, and lowercase letters a to z. The number 0 and uppercase letter O and lowercase letter o have great similarity and are classified into one category in the character classification stage. They can be corrected based on preset rules later. The preset rules include unifying the case of adjacent letters, etc. Similar situations also include uppercase letter S and lowercase letter s, uppercase letter P and lowercase letter p, etc. There are 53 classification categories in total.

[0112] In step 105, based on the character category and the field-level segmentation result, a box-structured field recognition result is determined.

[0113] In some embodiments, the field-level segmentation result includes a third feature map of multiple pixels. In step 104, the field recognition result of the box structure is determined based on the character category and the field-level segmentation result. This can be achieved by the following technical solution: maximum likelihood processing is performed on the third feature map of multiple pixels to obtain a second probability that each pixel is in the field area; pixels whose second probability is greater than a second probability threshold are determined as second pixels in the field area; connected domain analysis is performed based on the position of the second pixel to obtain multiple connected domains; edge information of each connected domain is obtained, and outward expansion processing is performed based on the edge information to obtain the field area; the overlap between pixels belonging to the character category and the field area is determined; when the overlap is greater than the overlap threshold, the character category is determined as the field recognition result.

[0114] As an example, for the third feature map output by the text area segmentation network, maximum likelihood processing is performed to obtain the second probability that each pixel is in the field area, and the pixels whose second probability is greater than the second probability threshold are determined as the second pixels in the field area, so that all the second pixels belonging to a certain field in the image can be obtained. The second pixels overlap with the first pixels, and some pixels are both first pixels and second pixels. The positions of multiple second pixels constitute the position of a candidate field area. Connected domain analysis is performed based on the positions of multiple second pixels to obtain multiple connected domains. For each connected domain, its edge information is obtained, and its edge is expanded outward to obtain the border coordinates of each field area. Finally, The field area is actually obtained by merging the candidate field areas. The set W is used to represent the border position information of all field areas. It is necessary to match each character with the field area to which it belongs. Each field area in the W set is traversed, and the degree of overlap between each character and the field area is calculated. If the degree of overlap is greater than the specified threshold (set to 0.8), it indicates that the character matches the field area successfully. Based on the character type of a single character and the pairing relationship between the character and the field area, the position information and text content corresponding to each field area can be obtained. For example, if the degree of overlap between the pixel belonging to character category A and the field area a is greater than the specified threshold, then character A belongs to the field area a, and thus A is the text content of the field area a.

[0115] In step 106, the field recognition results are processed for structured information extraction to obtain text information on the box structure.

[0116] In some embodiments, in step 106, structured information extraction processing is performed on the field recognition results to obtain text information on the box-type structure, which can be achieved through the following technical solutions: when there is a field recognition result in the image that meets the numbering rules of the box-type structure, the field recognition result is determined as a candidate number for the box-type structure; when there is no field recognition result that meets the numbering rules of the box-type structure in the image, multiple field recognition results in the image are combined to obtain a combined recognition result, and the combined recognition result that meets the numbering rules of the box-type structure is determined as a candidate number for the box-type structure; when the field recognition result is in the box type list of the box-type structure, the field recognition result is determined as a candidate box type for the box-type structure; when the field recognition result meets the weight identification rule, the field recognition result is determined as a candidate weight for the box-type structure; based on the candidate number, candidate box type and candidate weight, the text information on the box-type structure is determined.

[0117] As an example, extract the box number information. The standard box structure box number consists of 11 bits and follows specific verification rules. Traverse each field area. If the text content (a field recognition result) conforms to the box number numbering rules, terminate the traversal process and output the box number. If the field recognition results of all field areas do not conform to the rules, it is necessary to combine the field recognition results of different field areas for verification. This is because the box number text may consist of one row, one column, two rows, three rows, or even four rows of text. Extract the box type information. The box type of the box structure is an exhaustible list, and the arrangement on the box structure is one row or one column. Therefore, traverse each field area. If the field recognition result happens to be in the box type list, terminate the traversal process and output the box type. Extract the weight identification information of the total weight, load, dead weight, and volume. Each weight identification information has metric units and imperial units. For example, the volume information corresponds to The metric unit and imperial unit are cubic meter (CUM) and cubic foot (CUFT) respectively. There is a relationship between the values ​​corresponding to the total weight, load and dead weight: total weight = dead weight + load. Based on this prior information, the corresponding value is obtained according to the English name of the unit of each information. For example, from the field recognition result "28260KGS" in the field area, the value corresponding to the metric unit of the information can be obtained through KGS, which is 28260 kilograms. After traversing all KGS, 3 such information can be obtained. Based on the relationship between the total weight, load and dead weight information, it can be determined which information these 3 values ​​represent. Similarly, from the field recognition result "33.2CUM" in the field area, the value corresponding to the metric unit of volume can be obtained as 33.2 cubic meters, and from the field recognition result "1170CUFT" in the field area, the value corresponding to the imperial unit of volume can be obtained as 1170 cubic feet.

[0118] In some embodiments, when the images are multiple images captured from different surfaces of a box-type structure, the above-mentioned determination of the text information on the box-type structure based on the candidate numbers, candidate box types and candidate weights can be achieved through the following technical solutions: performing padding and combining processing on the candidate numbers obtained based on different images, and using the obtained padding and combining processing results as the number of the box-type structure; performing padding and combining processing on the candidate box types obtained based on different images, and using the obtained padding and combining processing results as the box type of the box structure; performing padding and combining processing on the candidate weights obtained based on different images, and using the obtained padding and combining processing results as the weight of the box-type structure; combining the number of the box structure, the box type of the box structure and the weight of the box structure into the text information on the box structure.

[0119] As an example, during the long-distance transfer of a box-type structure, the text on its surface is very likely to be worn, which will lead to the failure of character information extraction. The box number and box type of the box-type structure exist on the left side, right side, and door of the box structure. The extraction results of these three sides can be combined to output the final box number and box type results. The confidence of the combined result is higher. The combination is actually a filling process. For example, the number obtained based on the front image is 125802258 1, and the number obtained based on the left image is 1258 225851. The final number 12580225851 can be obtained through the filling process.

[0120] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0121] The embodiments of the present application can be applied to various application scenarios, for example, a container text recognition scenario, such as Figure 1 As shown, the terminal 400 is connected to the server 200 deployed in the cloud through the network 300, and the text recognition application of the container is installed on the terminal 400. After obtaining the recognition request for the image to be recognized, the container text recognition interface of the server 200 is called. The server 200 performs feature extraction processing on the image of the container according to the recognition request for the image to be recognized, and performs single character level detection processing on the image based on the extracted features to obtain character level detection results; based on the characteristics of the container, the image is subjected to field level text area segmentation processing to obtain field level segmentation results; based on the character level detection results, the character category corresponding to the character level detection results is determined; based on the character category and the field level segmentation results, the field recognition results of the container are determined; the field recognition results are subjected to structured information extraction processing on the field recognition results to obtain text information on the container, so as to respond to the text recognition request for the container, so as to subsequently perform accurate information entry of the container according to the recognition results, thereby improving the transshipment efficiency and utilization efficiency of the container.

[0122] A container is a steel box that is standardized according to specifications. It has a uniform format, a large volume, and can be stacked layer by layer. It is often used in the transportation and logistics industry and is a cost-effective cargo transportation container. The surface of the container contains several characters, which reflect the container number, box type, weight mark and other information. This information plays a very important identification role in the operation of the container. In related technologies, this information is manually entered, which is inefficient and requires a lot of manpower. It can be seen that in actual scenarios, there is an urgent need to automatically extract this character information.

[0123] However, in the embodiments of the present application, it is found that the problem of container text recognition has the following problems in practice:

[0124] 1) Due to the limitations and complexity of shooting conditions, there are large differences in images taken by different people using different hardware. The main problems are different brightness, size, angle, and the presence of occlusions. Box-structured text recognition needs to be highly robust in these situations to obtain accurate character structured information.

[0125] 2) The subsequent process has high requirements for the accuracy of character information extraction, and the characters on one side of the container may be worn. It is important to comprehensively utilize the character information on multiple sides.

[0126] 3) The arrangement of the box number and box type information on each side of the container is complex and diverse, and the style of the weight label information on the container door is not uniform. It is important to comprehensively utilize the character information on multiple sides.

[0127] In summary, the arrangement of the container number, box type, and weight characters is complex and diverse, and cannot be extracted by optical character recognition technology. At the same time, there are usually various uncontrollable interference factors when taking photos, such as image brightness changes, image distortion, inconsistent text scale, foreign object occlusion, etc. These problems make it particularly difficult to extract the text information of the container. See Figures 5A-5C , Figures 5A-5C is an image of a box-type structure provided in an embodiment of the present application, Figures 5A-5C Displays container images taken in real business scenarios. Figure 5A The image in the figure is a bottom-up shot of the container door. Figure 5B The image in is the image with watermark, Figure 5C The images in the figure are the side views of containers. These images have various styles, different brightness, and different text sizes. These problems have a great impact on the extraction of container character information.

[0128] An embodiment of the present application provides a text recognition method for a box-type structure. When the box-type structure is a container, the text of the container mainly includes information such as the box number, box type, and weight mark. The embodiment of the present application can simultaneously process the box number and box type information of different arrangements on each side of the container. The box number and box type results on different sides can be verified with each other, and the information such as the weight mark on the container door can be structured and output.

[0129] See also Fig.10 , Fig.10 is a schematic diagram of the result of the box-type structure text recognition method provided in the embodiment of the present application, Fig.10 The middle cabinet type is 22G1. Fig.10The container number is APZU3788699. Based on the input image of the container door taken in a natural scene, the text recognition method of the box structure provided in the embodiment of the present application can fully automatically output the container number (container number), box type (container type), gross weight, load, dead weight, volume and other information, which is convenient for subsequent entry into the online system and reduces a lot of manpower investment.

[0130] See also Figure 6 , Figure 6 is a logical schematic diagram of the box-type structure text recognition method provided in the embodiment of the present application. First, three images of the container (the left image of the container, the right image of the container, and the image of the container door) for which character information extraction is required are obtained as input data of the text recognition model. Then, character detection and text area segmentation are performed based on the input data, and then character classification is performed. Then, the characters are merged into fields, and finally structured information is extracted. See Figures 7A-7C , Figures 7A-7C is a partial image schematic diagram of a box-type structure text recognition method provided in an embodiment of the present application, Fig. 7A It is a local image that is input to the text recognition model as the image to be recognized. Based on deep learning network technology, the text recognition model simultaneously outputs the character-level detection results and the segmentation results of the field-level text area (field-level segmentation results). Specifically, for a single character detection branch, it outputs the value of each pixel of the classification feature map (first feature map), and the value represents the probability that the pixel is a character. The single character detection branch also outputs the value of each pixel of the positioning feature map (second feature map), and the value represents the horizontal and vertical distances of the pixel from the four vertices of the character border, thereby determining the border position of the character to which the point belongs, see Figure 7B , box 701B represents the detection result of a single character. For the segmentation branch of the field-level text region, the value of each pixel in its output feature map (third feature map) represents the probability that the pixel point is a field-level text region, see Figure 7C , the white area 701C represents the field-level text area. For the detected single characters, the single characters are cut out from the original image based on their border positions, and the characters are classified. The character categories include numbers 0 to 9, uppercase letters AZ, and lowercase letters a to z. Combined with the segmentation results of the field-level text area (field-level segmentation results), the detection and classification results of the single characters are merged into the field-level detection and classification results. Specifically, the segmentation results of the text area are vectorized to obtain the coordinate representation of each text area, so as to obtain the information of all characters belonging to this area, which are merged into the field-level text recognition results. After obtaining the field-level text recognition results, the structured information of the characters on the container is output, including the container number, container type, gross weight, load, dead weight, volume and other information.

[0131] In the embodiment of the present application, information of a total of six fields, including container number, container type, gross weight, load, dead weight, and volume, can be accurately output, and a variety of character arrangements can be adapted. Changes in external factors such as brightness and scale have little effect on the text recognition effect. Text information is easy to enter online, which is conducive to building a highly automated container information intelligent management system. The specific advantages are as follows: the input container image is passed through a dual-branch deep neural network (text recognition model) to obtain the detection results of single characters and the segmentation results of field-level text areas. Based on the detection results of single characters, single character areas are cropped from the original image, and single characters are classified and identified to determine which number or letter they belong to. Based on the field-level text area segmentation results and the detection and classification results of single characters, field-level detection and classification results are obtained. Based on the field-level detection and classification results, the character information of the container is structured extracted using a knowledge-based rule method. The image processing technology and deep learning technology in related technologies are combined to extract information from the captured container images. Compared with the image processing technology in related technologies, it has higher accuracy and robustness and is more adaptable to low-quality images. Through the deep neural network structure, the detection results of single characters and the segmentation results of field-level text areas can be output simultaneously, which not only makes the effects of the two branches promote each other, but also can adapt to the complex and diverse character arrangement methods compared with the method of directly outputting field-level detection results. Through the knowledge-based rule method, structured information extraction can be performed on the field-level detection and classification results, which is convenient for subsequent online archiving and use.

[0132] In some embodiments, see Figure 8 , the structures of single character detection network and text region segmentation network are as follows Figure 8 As shown in the figure, the feature network includes a multi-scale feature extraction network and a multi-scale feature fusion network. The multi-scale feature extraction network is used as the backbone network to continuously fuse multi-scale features upward to output a high-resolution response map. The multi-scale feature extraction network is used to obtain multi-scale information of the input image to be identified. The high-resolution feature map has a strong representation ability for small objects, and the low-resolution feature map has a stronger representation ability for large objects. Therefore, it is very necessary to use both high-resolution and low-resolution feature maps at the same time. The pixel category classification network and the pixel position regression network are both single character detection branches. The fused multi-scale features are passed through the pixel category classification network, the pixel position regression network, and the text area segmentation network.

[0133] The following will introduce the training process of the pixel classification network, pixel position regression network and text region segmentation network respectively. The output of the pixel classification network is a first feature map with 2 channels. The value of each pixel point on the first feature map after being processed by the Softmax function represents the probability that the pixel is classified as the character foreground. The pixel classification network obtains the probability that each pixel is inside and outside the character border. During the training process, the pixels located on or inside the character annotation box are regarded as positive samples (class 1), otherwise they are regarded as negative samples (class 0). The loss function adopts the standard cross entropy loss, see formula (3):

[0134]

[0135] In the example, N is the total number of pixels, xyc indicates that the pixel is located at the (x, y) position of the cth channel, and Y xyc Indicates the category of the pixel, with a value of 0 or 1. Indicates the probability of predicting that the pixel is a positive sample, L cls is the loss value of the pixel category classification network.

[0136] The output of the pixel position regression network is a second feature map with 8 channels, which represents the vertical and horizontal distances of the pixel point from the four vertices of the character border, thereby determining the border position of the character to which the point belongs. Different from the segmentation-based algorithm that only uses the information of the border center point, it can use the information of the four vertices of the border at the same time, and can obtain more accurate edge detection results. The training process of the pixel position regression network adopts the smooth_l1 loss function, which has high robustness to outliers and is defined as formula (4):

[0137]

[0138] Among them, N pos is the total number of pixels belonging to positive samples. Negative samples do not participate in the training of the position regression network. is the predicted distance, d is the actual marked distance, L reg is the loss value of the pixel position regression network.

[0139] The output of the text region segmentation network is a third feature map with 2 channels. The value of each pixel on the third feature map after being processed by the Softmax function represents the probability that the pixel is classified as a text region. The text region is defined as an area surrounded by adjacent characters. It is a field-level concept. A polygonal mask is usually used to represent a text region, which covers all characters in the field. On the other hand, if the distance between two text regions is close, their masks will overlap, which is not conducive to the subsequent segmentation of the mask. See Figures 9A-9B , Fig. 9A The distance between the two text areas in is relatively close, and their masks 901A and 902A will overlap. The masks can be shrunk in the direction of the shortest side, with each side shrinking by 0.2*r, where r is the length of the shortest side. Fig. 9B The shrunken masks 901B and 902B are easier to distinguish different text regions and are easier to converge during the training process. The training process adopts the binary cross entropy loss function shown in formula (3).

[0140] The sum of the losses of the pixel classification network, pixel position regression network, and text region segmentation network is calculated, and the gradient descent method is used to continuously backpropagate the losses, thereby updating the parameters of the pixel classification network, pixel position regression network, and text region segmentation network in the text recognition model. After several rounds of training iterations, the text recognition model reaches an optimal state and can be used for online reasoning.

[0141] In some embodiments, ResNet18 is used as the network structure of the single character classification network. The classification categories include numbers 0 to 9, uppercase letters AZ, and lowercase letters a to z. The number 0 and uppercase letter O and lowercase letter o have great similarity. They are classified into one category in the character classification stage and corrected later. Similar situations also include uppercase letter S and lowercase letter s, uppercase letter P and lowercase letter p, etc. There are 53 classification categories in total.

[0142] In the inference stage of the text recognition model, the image to be recognized is input, the pixel category classification network outputs the probability of each pixel being inside or outside the character border, the pixel position regression network outputs the horizontal and vertical distances from each pixel to the four vertices of the character border, and the text region segmentation network outputs the segmentation mask result for each text region. Based on the sub-feature graphs output by the pixel category classification network, pixel position regression network, and text region segmentation network, perform the following steps:

[0143] 1) Based on the output of the pixel classification network, the pixels whose probability value of belonging to the foreground character is greater than a specified threshold (usually set to 0.5) are obtained. These pixels (represented by the set P) are considered to be pixels with a high probability of containing characters;

[0144] 2) For each pixel point in the set P, the horizontal and vertical distances of these pixel points from the four vertices of the character border are obtained based on the output of the pixel position regression network, so as to determine the border position of the character to which the pixel point belongs, and the set B is used to represent all the character border information;

[0145] 3) There are likely to be repeated character borders in set B. Non-maximum suppression technology is used to remove those character borders whose probability values ​​of foreground are relatively small (less than a specified threshold, such as 0.6) and have a lot of overlap with character borders with large probability values. For each character, only one most accurate border information is output. Set C represents all the character border information after filtering.

[0146] 4) For the output of the text region segmentation network, a connected domain analysis is performed to obtain multiple connected domains. For each connected domain, its edge information is obtained and its edge is expanded outward to obtain the border coordinates of each field region. The set W represents the border position information of all field regions.

[0147] 5) Each character needs to be matched with the field area to which it belongs, and each field area in the W set is traversed to calculate the degree of overlap between each character and the field area. If the degree of overlap is greater than the specified threshold (set to 0.8), it indicates that the character matches the field area successfully.

[0148] 6) Classify individual characters one by one, and combine the pairing relationship between characters and field areas to obtain the position information and text content corresponding to each field area.

[0149] 7) Based on the position information and text content of each field area output in the above steps, the character information of the container can be structured extracted. The process of structured extraction is as follows: extract the container number information. The standard container number consists of 11-bit codes and follows specific verification rules. Traverse each field area. If its text content meets the verification rules of the container number, the traversal process is terminated and the container number is output. If all areas do not meet the rules, it is necessary to combine the contents of different field areas for verification. This is because the container number text may consist of one line, one column, two lines, three lines, or even four lines of text; extract the box type information. The box type of the container is an exhaustible list, and the arrangement on the container is one row or one column. Therefore, traverse each field area. If its text content happens to be in the box type list, the traversal process is terminated and the box type is output; extract the weight identification information of the total weight, load, dead weight, and volume. Each weight identification information has a metric system. Units and imperial units, such as the metric units and imperial units corresponding to volume information are cubic meters (CUM) and cubic feet (CUFT) respectively. There is a relationship between the values ​​corresponding to the total weight, load and dead weight, total weight = dead weight + load. Based on these prior information, the corresponding values ​​are obtained according to the English names of the units of each information. For example, from the text content "28260KGS" in the field area, the value corresponding to the metric unit of the information can be obtained through KGS, which is 28260 kilograms. After traversing all KGS, 3 such information can be obtained. According to the relationship between the total weight, load and dead weight information, it can be determined which information these 3 values ​​represent. Similarly, from the text content "33.2CUM" in the field area, the value corresponding to the metric unit of volume can be obtained as 33.2 cubic meters, and from the text content "1170CUFT" in the field area, the value corresponding to the imperial unit of volume can be obtained as 1170 cubic feet.

[0150] So far, the character information extraction process of one surface of the container has been completed. However, during the long-distance transit of the container, the characters on its surface are likely to be worn, which will lead to the failure of character information extraction. The container number and type exist on the left side, right side, and door of the container. The extraction results of these three sides can be combined to output the final container number and type results, and the confidence of the combined results is higher.

[0151] The embodiments of the present application are mainly aimed at the task of extracting character information of containers, and can be applied to intelligent management systems for container information. The recall rate and accuracy of character information extraction of the six fields of the container are both over 96%, which improves the accuracy while reducing a lot of manpower input. The information is extracted from the images of the captured containers by combining image processing technology and deep learning technology. Compared with image processing technology, it has higher accuracy and robustness, and is more adaptable to low-quality images. Through the deep neural network structure, character-level detection results and field-level segmentation results can be output simultaneously, which not only promotes the processing effects of the two branches, but also can adapt to complex and diverse character arrangement methods compared with the method of directly outputting field-level detection results. By extracting structured information from the field recognition results, subsequent online archiving and use are convenient.

[0152] In the embodiment of the present application, the data input into the text recognition model may be auxiliary text data in addition to images, so as to extract the character information of the container based on a multimodal data form.

[0153] See also Fig.10 , Fig.10 The figure shows the character information extraction results of the container in the natural scene. The extracted information of the six fields is Fig.10 It can be clearly seen that the box-type text recognition method provided in the embodiment of the present application can process texts of different shapes, including horizontal text, vertical text and inclined text, and can process texts of different sizes and images of different colors and brightness, and can achieve good results.

[0154] The following is a description of an exemplary structure of a box-type text recognition device 255 provided in an embodiment of the present application implemented as a software module. In some embodiments, for example, Figure 2 As shown, the software modules in the box-type structure text recognition device 255 stored in the memory 250 may include: a feature module 2551, which is used to perform feature extraction processing on the box-type structure image; a character detection and field segmentation module 2552, which is used to perform field-level text area segmentation processing on the image based on the features of the box-type structure to obtain a field-level segmentation result; the character detection and field segmentation module 2552 is also used to perform single-character level detection processing on the image based on the extracted features to obtain a character-level detection result; a character category determination module 2553, which is used to determine the character category corresponding to the character-level detection result based on the character-level detection result; a field recognition module 2554, which is used to determine the field recognition result of the box-type structure based on the character category and the field-level segmentation result; a text extraction module 2555, which is used to perform structured information extraction processing on the field recognition result to obtain text information on the box-type structure.

[0155] In some embodiments, feature extraction processing is achieved by calling a feature extraction network, which includes a multi-scale feature extraction network and a multi-scale feature fusion network. The feature module 2551 is also used to: perform multi-scale feature extraction processing on the image of the box-type structure through the multi-scale feature extraction network to obtain sub-features of multiple scales of the box-type structure; and perform fusion processing on sub-features of multiple scales through the multi-scale feature fusion network to obtain the features of the box-type structure.

[0156] In some embodiments, the multi-scale feature extraction network includes a convolution layer, a pooling layer, and a residual layer corresponding to multiple scales. The feature module 2551 is also used to: perform convolution processing on the image through the convolution layer to obtain a convolution processing result; perform pooling processing on the convolution processing result through the pooling layer to obtain a pooling result; perform residual processing on the pooling result through cascaded residual layers corresponding to multiple scales one by one to obtain sub-features of multiple scales of a box-type structure.

[0157] In some embodiments, the feature module 2551 is also used to: perform residual processing on the input of the nth residual layer through the nth residual layer in N cascaded residual layers, and transmit the nth residual processing result output by the nth residual layer to the n+1th residual layer to continue the residual processing; wherein n is an integer whose value increases from 1, and the value range of n satisfies 1≤n≤N-1, and N is an integer greater than or equal to 2; when n is 1, the input of the nth residual layer is the pooling result, and when n is 2≤n≤N-1, the input of the nth residual layer is the residual processing result of the n-1th residual layer, and the residual processing result of the nth residual layer is the feature of the corresponding scale.

[0158] In some embodiments, the multi-scale feature fusion network includes at least one fusion layer, and the feature module 2551 is also used to: when the number of fusion layers is 1, the sub-features output by the Nth residual layer and the sub-features output by the N-1th residual layer are fused through the fusion layer to obtain the features of the box structure; when the number of fusion layers is multiple, the input of the mth fusion layer is fused through the mth fusion layer in the M cascaded fusion layers, and the mth fusion processing result output by the mth fusion layer is transmitted to the m+1th fusion layer to continue The m-th fusion layer is subjected to fusion processing; wherein, m is an integer increasing from 1, and the value range of m satisfies 1≤m≤M-1, and M is N-1; when m is 1, the input of the m-th fusion layer is the sub-feature output by the N-th residual layer and the sub-feature output by the N-1-th residual layer; when m is 2≤m≤M-1, the input of the m-th fusion layer is the m-1-th fusion processing result of the m-1-th fusion layer and the sub-feature output by the Nm-th residual layer; when m is M-1, the fusion processing result of the m+1-th fusion layer is the feature of the box structure.

[0159] In some embodiments, single character detection processing is implemented through a pixel category classification network and a pixel position regression network. The character detection and field segmentation module 2551 is also used to: perform feature-based pixel detection processing on the image through a pixel category classification network to obtain a first feature map; perform feature-based pixel position regression processing on the image through a pixel position regression network to obtain a second feature map; and combine the first feature map and the second feature map into a character-level detection result.

[0160] In some embodiments, text region segmentation processing is implemented by calling a text region segmentation network. The character detection and field segmentation module 2552 is also used to: perform feature-based text region segmentation processing on multiple pixels of the image through the text region segmentation network to obtain a third feature map corresponding to the multiple pixels; and use the third feature map as the field-level segmentation result.

[0161] In some embodiments, the character category determination module 2553 is also used to: determine the first pixel belonging to a single character and the distance between the first pixel and the character border to which it belongs based on the character-level detection results; determine the character category of the single character based on the first pixel and the distance between the first pixel and the character border to which it belongs.

[0162] In some embodiments, the character category is determined by calling a single character classification network, and the character category determination module 2553 is also used to: determine the position of the character border based on the first pixel and the distance between the first pixel and the character border to which it belongs; crop the image according to the position of the character border to obtain the character area; perform character classification processing on the character area through the single character classification network to obtain the character category of the single character in the character area.

[0163] In some embodiments, the character-level detection results include a first feature map of multiple pixels and a second feature map of multiple pixels. The character category determination module 2553 is also used to: perform maximum likelihood processing on the first feature map of multiple pixels to obtain a first probability that each pixel belongs to a single character; determine pixels whose first probability is greater than a first probability threshold as first pixels belonging to a single character; and obtain the value of each first pixel in the second feature map, wherein the value represents the distance between the first pixel and the border of the character to which it belongs.

[0164] In some embodiments, the field-level segmentation result includes a third feature map of multiple pixels, and the field identification module 2554 is further used to: perform maximum likelihood processing on the third feature map of multiple pixels to obtain a second probability that each pixel is in the field area; determine the pixel whose second probability is greater than the second probability threshold as the second pixel in the field area; perform connected domain analysis based on the position of the second pixel to obtain multiple connected domains; obtain edge information of each connected domain, and perform outward expansion processing based on the edge information to obtain the field area; determine the overlap between pixels belonging to the character category and the field area; when the overlap is greater than the overlap threshold, determine the character category as the field identification result.

[0165] In some embodiments, the text extraction module 2555 is also used to: when there is a field recognition result in the image that satisfies the numbering rules of the box structure, determine the field recognition result as a candidate number for the box structure; when there is no field recognition result that satisfies the numbering rules of the box structure in the image, combine multiple field recognition results in the image to obtain a combined recognition result, and determine the combined recognition result that satisfies the numbering rules of the box structure as a candidate number for the box structure; when the field recognition result is in the box type list of the box structure, determine the field recognition result as a candidate box type for the box structure; when the field recognition result satisfies the weight identification rule, determine the field recognition result as a candidate weight for the box structure; and determine the text information on the box structure based on the candidate number, candidate box type and candidate weight.

[0166] In some embodiments, when the images are multiple images captured from different surfaces of a box-type structure, the text extraction module 2555 is further used to: perform padding and combining processing on candidate numbers obtained based on different images, and use the obtained padding and combining processing results as the number of the box-type structure; perform padding and combining processing on candidate box types obtained based on different images, and use the obtained padding and combining processing results as the box type of the box-type structure; perform padding and combining processing on candidate weights obtained based on different images, and use the obtained padding and combining processing results as the weight of the box-type structure; combine the number of the box-type structure, the box type of the box-type structure and the weight of the box-type structure into text information on the box-type structure.

[0167] The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned box-type structure text recognition method of the embodiment of the present application.

[0168] The present application embodiment provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the box-type text recognition method provided by the present application embodiment, for example, Figures 3A-3D The box-structured text recognition method is shown.

[0169] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EP ROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0170] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0171] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0172] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0173] To summarize, through the embodiments of the present application, single character level detection processing and field level text area segmentation processing are simultaneously performed on the image to obtain character level detection results and field level segmentation results, and then based on the character level detection results and field level segmentation results, the box-type structure field recognition results are determined. This can not only promote the effects of the dual-branch processing, but also adapt to the complex and diverse character arrangement methods, and effectively improve the accuracy of box-type structure text recognition.

[0174] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A box-type text recognition method, characterized in that: include: Performing multi-scale feature extraction processing on the image of the box-type structure through a multi-scale feature extraction network in the feature extraction network to obtain sub-features of multiple scales of the box-type structure; The sub-features of the multiple scales are fused through a multi-scale feature fusion network in the feature extraction network to obtain the features of the box-type structure; Performing pixel detection processing on the image based on the feature to obtain a first feature map; Performing pixel position regression processing on the image based on the feature to obtain a second feature map; Combining the first feature map and the second feature map into a character-level detection result; Performing text region segmentation processing based on the features on a plurality of pixels of the image to obtain a third feature map corresponding to the plurality of pixels, and using the third feature map as a field-level segmentation result; Based on the character-level detection result, determining a character category corresponding to the character-level detection result; Determining a field recognition result of the box-type structure based on the character category and the field-level segmentation result; The field recognition result is subjected to structured information extraction processing to obtain text information on the box structure.

2. The method according to claim 1, characterized in that The multi-scale feature extraction network includes a convolution layer, a pooling layer, and a residual layer corresponding to the multiple scales. The multi-scale feature extraction network is used to perform multi-scale feature extraction processing on the image of the box structure to obtain sub-features of multiple scales of the box structure, including: Performing convolution processing on the image through the convolution layer to obtain a convolution processing result; Performing pooling processing on the convolution processing result through the pooling layer to obtain a pooling result; The pooling result is subjected to residual processing by cascading residual layers corresponding to the multiple scales one by one, so as to obtain sub-features of multiple scales of the box-type structure.

3. The method according to claim 2, characterized in that The residual processing is performed on the pooling result through the cascaded residual layers corresponding to the multiple scales to obtain sub-features of multiple scales of the box structure, including: Through the nth residual layer in the N cascaded residual layers, residual processing is performed on the input of the nth residual layer, and Transmitting the nth residual processing result output by the nth residual layer to the (n+1)th residual layer to continue residual processing; Among them, n is an integer that increases from 1, and the value range of n satisfies 1≤n≤N-1, and N is an integer greater than or equal to 2; when n is 1, the input of the nth residual layer is the pooling result, when n is 2≤n≤N-1, the input of the nth residual layer is the residual processing result of the n-1th residual layer, and the residual processing result of the nth residual layer is the sub-feature corresponding to the scale.

4. The method according to claim 3, characterized in that The multi-scale feature fusion network includes at least one fusion layer, and the multi-scale feature fusion network is used to fuse the sub-features of the multiple scales to obtain the features of the box-type structure, including: When the number of the fusion layer is 1, the sub-features output by the Nth residual layer and the sub-features output by the N-1th residual layer are fused through the fusion layer to obtain the features of the box structure; When the number of the fusion layers is multiple, the input of the mth fusion layer is fused through the mth fusion layer in the M cascaded fusion layers, and Transmitting the mth fusion processing result output by the mth fusion layer to the (m+1)th fusion layer to continue the fusion processing; Among them, m is an integer increasing from 1, and the value range of m satisfies 1≤m≤M-1, M is N-1; when m is 1, the input of the mth fusion layer is the sub-features output by the Nth residual layer and the sub-features output by the N-1th residual layer; when m is 2≤m≤M-1, the input of the mth fusion layer is the m-1th fusion processing result of the m-1th fusion layer and the sub-features output by the Nmth residual layer; when m is M-1, the fusion processing result of the m+1th fusion layer is the feature of the box structure.

5. The method according to claim 1, characterized in that: The performing pixel detection processing on the image based on the feature to obtain a first feature map includes: Performing pixel detection processing based on the feature on the image through the pixel category classification network to obtain a first feature map; The performing pixel position regression processing on the image based on the feature to obtain a second feature map includes: The pixel position regression network performs pixel position regression processing on the image based on the feature to obtain a second feature map.

6. The method according to claim 1, characterized in that The step of performing text region segmentation processing on a plurality of pixels of the image based on the feature to obtain a third feature map corresponding to the plurality of pixels includes: A text region segmentation network is used to perform text region segmentation processing on a plurality of pixels of the image based on the features, so as to obtain a third feature map corresponding to the plurality of pixels.

7. The method according to claim 1, characterized in that The determining, based on the character-level detection result, a character category corresponding to the character-level detection result comprises: Based on the character-level detection result, determining a first pixel belonging to a single character and a distance between the first pixel and a border of the character to which it belongs; The character category of the single character is determined based on the first pixel and the distance between the first pixel and the border of the character to which it belongs.

8. The method according to claim 7, characterized in that The character-level detection result includes a first feature map of a plurality of pixels and a second feature map of the plurality of pixels, and determining a first pixel belonging to a single character and a distance between the first pixel and a border of the character to which it belongs based on the character-level detection result includes: Performing maximum likelihood processing on the first feature map of the plurality of pixels to obtain a first probability that each of the pixels belongs to the single character; Determine a pixel whose first probability is greater than a first probability threshold as a first pixel belonging to the single character; Obtain a value of each of the first pixels in the second feature map, wherein the value represents a distance between the first pixel and a character border to which it belongs.

9. The method according to claim 7, characterized in that: The character category is determined by calling a single character classification network, and the character category of the single character is determined based on the first pixel and the distance between the first pixel and the character border to which it belongs, including: Determine the position of the character border based on the first pixel and the distance between the first pixel and the character border to which it belongs; Crop the image according to the position of the character border to obtain a character area; The character region is subjected to character classification processing by the single character classification network to obtain the character category of the single character in the character region.

10. The method according to claim 1, characterized in that The field-level segmentation result includes a third feature map of multiple pixels, and determining the field recognition result of the box-type structure based on the character category and the field-level segmentation result includes: Performing maximum likelihood processing on the third feature map of the plurality of pixels to obtain a second probability that each of the pixels is in the field area; Determine a pixel whose second probability is greater than a second probability threshold as a second pixel in the field area; Performing a connected domain analysis based on the position of the second pixel to obtain a plurality of connected domains; Acquire edge information of each of the connected domains, and perform outward expansion processing based on the edge information to obtain the field area; Determining the degree of overlap between pixels belonging to the character category and the field area; When the degree of overlap is greater than an overlap threshold, the character category is determined as the field recognition result.

11. The method according to claim 1, characterized in that: The performing structured information extraction processing on the field recognition result to obtain text information on the box-type structure includes: When there is a field recognition result in the image that meets the numbering rule of the box-type structure, determining the field recognition result as a candidate number of the box-type structure; When there is no field recognition result satisfying the numbering rule of the box-type structure in the image, combining a plurality of field recognition results in the image to obtain a combined recognition result, and determining the combined recognition result satisfying the numbering rule of the box-type structure as a candidate number of the box-type structure; When the field recognition result is in the box type list of the box structure, determining the field recognition result as a candidate box type of the box structure; When the field recognition result satisfies the weight identification rule, determining the field recognition result as the candidate weight of the box-type structure; Based on the candidate numbers, the candidate box types, and the candidate weights, text information on the box structure is determined.

12. A box-type text recognition device, characterized in that: include: The feature module performs multi-scale feature extraction processing on the image of the box-type structure through a multi-scale feature extraction network in the feature extraction network to obtain sub-features of multiple scales of the box-type structure; and performs fusion processing on the sub-features of multiple scales through a multi-scale feature fusion network in the feature extraction network to obtain features of the box-type structure. A character detection and field segmentation module, used for performing pixel detection processing on the image based on the feature to obtain a first feature map; Performing pixel position regression processing on the image based on the feature to obtain a second feature map; combining the first feature map and the second feature map into a character-level detection result; The character detection and field segmentation module is further used to perform text area segmentation processing on multiple pixels of the image based on the features to obtain a third feature map corresponding to the multiple pixels, and use the third feature map as a field-level segmentation result; A character category determination module, used to determine the character category corresponding to the character level detection result based on the character level detection result; A field recognition module, used to determine the field recognition result of the box structure based on the character category and the field level segmentation result; The text extraction module is used to perform structured information extraction processing on the field recognition result to obtain text information on the box structure.

13. An electronic device, characterized in that: include: A memory for storing executable instructions; The processor is used to implement the box-type structure text recognition method described in any one of claims 1 to 12 when executing the executable instructions stored in the memory.

14. A computer-readable storage medium, characterized in that: Executable instructions are stored, which are used to implement the box-type structure text recognition method described in any one of claims 1 to 11 when executed by a processor.

15. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the box-type structure text recognition method described in any one of claims 1 to 11 is implemented.

Images