Label image processing method and device, equipment and storage medium
By using image segmentation and planarization processing technology in steel warehouses, rectangular label images are generated and text information is extracted, and the problem of poor recognition effect in the prior art is solved, and high-precision label information recognition is achieved.
Patent Information
- Application Number
- CN202411786419.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-05
- Publication Date
- 2025-04-18
AI Technical Summary
When the prior art recognizes tag information through OCR technology in steel warehouses, it is limited by the shooting angle and background interference, resulting in poor recognition effect and cannot realize practical application in industrial scenarios.
By obtaining the coordinate information of the target label in the image to be identified, image segmentation is performed to generate a quadrilateral bounding box, plane processing is performed to generate a rectangular label image, and text information is extracted.
It improves image recognition accuracy, solves the problem of shooting angle and background interference, and realizes label information recognition in industrial scenarios.
Smart Images
Figure CN119851004A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a label image processing method, device, equipment and storage medium. Background Art
[0002] At present, in the warehousing and warehouse supervision of steel, it is necessary to obtain steel label information and perform structured processing of the label information.
[0003] Compared with the traditional method of manually entering information, most of the information recognition and extraction in the market is realized through OCR (Optical Character Recognition) technology. OCR technology has high requirements for label sampling and shooting, but steel is usually stacked in multiple layers in the warehouse, which has problems such as occlusion and inability to shoot and sample at a specific angle. Therefore, it has a great impact on the final recognition effect and cannot be used in actual industrial scenarios. Summary of the invention
[0004] According to one aspect of the present application, a method for processing a label image is provided, comprising: obtaining coordinate information of a target label in an image to be identified to perform image segmentation of the target label; generating a quadrilateral bounding box of the target label after image segmentation; planarizing the quadrilateral bounding box of the target label to generate a rectangular label image corresponding to the target label; and extracting text information from the rectangular label image.
[0005] According to some embodiments, the coordinate information of the target label in the image to be identified is obtained to perform image segmentation of the target label, including: obtaining the coordinate information of the target label through a preset label detection model; based on the coordinate information of the target label, performing instance segmentation of the target label through a preset image segmentation model; and obtaining a mask image corresponding to the segmented target label.
[0006] According to some embodiments, generating a quadrilateral bounding box of a target label after image segmentation includes: generating a polygonal convex hull contour corresponding to the target label based on a mask image; optimizing the vertex positions of the polygonal convex hull contour to generate multiple candidate quadrilateral bounding boxes; and determining the quadrilateral bounding box of the target label among the multiple candidate quadrilateral bounding boxes.
[0007] According to some embodiments, a polygonal convex hull contour corresponding to a target label is generated based on a mask image, including: obtaining non-zero pixel points in the mask image; constructing an initial convex hull contour corresponding to the target label based on the non-zero pixel points; and performing boundary simplification and optimization on the initial convex hull contour to obtain a polygonal convex hull contour.
[0008] According to some embodiments, determining a quadrilateral bounding box of a target label among a plurality of candidate quadrilateral bounding boxes includes: calculating a difference loss of each of the plurality of candidate quadrilateral bounding boxes; determining a rectangular fit of each of the plurality of candidate quadrilateral bounding boxes according to the difference loss; and determining the quadrilateral bounding box of the target label according to the rectangular fit.
[0009] According to some embodiments, a quadrilateral bounding box of a target tag is planarized to generate a rectangular label image corresponding to the target tag, including: calculating the planarized output size of the quadrilateral bounding box of the target tag; constructing a perspective transformation matrix based on the planarized output size; and planarizing the quadrilateral bounding box of the target tag through the perspective transformation matrix to generate a rectangular label image.
[0010] According to some embodiments, extracting text information from a rectangular label image includes: acquiring the text information from the rectangular label image through a preset character recognition model; and converting the text information into a preset output format.
[0011] According to one aspect of the present application, a label image processing device is provided, including: an image segmentation module, used to obtain the coordinate information of the target label in the image to be identified, so as to perform image segmentation of the target label; a boundary optimization module, used to generate a quadrilateral bounding box of the target label after image segmentation; a rectangular correction module, used to planarize the quadrilateral bounding box of the target label, so as to generate a rectangular label image corresponding to the target label; an information extraction module, used to extract text information in the rectangular label image.
[0012] According to one aspect of the present application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.
[0013] According to one aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the aforementioned method is implemented.
[0014] According to the embodiments of the present application, image segmentation can be performed through a model based on deep learning technology, and a rectangular label image of the target can be generated through multiple optimization and correction methods to extract information, thereby solving the problems in the prior art of high requirements on image shooting angle and shooting quality, background interference, insufficient correction accuracy for small angles, and inability to achieve perspective transformation when the image label has multiple missing corners, thereby improving image recognition accuracy.
[0015] It should be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application.
[0017] Figure 1 A flowchart of a label image processing method according to an exemplary embodiment of the present application is shown.
[0018] Figure 2 A block diagram of a label image processing device according to an exemplary embodiment of the present application is shown.
[0019] Figure 3 A block diagram of an electronic device according to an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION
[0020] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this application will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.
[0021] The described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application may be put into practice without one or more of these specific details, or other modes, components, materials, devices or operations may be adopted. In these cases, known structures, methods, devices, realizations, materials or operations will not be shown or described in detail.
[0022] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.
[0023] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.
[0024] The present application provides a label image processing method, device, equipment and storage medium, which can improve image recognition accuracy.
[0025] A label image processing method, apparatus, device and storage medium according to an embodiment of the present application will be described in detail below with reference to the accompanying drawings.
[0026] Figure 1 A flowchart of a label image processing method according to an exemplary embodiment of the present application is shown.
[0027] like Figure 1 As shown, in step S100, the processing device obtains the coordinate information of the target label in the image to be identified to perform image segmentation of the target label.
[0028] For example, in step S100, the processing device obtains the coordinate information of the target label in the image to be identified through a preset label detection model, and performs instance segmentation on the target label through a preset image segmentation model.
[0029] The processing device obtains the image to be identified, and obtains the coordinate information of the target label in the image to be identified through a preset label detection model.
[0030] According to some embodiments, before obtaining the coordinate information of the target tag through a preset tag detection model, the processing device obtains a training data set formed by labeling a large number of object tags, and uses the training data set to train the preset tag detection model so that the preset tag detection model can obtain the coordinate information of the target tag.
[0031] Based on the acquired coordinate information of the target tag, the processing device performs instance segmentation on the target tag through a preset image segmentation model.
[0032] According to some embodiments, before the preset image segmentation model performs image segmentation, the processing device needs to train the preset image segmentation model.
[0033] According to some embodiments, the preset label detection model and the preset image segmentation model may adopt a CNN (Convolutional Neural Networks) model.
[0034] Furthermore, the processing device obtains a mask image corresponding to the target label after instance segmentation.
[0035] In step S200, the processing device generates a quadrilateral bounding box of the target label after image segmentation.
[0036] For example, in step S200, the processing device generates a polygonal convex hull contour corresponding to the target tag, and optimizes it to obtain a quadrilateral bounding box of the target tag.
[0037] Based on the mask image obtained after the target label instance segmentation, the processing device generates a polygonal convex hull contour corresponding to the target label.
[0038] According to some embodiments, the processing device identifies non-zero pixels in the mask image and obtains their coordinates to form a non-zero pixel set. Furthermore, the processing device constructs an initial convex hull contour corresponding to the target label based on the non-zero pixel set.
[0039] Furthermore, the processing device may optimize the initial convex hull contour by a boundary simplification algorithm, and gradually reduce the number of edges in the initial convex hull contour, so as to obtain a polygonal convex hull contour that fits the actual boundary of the target label to the greatest extent.
[0040] For example, the processing device calculates the angle formed by each edge in the initial convex hull contour and its two adjacent edges. When the sum of the angles formed by any edge in the initial convex hull contour and its two adjacent edges is greater than 180 degrees, the processing device determines that the edge can be deleted, and obtains a new vertex by interpolating its two adjacent edges to replace the two ends of the original three edges.
[0041] The processing device recursively optimizes the initial convex hull contour, starting from the edge that is most likely to be deleted, until the number of edges of the initial convex hull contour converges to four or the number of edges of the initial convex hull contour cannot be further reduced, and finally converges to an circumscribed convex hull that fits the actual boundary of the target label to the greatest extent, so as to form a polygonal convex hull contour corresponding to the target label.
[0042] According to some embodiments, the convex hull contour obtained after performing boundary simplification optimization on the initial convex hull contour may be a quadrilateral or a convex hull with more than four sides.
[0043] The processing device optimizes the vertex positions of the generated polygonal convex hull contour to generate a plurality of candidate quadrilateral bounding boxes.
[0044] According to some embodiments, when the number of sides of the polygonal convex hull contour exceeds four, the processing device calculates the angle between two adjacent sides of each vertex of the polygonal convex hull contour, and inserts one or more new vertices in the area between the extension lines of the two adjacent sides to generate multiple candidate quadrilateral bounding boxes.
[0045] Furthermore, the processing device determines a quadrilateral bounding box of the target label from among the plurality of candidate quadrilateral bounding boxes.
[0046] According to some embodiments, the processing device first calculates the difference loss of each of the plurality of candidate quadrilateral bounding boxes.
[0047] For example, the processing device calculates the deviation value of each inner angle of any candidate quadrilateral bounding box from 90°, and accumulates the absolute values thereof as the first type of difference value of the candidate quadrilateral bounding box.
[0048] For another example, the processing device calculates the length differences of the four sides of any candidate quadrilateral bounding box, and accumulates the absolute values thereof as the second type of difference of the candidate quadrilateral bounding box.
[0049] For another example, the processing device calculates the length difference between two diagonals of any candidate quadrilateral bounding box, and accumulates the absolute values thereof to serve as the third type of difference of the candidate quadrilateral bounding box.
[0050] The processing device normalizes and adds the first type difference, the second type difference and the third type difference of the candidate quadrilateral bounding box to obtain the difference loss of the candidate quadrilateral bounding box.
[0051] The processing device determines the rectangular fit of each of the plurality of candidate quadrilateral bounding boxes according to the difference loss of each of the plurality of candidate quadrilateral bounding boxes, wherein the smaller the difference loss of the candidate quadrilateral bounding box, the higher the rectangular fit of the candidate quadrilateral bounding box.
[0052] Furthermore, the processing device determines the final quadrilateral bounding box of the target label according to the rectangular fit of each of the plurality of candidate quadrilateral bounding boxes.
[0053] According to some embodiments, the processing device determines the candidate quadrilateral bounding box with the highest rectangular fitting degree (ie, the smallest difference loss) among the multiple candidate quadrilateral bounding boxes as the quadrilateral bounding box of the target label.
[0054] In step S300 , the processing device performs planarization processing on the quadrilateral bounding box of the target tag to generate a rectangular tag image corresponding to the target tag.
[0055] For example, in step S300, the processing device performs planarization processing on the quadrilateral bounding box of the target tag through a perspective transformation matrix to generate a rectangular tag image corresponding to the target tag.
[0056] The processing device first calculates the planarized output size of the quadrilateral bounding box of the target label.
[0057] According to some embodiments, the processing device estimates the width and height of the rectangle corresponding to the target tag by calculating the distance between opposite sides of the quadrilateral bounding box of the target tag to generate its flattened output size. The flattened output size of the quadrilateral bounding box of the target tag can ensure that the proportion of the rectangle corresponding to the transformed target tag conforms to its actual physical size.
[0058] Further, the processing device constructs a perspective transformation matrix according to the planarized output size of the quadrilateral bounding box of the target tag.
[0059] According to some embodiments, the processing device may establish a correspondence between the four vertices of the quadrilateral bounding box of the target tag and the rectangular box formed after planarization, so as to form a mapping relationship between the coordinate system where the quadrilateral bounding box of the target tag is located and the plane coordinate system, and thereby construct a perspective transformation matrix.
[0060] The processing device flattens the quadrilateral bounding box of the target label by using a perspective transformation matrix to generate a rectangular label image.
[0061] According to some embodiments, the processing device linearly transforms the coordinates of each pixel point in the quadrilateral bounding box of the target label through a perspective transformation matrix, maps the quadrilateral area of the target label to a standard rectangular plane, thereby completing the geometric reconstruction of the image and generating a planarized rectangular label image.
[0062] In step S400, the processing device extracts text information from the rectangular label image.
[0063] For example, in step S400, the processing device extracts text information from the rectangular label image through a preset character recognition model.
[0064] The processing device obtains a preset character recognition model, and uses it to obtain text information in the rectangular label image.
[0065] According to some embodiments, the preset character recognition model can be optimized and trained through the features of the target label so that the preset character recognition model can extract and parse text information in the image, such as the package number, origin, variety, brand and specification.
[0066] The processing device converts the extracted text information into a preset output format to achieve standardization and unification of the output data.
[0067] According to the embodiments of the present application, the problem of low label image recognition effect caused by factors such as shooting angle and background interference can be solved, and the recognition accuracy of label images can be improved.
[0068] Figure 2 A block diagram of a label image processing device according to an exemplary embodiment of the present application is shown.
[0069] like Figure 2 As shown, the processing device 100 includes an image segmentation module 110 , a boundary optimization module 120 , a rectangle correction module 130 and an information extraction module 140 .
[0070] The image segmentation module 110 obtains the image to be identified, and obtains the coordinate information of the target label in the image to be identified through a preset label detection model.
[0071] Based on the acquired coordinate information of the target tag, the image segmentation module 110 performs instance segmentation on the target tag through a preset image segmentation model.
[0072] The image segmentation module 110 obtains a mask image corresponding to the target label after instance segmentation.
[0073] Based on the mask image obtained after the target label instance segmentation, the boundary optimization module 120 generates the polygonal convex hull contour corresponding to the target label.
[0074] The boundary optimization module 120 optimizes the vertex positions of the generated polygonal convex hull contour to generate a plurality of candidate quadrilateral bounding boxes.
[0075] The boundary optimization module 120 determines a quadrilateral bounding box of the target label from among a plurality of candidate quadrilateral bounding boxes.
[0076] The rectangle correction module 130 first calculates the planarized output size of the quadrilateral bounding box of the target label.
[0077] The rectangle rectification module 130 constructs a perspective transformation matrix according to the planarized output size of the quadrilateral bounding box of the target tag.
[0078] The rectangle correction module 130 performs planarization of the quadrilateral bounding box of the target label by using a perspective transformation matrix to generate a rectangular label image.
[0079] The information extraction module 140 obtains a preset character recognition model, and uses it to obtain text information in the rectangular label image.
[0080] The information extraction module 140 converts the extracted text information into a preset output format.
[0081] Figure 3 A block diagram of an electronic device according to an exemplary embodiment of the present application is shown.
[0082] like Figure 3As shown, the electronic device 600 is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0083] like Figure 3 As shown, the electronic device 600 is in the form of a general-purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610), a display unit 640, etc. The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 executes the methods described in this specification according to various exemplary embodiments of the present application. For example, the processing unit 610 may execute the following: Figure 1 The method shown in .
[0084] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .
[0085] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.
[0086] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.
[0087] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0088] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by combining software with necessary hardware. The technical solution according to the embodiment of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal or a network device, etc.) to execute the method according to the embodiment of the present application.
[0089] The software product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.
[0090] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program codes contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.
[0091] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).
[0092] The computer-readable medium carries one or more programs. When the one or more programs are executed by a device, the computer-readable medium implements the aforementioned functions.
[0093] Those skilled in the art will appreciate that the above modules can be distributed in the device according to the description of the embodiment, or can be changed accordingly and only used in one or more devices different from the embodiment. The modules of the above embodiments can be combined into one module, or further divided into multiple sub-modules.
[0094] The embodiments of the present application are described in detail above, and the description of the above embodiments is only used to help understand the method and core idea of the present application. At the same time, changes or deformations made by those skilled in the art based on the idea of the present application, the specific implementation method and the scope of application of the present application, all belong to the scope of protection of the present application. In summary, the content of this specification should not be construed as limiting the present application.
Claims
1. A method for processing a label image, characterized in that: include: Obtaining coordinate information of a target label in an image to be identified to perform image segmentation of the target label; Generate a quadrilateral bounding box of the target label after image segmentation; Planarizing the quadrilateral bounding box of the target label to generate a rectangular label image corresponding to the target label; Extract text information from the rectangular label image.
2. The method according to claim 1, characterized in that Obtaining coordinate information of a target label in an image to be identified to perform image segmentation of the target label includes: Acquire the coordinate information of the target tag through a preset tag detection model; Based on the coordinate information of the target tag, performing instance segmentation on the target tag through a preset image segmentation model; Obtain a mask image corresponding to the segmented target label.
3. The method according to claim 2, characterized in that Generate a quadrilateral bounding box of the target label after image segmentation, including: Based on the mask image, generating a polygonal convex hull contour corresponding to the target label; Optimizing the vertex positions of the polygonal convex hull contour to generate a plurality of candidate quadrilateral bounding boxes; A quadrilateral bounding box of the target tag is determined among the multiple candidate quadrilateral bounding boxes.
4. The method according to claim 3, characterized in that Generating a polygonal convex hull contour corresponding to the target label based on the mask image, including: Obtaining non-zero pixel points in the mask image; Constructing an initial convex hull contour corresponding to the target label based on the non-zero pixel points; The initial convex hull contour is subjected to boundary simplification optimization to obtain the polygonal convex hull contour.
5. The method according to claim 3, characterized in that: Determining a quadrilateral bounding box of the target label among the plurality of candidate quadrilateral bounding boxes includes: Calculating a difference loss for each of the plurality of candidate quadrilateral bounding boxes; Determining a rectangle fit degree of each of the plurality of candidate quadrilateral bounding boxes according to the difference loss; A quadrilateral bounding box of the target label is determined according to the rectangle fitting degree.
6. The method according to claim 1, characterized in that The quadrilateral bounding box of the target label is planarized to generate a rectangular label image corresponding to the target label, including: Calculating a planarized output size of a quadrilateral bounding box of the target label; constructing a perspective transformation matrix based on the planarized output size; The quadrilateral bounding box of the target label is planarized by using the perspective transformation matrix to generate the rectangular label image.
7. The method according to claim 1, characterized in that Extracting text information from the rectangular label image includes: Acquire text information in the rectangular label image through a preset character recognition model; The text information is converted into a preset output format.
8. A label image processing device, characterized in that: include: An image segmentation module is used to obtain the coordinate information of the target label in the image to be identified so as to perform image segmentation of the target label; Boundary optimization module, used to generate quadrilateral bounding boxes of target labels after image segmentation; A rectangular correction module, used for performing planarization processing on the quadrilateral bounding box of the target label to generate a rectangular label image corresponding to the target label; The information extraction module is used to extract text information from the rectangular label image.
9. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.