Intelligent waybill information extraction method and system based on multi-faceted images

Abstract

The present invention provides a method and system for intelligent waybill information extraction based on multi-faceted images, the method comprising the steps of: obtaining face images of a package at multiple angles; scoring the waybill areas contained in the face images respectively to obtain the face images with the highest scores, and obtaining a waybill picture containing the waybill area according to the face images with the highest scores; recognizing and outputting text information in the waybill picture to obtain package information. The present invention can independently complete the recognition and extraction of waybill information, while improving the recognition accuracy and efficiency of the package waybill.

Description

Technical Field

[0001] The present invention relates to the technical field of logistics industry, and in particular to an intelligent waybill information extraction method and system based on multi-faceted images. Background Art

[0002] With the rapid development of the logistics industry and the concentrated outbreak of logistics during certain holidays, the rapid and accurate extraction of package information has become crucial. Traditional barcode scanning technology relies on single-sided scanning, that is, the package needs to be placed in accordance with the specifications so that the surface of the label faces the scanning device for identification. This not only improves the efficiency of placing the package on the transmission line, but also easily leads to identification failure when the package is not placed in a standardized manner or the label is not attached in a standardized manner.

[0003] In order to improve the reliability of information extraction, the existing technology usually adopts multi-faceted scanning technology. However, how to quickly and accurately locate and recognize the text information on the face sheet in the multi-faceted image is still a problem to be solved. Summary of the invention

[0004] In order to solve the above problems in the prior art, the present invention provides an intelligent waybill information extraction method and system based on multi-surface images, which can improve the accuracy and efficiency of waybill information extraction.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] In a first aspect, the present invention provides a method for intelligent face order information extraction based on multi-faceted images, comprising the steps of:

[0007] Get the surface images wrapped at multiple angles;

[0008] Scoring the face order areas contained in the face image respectively to obtain the face image with the highest score, and obtaining the face order picture containing the face order area according to the face image with the highest score;

[0009] Identify and output the text information in the shipping label image to obtain the package information.

[0010] The beneficial effects of the present invention are: by using multi-faceted scanning technology to capture the surface images of the package at multiple angles, and to score the surface images in relation to the face sheet, the surface image with the highest score is finally selected and cropped, and only the cropped face sheet portion is subjected to OCR recognition, which greatly increases the effective data volume for recognition, improves the recognition accuracy of text information, and improves the recognition accuracy and efficiency of the package face sheet as a whole. At the same time, the present invention does not rely on barcode scanning and the docking of back-end data, and can independently complete the recognition and extraction of face sheet information.

[0011] Optionally, scoring the face order areas contained in the face image respectively to obtain the face image with the highest score comprises:

[0012] Respectively identifying and semantically segmenting the face regions in the face image;

[0013] Based on the semantic segmentation result, the face image is comprehensively scored according to the face order area ratio and the face order area regularity of the face order area;

[0014] Output and get the face image with the highest score.

[0015] Optionally, the comprehensive scoring formula is:

[0016] ;

[0017] Among them, S is the comprehensive score of the face image; A 单 A is the pixel area of ​​the single area; 全 is the total pixel area of ​​the surface image; P 单 P is the regularity score of the face sheet area, which is calculated by the degree of fit between the edge of the face sheet area and the rectangle; 全 It is the maximum regularity score of the face order area; α and β are the weight coefficients configured for the current application scenario.

[0018] According to the above description, the accuracy and efficiency of face sheet recognition are significantly improved by comprehensively scoring the area ratio and regional regularity of the face sheet.

[0019] Optionally, the acquiring of surface images of the package at multiple angles specifically includes: acquiring surface images of five surfaces of the package except the bottom surface.

[0020] Optionally, the method further comprises the steps of:

[0021] If the face sheet is not recognized in the face image, the score of the face image is 0.

[0022] Optionally, the method further comprises the steps of:

[0023] When the package label recognition fails, all surface images are spliced ​​according to their positions to form a multi-surface unfolded image of the package;

[0024] The waybill image is re-identified based on the multi-sided unfolded images.

[0025] According to the above description, when the face sheet recognition fails, by splicing the surface images and then re-recognizing them, the phenomenon of face sheet recognition failure caused by the face sheet sticking to the corners can be effectively reduced, thereby further improving the accuracy of face sheet recognition.

[0026] Optionally, when the package bill recognition fails, the process includes:

[0027] When the area ratio of the parcel in all the face images is less than the area ratio threshold, or when the regional regularity of the parcel in all the face images is less than the regional regularity threshold, or the package information does not reach the minimum information threshold.

[0028] Optionally, the identifying and outputting the text information in the waybill image specifically includes: identifying and outputting the text information in the waybill image by using an OCR waybill character extraction algorithm.

[0029] Optionally, the identifying the face order areas in the face image respectively is specifically: sending the face image to a face order recognition AI algorithm program to identify the face order areas of the face image.

[0030] In a second aspect, the present invention provides an intelligent waybill information extraction system based on multi-sided images, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the intelligent waybill information extraction method based on multi-sided images of the first aspect is implemented.

[0031] Among them, the technical effect corresponding to the intelligent waybill information extraction system based on multi-surface images provided by the second aspect refers to the relevant description of the intelligent waybill information extraction method based on multi-surface images provided by the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Schematic diagram of the main process of the intelligent face order information extraction method based on multi-faceted images according to an embodiment of the present invention;

[0033] Figure 2 Schematic diagram of the overall process of the intelligent waybill information extraction method based on multi-faceted images according to an embodiment of the present invention;

[0034] Figure 3 is a schematic diagram of a surface image involved in an embodiment of the present invention;

[0035] Figure 4 A schematic diagram of a label image according to an embodiment of the present invention;

[0036] Figure 5 Schematic diagram of the framework of an intelligent face sheet information extraction system based on multi-faceted images according to an embodiment of the present invention.

[0037] Description of reference numerals:

[0038] 1. Intelligent face sheet information extraction system based on multi-faceted images;

[0039] 2. Processor;

[0040] 3. Memory. DETAILED DESCRIPTION

[0041] In order to better understand the above technical solution, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a clearer and more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0042] Embodiment 1

[0043] This embodiment is applicable to application scenarios such as logistics, warehousing and express delivery that require package label recognition. In particular, scenarios that require fast and accurate recognition of the label. Existing multi-faceted scanning technology still has the problem of low recognition rate and slow speed. This embodiment can improve the recognition accuracy and efficiency of the package label by optimizing and cutting out the label image for text recognition. See the following description for details.

[0044] Please refer to Figure 1 to Figure 2 , an intelligent face sheet information extraction method based on multi-faceted images, comprising the steps of:

[0045] S1, obtaining surface images of the package at multiple angles;

[0046] In this embodiment, obtaining the surface images of the package at multiple angles specifically includes: obtaining the surface images of five surfaces of the package except the bottom surface.

[0047] When the package flows on the transmission line, the five-sided barcode scanning camera obtains image data of the package at five different angles, that is, the surface images of the five surfaces of the package except the bottom surface, such as Figure 3 Shown is a frontal face image.

[0048] S2, scoring the face order areas contained in the face image respectively to obtain the face image with the highest score, and obtaining the face order picture containing the face order area according to the face image with the highest score;

[0049] In this embodiment, in step S2, the face order areas contained in the face image are scored respectively, and the face image with the highest score includes:

[0050] S21, identifying and semantically segmenting the face regions in the face image respectively;

[0051] In this embodiment, the face order areas in the face image are respectively identified by sending the face order image to the face order identification AI algorithm program to identify the face order areas in the face image.

[0052] Among them, the AI ​​algorithm program for package delivery bill recognition is an algorithm program trained for package delivery bills. AI is artificial intelligence. The algorithm model used can be an R-CNN target detection model formed by combining region candidates and convolutional neural networks, or a YOLOv4 target detection model, etc. In this embodiment, existing mature technologies can be used.

[0053] Among them, semantic segmentation is an image segmentation technology whose goal is to assign each pixel in the image to a semantic category. This means that it not only classifies each pixel in the image, but also indicates the semantic category to which each pixel belongs. In this way, the pixel area belonging to the single area of ​​the face is distinguished.

[0054] S22. Based on the semantic segmentation result, comprehensively score the face image according to the face sheet area ratio and the face sheet area regularity;

[0055] In this embodiment, the comprehensive scoring formula is:

[0056] ;

[0057] Among them, S is the comprehensive score of the face image; A 单 A is the pixel area of ​​the single area; 全 is the total pixel area of ​​the surface image; P 单 P is the regularity score of the face sheet area, which is calculated by the degree of fit between the edge of the face sheet area and the rectangle; 全 It is the maximum regularity score of the face order area; α and β are the weight coefficients configured for the current application scenario.

[0058] In this embodiment, α and β are 45% and 55% respectively. At this time, the proportion of the face order area in one face image is 37%, and the regularity of the face order area is 96%. The comprehensive score of the face image is S=45%*37%+55%*96%=69.45%.

[0059] Among them, α and β are weight coefficients, which are adjusted according to the actual application scenario to balance the impact of area ratio and regional regularity on the score. For example, α is 40% and β is 60%; or α is 70% and β is 30%, and so on.

[0060] In this embodiment, if the facial form is not recognized in the facial image, there is no facial form area, and the corresponding facial form area ratio and facial form area regularity are 0, and the facial image score is 0.

[0061] S23: Output and obtain the face image with the highest score.

[0062] Thus, this embodiment obtains the face image where the face bill is most likely to be located, and obtains the face bill picture containing the face bill area according to the face image with the highest score. Figure 3 The face image shown is the face image with the highest comprehensive score. The face order image cut out from it is as follows: Figure 4 As shown, Figure 4 The white blurred area in the figure is the delivery area. It should be noted that in order to avoid the privacy leakage of package information, Figure 3 and Figure 4 The middle surface area is blurred.

[0063] It should be noted that the cropping of the face bill image can be performed when the face bill area is identified in step S21, and then the face bill image cropped from the face image with the highest score can be directly selected; or after obtaining the face image with the highest score, cropping can be performed based on the recognition result.

[0064] S3. Identify and output the text information in the label image to obtain the package information.

[0065] In this embodiment, recognizing and outputting text information in the waybill image specifically includes: recognizing and outputting text information in the waybill image by using an OCR waybill character extraction algorithm.

[0066] Among them, OCR technology is a technology that recognizes characters in an image and converts them into text, accurately identifying text information on the waybill, such as the waybill number, recipient name, etc. Since OCR recognition is only performed on the cropped part of the waybill, the effective data volume for recognition is greatly increased, the recognition accuracy of text information is improved, and the recognition accuracy of the package waybill is improved overall.

[0067] Embodiment 2

[0068] Please refer to Figure 1 to Figure 2 The intelligent method for extracting information of a multi-faceted image based on the multi-faceted image, based on the above-mentioned embodiment 1, further comprises the following steps:

[0069] S4. When the package label recognition fails, all surface images are spliced ​​according to their positions to form a multi-surface unfolded image of the package;

[0070] In this embodiment, when the package label recognition fails, it includes:

[0071] When the area ratio of all face images is less than the area ratio threshold, or when the area regularity of all face images is less than the area regularity threshold, or the package information does not reach the minimum information threshold.

[0072] That is, when the above-mentioned area ratio is too small, the area regularity is too small, or the text information is too little, it may be because the face sheet is folded on two sides, resulting in too little face sheet area and face sheet information on both sides, resulting in the face sheet recognition failure. Among them, step S4 defines a variety of face sheet recognition failure situations, which may occur in step S3 or in step S2. Therefore, in this embodiment, step S4 can be executed when the area ratio is too small and the area regularity is too small in step S2, or step S4 can be executed after the text information is recognized in step S3. On the traditional transmission line, when the face sheet recognition of the package fails, it will return to the transmission line or be manually reviewed. When it returns to the transmission line, it needs to be recognized by the face sheet again, which not only reduces the recognition efficiency of the package, but also the failure rate of the secondary recognition is still very high. However, this embodiment can reduce the failure caused by the multi-sided pasting of the face sheet during recognition, and significantly reduce the situation where the package returns to the transmission line for secondary recognition, so it can further improve the accuracy of face sheet recognition.

[0073] Wherein, step S4 can be replaced by:

[0074] When recognition of the package label fails, surface images having a label area and the label area being located at least on an edge of the package are spliced ​​according to their locations to form a multi-surface unfolded image of the package.

[0075] That is, for the surface images that need to be spliced, it is necessary not only to consider whether there is a label area, but also to consider that the label area is at least located at one edge of the package to ensure that the above-mentioned spliced ​​labels are folded on adjacent surfaces. In some cases, it is avoided that advertising leaflets and flyers on different surfaces are mistakenly integrated into a "large label" when unfolded, resulting in recognition errors.

[0076] S5. Re-identify the face sheet image based on the multi-faceted unfolded images.

[0077] Among them, the technology used in step S5 to re-identify the face sheet image from the multi-faceted expanded image is the same as that used in step 2. The multi-faceted expanded image also participates in the scoring as the sixth "face image". If it has the highest score, the face sheet image is cropped to perform text information recognition in step S3, otherwise step S3 will not be executed.

[0078] In summary, this embodiment improves the recognition accuracy and efficiency of the package bill by optimizing and cutting out the bill image for text recognition, and performing secondary recognition by unfolding the image on multiple sides when recognition fails. At the same time, the present invention does not rely on barcode scanning and the docking of back-end data, and can independently complete the recognition and extraction of bill information.

[0079] Embodiment 3

[0080] Please refer to Figure 5 The intelligent face-bill information extraction system 1 based on multi-faceted images includes a memory 3, a processor 2, and a computer program stored in the memory 3 and executable on the processor 2. When the processor 2 executes the computer program, the steps in the above-mentioned embodiment 1 or 2 are implemented.

[0081] Since the system / device described in the above embodiments of the present invention is a system / device used to implement the method of the above embodiments of the present invention, a person skilled in the art can understand the specific structure and deformation of the system / device based on the method described in the above embodiments of the present invention, and thus will not be described in detail here. All systems / devices used in the method of the above embodiments of the present invention belong to the scope of protection of the present invention.

[0082] It should be understood by those skilled in the art that embodiments of the present invention may be provided as methods, apparatuses or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0083] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (apparatus) and computer program products according to embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions.

[0084] It should be noted that in the claims, any reference numerals placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention may be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In the claims enumerating several means, several of these means may be embodied by the same hardware. The use of the words first, second, third, etc., is for convenience of expression only and does not indicate any order. These words may be understood as part of the component name.

[0085] In addition, it should be noted that, in the description of this specification, the description of the terms "one embodiment", "some embodiments", "embodiment", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0086] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments after knowing the basic creative concept. Therefore, the claims should be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present invention.

[0087] Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention should also include these modifications and variations.

Claims

1. An intelligent face sheet information extraction method based on multi-faceted images, characterized in that: Includes steps: Get the surface images wrapped at multiple angles; Scoring the face order areas contained in the face image respectively to obtain the face image with the highest score, and obtaining the face order picture containing the face order area according to the face image with the highest score; Recognize and output the text information in the shipping label image to obtain package information; The scoring of the face order areas contained in the face image respectively to obtain the face image with the highest score comprises: Respectively identifying and semantically segmenting the face regions in the face image; Based on the semantic segmentation result, comprehensively scoring the face image according to the face order area ratio and the face order area regularity of the face order area; Output and obtain the face image with the highest score; The formula for the comprehensive score is: ； Among them, S is the comprehensive score of the face image; A 单 A is the pixel area of ​​the single area; 全 is the total pixel area of ​​the surface image; P 单 P is the regularity score of the face sheet area, which is calculated by the degree of fit between the edge of the face sheet area and the rectangle; 全 The maximum regularity score of the face order area; α and β are the weight coefficients configured for the current application scenario; The obtaining of the surface images of the package at multiple angles specifically includes: obtaining the surface images of five surfaces of the package except the bottom surface.

2. The intelligent face order information extraction method based on multi-faceted images according to claim 1 is characterized in that: Also includes the steps: If the face sheet is not recognized in the face image, the score of the face image is 0.

3. The intelligent face order information extraction method based on multi-faceted images according to claim 1 or 2, characterized in that: Also includes the steps: When the package label recognition fails, all surface images are spliced ​​according to their positions to form a multi-surface unfolded image of the package; The waybill image is re-identified based on the multi-sided unfolded images.

4. The intelligent face order information extraction method based on multi-faceted images according to claim 3 is characterized in that: When the package label recognition fails, the process includes: When the area ratio of the parcel in all the face images is less than the area ratio threshold, or when the regional regularity of the parcel in all the face images is less than the regional regularity threshold, or the package information does not reach the minimum information threshold.

5. The intelligent face order information extraction method based on multi-faceted images according to claim 1 or 2, characterized in that: The identifying and outputting the text information in the waybill image specifically includes: identifying and outputting the text information in the waybill image by using an OCR waybill character extraction algorithm.

6. The intelligent face order information extraction method based on multi-faceted images according to claim 1 is characterized in that: The identifying of the face order areas in the face image respectively is specifically as follows: sending the face order image to a face order recognition AI algorithm program to identify the face order areas of the face image.

7. An intelligent face-bill information extraction system based on multi-faceted images, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for intelligent face sheet information extraction based on multi-faceted images as described in any one of claims 1 to 6 is implemented.

Images