Express bill information extraction method, system and related device

Abstract

The invention discloses an express sheet information extraction method and system, and a related device. The method comprises the steps of obtaining a to-be-recognized image; wherein the to-be-identified image comprises express parcels, and each express parcel corresponds to an express sheet; based on the to-be-recognized image, obtaining an express sheet, and determining a category label corresponding to the express sheet; wherein the category label is matched with the integrity and definition of the express sheet; identifying the express waybill to obtain a plurality of characters and confidence degrees corresponding to the characters, determining to-be-confirmed characters on the express waybill and positions corresponding to the to-be-confirmed characters on the express waybill based on the category labels, the characters and the confidence degrees corresponding to the characters, and generating a predicted waybill number of the express waybill by utilizing the identified characters and candidate characters corresponding to the to-be-confirmed characters; and obtaining a target bill number corresponding to the express bill based on the predicted bill number and a pre-stored bill number in a database. Through the method, the accuracy of express information extraction can be improved.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, and in particular to a method, system and related apparatus for extracting information from express delivery waybills. Background Technology

[0002] The application of perception algorithms and deep learning technologies in express delivery sorting has significantly reduced labor costs and improved sorting efficiency. However, the mainstream approach to extracting express delivery information using machines still relies on extracting information from the QR codes or barcodes on the packages. This method is highly dependent on the integrity of the QR codes or barcodes; incomplete codes lead to a decrease in the accuracy of information extraction, resulting in numerous cases of missorted packages. Therefore, improving the accuracy of express delivery information extraction has become an urgent problem to be solved. Summary of the Invention

[0003] The main technical problem addressed by this application is to provide a method, system, and related apparatus for extracting express delivery waybill information, which can improve the accuracy of express delivery information extraction.

[0004] To address the aforementioned technical problems, this application provides a method for extracting express delivery waybill information, comprising: acquiring an image to be identified; wherein the image to be identified includes express parcels, and each express parcel corresponds to an express delivery waybill; obtaining the express delivery waybill based on the image to be identified, and determining a category label corresponding to the express delivery waybill; wherein the category label matches the completeness and clarity of the express delivery waybill; identifying multiple characters and their corresponding confidence scores on the express delivery waybill; determining the characters to be confirmed and their corresponding positions on the express delivery waybill based on the category label and the characters and their corresponding confidence scores; generating a predicted tracking number for the express delivery waybill using the identified characters and candidate characters corresponding to the characters to be confirmed; and obtaining a target tracking number corresponding to the express delivery waybill based on the predicted tracking number and pre-stored tracking numbers in a database.

[0005] To address the aforementioned technical problems, a second aspect of this application provides a system for extracting express delivery waybill information. This system includes an acquisition module, a determination module, an identification module, and a matching module. The acquisition module acquires an image to be identified, wherein the image includes express parcels, and each parcel corresponds to an express delivery waybill. The determination module obtains the express delivery waybill based on the image to be identified and determines the category label corresponding to the waybill. The category label matches the completeness and clarity of the waybill. The identification module identifies multiple characters and their corresponding confidence levels on the waybill. Based on the category label and the characters and their corresponding confidence levels, it determines the character to be confirmed on the waybill and its corresponding position. Using the identified characters and candidate characters corresponding to the character to be confirmed, it generates a predicted tracking number for the waybill. The matching module obtains the target tracking number corresponding to the waybill based on the predicted tracking number and pre-stored tracking numbers in a database.

[0006] To address the aforementioned technical problems, a third aspect of this application provides an electronic device comprising: a memory and a processor coupled to each other, wherein the memory stores program data, and the processor invokes the program data to execute the method described in the first aspect.

[0007] To address the aforementioned technical problems, a fourth aspect of this application provides a computer-readable storage medium storing program data thereon, wherein the program data, when executed by a processor, implements the method described in the first aspect.

[0008] The beneficial effects of this application are as follows: Unlike existing technologies, this method acquires images to be identified, each image containing a courier package, and each package corresponding to a waybill. Based on the images, the courier packages are processed to obtain the waybills. The completeness and clarity of the waybills determine their corresponding category labels. The waybills are then processed to obtain multiple characters and their corresponding confidence scores. Based on the category labels, characters, and their confidence scores, the characters to be confirmed on the waybill and their corresponding positions are determined. Candidate characters corresponding to the characters to be confirmed are obtained. Using the identified characters and the candidate characters, a predicted tracking number for the waybill is generated. Based on the predicted tracking number and pre-stored tracking numbers in the database, the target tracking number is obtained. This method adds a text information extraction process to the information extraction process. Furthermore, by combining category labels, characters, their corresponding confidence scores, and position markers during text information extraction, the impact of incomplete or blurry waybills on courier information extraction is reduced, thereby improving the accuracy of courier information extraction. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a flowchart illustrating one implementation method of the express waybill information extraction method of this application; Figure 2 This is a flowchart illustrating another implementation of the express waybill information extraction method of this application; Figure 3 This is a schematic diagram of one embodiment of the express waybill information extraction system of this application; Figure 4 This is a schematic diagram of the structure of one embodiment of the electronic device of this application; Figure 5 This is a schematic diagram of one embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0010] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments, and different implementation methods can be adaptively combined. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0011] In this paper, the terms "system" and "network" are often used interchangeably. The term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this paper means two or more.

[0012] The express delivery waybill information extraction method provided in this application is used to extract the express delivery tracking number of express parcels. The corresponding execution entity is a processing unit capable of data processing. This processing unit is integrated into the smart terminal or exists independently of the smart terminal and interacts with the smart terminal for data.

[0013] Please see Figure 1 , Figure 1 This is a flowchart illustrating one implementation method of the express waybill information extraction method of this application. The method includes: S101: Obtain the image to be recognized; wherein, the image to be recognized includes express parcels, and each express parcel corresponds to an express waybill.

[0014] Specifically, images to be identified are obtained, each image including a courier package, and each courier package has a corresponding waybill.

[0015] In one embodiment, a video stream matching the image to be identified is acquired, a specific frame is extracted from the video stream and saved, the specific frame contains express parcels, and each express parcel has a corresponding express waybill, and the specific frame is used as the image to be identified.

[0016] In one embodiment, when a courier package and its corresponding waybill are identified, the corresponding image is captured in real time and saved as the image to be identified.

[0017] In some implementation scenarios, a video stream matching the image to be identified is acquired, and a video processing library is used to read the video stream frame by frame. Specific frames are extracted from the video stream and saved. The specific frame contains a complete express package and has a corresponding express waybill. The specific frame is used as the image to be identified in subsequent processing.

[0018] In some implementation scenarios, a camera device is used to detect express parcels. When the express parcel and the corresponding waybill on the parcel are identified, the camera device captures and saves the corresponding image in real time as the image to be identified in subsequent processing.

[0019] Optionally, an image to be identified may contain one or more express delivery packages. This application does not impose a specific limit on the number of express delivery packages in the image to be identified.

[0020] S102: Based on the image to be identified, obtain the express delivery waybill and determine the category label corresponding to the express delivery waybill; wherein, the category label is matched with the completeness and clarity of the express delivery waybill.

[0021] Specifically, based on the image to be identified, the express parcel in the image is processed to obtain the express waybill, and the category label corresponding to the express waybill is determined according to the completeness and clarity of the express waybill.

[0022] In one embodiment, based on the image to be identified, a segmentation network is used to process the express parcel in the image to obtain the corresponding express waybill, and the category label corresponding to the express waybill is determined according to the completeness and clarity of the express waybill.

[0023] In one embodiment, based on the image to be identified, edge detection algorithms and shape analysis methods are used to process the express parcels in the image to be identified, obtain the corresponding express waybill, and determine the category label corresponding to the express waybill based on the completeness and clarity of the express waybill.

[0024] It should be noted that due to factors such as transportation, packaging, and the environment, express parcels often have stains and wrinkles. Therefore, the completeness and clarity of the express waybill will vary, such as complete and clear, complete but blurry, missing but clear, missing and blurry, etc. Correspondingly, different category labels will be assigned to the express waybill based on these conditions.

[0025] In some implementation scenarios, the segmentation network includes a visual feature extraction module, a feature fusion module, and an output module. The visual feature extraction module is used to extract features from the image to be recognized, obtaining multiple visual features at different scales. The feature fusion module is used to fuse multiple visual features at different scales to obtain a fused feature. The output module is used to analyze and process the fused feature to obtain the corresponding express delivery waybill, and at the same time, classify it according to the status of the express delivery waybill to determine the corresponding category label.

[0026] In some implementation scenarios, edge detection methods are used to calculate the gradient magnitude and gradient direction of pixels in the image to be identified in order to determine the edges of the express delivery waybill. Then, shape analysis methods are used to analyze the contour shape of the edges of the express delivery waybill to determine the position of the express delivery waybill, thereby obtaining the corresponding express delivery waybill. At the same time, the waybill is classified according to its status to determine the corresponding category label.

[0027] S103: Recognize multiple characters and their corresponding confidence scores on the express delivery waybill. Based on the category label and the characters and their corresponding confidence scores, determine the characters to be confirmed on the express delivery waybill and their corresponding positions. Use the recognized characters and the candidate characters corresponding to the characters to be confirmed to generate the predicted tracking number of the express delivery waybill.

[0028] Specifically, the express delivery waybill is identified to obtain multiple characters and their corresponding confidence scores. Based on the category labels and the characters and their corresponding confidence scores, the characters to be confirmed on the express delivery waybill and their corresponding positions are determined. Candidate characters corresponding to the characters to be confirmed are obtained. Using the identified characters and the candidate characters corresponding to the characters to be confirmed, the predicted tracking number of the express delivery waybill is generated.

[0029] Understandably, the confidence level corresponding to a character is used to determine the credibility of the character.

[0030] In one embodiment, optical character recognition (OCR) technology is used to identify the express delivery waybill, obtaining multiple characters and their corresponding confidence levels. Characters with confidence levels greater than or equal to a preset confidence threshold are identified as identified characters, while characters with confidence levels less than the preset confidence threshold are identified as characters to be confirmed. Based on the corresponding category labels, it is determined whether there are missing or blurred characters among the multiple characters. Characters with missing or blurred characters are identified as characters to be confirmed. The corresponding positions are determined according to the waybill features and the order of the multiple characters, and candidate characters corresponding to the characters to be confirmed are obtained. Finally, using the identified characters and the candidate characters corresponding to the characters to be confirmed, a predicted tracking number for the express delivery waybill is generated.

[0031] In one embodiment, optical character recognition (OCR) technology is used to identify the express delivery waybill, obtaining multiple characters and their corresponding confidence levels. A matching dynamic confidence threshold is determined based on the category label. For example, when the category label indicates that the express delivery waybill is complete and clear, the corresponding dynamic confidence threshold is higher and can be set to 90%. When the category label indicates that the express delivery waybill is missing and blurry, the corresponding dynamic confidence threshold is lower and can be set to 70%. Characters with a confidence level greater than or equal to the dynamic confidence threshold matching the category label are identified as identified characters, and characters with a confidence level less than the dynamic confidence threshold matching the category label are identified as characters to be confirmed. Based on the corresponding category label, it is determined whether there are missing or blurry characters among the multiple characters. Characters with missing or blurry characters are identified as characters to be confirmed. According to the characteristics of the express delivery waybill and using a pre-stored express delivery waybill template, the position corresponding to the character to be confirmed is obtained, and candidate characters corresponding to the character to be confirmed are obtained. Using the identified characters and the candidate characters corresponding to the character to be confirmed, a predicted tracking number for the express delivery waybill is generated.

[0032] S104: Based on the predicted tracking number and the pre-stored tracking number in the database, obtain the target tracking number corresponding to the express waybill.

[0033] Specifically, based on the predicted tracking number and the pre-stored tracking numbers of express waybills in the database, the target tracking number corresponding to the express waybill is obtained.

[0034] In one implementation, the predicted tracking number is directly input into the database as a query condition and matched with the pre-stored tracking number. If a pre-stored tracking number in the database is exactly the same as the predicted tracking number, the query result of the pre-stored tracking number is returned, and the pre-stored tracking number is used as the target tracking number corresponding to the express waybill.

[0035] In one embodiment, the predicted tracking number is input into the database, the similarity between the predicted tracking number and the pre-stored tracking number in the database is calculated, the pre-stored tracking number with the highest similarity value is returned, and this pre-stored tracking number is used as the target tracking number corresponding to the express waybill.

[0036] The above scheme acquires images to be identified, each image containing a courier package, and each package has a corresponding waybill. Based on the images, the courier packages are processed to obtain the waybills. The completeness and clarity of the waybills determine the corresponding category labels. The waybills are then processed to obtain multiple characters and their corresponding confidence scores. Based on the category labels, characters, and their corresponding confidence scores, the characters to be confirmed and their corresponding positions on the waybills are determined. Candidate characters corresponding to the characters to be confirmed are obtained. Using the identified characters and the candidate characters corresponding to the characters to be confirmed, a predicted tracking number for the waybill is generated. Based on the predicted tracking number and the pre-stored tracking numbers in the database, the target tracking number for the waybill is obtained. This approach adds a text information extraction process to the information extraction process, and combines category labels, characters, their corresponding confidence scores, and position markers during text information extraction. This reduces the impact of incomplete or blurry waybills on the extraction of courier information, thereby improving the accuracy of courier information extraction.

[0037] Please see Figure 2 , Figure 2 This is a flowchart illustrating another embodiment of the express delivery waybill information extraction method of this application. The method includes: S201: Obtain the image to be recognized; wherein, the image to be recognized includes express parcels, and each express parcel corresponds to an express waybill.

[0038] Specifically, images to be identified are obtained, each image including a courier package, and each courier package has a corresponding waybill.

[0039] S202: Extract features from the image to be recognized to obtain multi-scale first visual features, and fuse the multi-scale first visual features to obtain the first fused features.

[0040] Specifically, feature extraction is performed on the image to be recognized to obtain multiple first visual features at different scales, thereby enriching the semantic content of the image. Feature fusion is then performed on the multiple first visual features at different scales to obtain the first fused feature.

[0041] In one implementation scenario, a lightweight backbone extracts visual features layer by layer from the image to be recognized, outputting multiple first visual features at different scales at different network stages. The first visual features at multiple scales are then fused through dynamic weight adjustment and interaction to obtain the first fused feature.

[0042] In one implementation scenario, after the encoder extracts multiple first visual features of different scales from the image to be recognized layer by layer through downsampling, the decoder fuses the multiple first visual features of different scales extracted by the encoder to obtain the first fused feature.

[0043] S203: Use the first fusion feature to perform image segmentation, obtain the express delivery waybill within the segmented area, and determine the category label corresponding to the express delivery waybill based on the features within the segmented area.

[0044] Specifically, the image is segmented using the first fusion feature to obtain the express delivery waybill within the segmented area, and the completeness and clarity of the express delivery waybill are determined based on the features within the segmented area, thereby obtaining the category label corresponding to the express delivery waybill.

[0045] In one implementation scenario, the bounding box of the express delivery waybill is predicted using the first fusion feature, generating multiple candidate regions. Overlapping or low-confidence candidate regions are filtered out, and the candidate region with the highest confidence is retained as the final express delivery waybill region. The completeness and clarity are scored according to the features within the final express delivery waybill region to obtain corresponding score values. The category label corresponding to the express delivery waybill is determined based on their respective score values. For example, if the completeness score value is higher than a preset value and the clarity score value is also higher than a preset value, then the category label corresponding to the express delivery waybill is complete and clear, and so on.

[0046] In one implementation scenario, multiple training samples are acquired. Image segments containing the express delivery waybill area are selected as positive samples, and background segments not containing the express delivery waybill area are selected as negative samples. The first fused feature is input into the contrastive learning module. By calculating feature similarity, the feature distance of positive samples is reduced, and the feature distance of negative samples is reduced. The loss function is used for optimization and adjustment. Based on the feature similarity score after training, the binarization threshold is dynamically adjusted to generate the final segmentation mask. For example, 0 represents the background area and 1 represents the waybill area. Then, through morphological processing, such as performing closing and opening operations on the mask, the contrastive learning similarity score and segmentation confidence are combined to obtain the final express delivery waybill area. Based on the final express delivery waybill area, the probability value of each category label is predicted. For example, [0.7, 0.1, 0.1, 0.1] can represent a probability of 70% for complete and clear, 10% for complete but blurry, 10% for missing but clear, and 10% for missing and blurry. At this time, the category label corresponding to the express delivery waybill is complete and clear.

[0047] In a specific implementation scenario, a segmentation network is used to process the image to be recognized, obtaining the express delivery waybill and its corresponding category label. The segmentation network includes a first visual feature extraction module, a first feature fusion module, and a first output module. The first visual feature extraction module extracts features from the image to be recognized, obtaining multiple first visual features at different scales. The first feature fusion module fuses the multiple first visual features at different scales to obtain the first fused features. In order to extract more detailed information, the original upsampling layer in the output module is replaced with a deconvolution layer. The first output module then analyzes and processes the first fused features to segment the express delivery waybill from the express package. Since the express delivery waybill has different levels of completeness and clarity during the sorting process, the output module also includes a classifier. Thus, while segmenting the waybill, the classifier can classify the express delivery waybill according to its corresponding state to obtain the corresponding category label, and these category labels are used to correspond to different subsequent processing flows.

[0048] Optionally, in this implementation scenario, a pre-trained YOLO v8-seg network can be used as the instance segmentation network. In other implementation scenarios, other types of segmentation networks can also be used for waybill segmentation. This application does not impose any specific restrictions here.

[0049] In a specific implementation scenario, the image to be identified is processed by an image processing algorithm to extract visual features from the image at different granularities, resulting in multiple first visual features at different scales. These first visual features at different scales are then normalized and weighted to generate first fusion features. Next, a pre-trained machine learning model is used to match preset rules or training labels based on the first fusion features to determine the location and state category of the waybill area, thereby obtaining the express waybill and its corresponding category label. These category labels are then used to correspond to different subsequent processing flows.

[0050] S204: Identify multiple characters and their corresponding confidence scores on the express delivery waybill. Based on the category label and the characters and their corresponding confidence scores, determine the characters to be confirmed on the express delivery waybill and their corresponding positions. Using the identified characters and the candidate characters corresponding to the characters to be confirmed, generate the predicted tracking number of the express delivery waybill.

[0051] Specifically, the express delivery waybill is identified to obtain multiple characters and their corresponding confidence scores. Based on the category labels and the characters and their corresponding confidence scores, the characters to be confirmed on the express delivery waybill and their corresponding positions are determined. Candidate characters corresponding to the characters to be confirmed are obtained. Using the identified characters and the candidate characters corresponding to the characters to be confirmed, the predicted tracking number of the express delivery waybill is generated.

[0052] In one embodiment, step S204 specifically includes: acquiring the waybill features of the express delivery waybill; obtaining multiple characters and their corresponding confidence scores based on the waybill features; acquiring characters whose confidence scores meet a preset confidence threshold as identification characters, and identifying characters whose confidence scores do not meet the confidence threshold as characters to be confirmed; in response to the category label indicating that the express delivery waybill is blurred or missing, identifying blurred or missing characters on the express delivery waybill as characters to be confirmed; determining the position corresponding to the character to be confirmed based on the waybill features and the sorting of multiple characters; and generating a predicted tracking number based on the identification characters and the candidate characters corresponding to the character to be confirmed at the corresponding position.

[0053] Specifically, the process involves acquiring the features of the express delivery waybill, obtaining multiple characters and their corresponding confidence levels based on these features, and identifying characters whose confidence levels meet a preset confidence threshold as the recognized characters. Characters with high confidence levels are considered correctly identified, while those whose confidence levels do not meet the threshold are considered characters to be identified. When the category label indicates that the express delivery waybill is blurred or missing, the blurred or missing characters on the waybill are considered characters to be identified. Based on the waybill features and the order of multiple characters, the position of the character to be identified is determined. Then, the identified character and the candidate characters corresponding to the character to be identified at the corresponding position are combined to obtain the predicted tracking number. On the one hand, directly identifying high-confidence characters as correctly identified characters allows for quick locking of reliable information and avoids repeated verification of characters with a high probability of being correct, thus ensuring the accuracy of the identification results. On the other hand, for low-confidence, blurred, or missing characters to be identified, by locating their positions and combining them with candidate characters, reasonable combinations can be selected based on tracking number rules, reducing tracking number errors caused by character blurring or recognition deviations, thereby improving the accuracy of express delivery tracking number identification.

[0054] In a specific implementation scenario, the tracking number on a courier waybill is a 13-digit consecutive number. Therefore, during the extraction of courier information, it is first necessary to obtain the number of characters in the text information. When the number of characters meets the preset threshold condition, such as 12 characters, the credibility of the characters is judged based on the confidence level of recognition, and characters with high confidence are used as recognition characters. For example, if the number of characters obtained is 13, and 12 of them have high confidence, then the 12 characters with high confidence are used as recognition characters. For the remaining character, if the confidence level of the character is low, then the character is determined as a character to be confirmed. Alternatively, if the category label on the courier waybill is ambiguous, the ambiguous character on the courier waybill is determined as a character to be confirmed, and a specific character mark is given to determine the position corresponding to the character to be confirmed. Alternatively, if 12 characters are obtained, and 12 of them have high confidence, then these 12 characters with high confidence are used as the identification characters. For the remaining character, the corresponding category label on the waybill is missing. For example, if the character is missing at a certain position in the middle due to wrinkles on the waybill, then the approximate position of the missing character can be obtained based on the characteristics of the waybill, such as the significant difference between the gray value corresponding to the wrinkled area and the gray value corresponding to the normal area, combined with the rotation angle of the waybill information (e.g., after the waybill information is turned upright, there may be a tilt starting from the angle corresponding to the wrinkled area), and the order of multiple characters. The missing character on the waybill can then be identified as the character to be confirmed, and a specific character label is given to determine the position corresponding to the character to be confirmed.

[0055] It should be noted that the recognition of fuzzy characters will only be numbers, and will not include other types of characters.

[0056] Understandably, when too many characters are missing, it means that the waybill information is unusable and the tracking number cannot be recognized. In this case, manual intervention is required to sort the express delivery.

[0057] Furthermore, for characters to be confirmed that have low confidence, are ambiguous, or are missing, the recognition module will provide multiple possible candidate characters, thereby enabling the prediction of the order number by combining the recognized character and the candidate characters.

[0058] In one implementation scenario, the steps of obtaining the waybill features of an express delivery waybill and obtaining multiple characters and their corresponding confidence scores based on the waybill features specifically include: extracting features from the express delivery waybill to obtain multi-scale second visual features; fusing the multi-scale second visual features to obtain second fused features; obtaining the waybill information and rotation angle based on the second fused features; and reversing and recognizing the waybill information based on the rotation angle to obtain multiple characters and their corresponding confidence scores.

[0059] Specifically, features are extracted from the express delivery waybill to obtain multiple second visual features at different scales, thereby enriching the semantic content of the express delivery waybill. The multiple second visual features at different scales are then fused to obtain a second fused feature.

[0060] In one implementation scenario, visual features are extracted layer by layer from express delivery waybills through a lightweight backbone. Multiple second visual features of different scales are output at different network stages. The second visual features of different scales are fused through dynamic weight adjustment and interaction to obtain the second fused feature.

[0061] In one implementation scenario, after the encoder extracts multiple second visual features of different scales from the express waybill layer by layer through downsampling, the decoder fuses the multiple second visual features of different scales extracted by the encoder to obtain the second fused feature.

[0062] Furthermore, based on the second fusion feature, the information included in the express delivery waybill and the position of the express delivery waybill are analyzed to obtain the waybill information and the rotation angle of the express delivery waybill.

[0063] In one implementation scenario, the text region on the express delivery waybill is detected based on the second fusion feature, the bounding box corresponding to the text region is predicted, and the spatial relationship between the bounding box and the express delivery waybill as a whole is obtained, thereby obtaining the waybill information and the rotation angle of the express delivery waybill.

[0064] In one implementation scenario, based on the global context information in the second fusion feature, the association between the text region on the express delivery waybill and the waybill as a whole is established through a self-attention mechanism, and the distribution pattern of the text region is inferred through the attention weight distribution, thereby predicting the waybill information and the rotation angle of the express delivery waybill through the distribution pattern.

[0065] Furthermore, the waybill information is rotated based on the rotation angle, so that the orientation of the waybill information becomes the correct orientation. This helps to reduce misjudgments and omissions in text recognition, improve the accuracy of recognition, and recognize the waybill information after the angle is corrected to obtain multiple characters and their corresponding confidence scores.

[0066] In one implementation scenario, after correcting the orientation of the waybill information, the text in the waybill information is divided into lines, and finally the text is recognized line by line to obtain the multiple characters to be extracted and their corresponding confidence scores.

[0067] In a specific implementation scenario, an information recognition network is used to process express delivery waybills to obtain multiple characters and their corresponding confidence scores. The information recognition network includes a second visual feature extraction module, a second feature fusion module, a second output module, and a recognition module. The second visual feature extraction module extracts features from the express delivery waybill to obtain multiple second visual features at different scales. To extract finer-grained and more subtle visual information from the waybill, the number of layers in the second visual feature extraction module is increased. The second feature fusion module fuses the multiple second visual features at different scales to obtain second fused features. Furthermore, to extract subtle and high-density text information in the waybill space, the original upsampling layer in the output module is replaced with a deconvolution layer. The second output module then analyzes and processes the second fused features to obtain the waybill information and rotation angle. The recognition module then uses the rotation angle to orthogonalize and recognize the waybill information, obtaining multiple characters and their corresponding confidence scores.

[0068] Optionally, in this implementation scenario, a pre-trained YOLO v8-obb ​​network and an optical character recognition module can be used to form an information recognition network. In other implementation scenarios, other types of information recognition networks can also be used for information extraction and recognition operations. This application does not impose any specific restrictions here.

[0069] S205: Based on the predicted tracking number and the pre-stored tracking number in the database, obtain the target tracking number corresponding to the express waybill.

[0070] Specifically, based on the predicted tracking number and the pre-stored tracking numbers of express waybills in the database, the target tracking number corresponding to the express waybill is obtained.

[0071] In one implementation scenario, step S205 specifically includes: matching the predicted tracking number with the pre-stored tracking numbers in the database; in response to a successful match, obtaining the first tracking number corresponding to the express waybill, and using the first tracking number as the target tracking number; in response to a failed match, obtaining at least one second tracking number from the database that meets the similarity condition with the predicted tracking number that failed to match, and determining the target tracking number corresponding to the express waybill based on all the second tracking numbers.

[0072] Specifically, the predicted tracking number is matched with pre-stored tracking numbers in the database. When a match is successful, the first tracking number corresponding to the express waybill is obtained, and this first tracking number is used as the target tracking number for the express waybill. When a match fails, at least one second tracking number that meets the similarity condition with the predicted tracking number that failed to match is retrieved from the database. Based on all the second tracking numbers, the target tracking number corresponding to the express waybill is determined. For example, the second tracking number with the highest similarity is used as the target tracking number, and a special mark is given to the express package corresponding to the second tracking number for subsequent filtering and review.

[0073] In one implementation scenario, the characters also include address information. The steps to determine the target tracking number corresponding to the express waybill based on all the second tracking numbers specifically include: obtaining the address information corresponding to each second tracking number; matching the address information of the second tracking number with the pre-stored addresses in the database; and using the successfully matched second tracking number as the target tracking number.

[0074] Specifically, when a match fails, the system retrieves a second tracking number from the database that meets the similarity criteria to the predicted tracking number that failed to match. For example, it retrieves the top 3 most similar second tracking numbers from the database, confirms the address information of these 3 second tracking numbers, and matches these 3 address information with the pre-stored addresses in the database. The successfully matched second tracking number is used as the target tracking number, and a special mark is given to the express package corresponding to the second tracking number for subsequent filtering and review. By using address information to assist in determining the tracking number corresponding to the express waybill, the accuracy of tracking number identification can be further improved.

[0075] It should be noted that when decoding and recognizing address information, the characters obtained will only be Chinese characters, numbers, or letters; no other types of characters will appear during the decoding process.

[0076] Optionally, the express delivery waybill and its corresponding category label are obtained based on a segmentation network, the predicted tracking number of the express delivery waybill is obtained based on an information recognition network, and the target tracking number corresponding to the express delivery waybill is obtained based on a matching network. The segmentation network, the information recognition network, and the matching network constitute an information extraction model.

[0077] Specifically, the target tracking number is obtained based on an end-to-end information extraction model, which includes a segmentation network, an information recognition network, and a matching network. The express waybill and its corresponding category label are obtained based on the segmentation network, the predicted tracking number of the express waybill is obtained based on the information recognition network, and the target tracking number corresponding to the express waybill is obtained based on the matching network, thereby achieving efficient and accurate end-to-end output of the target tracking number corresponding to the express waybill.

[0078] Please see Figure 3 , Figure 3This is a schematic diagram of one embodiment of the express delivery waybill information extraction system of this application. The express delivery waybill information extraction system 30 includes an acquisition module 301, a determination module 302, an identification module 303, and a matching module 304. The acquisition module 301 is used to acquire an image to be identified, wherein the image to be identified includes express parcels, and each express parcel corresponds to an express delivery waybill. The determination module 302 is used to obtain the express delivery waybill based on the image to be identified and determine the category label corresponding to the express delivery waybill. The category label is matched with the completeness and clarity of the express delivery waybill. The identification module 303 is used to identify multiple characters and their corresponding confidence scores on the express delivery waybill. Based on the category label and the characters and their corresponding confidence scores, it determines the characters to be confirmed on the express delivery waybill and their corresponding positions. Using the identified characters and the candidate characters corresponding to the characters to be confirmed, it generates a predicted tracking number for the express delivery waybill. The matching module 304 is used to obtain the target tracking number corresponding to the express delivery waybill based on the predicted tracking number and the pre-stored tracking numbers in the database.

[0079] The above scheme involves the following steps: The acquisition module 301 acquires images to be recognized, each image containing a courier package and a corresponding waybill. The determination module 302 processes the courier packages in the images to obtain the waybills and determines the category label based on the completeness and clarity of the waybills. The recognition module 303 performs recognition processing on the waybills to obtain multiple characters and their corresponding confidence scores. Based on the category labels and the characters and their corresponding confidence scores, it determines the characters to be confirmed on the waybill and their corresponding positions, obtains candidate characters corresponding to the characters to be confirmed, and generates a predicted tracking number for the waybill using the recognized characters and the candidate characters corresponding to the characters to be confirmed. The matching module 304 obtains the target tracking number corresponding to the waybill based on the predicted tracking number and the pre-stored tracking numbers in the database. This approach adds a text information extraction process to the information extraction process and combines category labels, characters, their corresponding confidence scores, and position markers during text information extraction. This reduces the impact of incomplete or blurry waybills on courier information extraction, thereby improving the accuracy of courier information extraction.

[0080] In one embodiment, the determining module 302 is further configured to extract features from the image to be identified, obtain multi-scale first visual features, perform feature fusion on the multi-scale first visual features to obtain first fused features, use the first fused features to perform image segmentation, obtain express delivery slips within the segmented area, and determine the category label corresponding to the express delivery slip based on the features within the segmented area.

[0081] In one embodiment, the identification module 303 is further configured to acquire the waybill features of the express delivery waybill, obtain multiple characters and their corresponding confidence scores based on the waybill features; acquire characters whose confidence scores meet a preset confidence threshold as identification characters, and acquire characters whose confidence scores do not meet the confidence threshold as characters to be confirmed; in response to the category label indicating that the express delivery waybill is blurred or missing, acquire blurred or missing characters on the express delivery waybill as characters to be confirmed; determine the position corresponding to the character to be confirmed based on the waybill features and the sorting of multiple characters; and generate a predicted tracking number based on the identification characters and the candidate characters corresponding to the character to be confirmed at the corresponding position.

[0082] In one embodiment, the recognition module 303 is further configured to extract features from the express delivery waybill to obtain multi-scale second visual features, fuse the multi-scale second visual features to obtain second fused features, obtain the waybill information and rotation angle based on the second fused features, and correct and recognize the waybill information based on the rotation angle to obtain multiple characters and their corresponding confidence scores.

[0083] In one embodiment, the matching module 304 is further configured to match the predicted tracking number with the pre-stored tracking number in the database; in response to a successful match, obtain the first tracking number corresponding to the express waybill and use the first tracking number as the target tracking number; in response to a failed match, obtain at least one second tracking number from the database that meets the similarity condition with the predicted tracking number that failed to match, and determine the target tracking number corresponding to the express waybill based on all the second tracking numbers.

[0084] In one embodiment, the characters also include address information. The matching module 304 is further used to obtain the address information corresponding to each second express tracking number; match the address information of the second express tracking number with the pre-stored address in the database, and use the successfully matched second express tracking number as the target tracking number.

[0085] In one embodiment, the express delivery waybill and its corresponding category label are obtained based on a segmentation network, the predicted tracking number of the express delivery waybill is obtained based on an information recognition network, and the target tracking number corresponding to the express delivery waybill is obtained based on a matching network. The segmentation network, the information recognition network, and the matching network constitute an information extraction model.

[0086] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of an embodiment of the electronic device of this application. The electronic device 40 includes a memory 401 and a processor 402 coupled to each other. The memory 401 stores program data (not shown in the figure). The processor 402 calls the program data to implement the method in any of the above embodiments. For the description of the relevant content, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0087] Please see Figure 5 , Figure 5 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application. The computer-readable storage medium 50 stores program data 500. When the program data 500 is executed by a processor, it implements the method in any of the above embodiments. For related descriptions, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0088] It should be noted that the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0089] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0090] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0091] The above description is merely an embodiment of this application and does not limit the scope of protection of this application. Any equivalent structural or procedural transformations made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of protection of this application.

Claims

1. A method for extracting express waybill information, characterized in that, The method includes: Obtain an image to be identified; wherein, the image to be identified includes express parcels, and each express parcel corresponds to an express waybill; Based on the image to be identified, the express delivery slip is obtained, and the category label corresponding to the express delivery slip is determined; wherein, the category label matches the completeness and clarity of the express delivery slip; The express waybill is identified to obtain multiple characters and their corresponding confidence scores. Based on the category label and the characters and their corresponding confidence scores, the characters to be confirmed on the express waybill and their corresponding positions are determined. The predicted tracking number of the express waybill is generated using the identified characters and the candidate characters corresponding to the characters to be confirmed. Based on the predicted tracking number and the pre-stored tracking number in the database, the target tracking number corresponding to the express waybill is obtained.

2. The method according to claim 1, characterized in that, The process of obtaining the express delivery waybill based on the image to be identified and determining the category label corresponding to the express delivery waybill includes: Feature extraction is performed on the image to be identified to obtain multi-scale first visual features, and feature fusion is performed on the multi-scale first visual features to obtain first fused features; The image is segmented using the first fusion feature to obtain the express delivery waybill within the segmented area, and the category label corresponding to the express delivery waybill is determined based on the features within the segmented area.

3. The method according to claim 1, characterized in that, The process involves identifying multiple characters and their corresponding confidence levels on the express delivery waybill, determining the characters to be confirmed and their corresponding positions on the waybill based on the category labels and the characters and their corresponding confidence levels, and generating a predicted tracking number for the express delivery waybill using the identified characters and the candidate characters corresponding to the characters to be confirmed. Obtain the label features of the express delivery label, and based on the label features, obtain multiple characters and their corresponding confidence scores; Characters whose confidence level meets a preset confidence threshold are identified as characters, and characters whose confidence level does not meet the confidence threshold are identified as characters to be confirmed. In response to the category label indicating that the express delivery waybill is blurred or missing, the blurred or missing characters on the express delivery waybill are taken as the characters to be confirmed; Based on the features of the waybill and the sorting of multiple characters, the position corresponding to the character to be confirmed is determined; The predicted order number is generated based on the identified character and the candidate characters corresponding to the character to be confirmed at the corresponding position.

4. The method according to claim 3, characterized in that, The step of obtaining the waybill features and, based on the waybill features, obtaining multiple characters and their corresponding confidence scores includes: Feature extraction is performed on the express delivery waybill to obtain multi-scale second visual features. Feature fusion is then performed on the multi-scale second visual features to obtain second fused features. Based on the second fusion feature, the waybill information and rotation angle of the express waybill are obtained; Based on the rotation angle, the waybill information is straightened and recognized to obtain the multiple characters and their corresponding confidence levels.

5. The method according to claim 1, characterized in that, The process of obtaining the target tracking number corresponding to the express waybill based on the predicted tracking number and the pre-stored tracking numbers in the database includes: Match the predicted order number with the pre-stored order number in the database; In response to a successful match, the first tracking number corresponding to the express waybill is obtained, and the first tracking number is used as the target tracking number. In response to a matching failure, at least one second express tracking number that meets the similarity condition with the predicted tracking number that failed to match is obtained from the database. Based on all the second express tracking numbers, the target tracking number corresponding to the express waybill is determined.

6. The method according to claim 5, characterized in that, The characters also include address information; The step of determining the target tracking number corresponding to the express waybill based on all the second express tracking numbers includes: Obtain the address information corresponding to each of the second express tracking numbers; The address information of the second tracking number is matched with the pre-stored address in the database, and the second tracking number that matches successfully is used as the target tracking number.

7. The method according to any one of claims 1-6, characterized in that, The express delivery waybill and its corresponding category label are obtained based on a segmentation network, the predicted tracking number of the express delivery waybill is obtained based on an information recognition network, and the target tracking number corresponding to the express delivery waybill is obtained based on a matching network. The segmentation network, the information recognition network, and the matching network constitute an information extraction model.

8. A system for extracting express waybill information, characterized in that, include: An acquisition module is used to acquire an image to be identified; wherein, the image to be identified includes express parcels, and each express parcel corresponds to an express waybill; The determining module is used to obtain the express delivery slip based on the image to be identified, and to determine the category label corresponding to the express delivery slip; wherein the category label matches the completeness and clarity of the express delivery slip; The identification module is used to identify multiple characters and their corresponding confidence scores on the express delivery waybill, determine the characters to be confirmed and their corresponding positions on the express delivery waybill based on the category label and the characters and their corresponding confidence scores, and generate the predicted tracking number of the express delivery waybill using the identified characters and the candidate characters corresponding to the characters to be confirmed. The matching module is used to obtain the target tracking number corresponding to the express waybill based on the predicted tracking number and the pre-stored tracking numbers in the database.

9. An electronic device, characterized in that, include: A memory and a processor are coupled to each other, wherein the memory stores program data, and the processor invokes the program data to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium storing program data thereon, characterized in that, When the program data is executed by the processor, the method as described in any one of claims 1-7 is implemented.

Images