Method, device and computer equipment for extracting bill of lading information

Abstract

This application relates to a method, apparatus, computer device, computer-readable storage medium, and computer program product for extracting delivery note information. The method includes: acquiring a delivery note image; performing text box detection on the delivery note image based on a first model to obtain the coordinate information of the text boxes in the delivery note image; inputting the coordinate information of the text boxes and the delivery note image into a second model to obtain the text information corresponding to the text boxes; determining a target delivery note template matching the delivery note image based on the text information, coordinate information, and a template library; generating a reference delivery note based on the text information and the target delivery note template; and extracting key information from the reference delivery note to obtain the delivery note information corresponding to the delivery note image. The delivery note information extraction method provided by this application can achieve the technical effect of improving the accuracy of delivery note image recognition.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, computer equipment, storage medium and computer program product for extracting bill of lading information. Background Art

[0002] In international trade, a bill of lading is a document that is handed over by a transport company to a carrier or consignee so that the carrier or consignee can pick up the goods from a ship, plane or warehouse. In the daily work of international trade document business, a large amount of data entry needs to be generated every day.

[0003] When the carrier or consignee goes to pick up the goods, the information on the bill of lading needs to be identified and a document needs to be kept based on the identification results. However, due to problems such as too dark ink, changes in the structure between characters, and interference from signatures on paper bills of lading, the accuracy of identifying valid information in paper bills is low. Summary of the invention

[0004] The present application provides a method, apparatus, computer device, computer-readable storage medium and computer program product for extracting bill of lading information, which recognizes bill of lading images based on multiple models, can effectively recognize the information in the bill of lading images, and improves the recognition accuracy of the bill of lading images.

[0005] In a first aspect, the present application provides a method for extracting bill of lading information, the method comprising:

[0006] Obtain a delivery note image, perform text box detection on the delivery note image based on the first model, and obtain coordinate information of the text box in the delivery note image;

[0007] Input the coordinate information of the text box and the picture of the delivery note into the second model to obtain the text information corresponding to the text box;

[0008] Perform template library matching based on text information and coordinate information to determine a target delivery note template that matches the delivery note image;

[0009] Generate a reference delivery note based on the text information and the target delivery note template, extract key information from the reference delivery note, and obtain the delivery note information corresponding to the delivery note image.

[0010] In a second aspect, the present application further provides a device for extracting bill of lading information, the device comprising:

[0011] An acquisition detection module is used to acquire a delivery note image, perform text box detection on the delivery note image based on the first model, and obtain coordinate information of the text box in the delivery note image;

[0012] A recognition module, used to input the coordinate information of the text box and the picture of the delivery note into the second model to obtain the text information corresponding to the text box;

[0013] A matching determination module is used to match the template library according to the text information and the coordinate information to determine the target delivery note template that matches the delivery note image;

[0014] The extraction module is used to generate a reference delivery note based on the text information and the target delivery note template, extract key information from the reference delivery note, and obtain the delivery note information corresponding to the delivery note image.

[0015] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any of the above methods when executing the computer program:

[0016] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of any of the above methods when the computer program is executed by a processor.

[0017] In a fifth aspect, the present application also provides a computer program product, including a computer program, which implements the steps of any of the above methods when the computer program is executed by a processor.

[0018] The present application provides a method, apparatus, computer device, computer-readable storage medium and computer program product for extracting bill of lading information, the method comprising: obtaining a bill of lading picture, performing text box detection on the bill of lading picture based on a first model, and obtaining coordinate information of the text box in the bill of lading picture; inputting the coordinate information of the text box and the bill of lading picture into a second model, and obtaining text information corresponding to the text box; determining a target bill of lading template that matches the bill of lading picture based on the text information, the coordinate information and the template library; generating a reference bill of lading based on the text information and the target bill of lading template, extracting key information from the reference bill of lading, and obtaining bill of lading information corresponding to the bill of lading picture. The method for extracting delivery note information provided in the present application locates the text box in the delivery note image based on the first model, and recognizes the text information in the corresponding text box based on the second model. Since the first model and the second model are both obtained by training the historical delivery notes after adding interference information, the positioning of the text box and the recognition of the corresponding text information are relatively accurate. At the same time, matching with the delivery note image based on a rich template library improves the matching rate of the delivery note image, and further can extract key information based on the template of the delivery note. After comprehensive processing, the technical effect of improving the recognition accuracy of the delivery note image can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is an application environment diagram of a method for extracting bill of lading information in one embodiment;

[0020] Figure 2 A schematic diagram of a process of extracting bill of lading information in one embodiment;

[0021] Figure 3 A schematic diagram of a process of extracting bill of lading information in one embodiment;

[0022] Figure 4 A schematic diagram of a flow chart of a method for extracting bill of lading information in another embodiment;

[0023] Figure 5 A schematic diagram of a flow chart of a method for extracting bill of lading information in another embodiment;

[0024] Figure 6 A schematic diagram of a flow chart of a method for extracting bill of lading information in another embodiment;

[0025] Figure 7 A schematic diagram of a flow chart of a method for extracting bill of lading information in another embodiment;

[0026] Figure 8 It is a structural block diagram of a device for extracting bill of lading information in one embodiment;

[0027] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0029] The method for extracting bill of lading information provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 collects the bill of lading picture, and sends the acquired bill of lading picture to the server 104 through the network. The server 104 inputs the bill of lading picture into the first model for text box detection, obtains the coordinate information of the text box in the bill of lading, and then inputs the coordinate information of the text box and the bill of lading picture into the second model for text recognition, obtains the text information corresponding to each text box, and determines the target bill of lading template matching the bill of lading picture according to the text information, coordinate information and template library; generates a reference bill of lading according to the text information and the target bill of lading template, extracts key information from the reference bill of lading, obtains the bill of lading information corresponding to the bill of lading picture, and can achieve the technical effect of improving the accuracy of bill of lading picture recognition. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, cameras, video cameras, scanners and portable wearable devices. The portable wearable device may be a smart watch, a smart bracelet, a head-mounted device, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0030] In one embodiment, Figure 2 As shown, a method for extracting bill of lading information is provided, and the method is applied to Figure 1 The server in the example is used to illustrate the following steps:

[0031] Step S202, obtain the delivery note image, perform text box detection on the delivery note image based on the first model, and obtain coordinate information of the text box in the delivery note image.

[0032] Among them, the bill of lading is a voucher given by the transport company to the carrier or consignee so that the carrier or consignee can pick up the goods from the ship, plane or warehouse. The bill of lading can be a paper document issued by the transport company, and the bill of lading can include various types of information such as consignee information, consignor information, port information, warehouse information, carrier information, agent information, waybill number information, etc. The bill of lading is used to pick up the goods on the one hand, and on the other hand, the relevant information on the bill of lading needs to be provided to the financial institution so that the financial institution can process the payment between the shipper and the consignee.

[0033] Then, information extraction from the delivery note can be applied in the process of picking up goods, and can also be applied in the process of payment processing by financial institutions, and this application does not limit this.

[0034] The delivery note image can be collected by a terminal device such as a scanner, camera, camcorder, mobile phone, laptop computer, desktop computer, etc. that can communicate with the server and sent to the server.

[0035] The server may receive the bill of lading image through the receiving module, and then transmit the receiving module to the processing module. The processing module then calls the first model set in the server and inputs the bill of lading image into the first model so that the first model detects the coordinate information of each text box in the bill of lading image. The detection may be to determine the coordinate information of each text box by obtaining the coordinates of the two vertices in each text box, or to determine the coordinate information of each text box by obtaining the coordinates of the center point of each text box, or to determine the coordinate information of each text box by obtaining the coordinates of three or four vertices of each text box. This application does not limit this. The text box is used to limit the location of specific text information such as consignee information, consignor information, port information, warehouse information, carrier information, agent information, waybill number information, etc. in the bill of lading.

[0036] Here we need to explain the key value and value value in the picture. The key value is equivalent to the consignee's name, gender, warehouse address, carrier name, consignor name, waybill number, etc. in each bill of lading picture, while the value value indicates the specific content corresponding to the consignee's name, Zhang San, the specific content corresponding to the gender, male, and the specific content corresponding to the warehouse address, Guanghua Road No. 128, etc. Since the key value and value value in the bill of lading picture need to be restricted by the text box, after the processing module in the server inputs the bill of lading picture into the first model, the first model performs overall coordinate recognition of the text boxes corresponding to all the key values ​​and value values ​​in the bill of lading picture, and obtains the coordinate information of the text boxes corresponding to all the key values ​​and value values ​​in the bill of lading picture.

[0037] Step S204, input the coordinate information of the text box and the delivery note image into the second model to obtain the text information corresponding to the text box.

[0038] Among them, after the coordinate information of the text boxes corresponding to all the key values ​​and value values ​​in the delivery note image is output based on the above-mentioned first model, the output can be sent to the processor, and the processor can input the coordinate information of the text boxes corresponding to all the key values ​​and value values ​​and the delivery note image into the second model for text information recognition. That is, the key value and value value are recognized. The second model can output text information corresponding to each coordinate information. Exemplarily, the second model can output [(1, 2)(4, 2)-name], [(4, 2)(2, 2)-Zhang San] and other similar information. Then the text box can be matched one-to-one with the corresponding text information. After obtaining the coordinate information of the text box and the text information corresponding to the text box, the processor in the server can mark the coordinates and text of the delivery note image to facilitate subsequent information extraction. Since the detection and recognition scope of the first model and the second model is the key value, value value, coordinates of the key value and coordinates of the value value in the entire delivery note image, what is obtained is a single text box and the key value or value value in the text box. The key value and value value cannot be matched, so further determination is required through the subsequent delivery note template.

[0039] Step S206, performing template library matching according to the text information and coordinate information, and determining a target delivery note template that matches the delivery note image.

[0040] Among them, based on the text information of the text box obtained by the second model and the coordinate information of the text box obtained by the first model, template matching can be performed to match the target bill of lading template similar to the text information of the text box in the bill of lading picture and the coordinate information of the text box, that is, a bill of lading template similar to the layout and format selected for the paper bill of lading. Since there may be many companies that can trade goods, and the bill of lading templates set by each company may be different, the server may face the identification of bills of lading with thousands of different templates, so the template library provided by this application may include all types of templates currently circulating in the market, and the templates may be from different companies, and the same company may also issue different bill of lading templates. Then by matching with the template library, a bill of lading template with the same template as the bill of lading can be matched. In general, if it is not a new bill of lading template provided by a certain company or a new company, one or more corresponding target bill of lading templates can be matched from the template library. Then, after a new delivery note template appears, the server may also expand the template library according to the new delivery note template to further enrich the template library, so as to facilitate subsequent matching of the target delivery note template according to the template library.

[0041] Step S208, generating a reference delivery note according to the text information and the target delivery note template, extracting key information from the reference delivery note, and obtaining delivery note information corresponding to the delivery note image.

[0042] Among them, according to the above-mentioned template library matching step, the processing module of the server may be to obtain one or more target delivery note templates, or may continue to use the text information corresponding to the text box identified by the second model, and map the text information to the target delivery note template through the transformation matrix method. Because there is no corresponding value value in the target delivery note template, only the key value exists, so after mapping, it may be implemented to fill the value value in the delivery note image into the area where the value value corresponding to the key value is located in the target delivery note template, and obtain the reference delivery note (equivalent to the process of matching the key value and the value value one by one). Then, the server may extract the information in the area where the value value is located through the extraction module to obtain the key information in the reference delivery note, and then obtain the delivery note information needed by the financial institution or warehouse manager in the delivery note image.

[0043] The present application provides a method for extracting delivery note information, the method comprising: obtaining a delivery note image, performing text box detection on the delivery note image based on a first model, and obtaining coordinate information of the text box in the delivery note image; inputting the coordinate information of the text box and the delivery note image into a second model to obtain text information corresponding to the text box; determining a target delivery note template that matches the delivery note image based on the text information, coordinate information, and a template library; generating a reference delivery note based on the text information and the target delivery note template, extracting key information from the reference delivery note, and obtaining delivery note information corresponding to the delivery note image. The method for extracting delivery note information provided in the present application locates the text box in the delivery note image based on the first model, and recognizes the text information in the corresponding text box based on the second model. Since the first model and the second model are both obtained by training the historical delivery notes after adding interference information, the positioning of the text box and the recognition of the corresponding text information are relatively accurate. At the same time, matching with the delivery note image based on a rich template library improves the matching rate of the delivery note image, and further can extract key information based on the template of the delivery note. After comprehensive processing, the technical effect of improving the recognition accuracy of the delivery note image can be achieved.

[0044] In one embodiment, this embodiment is an optional method embodiment after template library matching fails, and the method includes:

[0045] If the text information, coordinate information and the template library are matched unsuccessfully, the text information and coordinate information are input into the universal matching model to obtain the delivery note information corresponding to the delivery note image.

[0046] Among them, according to the above, if the delivery order picture comes from a new company, or the previous company used a new template when making the delivery order, and the template library has not been updated or supplemented, then when the server matches the template library through the text information in the delivery order picture and the coordinate information corresponding to the text information, there will be a situation where the corresponding template library cannot be matched, then the processing module in the server will call the universal matching model to perform a one-to-one matching of key values ​​and value values. The universal matching model is obtained by training a neural network model based on a large number of key values ​​and a large number of value values. For example, it is obtained by training a key value of the name and a large number of value values ​​such as Zhang San, Li Si, and Wang Er, so a one-to-one matching of key values ​​and value values ​​can be achieved through a universal matching model. This application uses a universal matching model to make up for the deficiency that the delivery order information cannot be extracted from the delivery order picture due to the failure to match the template library, so that the delivery order information extraction method provided by this application can be applied to more usage scenarios, and users can also select a suitable extraction method to extract the delivery order information from the delivery order according to the specific situation of the delivery order, and the extraction of the delivery order information is more flexible. It should be noted that the accuracy of extracting bill of lading information by the general model is slightly lower than that of extracting bill of lading information by mapping the target bill of lading template as mentioned above. Therefore, when extracting bill of lading information, it is preferred to use template mapping to extract bill of lading information, which can obtain more accurate bill of lading information.

[0047] In one embodiment, Figure 3 As shown, this embodiment is an optional method embodiment for training the first model, and the method embodiment includes the following steps:

[0048] Step S302, obtaining a plurality of first historical delivery note images, performing interference adding processing on the first historical delivery note images, and obtaining an initial image sample corresponding to each first historical delivery note image;

[0049] Step S304, performing text box calibration processing on each initial image sample, performing model training based on the processed initial image samples, and obtaining a first model.

[0050] Among them, multiple first historical delivery order images can be obtained from the server's memory, or from the historical delivery order database of each company, and this application does not limit this. After obtaining a large number of first historical delivery order images, the server can perform interference addition on multiple first historical delivery order images one by one to increase the difficulty of recognizing the text box of the delivery order image and enhance the detection effect and generalization ability of the model. Interference addition can be to increase the background of the delivery order image so that the delivery order image presents multiple backgrounds; it can be to increase the clarity of the weakened text box and increase the difficulty of recognizing the text box; it can also be to add symbols, text, etc. on the frame line of the text box to increase the difficulty of recognizing the text box, etc., and this application does not limit this. After interference processing, the initial image sample corresponding to each first historical delivery order image is obtained, and the text box calibration processing is performed on each initial image sample. Model training is performed based on the processed initial image sample to obtain the first model.

[0051] The method for extracting delivery note information provided in the present application obtains a first model by performing model training on multiple first historical delivery note images that have been subjected to interference processing. Since the text box of the first model has undergone various types of interference processing during training, the first model's ability to recognize the text box is greatly enhanced. By detecting the coordinate information of the text box of the delivery note image based on the first model, the coordinate information of the text box in the delivery note image can be obtained more accurately.

[0052] In one embodiment, Figure 4 As shown, this embodiment is an optional method embodiment for training the second model, and the method embodiment includes the following steps:

[0053] Step S402, obtaining a plurality of second historical delivery note images, performing text interference adding processing on the second historical delivery note images, and obtaining an initial image sample corresponding to each second historical delivery note image;

[0054] Step S404: perform text calibration processing on each initial image sample, perform model training based on the processed initial image samples, and obtain a second model.

[0055] Among them, multiple second historical delivery order pictures can be obtained from the server's memory, or from the historical delivery order database of each company. The second historical delivery order picture can also be the same as the first historical delivery order picture mentioned above, and this application does not limit this. After obtaining a large number of second historical delivery order pictures, the server can perform text interference addition on multiple second historical delivery order pictures one by one to increase the difficulty of recognizing text information in the delivery order pictures, and enhance the recognition effect and generalization ability of the model. Text addition can be to add the background of the delivery order picture so that the delivery order picture presents multiple backgrounds; it can be to change the font of the text; it can be to change the thickness of the text; it can be to change the structure between the text; it can be to overlap the text; it can be to add a seal to the text; it can be to add a signature to the text, etc. to increase the difficulty of recognizing text information, etc., and this application does not limit this. After interference processing, the initial image sample corresponding to each second historical delivery order picture is obtained, and text calibration processing is performed on each initial image sample. Model training is performed based on the processed initial image sample to obtain a second model.

[0056] The method for extracting delivery note information provided in the present application obtains a second model by performing model training on multiple second historical delivery note images that have been subjected to interference processing. Since the second model has undergone various types of interference processing on the text during training, the second model's ability to recognize text is greatly enhanced. By recognizing the text information of the delivery note image based on the second model, the text information in the delivery note image can be obtained more accurately.

[0057] In one embodiment, Figure 5 As shown, this embodiment is an optional method embodiment for determining a target bill of lading template, and the method embodiment includes the following steps:

[0058] Step 502, obtaining a unique tag of the delivery note image, where the unique tag is used to indicate the source of the delivery note image.

[0059] The unique mark of the delivery note image can be an identifier set by the server for different companies. The identifier can be an English character, a Chinese character, a number, etc., or a combination of several characters. This application does not limit this. The unique mark indicates whether the delivery note image comes from Company A or Company B.

[0060] Step 504, based on the unique tag and the unique tags of each bill of lading template in the template library, multiple candidate bill of lading templates are matched from the template library.

[0061] Based on the unique mark of the acquired bill of lading image, the unique mark can be compared with all bill of lading templates in the template library. Since each company may have multiple different bill of lading templates, multiple candidate bill of lading templates may be matched.

[0062] Step 506: Determine a target bill of lading template that matches the bill of lading image from a plurality of candidate bill of lading templates based on the text information and the coordinate information.

[0063] Among them, based on the multiple candidate bill of lading templates obtained above, template matching can be continued according to text information and coordinate information to obtain the target bill of lading template. Template matching can be performed through text information and coordinate information, and the text information in the candidate bill of lading template can be matched with the text information in the bill of lading. The candidate bill of lading template with the most matched text information is determined as the target bill of lading template; or the coordinate information corresponding to each text box in the candidate bill of lading template can be matched with the coordinate information of each text box in the bill of lading picture, and the candidate bill of lading template with the most matched coordinate information is determined as the target bill of lading template; or the coordinate information in the candidate bill of lading template can be matched with the coordinate information in the bill of lading picture, and then the text information corresponding to the coordinate information can be matched, or the candidate bill of lading template with the most matched text information can be determined as the target bill of lading template.

[0064] The method for extracting delivery note information provided in the present application preliminarily screens the delivery note templates in the template library through the unique mark of the delivery note image to obtain candidate delivery note templates, and then determines the target delivery note template from the candidate delivery note templates. This can avoid comparing the text information, coordinate information, etc. of the delivery note with each delivery note template in the huge template library one by one, which causes heavy burden on the server and low comparison rate.

[0065] In one embodiment, Figure 6 As shown, this embodiment is an optional method embodiment for determining a target bill of lading template, and the method embodiment includes the following steps:

[0066] Step S602, calculating the similarity between the delivery note image and the multiple candidate delivery note templates according to the coordinate information corresponding to the text information and the coordinate information corresponding to the text information of each delivery note template in the multiple candidate templates;

[0067] Step S604: determine the candidate delivery note template with a similarity greater than a threshold as the target delivery note template.

[0068] Among them, the present application may be to match the text information in the delivery note image with the text information in multiple candidate templates, and after the match is successful, calculate the similarity of the coordinate information of the corresponding text information. Since the delivery note image may include multiple text information, the text information in the delivery note image may be compared with the text information in the candidate templates one by one, and then the similarity of the coordinate information of the matched text information is calculated to obtain multiple similarities, which may be the average of multiple similarities, or the one with the smallest similarity may be compared with a preset similarity threshold. If the similarity is greater than the preset threshold, it means that the delivery note template is similar or identical to the template selected for the delivery note image. If the delivery note template is similar or identical, it means that the text box layout in the delivery note image and the key value layout in the text box are similar or identical. Then, by mapping the value value through the target delivery note template, the value value obtained is more accurate.

[0069] In one embodiment, Figure 7 As shown, this embodiment is an optional method embodiment of obtaining the bill of lading information corresponding to the bill of lading picture, and the method embodiment includes:

[0070] Step S702, determining description information of a target bill of lading template, the description information including a plurality of bill of lading parameters;

[0071] Step S704, determining the target bill of lading parameter corresponding to the text information, and mapping the text information to the value area corresponding to the target bill of lading parameter in the description information.

[0072] Among them, the description information of the target bill of lading template includes the key value, the coordinates of the key value and the coordinates of the value value in the target bill of lading template. The target bill of lading parameters of the corresponding text information are also the key value, the coordinates of the key value and the coordinates of the value value corresponding to the text information. The text information is mapped to the description information, that is, the bill of lading parameters of the text information are matched with the bill of lading parameters of the target bill of lading template. After the match is successful, the text information falls into the corresponding target bill of lading template value area. Exemplarily, the text information is Zhang San, and the target bill of lading parameters of Zhang San are: name, [(1,2)(4,2)], [(4,2)(2,2)], then Zhang San can be mapped into the area of ​​[(4,2)(2,2)] corresponding to the bill of lading parameters of the target bill of lading template as name, [(1,2)(4,2)], [(4,2)(2,2)], then when performing subsequent key information extraction, it can be extracted. Information such as name-Zhang San can be obtained to obtain the bill of lading information corresponding to the bill of lading picture.

[0073] Optionally, the second model library includes a general corpus and a professional corpus.

[0074] Among them, the general corpus includes a variety of general corpus information, and the professional corpus includes targeted corpus information such as the names of various companies, the names of various receiving companies, the names of various ports, etc., which can improve the recognition accuracy of text information.

[0075] Optionally, after obtaining the delivery note information corresponding to the delivery note image, the delivery note information may also be verified using business rules.

[0076] Among them, for example, if the value of the ID card can only be 18 digits, then the text information of the ID card in the delivery order information is 20 digits, which means that the delivery order information is incorrect; the value of the waybill number can only be 10 digits, then the text information of the waybill number in the delivery order information is 12 digits, which means that the delivery order information is incorrect; the value of the company name can only be Chinese characters, then the text information of the company name in the delivery order information includes numbers, which means that the delivery order information is incorrect, etc. Business rules can be some currently accepted specifications. By verifying the delivery order information through property specifications, the delivery order information can be effectively verified to avoid outputting incorrect delivery order information.

[0077] Optionally, after acquiring the delivery note image, the server may also resize the delivery note image and input the resized delivery note image into the first model.

[0078] Since the server can detect and identify pictures of a predetermined size locally when detecting and identifying pictures, it is convenient for the server to carry out a patterned processing flow and speed up the processing rate of the server. Then, in order to facilitate the subsequent detection and identification of the server, the detection and identification can only detect and identify pictures within a preset range. If the size of the picture is too large, the server may not be able to detect and identify the part of the picture that exceeds the range, so that the subsequent extracted bill of lading information is inaccurate. Therefore, the obtained bill of lading picture can be first transformed into a predetermined size to facilitate subsequent server processing. Different servers can be set with different predetermined sizes for the predetermined size, and this application does not limit this.

[0079] Optionally, the present application may package the above-mentioned method for extracting the delivery note information and generate an easy-to-call RESTFUL API microservice for the front-end system to call. In order to meet the delivery note information extraction requirement within two seconds of a single request from the external system, a multi-GPU polling mechanism may be adopted. A separate sub-thread will be started for each delivery note image for identification to complete the entire extraction step.

[0080] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0081] Based on the same inventive concept, the embodiment of the present application also provides a bill of lading information extraction device for implementing the bill of lading information extraction method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more bill of lading information extraction device embodiments provided below can refer to the limitations of the bill of lading information extraction method above, and will not be repeated here.

[0082] In one embodiment, Figure 8 As shown, a device for extracting bill of lading information is provided, including: an acquisition detection module 802, an identification module 804, a determination module 806 and an extraction module 808, wherein:

[0083] The acquisition detection module 802 is used to acquire a delivery note image, perform text box detection on the delivery note image based on the first model, and obtain coordinate information of the text box in the delivery note image;

[0084] Recognition module 804, used to input the coordinate information of the text box and the picture of the delivery note into the second model to obtain text information corresponding to the text box;

[0085] A matching determination module 806 is used to perform template library matching based on the text information and the coordinate information to determine a target bill of lading template that matches the bill of lading image;

[0086] The extraction module 808 is used to generate a reference delivery note according to the text information and the target delivery note template, extract key information from the reference delivery note, and obtain the delivery note information corresponding to the delivery note image.

[0087] In one embodiment, the matching determination module 806 is also used to input the text information and coordinate information into a universal matching model to obtain the delivery note information corresponding to the delivery note image when the matching between the text information and coordinate information and the template library fails.

[0088] In one embodiment, the above device further comprises a training module.

[0089] A training module is used to obtain multiple first historical delivery note images, perform interference adding processing on the first historical delivery note images, and obtain initial image samples corresponding to each first historical delivery note image; perform text box calibration processing on each initial image sample, and perform model training based on the processed initial image samples to obtain a first model.

[0090] In one embodiment, the above-mentioned device also includes a training module, which is also used to obtain multiple second historical delivery note images, perform text interference adding processing on the second historical delivery note images, and obtain initial image samples corresponding to each second historical delivery note image; perform text calibration processing on each initial image sample, and perform model training based on the processed initial image samples to obtain a second model.

[0091] In one embodiment, the above-mentioned matching determination module 806 is also used to obtain a unique tag of the delivery note image, and the unique tag is used to characterize the source of the delivery note image; based on the unique tag and the unique tags of each delivery note template in the template library, multiple candidate delivery note templates are matched from the template library; and a target delivery note template that matches the delivery note image is determined from multiple candidate delivery note templates based on text information and coordinate information.

[0092] In one embodiment, the matching determination module 806 includes a calculation determination unit,

[0093] A calculation and determination unit is used to calculate the similarity between the delivery note image and multiple candidate delivery note templates based on the coordinate information corresponding to the text information and the coordinate information corresponding to the text information of each delivery note template in the multiple candidate templates; and determine the candidate delivery note template with a similarity greater than a threshold as the target delivery note template.

[0094] In one embodiment, the extraction module 808 is specifically used to determine the description information of the target bill of lading template, the description information including multiple bill of lading parameters; determine the target bill of lading parameters corresponding to the text information, and map the text information to the value area corresponding to the target bill of lading parameters in the description information.

[0095] Each module in the above-mentioned bill of lading information extraction device can be implemented in whole or in part by software, hardware and their combination. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0096] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 9As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store coordinate information and text information data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for extracting bill of lading information is implemented.

[0097] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0098] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0099] Obtain a delivery note image, perform text box detection on the delivery note image based on the first model, and obtain coordinate information of the text box in the delivery note image;

[0100] Input the coordinate information of the text box and the picture of the delivery note into the second model to obtain the text information corresponding to the text box;

[0101] Perform template library matching based on text information and coordinate information to determine a target delivery note template that matches the delivery note image;

[0102] Generate a reference delivery note based on the text information and the target delivery note template, extract key information from the reference delivery note, and obtain the delivery note information corresponding to the delivery note image.

[0103] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0104] When the text information and coordinate information fail to match the template library, the text information and coordinate information are input into the universal matching model to obtain the delivery note information corresponding to the delivery note image.

[0105] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0106] Acquire multiple first historical delivery note images, perform interference adding processing on the first historical delivery note images, and obtain initial image samples corresponding to each first historical delivery note image; perform text box calibration processing on each initial image sample, perform model training based on the processed initial image samples, and obtain a first model.

[0107] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0108] Acquire multiple second historical delivery note images, perform text interference adding processing on the second historical delivery note images, and obtain initial image samples corresponding to each second historical delivery note image; perform text calibration processing on each initial image sample, perform model training based on the processed initial image samples, and obtain a second model.

[0109] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0110] A unique tag of the delivery note image is obtained, where the unique tag is used to characterize the source of the delivery note image; multiple candidate delivery note templates are matched from the template library based on the unique tag and the unique tags of each delivery note template in the template library; a target delivery note template that matches the delivery note image is determined from the multiple candidate delivery note templates based on the text information and the coordinate information.

[0111] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0112] Based on the coordinate information corresponding to the text information and the coordinate information corresponding to the text information of each delivery note template in multiple candidate templates, the similarity between the delivery note image and multiple candidate delivery note templates is calculated; and the candidate delivery note template with a similarity greater than a threshold is determined as the target delivery note template.

[0113] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0114] Determine description information of a target bill of lading template, the description information including multiple bill of lading parameters;

[0115] Determine the target bill of lading parameter corresponding to the text information, and map the text information to the value area corresponding to the target bill of lading parameter in the description information.

[0116] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0117] Obtain a delivery note image, perform text box detection on the delivery note image based on the first model, and obtain coordinate information of the text box in the delivery note image;

[0118] Input the coordinate information of the text box and the picture of the delivery note into the second model to obtain the text information corresponding to the text box;

[0119] Perform template library matching based on text information and coordinate information to determine a target delivery note template that matches the delivery note image;

[0120] Generate a reference delivery note based on the text information and the target delivery note template, extract key information from the reference delivery note, and obtain the delivery note information corresponding to the delivery note image.

[0121] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0122] When the text information and coordinate information fail to match the template library, the text information and coordinate information are input into the universal matching model to obtain the delivery note information corresponding to the delivery note image.

[0123] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0124] Acquire multiple first historical delivery note images, perform interference adding processing on the first historical delivery note images, and obtain initial image samples corresponding to each first historical delivery note image; perform text box calibration processing on each initial image sample, perform model training based on the processed initial image samples, and obtain a first model.

[0125] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0126] Acquire multiple second historical delivery note images, perform text interference adding processing on the second historical delivery note images, and obtain initial image samples corresponding to each second historical delivery note image; perform text calibration processing on each initial image sample, perform model training based on the processed initial image samples, and obtain a second model.

[0127] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0128] A unique tag of the delivery note image is obtained, where the unique tag is used to characterize the source of the delivery note image; multiple candidate delivery note templates are matched from the template library based on the unique tag and the unique tags of each delivery note template in the template library; a target delivery note template that matches the delivery note image is determined from the multiple candidate delivery note templates based on the text information and the coordinate information.

[0129] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0130] Based on the coordinate information corresponding to the text information and the coordinate information corresponding to the text information of each delivery note template in multiple candidate templates, the similarity between the delivery note image and multiple candidate delivery note templates is calculated; and the candidate delivery note template with a similarity greater than a threshold is determined as the target delivery note template.

[0131] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0132] Determine description information of a target bill of lading template, the description information including multiple bill of lading parameters;

[0133] Determine the target bill of lading parameter corresponding to the text information, and map the text information to the value area corresponding to the target bill of lading parameter in the description information.

[0134] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0135] Obtain a delivery note image, perform text box detection on the delivery note image based on the first model, and obtain coordinate information of the text box in the delivery note image;

[0136] Input the coordinate information of the text box and the picture of the delivery note into the second model to obtain the text information corresponding to the text box;

[0137] Perform template library matching based on text information and coordinate information to determine a target delivery note template that matches the delivery note image;

[0138] Generate a reference delivery note based on the text information and the target delivery note template, extract key information from the reference delivery note, and obtain the delivery note information corresponding to the delivery note image.

[0139] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0140] When the text information and coordinate information fail to match the template library, the text information and coordinate information are input into the universal matching model to obtain the delivery note information corresponding to the delivery note image.

[0141] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0142] Acquire multiple first historical delivery note images, perform interference adding processing on the first historical delivery note images, and obtain initial image samples corresponding to each first historical delivery note image; perform text box calibration processing on each initial image sample, perform model training based on the processed initial image samples, and obtain a first model.

[0143] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0144] Acquire multiple second historical delivery note images, perform text interference adding processing on the second historical delivery note images, and obtain initial image samples corresponding to each second historical delivery note image; perform text calibration processing on each initial image sample, perform model training based on the processed initial image samples, and obtain a second model.

[0145] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0146] A unique tag of the delivery note image is obtained, where the unique tag is used to characterize the source of the delivery note image; multiple candidate delivery note templates are matched from the template library based on the unique tag and the unique tags of each delivery note template in the template library; a target delivery note template that matches the delivery note image is determined from the multiple candidate delivery note templates based on the text information and the coordinate information.

[0147] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0148] Based on the coordinate information corresponding to the text information and the coordinate information corresponding to the text information of each delivery note template in multiple candidate templates, the similarity between the delivery note image and multiple candidate delivery note templates is calculated; and the candidate delivery note template with a similarity greater than a threshold is determined as the target delivery note template.

[0149] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0150] Determine description information of a target bill of lading template, the description information including multiple bill of lading parameters;

[0151] Determine the target bill of lading parameter corresponding to the text information, and map the text information to the value area corresponding to the target bill of lading parameter in the description information.

[0152] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0153] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0154] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0155] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for extracting bill of lading information, characterized in that: The method comprises: Acquire a delivery note picture, perform text box detection on the delivery note picture based on the first model, and obtain coordinate information of the text box in the delivery note picture; Input the coordinate information of the text box and the picture of the bill of lading into the second model to obtain the text information corresponding to the text box; Perform template library matching based on the text information and the coordinate information to determine a target delivery note template that matches the delivery note image; Generate a reference bill of lading according to the text information and the target bill of lading template, extract key information from the reference bill of lading, and obtain bill of lading information corresponding to the bill of lading image; The method further comprises: If the text information, the coordinate information and the template library fail to match, the text information and the coordinate information are input into a universal matching model to obtain the bill of lading information corresponding to the bill of lading image; the universal matching model is obtained by training a neural network model based on key values ​​and value values; the universal matching model is used to match key values ​​and value values; the key value is used to represent one of the consignee's name, gender, warehouse address, carrier name, consignor's name, and waybill number in each of the bill of lading images, and the value value is used to represent the specific content corresponding to the consignee's name, the specific content corresponding to the gender, the specific content corresponding to the warehouse address, the specific content corresponding to the carrier's name, and the specific content corresponding to the consignor's name.

2. The method according to claim 1, characterized in that The training process of the first model includes: Acquire multiple first historical delivery note images, perform interference adding processing on the first historical delivery note images, and obtain an initial image sample corresponding to each first historical delivery note image; Perform text box calibration processing on each of the initial image samples, perform model training based on the processed initial image samples, and obtain the first model.

3. The method according to claim 1, characterized in that The training process of the second model includes: Acquire multiple second historical delivery note images, perform text interference adding processing on the second historical delivery note images, and obtain an initial image sample corresponding to each second historical delivery note image; Perform text calibration processing on each of the initial image samples, perform model training based on the processed initial image samples, and obtain the second model.

4. The method according to claim 1, characterized in that: The step of determining a target bill of lading template that matches the bill of lading image according to the text information, the coordinate information, and the template library includes: Obtaining a unique tag of the bill of lading image, wherein the unique tag is used to indicate the source of the bill of lading image; Based on the unique mark and the unique mark of each bill of lading template in the template library, matching multiple candidate bill of lading templates from the template library; A target bill of lading template matching the bill of lading image is determined from the multiple candidate bill of lading templates according to the text information and the coordinate information.

5. The method according to claim 4, characterized in that The determining a target bill of lading template that matches the bill of lading image from the plurality of candidate bill of lading templates according to the text information and the coordinate information comprises: Calculate the similarity between the delivery note image and the multiple candidate delivery note templates according to the coordinate information corresponding to the text information and the coordinate information corresponding to the text information of each delivery note template in the multiple candidate templates; The candidate bill of lading template whose similarity is greater than a threshold is determined as the target bill of lading template.

6. The method according to claim 1, characterized in that The generating a reference bill of lading according to the text information and the target bill of lading template includes: Determine description information of the target bill of lading template, wherein the description information includes a plurality of bill of lading parameters; Determine the target bill of lading parameter corresponding to the text information, and map the text information to the value area corresponding to the target bill of lading parameter in the description information.

7. A device for extracting bill of lading information, characterized in that: The device comprises: An acquisition detection module is used to acquire a delivery note picture, perform text box detection on the delivery note picture based on the first model, and obtain coordinate information of the text box in the delivery note picture; A recognition module, used for inputting the coordinate information of the text box and the picture of the bill of lading into a second model to obtain text information corresponding to the text box; A matching determination module, configured to perform template library matching according to the text information and the coordinate information, and determine a target bill of lading template that matches the bill of lading image; An extraction module, used to generate a reference bill of lading according to the text information and the target bill of lading template, extract key information from the reference bill of lading, and obtain bill of lading information corresponding to the bill of lading image; The device also includes: If the text information, the coordinate information and the template library fail to match, the text information and the coordinate information are input into a universal matching model to obtain the bill of lading information corresponding to the bill of lading image; the universal matching model is obtained by training a neural network model based on key values ​​and value values; the universal matching model is used to match key values ​​and value values; the key value is used to represent one of the consignee's name, gender, warehouse address, carrier name, consignor's name, and waybill number in each of the bill of lading images, and the value value is used to represent the specific content corresponding to the consignee's name, the specific content corresponding to the gender, the specific content corresponding to the warehouse address, the specific content corresponding to the carrier's name, and the specific content corresponding to the consignor's name.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images