Logistics receipt intelligent input method and system based on multi-model cooperation

Abstract

The invention relates to a logistics receipt intelligent input method and system based on multi-model collaboration, and the method comprises the steps: obtaining a logistics receipt image through an image collection device, and carrying out the preprocessing of the logistics receipt image; identifying the preprocessed logistics document image by using a multi-modal identification model, and starting an independent thread for synchronous processing to obtain an identification result; the inference model classifies the identification results based on a business rule base to obtain business keywords, and ledger information is generated based on a ledger template and the business keywords; monitoring arrangement and scheduling conditions of the multi-modal recognition model and the reasoning model, adjusting an execution sequence based on the arrangement and scheduling conditions, and optimizing a distribution path; and encrypting the generated standing book, and connecting the encrypted standing book to a service system through an interface to complete input. Through cooperative operation of a multi-modal identification model and an inference model and an innovative interaction mode, end-to-end automation of logistics document data from acquisition to input is realized.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and system for intelligent entry of logistics documents based on multi-model collaboration. Background Technology

[0002] In the document entry stage of the logistics industry, existing technologies improve entry efficiency by introducing Optical Character Recognition (OCR) and template generation, but these methods still have many limitations. In the image recognition stage, OCR technology is primarily relied upon. This method is greatly affected by document quality; when documents are handwritten, unclearly printed, or have stains, the recognition error rate is high, making it difficult to guarantee data accuracy. In the ledger generation stage, traditional code-based fixed programs are used to generate ledgers, adapting to fixed templates. When document content changes or special ledger requirements arise, the code needs to be rewritten to generate compliant ledgers. Regarding user interaction, operations are initiated by clicking buttons, resulting in a cumbersome and inflexible process and a poor user experience. With increasingly complex and diverse logistics operations, the existing technologies can no longer meet the needs of enterprises for efficient and accurate processing of logistics documents. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention utilizes the orchestration capabilities of an artificial intelligence platform to connect different large models. Through the collaborative operation of multimodal recognition and inference models, as well as innovative interaction methods, it solves problems such as large image recognition errors, inflexible ledger generation, and poor user experience in existing technologies. This achieves automation, efficiency, and accuracy in logistics document processing, thereby enhancing the data processing capabilities and overall competitiveness of logistics operations.

[0004] To achieve the above objectives, the technical solution adopted by the present invention includes:

[0005] A method for intelligent entry of logistics documents based on multi-model collaboration, characterized by comprising:

[0006] S1. Acquire logistics document images through image acquisition devices, and preprocess the logistics document images, wherein the logistics document images include a first sub-module, a second sub-module and a third sub-module;

[0007] S2. Use a multimodal recognition model to recognize the preprocessed logistics document image, start an independent thread to synchronously process the recognition tasks of the first submodule, the second submodule and the third submodule, and obtain the recognition results;

[0008] S3. The reasoning model classifies the identification results based on the business rule base to obtain business keywords, and generates ledger information based on the ledger template and the business keywords;

[0009] S4. Monitor the orchestration and scheduling of the multimodal recognition model and the inference model. The first platform adjusts the execution order and optimizes the allocation path based on the orchestration and scheduling. The first platform includes an intelligent module, which is used to parse natural language.

[0010] S5. Encrypt the generated ledger and then connect the encrypted ledger to the business system via an interface to complete the data entry.

[0011] Furthermore, the preprocessing of the logistics document image includes image enhancement, noise reduction, and tilt correction operations.

[0012] Furthermore, a multimodal recognition model is used to recognize the preprocessed logistics document images, and the recognition results are obtained, including:

[0013] A convolutional neural network and a long short-term memory network architecture are constructed. The feature information of the first mode is extracted by the convolutional neural network, and the long short-term memory network is used for temporal modeling. Based on the historical feature information of the first mode, the recognition result of the first mode is obtained. The convolutional neural network is a residual neural network.

[0014] The feature information of the second modality is extracted using OCR, and then combined with an attention mechanism to perform semantic verification on the feature information of the second modality, thereby obtaining the recognition result of the second modality.

[0015] The third modality is labeled using an object detection algorithm, and its shape feature information is extracted to obtain the recognition result of the third modality.

[0016] Furthermore, the first modality is handwritten text, the second modality is printed text, and the third modality is a symbolic graphic.

[0017] Furthermore, after using a multimodal recognition model to recognize the preprocessed logistics document image, starting an independent thread to synchronously process the recognition tasks of the first sub-module, the second sub-module, and the third sub-module, and obtaining the recognition result, the method further includes:

[0018] The confidence level of the recognition result is evaluated. When the confidence level of the recognition result is lower than the preset confidence level, a second recognition operation is triggered.

[0019] Furthermore, the inference model classifies the identification results based on a business rule base to obtain business keywords, and generates ledger information based on the ledger template and the business keywords, including:

[0020] The first sub-module of the logistics document image is processed using a multimodal recognition model to obtain the logistics document category;

[0021] The inference model retrieves matching business rules from the business rule base based on the logistics document category to obtain business keywords corresponding to the business requirements.

[0022] The business requirements are analyzed based on the first natural language instruction information, and the inference model generates a custom ledger template or matches a preset ledger template based on the business requirements.

[0023] The business keywords are combined with the custom ledger template or the preset ledger template to generate ledger information.

[0024] Furthermore, the method also includes:

[0025] The received ledgers are verified according to the business rule base;

[0026] If the verification passes, the ledger information will be stored in the database;

[0027] If the verification fails, the information is sent to the first platform. The intelligent module of the first platform uses natural language to interact with the user to confirm the correction method of the ledger based on the error message. After correcting the ledger, it is re-entered into the business system.

[0028] If the verification fails, the process also includes: marking abnormal logistics documents according to the business rule base and generating pending items for the approval process.

[0029] This invention also relates to an intelligent data entry system for logistics documents based on multi-model collaboration, characterized in that it includes:

[0030] The preprocessing module is used to acquire logistics document images through image acquisition devices and preprocess the logistics document images. The logistics document images include a first submodule, a second submodule, and a third submodule.

[0031] The recognition module is used to recognize the preprocessed logistics document images using a multimodal recognition model. It starts an independent thread to synchronously process the recognition tasks of the first sub-module, the second sub-module, and the third sub-module to obtain the recognition results.

[0032] The reasoning module is used to classify the recognition results based on the business rule base of the reasoning model, obtain business keywords, and generate ledger information based on the ledger template and the business keywords;

[0033] The scheduling module is used to monitor the orchestration and scheduling of the multimodal recognition model and the inference model. The first platform adjusts the execution order and optimizes the allocation path based on the orchestration and scheduling. The first platform includes an intelligent module, which is used to parse natural language.

[0034] The data entry module is used to encrypt the generated ledgers and then connect the encrypted ledgers to the business system via an interface to complete the data entry.

[0035] The present invention also relates to a computer-readable storage medium, characterized in that the storage medium stores a computer program, which, when executed by a processor, implements the above-described method.

[0036] The present invention also relates to an electronic device, characterized in that it includes a processor and a memory;

[0037] The memory is used to store program instructions;

[0038] The processor is used to execute the above-described method by invoking program instructions.

[0039] The present invention also relates to a computer program product, including a computer program and / or instructions, characterized in that the computer program and / or instructions, when executed by a processor, implement the steps of the above-described method.

[0040] The beneficial effects of this invention are as follows:

[0041] This invention applies multimodal data processing and intelligent reasoning technologies to real-world business scenarios, meeting diverse business needs and effectively solving the problems of low efficiency, poor accuracy, and high cost associated with traditional manual data entry. By integrating the platform with intelligent modules and orchestrating and connecting different large models, it provides a new technical path for multi-model collaborative processing of complex business processes. This avoids efficiency losses and data connectivity issues that may arise from individual model operation, enabling complementary advantages among models and further improving the efficiency and quality of logistics document processing. The use of natural language processing makes operation more convenient and intuitive, better addressing complex and ever-changing logistics business scenarios and ensuring continuous and efficient business operations. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0043] Figure 1 This is a schematic diagram of the process of a smart data entry method for logistics documents based on multi-model collaboration according to the present invention.

[0044] Figure 2 This is a schematic diagram of the structure of a logistics document intelligent entry system based on multi-model collaboration according to the present invention;

[0045] Figure 3 This is a schematic diagram of the structure of an electronic device according to the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. It should be understood that the described embodiments are only some embodiments of this application, not all embodiments, and these embodiments are only used to illustrate this application and not to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0047] like Figure 1 As shown in the figure, this invention discloses an intelligent data entry method for logistics documents based on multi-model collaboration, the steps of which include:

[0048] Step S1. Acquire logistics document images through image acquisition devices, and preprocess the logistics document images, which include a first sub-module, a second sub-module, and a third sub-module.

[0049] Logistics documents include, but are not limited to, weighbridge slips, invoices, and contracts. Each logistics document image can be divided into sub-modules such as header area, table area, and signature area. Image data of logistics documents such as weighbridge slips, invoices, and contracts are collected using scanning devices or high-definition cameras. Common formats such as JPEG and PNG are supported during collection, and the image resolution must be greater than or equal to 300 dpi. During image collection, interference such as reflections and shadows must be avoided. For example, when collecting weighbridge slip images, the entire header, cargo information, and signature area should be captured to obtain the weighbridge slip image information.

[0050] In this embodiment, a unique identifier (such as timestamp + serial number) is generated for each logistics document image and associated with basic logistics information, including the type of logistics document (such as weighbridge slip, invoice, contract, etc.), the company to which it belongs, the transportation batch, etc., to facilitate subsequent business tracking.

[0051] The logistics document image undergoes preprocessing, including image enhancement, denoising, and tilt correction. Histogram equalization is used to adjust brightness and contrast, highlighting text areas; an adaptive threshold algorithm is employed to enhance blurred text, ensuring the clarity of handwritten numbers (such as weights on weighbridges) and printed text (such as invoice amounts); median filtering is used to remove paper stains and scanning noise; and Hough transform is used to detect the tilt angle of text lines, automatically rotating and correcting it (error ≤ 1°) to ensure the logistics document image is horizontally aligned, guaranteeing the input quality for subsequent recognition models and facilitating accurate model recognition.

[0052] Step S2. Use a multimodal recognition model to recognize the preprocessed logistics document image, start an independent thread to synchronously process the recognition tasks of the first sub-module, the second sub-module and the third sub-module, and obtain the recognition results.

[0053] A multimodal recognition model is used to recognize the preprocessed logistics document images, and the recognition results are obtained, including:

[0054] A convolutional neural network and a long short-term memory network architecture are constructed. The feature information of the first mode is extracted by the convolutional neural network, and the long short-term memory network is used for temporal modeling. Based on the historical feature information of the first mode, the recognition result of the first mode is obtained. The convolutional neural network is a residual neural network.

[0055] The feature information of the second modality is extracted using OCR, and then combined with an attention mechanism to perform semantic verification on the feature information of the second modality, thereby obtaining the recognition result of the second modality.

[0056] The third modality is labeled using an object detection algorithm, and its shape feature information is extracted to obtain the recognition result of the third modality.

[0057] For example, the weighbridge image is segmented into sub-modules such as a header area, a table area, and a signature area. Multiple threads are then used to process the company name in the header area, the weight of the goods in the table area, and the signature in the signature area, respectively. Simultaneously, text and symbols in the weighbridge image are recognized. In this embodiment, the first modality can be handwritten text, the second modality can be printed text, and the third modality can be symbolic graphics. For handwritten text, a deep learning network model is invoked. A convolutional neural network (CNN) extracts features such as stroke curvature and continuous strokes of the handwritten characters. These features are then used for temporal modeling via a recurrent neural network (RNN) or a long short-term memory network (LSTM). The model is compared against a historical handwritten sample database to recognize handwritten signatures and temporary annotations on the weighbridge, yielding the handwritten text recognition result. For printed text, an OCR engine and attention mechanism are combined to perform semantic verification. The OCR engine can also be combined with other image analysis algorithms (such as template matching-based correction algorithms and deep learning-based error detection algorithms) to optimize the recognition results. For symbols and graphics, special symbols in the bill image (such as the seal across the contract on the bill) are located using object detection algorithms (such as YOLOv8), and shape features (such as contour matching of circular seals) are extracted to fully extract non-textual information.

[0058] The preprocessed logistics document image is recognized using a multimodal recognition model. An independent thread is started to synchronously process the recognition tasks of the first, second and third sub-modules. After obtaining the recognition result, the following steps are also taken: the confidence of the recognition result is evaluated. When the confidence of the recognition result is lower than the preset confidence, a second recognition operation is triggered.

[0059] During the identification process, the confidence level of the identification results is assessed, generating a confidence score of 0-1 for each result. The preset confidence level can be 0.9. When the confidence score of the identification result is greater than or equal to 0.9, it is considered reliable. When it is lower than the preset confidence level, a second identification operation is automatically triggered to ensure data accuracy. Furthermore, for questionable fields, the intelligent module of the first platform uses natural language interaction to prompt the user for confirmation, such as "The weighbridge weight detected is '5.68 tons', please confirm whether it is '5.68 tons'".

[0060] Step S3. The inference model classifies the identification results based on the business rule base to obtain business keywords, and generates ledger information based on the ledger template and the business keywords.

[0061] The inference model classifies the identification results based on a business rule base to obtain business keywords. Based on the ledger template and the business keywords, it generates ledger information, including:

[0062] The first sub-module of the logistics document image is processed using a multimodal recognition model to obtain the logistics document category;

[0063] The inference model retrieves matching business rules from the business rule base based on the logistics document category to obtain business keywords corresponding to the business requirements.

[0064] The business requirements are analyzed based on the first natural language instruction information, and the inference model generates a custom ledger template or matches a preset ledger template based on the business requirements.

[0065] The business keywords are combined with the custom ledger template or the preset ledger template to generate ledger information.

[0066] This embodiment includes pre-processing: a multimodal recognition model processes the header area of ​​the logistics document image to obtain the logistics document category (such as weighbridge slip, invoice, contract, etc.). By recognizing the header information of the logistics document, the corresponding processing flow can be automatically matched.

[0067] The multimodal recognition model transmits the recognition results to the inference model, which then classifies the data based on the built-in business rule library and ledger template library.

[0068] Users can input natural language commands, and the inference model parses these commands to obtain business requirements. Based on a business rule base, it matches corresponding rules for different logistics document categories. Specifically, for weighbridge slips, they are categorized according to cargo type (e.g., coal / steel), transport vehicle, weighing time, customer, etc., and the "weight-unit price-total price" calculation formula (total price = gross weight - tare weight × unit price) is called; for invoices, tax rules are matched based on tax number and commodity number to automatically calculate the tax amount (e.g., VAT = amount × tax rate); for contracts, key clauses such as delivery date and liability for breach of contract are extracted, compared with preset contract template specifications, and missing items are marked.

[0069] For standardized logistics documents, templates from the built-in template library are directly called to fill the recognition results into the corresponding cells. This embodiment also supports custom templates. The inference model receives user natural language instructions, automatically parses business requirements, and automatically associates business rules (such as calculation formulas) in the business rule library with field mapping relationships to form a complete ledger report.

[0070] For example, inputting the natural language command "Generate this week's ledger by customer category" triggers the inference model to parse the business requirements. First, it categorizes the data according to dimensions such as cargo type, transport vehicle, weighing time, and customer. Then, it performs calculations based on preset formulas, such as calculating the total weight and total price of the goods. The inference model generates a template based on the business requirements parsed from the natural language, combining the fields "'Customer Name' + 'Cargo Type' + 'Transport Quantity'" from the template library to generate a ledger report (e.g., Excel, PDF).

[0071] The inference model in this embodiment has a built-in rule base and template base, which supports dynamic configuration of fields and logical rules for natural language instructions, and improves template adaptation efficiency to 90%.

[0072] This invention combines a multimodal recognition model and an inference model to form a unique collaborative working mode. Compared with the single OCR recognition of existing technologies, the data entry accuracy of this invention is as high as 99% or more, effectively avoiding problems caused by logistics document quality issues and technical limitations, ensuring the reliability and authenticity of logistics data, and realizing the fully automated processing of logistics documents from data recognition and ledger generation to system entry. It automatically parses requirements through natural language interaction, greatly reducing operation time.

[0073] Step S4. Monitor the orchestration and scheduling of the multimodal recognition model and the inference model. The first platform adjusts the execution order and optimizes the allocation path based on the orchestration and scheduling. The first platform includes an intelligent module, which is used to parse natural language.

[0074] A monitoring module is deployed to monitor the operational status and performance in real time, including the recognition accuracy of the multimodal model, the processing speed of the inference model, system resource consumption, and the response accuracy of natural language interaction. Simultaneously, the system monitors the orchestration and scheduling of each model by the primary platform, analyzing metrics such as data transmission efficiency and the rationality of task execution order during model concatenation. By collecting and analyzing operational data, the multimodal recognition model, inference model, and natural language processing module are periodically optimized and trained. For example, the training sample set of the multimodal recognition model is updated and model parameters are adjusted based on newly emerging logistics document formats and data types; the logical algorithm of the inference model is optimized based on changes in business rules; the corpus and semantic understanding model of the natural language processing module are expanded based on the diversity of user natural language commands; and the scheduling strategy of the primary platform is optimized based on problems discovered during model orchestration, ensuring that the system always maintains a high-efficiency and accurate operating state, adapting to the continuous development and changes in logistics operations.

[0075] This invention utilizes the model orchestration and scheduling capabilities of an artificial intelligence platform to connect different large models, achieving multimodal collaboration. The artificial intelligence platform includes functional modules such as data storage, algorithm computation, and interface calls. During model collaboration, based on the requirements of each stage of logistics document entry, different types of large models, such as multimodal recognition models and inference models, are intelligently orchestrated and connected to ensure orderly collaboration among models, achieving efficient data processing and system integration. This invention also utilizes the artificial intelligence platform to develop an intelligent agent, which serves as the core execution unit responsible for the intelligent operation of the entire logistics document entry process.

[0076] This invention utilizes the weights of a trained weighbridge identification model and leverages the platform's model orchestration capabilities to quickly adapt to new logistics document categories (such as invoices, contracts, etc.) through template adaptation and transfer learning, reducing the training cycle and rapidly enabling the expansion of multiple document categories.

[0077] Step S5. Encrypt the generated ledger and then connect the encrypted ledger to the business system via an interface to complete the data entry.

[0078] Sensitive data (such as invoice amounts and contract amounts) is encrypted using AES-256 to ensure data transmission security. After receiving the data, the business system performs hash value verification to ensure that the data has not been tampered with.

[0079] The method of the present invention further includes:

[0080] The received ledgers are verified according to the business rule base;

[0081] If the verification passes, the ledger information will be stored in the database;

[0082] If the verification fails, the information is sent to the first platform. The intelligent module of the first platform uses natural language to interact with the user to confirm the correction method of the ledger based on the error message. After correcting the ledger, it is re-entered into the business system.

[0083] If the verification fails, the process also includes: marking abnormal logistics documents according to the business rule base and generating pending items for the approval process.

[0084] Data is validated based on a business rule base. For example, according to the business rules of the calculation formula, "gross weight - tare weight" in the weighbridge slip should equal "net weight", and "quantity × unit price" in the invoice should equal "amount". The system compares the current data with historical data to see if the change in the current weighbridge weight is within a preset range. If the data validation fails, an anomaly flag is triggered and pushed to the user for confirmation. The system then interacts with the user through natural language to confirm the ledger information.

[0085] like Figure 2 As shown in the figure, this invention discloses an intelligent data entry system for logistics documents based on multi-model collaboration, the structure of which includes:

[0086] The preprocessing module is used to acquire logistics document images through image acquisition devices and preprocess the logistics document images. The logistics document images include a first submodule, a second submodule, and a third submodule.

[0087] The recognition module is used to recognize the preprocessed logistics document images using a multimodal recognition model. It starts an independent thread to synchronously process the recognition tasks of the first sub-module, the second sub-module, and the third sub-module to obtain the recognition results.

[0088] The reasoning module is used to classify the recognition results based on the business rule base of the reasoning model, obtain business keywords, and generate ledger information based on the ledger template and the business keywords;

[0089] The scheduling module is used to monitor the orchestration and scheduling of the multimodal recognition model and the inference model. The first platform adjusts the execution order and optimizes the allocation path based on the orchestration and scheduling. The first platform includes an intelligent module, which is used to parse natural language.

[0090] The data entry module is used to encrypt the generated ledgers and then connect the encrypted ledgers to the business system via an interface to complete the data entry.

[0091] By using this system, the aforementioned computational processing methods can be executed and the corresponding technical effects can be achieved.

[0092] Embodiments of the present invention also provide a computer-readable storage medium capable of implementing all the steps of the methods in the above embodiments, wherein the computer-readable storage medium stores a computer program that, when executed by a processor, implements all the steps of the methods in the above embodiments.

[0093] Embodiments of the present invention also provide an electronic device for performing the above-described method, which, as an implementation apparatus for the method, has the following structure. Figure 3 As shown, the electronic device 300 includes a processor 301 and a memory 303. The processor 301 and the memory 303 are connected, for example, via a bus 302. Optionally, the electronic device 300 may also include a transceiver 304. It should be noted that in practical applications, the transceiver 304 is not limited to one. It is understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on the specific structure of the electronic device 300. In other embodiments of this application, the electronic device 300 may include more or fewer components than illustrated, or combine some components, or split some components, or arrange different components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware. Optionally, the electronic device may also include a display screen 305 for displaying images or receiving user operation commands when needed.

[0094] In this embodiment, processor 301 is used to implement the method shown in the above method embodiment. Transceiver 304 may include a receiver and a transmitter. Transceiver 304 is used in this embodiment to enable the electronic device of this embodiment to communicate with other devices during execution.

[0095] Processor 301 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 301 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0096] Processor 301 may also include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. The controller may serve as the central nervous system and command center of the electronic device 300. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution. Processor 301 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 301 is a cache memory. This memory can store instructions or data that processor 301 has just used or is reusing. If processor 301 needs to reuse an instruction or data, it can directly retrieve it from the memory. This avoids repeated access, reduces the waiting time of processor 301, and thus improves system efficiency.

[0097] The processor 301 can run the methods provided in the embodiments of this application. The processor 301 may include different devices. For example, when integrating a CPU and a GPU, the CPU and GPU can cooperate to execute the methods provided in the embodiments of this application. For example, some algorithms in the method are executed by the CPU, and other algorithms are executed by the GPU to obtain faster processing efficiency.

[0098] Bus 302 may include a pathway for transmitting information between the aforementioned components. Bus 302 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 302 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0099] The memory 303 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory), or high-speed random access memory. It may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), or other optical disc storage, optical disk storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0100] Optionally, the memory 303 is used to store application code that executes the scheme of this application, and the execution is controlled by the processor 301. The processor 301 is used to execute the application code stored in the memory 303 to implement the method provided in any embodiment of this application.

[0101] The memory 303 can be used to store computer executable program code, which includes instructions. The processor 301 executes various functional applications and data processing of the electronic device 300 by running the instructions stored in the memory 303. The memory 303 may include a program storage area and a data storage area. The program storage area can store the operating system, application code, etc. The data storage area can store data created during the use of the electronic device 300 (such as images and videos captured by a camera application).

[0102] The memory 303 may also store one or more computer programs corresponding to the methods provided in the embodiments of this application. The one or more computer programs are stored in the memory 303 and configured to be executed by the one or more processors 301. The one or more computer programs include instructions that can be used to perform the various steps in the corresponding embodiments described above.

[0103] Of course, the code for the method provided in this application embodiment can also be stored in external memory. In this case, the processor 301 can run the code for the method stored in external memory through the external memory interface, and the processor 301 can control the execution flow.

[0104] The display screen 305 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a minimized LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 300 may include one or N displays 305, where N is a positive integer greater than 1. The display screen 305 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces (GUIs). For example, the display screen 305 can display photos, videos, web pages, or documents.

[0105] The electronic device provided in this application is applicable to any of the above-described methods. Therefore, the beneficial effects it can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0106] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0107] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.

[0108] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction set implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0109] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the functions specified in one or more boxes. Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0110] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for intelligent entry of logistics documents based on multi-model collaboration, characterized in that, include: S1. Acquire logistics document images through image acquisition devices, and preprocess the logistics document images, wherein the logistics document images include a first sub-module, a second sub-module and a third sub-module; S2. Use a multimodal recognition model to recognize the preprocessed logistics document image, start an independent thread to synchronously process the recognition tasks of the first submodule, the second submodule and the third submodule, and obtain the recognition results; S3. The reasoning model classifies the identification results based on the business rule base to obtain business keywords, and generates ledger information based on the ledger template and the business keywords; S4. Monitor the orchestration and scheduling of the multimodal recognition model and the inference model. The first platform adjusts the execution order and optimizes the allocation path based on the orchestration and scheduling. The first platform includes an intelligent module, which is used to parse natural language. S5. Encrypt the generated ledger and then connect the encrypted ledger to the business system via an interface to complete the data entry.

2. The method as described in claim 1, characterized in that, The preprocessing of the logistics document image includes image enhancement, noise reduction, and tilt correction operations.

3. The method as described in claim 1, characterized in that, A multimodal recognition model is used to recognize the preprocessed logistics document images, and the recognition results are obtained, including: A convolutional neural network and a long short-term memory network architecture are constructed. The feature information of the first mode is extracted by the convolutional neural network, and the long short-term memory network is used for temporal modeling. Based on the historical feature information of the first mode, the recognition result of the first mode is obtained. The convolutional neural network is a residual neural network. The feature information of the second modality is extracted using OCR, and then combined with an attention mechanism to perform semantic verification on the feature information of the second modality, thereby obtaining the recognition result of the second modality. The third modality is labeled using an object detection algorithm, and its shape feature information is extracted to obtain the recognition result of the third modality.

4. The method as described in claim 3, characterized in that, The first modality is handwritten text, the second modality is printed text, and the third modality is a symbolic graphic.

5. The method as described in claim 1, characterized in that, The process of using a multimodal recognition model to recognize the preprocessed logistics document image involves starting an independent thread to synchronously process the recognition tasks of the first, second, and third sub-modules. After obtaining the recognition results, the process further includes: The confidence level of the recognition result is evaluated. When the confidence level of the recognition result is lower than the preset confidence level, a second recognition operation is triggered.

6. The method as described in claim 1, characterized in that, The inference model classifies the identification results based on a business rule base to obtain business keywords, and generates ledger information based on the ledger template and the business keywords, including: The first sub-module of the logistics document image is processed using a multimodal recognition model to obtain the logistics document category; The inference model retrieves matching business rules from the business rule base based on the logistics document category to obtain business keywords corresponding to the business requirements. The business requirements are analyzed based on the first natural language instruction information, and the inference model generates a custom ledger template or matches a preset ledger template based on the business requirements. The business keywords are combined with the custom ledger template or the preset ledger template to generate ledger information.

7. The method as described in claim 6, characterized in that, The method further includes: The received ledgers are verified according to the business rule base; If the verification passes, the ledger information will be stored in the database; If the verification fails, the information is sent to the first platform. The intelligent module of the first platform uses natural language to interact with the user to confirm the correction method of the ledger based on the error message. After correcting the ledger, it is re-entered into the business system. If the verification fails, the process also includes: marking abnormal logistics documents according to the business rule base and generating pending items for the approval process.

8. A logistics document intelligent entry system based on multi-model collaboration, characterized in that, include: The preprocessing module is used to acquire logistics document images through image acquisition devices and preprocess the logistics document images. The logistics document images include a first submodule, a second submodule, and a third submodule. The recognition module is used to recognize the preprocessed logistics document images using a multimodal recognition model. It starts an independent thread to synchronously process the recognition tasks of the first sub-module, the second sub-module, and the third sub-module to obtain the recognition results. The reasoning module is used to classify the recognition results based on the business rule base of the reasoning model, obtain business keywords, and generate ledger information based on the ledger template and the business keywords; The scheduling module is used to monitor the orchestration and scheduling of the multimodal recognition model and the inference model. The first platform adjusts the execution order and optimizes the allocation path based on the orchestration and scheduling. The first platform includes an intelligent module, which is used to parse natural language. The data entry module is used to encrypt the generated ledgers and then connect the encrypted ledgers to the business system via an interface to complete the data entry.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method of any one of claims 1 to 7.

10. An electronic device, characterized in that, Including processor and memory; The memory is used to store program instructions; The processor is configured to execute the method of any one of claims 1 to 7 by invoking program instructions.

11. A computer program product, comprising a computer program and / or instructions, characterized in that, When the computer program and / or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 7.

Images