Multi-modal model-based reimbursement process optimization method and device

Abstract

The invention discloses a reimbursement process optimization method and device based on a multi-modal model. The method comprises the following steps: receiving a to-be-reimbursed material submitted by a target object; performing image recognition and text extraction on the to-be-reimbursed material to obtain document classification information and reimbursement text information of the to-be-reimbursed material; determining a reimbursement type based on the information by using a reimbursement classification model; comparing the feature information of the target object and the above information with fields in the reimbursement application form, and if missing information does not exist, directly generating a target reimbursement application form according to the above information; otherwise, filling the missing information submitted by the target object into the reimbursement application form to generate a target reimbursement application form; and submitting the target reimbursement application form to enter a reimbursement process. The technical problem that the efficiency and accuracy of the reimbursement process are low due to the fact that the traditional financial reimbursement process generally depends on manual operation, the operation process is tedious and time-consuming, and errors are prone to occurring in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of enterprise reimbursement, and in particular to a method and device for optimizing a reimbursement process based on a multimodal model. Background Art

[0002] In the traditional financial reimbursement process, it is usually carried out by manually filling out reimbursement forms, manually checking invoices, and reviewing and approving. In recent years, with the development of artificial intelligence and data processing technology, some intelligent financial reimbursement systems based on rule engines, machine learning and other technologies have also begun to appear. These systems can automatically identify invoice information, classify expenses, manage approval processes, etc., but there are still problems such as low recognition accuracy and limited ability to handle complex situations; the cumbersome, time-consuming, and error-prone problems in the traditional financial reimbursement process have seriously affected the financial efficiency and management level of the company. The traditional financial reimbursement process often relies on manual operations, which are easily affected by individual ability and subjective consciousness, resulting in many problems such as inefficient processes, lax review, and prone to errors.

[0003] In view of the problem that the traditional financial reimbursement process in the above-mentioned related technologies usually relies on manual operation, the operation process is not only cumbersome and time-consuming, but also prone to errors, resulting in low efficiency and accuracy of the reimbursement process. No effective solution has been proposed so far. Summary of the invention

[0004] The embodiments of the present invention provide a method and device for optimizing the reimbursement process based on a multimodal model, so as to at least solve the technical problem that the traditional financial reimbursement process in the related technology usually relies on manual operation, the operation process is not only cumbersome and time-consuming, but also prone to errors, resulting in low efficiency and accuracy of the reimbursement process.

[0005] According to one aspect of an embodiment of the present invention, a method for optimizing a reimbursement process based on a multimodal model is provided, comprising: receiving materials to be reimbursed submitted by a target object, wherein the target object is an object that makes a reimbursement request, and the materials to be reimbursed refer to certification materials required for reimbursement; performing image recognition and text extraction on the materials to be reimbursed to obtain document classification information and reimbursement text information of the materials to be reimbursed, wherein the document classification information refers to information on the type of reimbursement voucher in the materials to be reimbursed, and the reimbursement text information refers to text information extracted from the reimbursement voucher; combining the document classification information and the reimbursement text information; The reimbursement text information is input into a reimbursement classification model, so as to use the reimbursement classification model to determine the reimbursement type of the material to be reimbursed based on the document classification information and the reimbursement text information, wherein the reimbursement classification model is trained by deep learning using multiple groups of first training data, and each of the multiple groups of first training data includes: sample document classification information, sample reimbursement text information, and sample reimbursement types corresponding to the sample document classification information and the sample reimbursement text information; the feature information of the target object, the document classification information, the reimbursement text information and the reimbursement type are compared with The fields in the reimbursement application form are compared to obtain a comparison result, wherein the characteristic information is information obtained by searching in a database according to the identifier of the target object, the reimbursement application form refers to a reimbursement form template, and the fields represent the reimbursement information that the target object needs to provide when making the reimbursement request; if the comparison result indicates that there is no missing information, the characteristic information, the document classification information, the reimbursement text information and the reimbursement type are filled into the reimbursement application form to generate a target reimbursement application form corresponding to the material to be reimbursed, wherein the missing information refers to the reimbursement information corresponding to the target field. reimbursement information, the target field refers to the field in the reimbursement application form that does not match the reimbursement information; when the comparison result indicates that there is missing information, after receiving the missing information submitted by the target object according to the prompt information, the feature information, the document classification information, the reimbursement text information, the reimbursement type and the missing information are filled in the reimbursement application form to generate the target reimbursement application form corresponding to the material to be reimbursed, wherein the prompt information is used to prompt the target object to submit the prompt information; submit the target reimbursement application form to enter the reimbursement process.

[0006] Optionally, image recognition and text extraction are performed on the materials to be reimbursed to obtain document classification information and reimbursement text information of the materials to be reimbursed, including: using a multimodal image classification model to identify and process the image information in the materials to be reimbursed to obtain the document classification information, wherein the image information is the information corresponding to the reimbursement voucher, and the multimodal image classification model is obtained by training the multimodal model through deep learning using multiple groups of second training data, and each of the multiple groups of second training data includes: sample image information, sample document classification information corresponding to the sample image information; using text recognition technology to identify and extract the text information in the image information, and determine that the extracted text information is the reimbursement text information of the materials to be reimbursed.

[0007] Optionally, before inputting the document classification information and the reimbursement text information into the reimbursement classification model so as to use the reimbursement classification model to determine the reimbursement type of the material to be reimbursed based on the document classification information and the reimbursement text information, the reimbursement process optimization method based on the multimodal model also includes: obtaining multiple groups of sample reimbursement information, wherein the sample reimbursement information includes: the sample document classification information, the sample reimbursement text information, and the sample reimbursement type corresponding to the sample document classification information and the sample reimbursement text information, and the sample document classification information is information obtained by performing image recognition on the image information in the sample material to be reimbursed by the multimodal image classification model; dividing the multiple groups of the sample reimbursement information to obtain sample training reimbursement information and sample test reimbursement information, wherein the sample training reimbursement information is used to train the reimbursement classification model. model, the sample test reimbursement information is used to test the accuracy of the reimbursement classification model; the natural language processing model containing the classifier is iteratively trained using the sample training reimbursement information until the iteration termination condition is reached, and the model obtained by the current training is determined to be the initial reimbursement classification model, wherein the classifier is located at the top of the natural language processing model, and the iteration termination condition at least includes: the number of iterations reaches a preset number of iterations, and the classification error does not exceed the error threshold; the initial reimbursement classification model is tested using the sample test reimbursement information to obtain a test result; when the test result indicates that the initial reimbursement classification model meets the accuracy requirement, the initial reimbursement classification model is determined to be the reimbursement classification model, wherein the meeting of the accuracy requirement means that the classification error of the initial reimbursement classification model does not exceed the error threshold.

[0008] Optionally, the characteristic information of the target object, the document classification information, the reimbursement text information and the reimbursement type are compared with the fields in the reimbursement application form to obtain a comparison result, including: obtaining the reimbursement information corresponding to each of the fields in the reimbursement application form; determining the required fields and non-required fields in the reimbursement application form, wherein the required fields refer to the fields that must be filled in to generate the target reimbursement application form, and the non-required fields refer to the fields that are not required to be filled in to generate the target reimbursement application form; matching the characteristic information of the target object, the document classification information, the reimbursement text information and the reimbursement type with the reimbursement information corresponding to each of the fields to obtain a matching result; when the matching result indicates that all of the required fields in the reimbursement application form are matched to the corresponding reimbursement information, determining that the comparison result is that the missing information does not exist; when the matching result indicates that at least one of the required fields in the reimbursement application form is not matched to the corresponding reimbursement information, determining that the comparison result is that the missing information exists.

[0009] Optionally, when the comparison result indicates the existence of the missing information, after receiving the missing information submitted by the target object according to the prompt information, the characteristic information, the document classification information, the reimbursement text information, the reimbursement type and the missing information are filled in the reimbursement application form to generate the target reimbursement application form corresponding to the material to be reimbursed, including: when the comparison result indicates the existence of the missing information, determining that the field in the reimbursement application form that is not matched with the reimbursement information is the target field; determining that the reimbursement information corresponding to the target field is the missing information; generating a prompt information according to the missing information to prompt the target object to submit the missing information according to the prompt information; after receiving the missing information submitted by the target object, filling in the characteristic information, the document classification information, the reimbursement text information, the reimbursement type and the missing information into the reimbursement application form to generate the target reimbursement application form corresponding to the material to be reimbursed.

[0010] Optionally, submitting the target reimbursement application form to enter the reimbursement process includes: pre-reviewing the target reimbursement application form according to preset rules to obtain a pre-review result, wherein the preset rules are rules pre-set according to reimbursement requirements; when the pre-review result indicates that the target reimbursement application form has passed the pre-review, sending the target reimbursement application form to the front end so that the target object can review the target reimbursement application form; and when receiving a submission request information, submitting the target reimbursement application form to enter the reimbursement process, wherein the submission request information refers to a submission request initiated by the target object after reviewing the target reimbursement application form.

[0011] Optionally, the reimbursement process optimization method based on a multimodal model also includes: in the process of generating the target reimbursement application form corresponding to the material to be reimbursed, optimizing the reimbursement classification model and the multimodal image classification model using the material to be reimbursed.

[0012] According to another aspect of an embodiment of the present invention, there is also provided a reimbursement process optimization device based on a multimodal model, comprising: a receiving unit, used to receive materials to be reimbursed submitted by a target object, wherein the target object is the object that makes a reimbursement request, and the materials to be reimbursed refer to the certification materials required for reimbursement; a first acquisition unit, used to perform image recognition and text extraction on the materials to be reimbursed, and obtain document classification information and reimbursement text information of the materials to be reimbursed, wherein the document classification information refers to information on the type of reimbursement voucher in the materials to be reimbursed, and the reimbursement text information refers to text information extracted from the reimbursement voucher; a first determination unit, used to The document classification information and the reimbursement text information are input into a reimbursement classification model, so as to utilize the reimbursement classification model to determine the reimbursement type of the material to be reimbursed based on the document classification information and the reimbursement text information, wherein the reimbursement classification model is trained by deep learning using multiple sets of first training data, each of the multiple sets of first training data comprising: sample document classification information, sample reimbursement text information, and sample reimbursement types corresponding to the sample document classification information and the sample reimbursement text information; a second acquisition unit is used to obtain the feature information of the target object, the document classification information, the reimbursement text information and the sample reimbursement text information The reimbursement type is compared with the fields in the reimbursement application form to obtain a comparison result, wherein the characteristic information is information obtained by searching in a database according to the identification of the target object, the reimbursement application form refers to a reimbursement form template, and the fields represent the reimbursement information that the target object needs to provide when making the reimbursement request; a first generating unit is used to fill the characteristic information, the document classification information, the reimbursement text information and the reimbursement type into the reimbursement application form when the comparison result indicates that there is no missing information, so as to generate a target reimbursement application form corresponding to the material to be reimbursed, wherein the missing information refers to the reimbursement corresponding to the target field. information, the target field refers to the field in the reimbursement application form that does not match the reimbursement information; a second generating unit is used to fill in the feature information, the document classification information, the reimbursement text information, the reimbursement type and the missing information into the reimbursement application form after receiving the missing information submitted by the target object according to the prompt information when the comparison result indicates that the missing information exists, so as to generate the target reimbursement application form corresponding to the material to be reimbursed, wherein the prompt information is used to prompt the target object to submit the prompt information; a submitting unit is used to submit the target reimbursement application form to enter the reimbursement process.

[0013] Optionally, the first acquisition unit includes: a first acquisition module, used to use a multimodal image classification model to identify and process the image information in the material to be reimbursed to obtain the document classification information, wherein the image information is the information corresponding to the reimbursement voucher, and the multimodal image classification model is obtained by training the multimodal model through deep learning using multiple groups of second training data, and each of the multiple groups of second training data includes: sample image information, sample document classification information corresponding to the sample image information; an extraction module, used to use text recognition technology to identify and extract the text information in the image information, and determine that the extracted text information is the reimbursement text information of the material to be reimbursed.

[0014] Optionally, the reimbursement process optimization device based on a multimodal model also includes: a third acquisition unit, used to acquire multiple groups of sample reimbursement information before inputting the document classification information and the reimbursement text information into the reimbursement classification model to determine the reimbursement type of the material to be reimbursed based on the document classification information and the reimbursement text information using the reimbursement classification model, wherein the sample reimbursement information includes: the sample document classification information, the sample reimbursement text information, and the sample reimbursement type corresponding to the sample document classification information and the sample reimbursement text information, and the sample document classification information is information obtained by performing image recognition on the image information in the sample material to be reimbursed by the multimodal image classification model; a fourth acquisition unit, used to divide the multiple groups of sample reimbursement information to obtain sample training reimbursement information and sample test reimbursement information, wherein the sample training reimbursement information is used to train the reimbursement classification model. The sample test reimbursement information is used to test the accuracy of the reimbursement classification model; the second determination unit is used to use the sample training reimbursement information to iteratively train the natural language processing model containing the classifier until the iteration termination condition is reached, and determine that the model obtained by the current training is the initial reimbursement classification model, wherein the classifier is located at the top of the natural language processing model, and the iteration termination condition at least includes: the number of iterations reaches a preset number of iterations, and the classification error does not exceed the error threshold; the fifth acquisition unit is used to use the sample test reimbursement information to test the initial reimbursement classification model to obtain a test result; the third determination unit is used to determine that the initial reimbursement classification model is the reimbursement classification model when the test result indicates that the initial reimbursement classification model meets the accuracy requirement, wherein the meeting the accuracy requirement means that the classification error of the initial reimbursement classification model does not exceed the error threshold.

[0015] Optionally, the second acquisition unit includes: a second acquisition module, used to acquire the reimbursement information corresponding to each of the fields in the reimbursement application form; a first determination module, used to determine the required fields and non-required fields in the reimbursement application form, wherein the required fields refer to the fields that must be filled in to generate the target reimbursement application form, and the non-required fields refer to the fields that are not required to be filled in to generate the target reimbursement application form; a third acquisition module, used to match the characteristic information of the target object, the document classification information, the reimbursement text information and the reimbursement type with the reimbursement information corresponding to each of the fields to obtain a matching result; a second determination module, used to determine that the comparison result is that the missing information does not exist when the matching result indicates that all of the required fields in the reimbursement application form are matched to the corresponding reimbursement information; a third determination module, used to determine that the comparison result is that the missing information exists when the matching result indicates that at least one of the required fields in the reimbursement application form is not matched to the corresponding reimbursement information.

[0016] Optionally, the second generating unit includes: a fourth determining module, used to determine that the field in the reimbursement application form that does not match the reimbursement information is the target field when the comparison result indicates the existence of the missing information; a fifth determining module, used to determine that the reimbursement information corresponding to the target field is the missing information; a first generating module, used to generate prompt information according to the missing information to prompt the target object to submit the missing information according to the prompt information; and a second generating module, used to fill in the characteristic information, the document classification information, the reimbursement text information, the reimbursement type and the missing information into the reimbursement application form after receiving the missing information submitted by the target object, so as to generate the target reimbursement application form corresponding to the material to be reimbursed.

[0017] Optionally, the submission unit includes: a fourth acquisition module, used to pre-review the target reimbursement application form according to preset rules to obtain a pre-review result, wherein the preset rules are rules pre-set according to reimbursement requirements; a sending module, used to send the target reimbursement application form to the front end when the pre-review result indicates that the target reimbursement application form has passed the pre-review, so that the target object can review the target reimbursement application form; a submission module, used to submit the target reimbursement application form when receiving a submission request information to enter the reimbursement process, wherein the submission request information refers to a submission request initiated by the target object after reviewing the target reimbursement application form.

[0018] Optionally, the reimbursement process optimization device based on a multimodal model also includes: an optimization unit, which is used to optimize the reimbursement classification model and the multimodal image classification model using the materials to be reimbursed in the process of generating the target reimbursement application form corresponding to the materials to be reimbursed.

[0019] According to another aspect of an embodiment of the present invention, a reimbursement process optimization system based on a multimodal model is further provided. The reimbursement process optimization system based on a multimodal model uses any of the above-mentioned reimbursement process optimization methods based on a multimodal model.

[0020] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium includes a stored program, wherein the program executes any one of the above-mentioned reimbursement process optimization methods based on a multimodal model.

[0021] According to another aspect of an embodiment of the present invention, a processor is further provided, wherein the processor is used to run a program, wherein when the program is run, any one of the above-mentioned reimbursement process optimization methods based on a multimodal model is executed.

[0022] According to another aspect of an embodiment of the present invention, a computer program product is also provided, comprising computer instructions, which, when executed by a processor, execute any one of the above-mentioned methods for optimizing the reimbursement process based on a multimodal model.

[0023] In an embodiment of the present invention, materials to be reimbursed submitted by a target object are received, wherein the target object is the object that makes a reimbursement request, and the materials to be reimbursed refer to the certification materials required for reimbursement; image recognition and text extraction are performed on the materials to be reimbursed to obtain document classification information and reimbursement text information of the materials to be reimbursed, wherein the document classification information refers to the information on the type of reimbursement voucher in the materials to be reimbursed, and the reimbursement text information refers to the text information extracted from the reimbursement voucher; the document classification information and the reimbursement text information are input into a reimbursement classification model, so as to use the reimbursement classification model to determine the reimbursement type of the materials to be reimbursed based on the document classification information and the reimbursement text information, wherein the reimbursement classification model is trained by deep learning using multiple groups of first training data, and each of the multiple groups of first training data includes: sample document classification information, sample reimbursement text information, and sample reimbursement types corresponding to the sample document classification information and the sample reimbursement text information; the feature information, document classification information, reimbursement text information and reimbursement type of the target object are compared with the reimbursement The fields in the application form are compared to obtain a comparison result, wherein the characteristic information is the information obtained by searching in the database according to the identification of the target object, the reimbursement application form refers to the reimbursement form template, and the fields represent the reimbursement information that the target object needs to provide when making a reimbursement request; when the comparison result indicates that there is no missing information, the characteristic information, document classification information, reimbursement text information and reimbursement type are filled into the reimbursement application form to generate a target reimbursement application form corresponding to the material to be reimbursed, wherein the missing information refers to the reimbursement information corresponding to the target field, and the target field refers to the field in the reimbursement application form that is not matched with the reimbursement information; when the comparison result indicates that there is missing information, after receiving the missing information submitted by the target object according to the prompt information, the characteristic information, document classification information, reimbursement text information, reimbursement type and missing information are filled into the reimbursement application form to generate a target reimbursement application form corresponding to the material to be reimbursed, wherein the prompt information is used to prompt the target object to submit the prompt information; submit the target reimbursement application form to enter the reimbursement process. Through the above technical scheme, the purpose of using the trained model to identify and analyze the materials to be reimbursed submitted by the user from multiple angles and automatically generating the corresponding reimbursement form according to the materials submitted by the user is achieved. It is realized that the user only needs to submit the relevant reimbursement materials, and there is no need to fill out the reimbursement form manually. The model can be used for identification and analysis, and the reimbursement form can be automatically generated in the background according to the materials submitted by the user and the analysis results. The technical effect simplifies the reimbursement process and improves the accuracy and efficiency of filling out the reimbursement form, thereby solving the technical problem that the traditional financial reimbursement process in the relevant technology usually relies on manual operation, and the operation process is not only cumbersome and time-consuming, but also prone to errors, resulting in low efficiency and accuracy of the reimbursement process. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0025] Figure 1 It is a hardware structure block diagram of a mobile terminal of a reimbursement process optimization method based on a multimodal model according to an embodiment of the present invention;

[0026] Figure 2 is a flowchart of a method for optimizing a reimbursement process based on a multimodal model according to an embodiment of the present invention;

[0027] Figure 3 is a flowchart of an optional reimbursement process optimization method based on a multimodal model according to an embodiment of the present invention;

[0028] Figure 4 is a schematic diagram of a reimbursement process optimization device based on a multimodal model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] The following are explanations of the specific terms or terms involved in the embodiments of the present invention:

[0032] Multi-modal Model: refers to a deep learning model that combines different types of data (such as text, images, voice, etc.) and processes and analyzes the data together to improve the accuracy and robustness of the model. It is often used to process complex information or situations.

[0033] Pretrained model fine-tuning: refers to performing additional training on a model that has been pre-trained with a large-scale dataset using a task-specific dataset to adapt the model to a specific task or field.

[0034] Financial Reporting: refers to the procedures and norms used within an organization to reimburse expenses, review and approve related processes, and the process by which employees apply for reimbursement for expenses incurred during work.

[0035] OCR (Optical Character Recognition): is an optical character recognition technology used to convert printed or handwritten text into an editable and searchable digital text format. OCR systems typically use image processing and pattern recognition technology to identify characters in text and convert them into computer-understandable codes, such as ASCII or Unicode. This technology is often used in scenarios such as digitizing documents, automated data input, and recognizing printed text.

[0036] CLIP (Contrastive Language-Image Pre-Training): is a multimodal learning method that aims to learn the relationship between vision and language by jointly training image and text data. The CLIP model uses a contrastive learning approach to pre-train by maximizing the similarity between related image and text pairs. This enables the model to understand the semantic connection between images and text, thereby demonstrating good generalization capabilities on a variety of vision and language tasks.

[0037] BERT (Bidirectional Encoder Representations from Transformers): is a natural language processing model that uses the Transformer architecture to perform unsupervised pre-training on large-scale text corpora to learn the semantics and contextual information in text. The BERT model uses bidirectional contextual information to represent the meaning of words, thus achieving outstanding performance in various natural language processing tasks, such as text classification, named entity recognition, and semantic similarity calculation.

[0038] As introduced in the background technology, the traditional financial reimbursement process in the related technology usually relies on manual operation, which is not only cumbersome and time-consuming, but also prone to errors, resulting in low efficiency and accuracy of the reimbursement process. In view of the above defects, a reimbursement process optimization method and device based on a multimodal model are provided in an embodiment of the present invention.

[0039] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0040] The method embodiments provided in the embodiments of the present invention can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 1 is a hardware structure block diagram of a mobile terminal of a method for optimizing a reimbursement process based on a multimodal model according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0041] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the reimbursement process optimization method based on the multimodal model in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The transmission device 106 is used to receive or send data via a network. The above-mentioned specific example of the network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0042] According to an embodiment of the present invention, a method embodiment of a reimbursement process optimization method based on a multimodal model is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0043] Figure 2 is a flowchart of a method for optimizing a reimbursement process based on a multimodal model according to an embodiment of the present invention, such as Figure 2 As shown, the method comprises the following steps:

[0044] Step S202, receiving materials to be reimbursed submitted by a target object, wherein the target object is the object that makes a reimbursement request, and the materials to be reimbursed refer to the certification materials required for reimbursement.

[0045] Combine the following Figure 3 The above embodiments of the present invention are described in detail. Figure 3 is a flowchart of an optional reimbursement process optimization method based on a multimodal model according to an embodiment of the present invention; Figure 3 As shown, users can directly submit the relevant materials that need to be reimbursed (i.e., materials to be reimbursed) without confirming the reimbursement type or the type of materials required in advance.

[0046] Step S204, performing image recognition and text extraction on the materials to be reimbursed, obtaining document classification information and reimbursement text information of the materials to be reimbursed, wherein the document classification information refers to the information on the type of reimbursement vouchers in the materials to be reimbursed, and the reimbursement text information refers to the text information extracted from the reimbursement vouchers.

[0047] In this embodiment, image information in the materials to be reimbursed can be identified, including but not limited to invoices, receipts, reimbursement forms, special tickets, travel vouchers, etc. (i.e., reimbursement vouchers in the embodiment of the present invention), so as to identify the document classification information corresponding to the reimbursement vouchers in the materials to be reimbursed, i.e., the type of reimbursement vouchers; text recognition can also be performed on the images of the materials to be reimbursed, so as to use the identified text information as the reimbursement text information of the materials to be reimbursed.

[0048] According to the above embodiment of the present invention, in the above step S204, image recognition and text extraction are performed on the materials to be reimbursed to obtain document classification information and reimbursement text information of the materials to be reimbursed, including: using a multimodal image classification model to identify and process the image information in the materials to be reimbursed to obtain document classification information, wherein the image information is information corresponding to the reimbursement voucher, and the multimodal image classification model is obtained by training the multimodal model through deep learning using multiple groups of second training data, and each of the multiple groups of second training data includes: sample image information, sample document classification information corresponding to the sample image information; using text recognition technology to identify and extract text information in the image information, and determining that the extracted text information is the reimbursement text information of the materials to be reimbursed.

[0049] As above Figure 3 As shown, a multimodal image classification model can be called to identify image information in materials to be reimbursed to obtain document classification information corresponding to reimbursement vouchers in the materials to be reimbursed (in the materials to be reimbursed submitted by the user, these are usually image information of reimbursement vouchers such as invoices, receipts, reimbursement forms, special tickets, travel vouchers, etc.), and OCR technology can also be used to perform text recognition on images to extract text information.

[0050] Specifically, the multimodal image classification model can be trained in the following ways: 1) Prepare the data set: Collect multimodal data sets, mainly including image data sets classified into invoices, receipts, reimbursement forms, special tickets, travel vouchers, etc.; preprocess the data, including image scaling, cropping, normalization, text segmentation, vectorization and other operations to ensure that each image has a corresponding label or category; manually label each image to assign a category label so that classification can be learned when training the model; divide the data set into training set, validation set and test set; 2) Select the CLIP model: The CLIP model is a pre-trained multimodal model that can process images and text at the same time. According to the business needs of the financial reimbursement process, select a CLIP model (Taiyi-CLIP-Roberta-102M-Chinese) that performs well in image classification tasks in terms of model size, domain adaptability, resource availability, etc.; 3) Pre-train CLIP model: Use the image dataset prepared in the first step to pre-train the selected CLIP model. Through pre-training, the model can better understand the relationship between reimbursement-related images and corresponding labels in the financial reimbursement scenario; 4) Train image classification model: Build an image classifier based on the fine-tuned CLIP model. Add one or more additional layers on top of the fine-tuned model to classify according to the features of the image, and use the image dataset to train this newly added classifier; 5) Model evaluation and optimization: Use the reserved test set to evaluate the trained image classifier to determine its performance on unseen data. Common evaluation indicators include accuracy, precision, recall, and F1 score; adjust the model's hyperparameters based on the evaluation results, or perform further fine-tuning to improve performance. Try different learning rates, optimization algorithms, model structures, etc. to optimize the model.

[0051] Step S206, input the document classification information and the reimbursement text information into the reimbursement classification model, so as to utilize the reimbursement classification model to determine the reimbursement type of the material to be reimbursed based on the document classification information and the reimbursement text information, wherein the reimbursement classification model is trained by deep learning using multiple groups of first training data, and each of the multiple groups of first training data includes: sample document classification information, sample reimbursement text information, and sample reimbursement types corresponding to the sample document classification information and the sample reimbursement text information.

[0052] As above Figure 3 As shown, the reimbursement classification model may be called to analyze and process the document classification information and reimbursement text information acquired in the above step S204 to determine the reimbursement type of the material to be reimbursed.

[0053] According to the above-mentioned embodiment of the invention, before the document classification information and the reimbursement text information are input into the reimbursement classification model so as to use the reimbursement classification model to determine the reimbursement type of the materials to be reimbursed based on the document classification information and the reimbursement text information, the reimbursement process optimization method based on the multimodal model also includes: obtaining multiple sets of sample reimbursement information, wherein the sample reimbursement information includes: sample document classification information, sample reimbursement text information, and sample reimbursement types corresponding to the sample document classification information and the sample reimbursement text information, and the sample document classification information is information obtained by performing image recognition on the image information in the sample materials to be reimbursed by the multimodal image classification model; dividing the multiple sets of sample reimbursement information to obtain sample training reimbursement information and sample test reimbursement information, wherein the sample training reimbursement information is used for training A reimbursement classification model is trained, and the sample test reimbursement information is used to test the accuracy of the reimbursement classification model; the natural language processing model containing the classifier is iteratively trained using the sample training reimbursement information until the iteration termination condition is reached, and the model obtained by the current training is determined to be the initial reimbursement classification model, wherein the classifier is located at the top of the natural language processing model, and the iteration termination condition at least includes: the number of iterations reaches a preset number of iterations, and the classification error does not exceed the error threshold; the initial reimbursement classification model is tested using the sample test reimbursement information to obtain a test result; when the test result indicates that the initial reimbursement classification model meets the accuracy requirements, the initial reimbursement classification model is determined to be the reimbursement classification model, wherein meeting the accuracy requirements means that the classification error of the initial reimbursement classification model does not exceed the error threshold.

[0054] Specifically, as above Figure 3As shown in the figure, the reimbursement classification model can be trained in the following way: 1) Prepare the data set: Collect all data related to the reimbursement classification data, use OCR to obtain text information for image data, and finally build a text data set including: user information (structured information), all document categories, image recognition content, etc., ensure that each text sample has a corresponding label or category, pre-process the text data, including word segmentation, tokenization, sentence length limit, etc., and convert the text data into an input format acceptable to the model according to the input requirements of the selected BERT model; 2) Select the BERT model: The BERT model has different variants and scales. From the aspects of model scale, Chinese language support, fine-tuning performance, etc., select a pre-trained BERT model suitable for text classification tasks and resource constraints (t ext2vec-base-chinese); 3) Pre-train BERT model: Use the dataset prepared in the first step to pre-train the selected BERT model so that the model can better understand the text style of the classification task in the financial reimbursement scenario; 4) Train the reimbursement classification model: Add a classifier on top of the BERT model, which is a fully connected layer or multi-layer perceptron, to classify text based on the output of BERT; adjust the structure and parameters of the classifier to adapt to the reimbursement classification task and dataset; train the entire model (including the BERT model and the top classifier) ​​using the prepared dataset; 5) Model evaluation and optimization: Evaluate the trained financial reimbursement model with the reserved test set to determine the performance of the model on unseen data. Common evaluation indicators include accuracy, precision, recall, and F1 score; During the training process, use appropriate loss functions and optimization algorithms, such as cross entropy loss and Adam optimizer; adjust training hyperparameters, such as learning rate, batch size, and number of training rounds, to obtain the best performance of the model.

[0055] For the evaluation and optimization of the multimodal image classification model and the reimbursement classification model after the initial training, the performance of the model can be evaluated from the following aspects: 1) Accuracy: Accuracy refers to the proportion of the classification results of the model on the test data set that match the actual classification results. High accuracy means that the model can correctly identify and classify reimbursement information when processing new data; 2) Precision: Precision refers to the proportion of data correctly classified as a certain category by the model to all data classified as that category. In the financial reimbursement scenario, high precision means that the reimbursement category identified by the system rarely has false positives, that is, the reimbursement information identified by the system as a specific type actually belongs to that type; 3) Recall l): Recall rate refers to the proportion of data that is correctly identified and classified by the model among all data that actually belong to a certain category. A high recall rate means that the system rarely misses the expense category to be reimbursed, that is, all actual specific types of reimbursement information are almost correctly identified; 4) F1 score: The F1 score is the harmonic mean of precision and recall, which comprehensively measures the accuracy and comprehensiveness of the model. A high F1 score indicates that the model has good coverage while ensuring classification accuracy, and is suitable for use in the actual reimbursement process. 5) Processing speed: Evaluate the model's processing speed for new data to ensure that in actual applications, reimbursement information can be quickly identified and pre-filled, thereby improving the efficiency of the overall process; 6) Stability: Test the stability of the model in different environments and over long periods of time to prevent model performance degradation due to factors such as hardware resources and data format changes; 7) Interpretability: Perform interpretability verification on the model's prediction results to ensure that the prediction logic is clear, easy for users to understand, and its rationality can be judged through manual review; 8) Robustness: Test the model's ability to handle abnormal data, noise, or images of different qualities to ensure that the model can maintain high performance in various actual situations; 9) Generalization ability: Verify whether the model can handle reimbursement types or special situations that did not appear in the training set to ensure its adaptability and flexibility in actual applications.

[0056] Step S208, compare the target object's characteristic information, document classification information, reimbursement text information and reimbursement type with the fields in the reimbursement application form to obtain a comparison result, wherein the characteristic information is information obtained by searching the database according to the target object's identification, the reimbursement application form refers to the reimbursement form template, and the fields represent the reimbursement information that the target object needs to provide when making a reimbursement request.

[0057] In this embodiment, relevant information of the target object (i.e., characteristic information in the embodiment of the present invention) can be obtained in the database according to the identification of the target object, and then the relevant information of the target object, as well as the document classification information, reimbursement text information and reimbursement type obtained in the above steps are compared with the fields in the reimbursement application form to obtain a comparison result. The reimbursement application form here can be understood as a reimbursement form template, and the fields therein represent the relevant reimbursement information that needs to be filled in when reimbursing.

[0058] It should be noted that the above-obtained relevant information of the target object is the information required when filling out the reimbursement form to ensure that the reimbursement form can correspond to the target object and that the subsequent reimbursement fees can be smoothly reimbursed to the target object.

[0059] According to the above embodiment of the present invention, in the above step S208, the characteristic information, document classification information, reimbursement text information and reimbursement type of the target object are compared with the fields in the reimbursement application form to obtain a comparison result, including: obtaining the reimbursement information corresponding to each field in the reimbursement application form; determining the required fields and non-required fields in the reimbursement application form, wherein the required fields refer to the fields that must be filled in to generate the target reimbursement application form, and the non-required fields refer to the fields that are not required to be filled in to generate the target reimbursement application form; matching the characteristic information, document classification information, reimbursement text information and reimbursement type of the target object with the reimbursement information corresponding to each field to obtain a matching result; when the matching result indicates that all required fields in the reimbursement application form are matched to the corresponding reimbursement information, determining that the comparison result is that there is no missing information; when the matching result indicates that at least one required field in the reimbursement application form is not matched to the corresponding reimbursement information, determining that the comparison result is that there is missing information.

[0060] The above required fields are optional and refer to the fields that must be filled in when submitting the reimbursement application form mentioned when submitting the reimbursement request.

[0061] The above non-mandatory fields are optional fields that can be filled in or not filled in when submitting the reimbursement application form mentioned in the reimbursement request.

[0062] In this embodiment, the acquired information (including: characteristic information of the target object, document classification information, reimbursement text information and reimbursement type) can be matched with the reimbursement information corresponding to each field in the reimbursement application form. If all required fields are matched with the corresponding reimbursement information, it is considered that there is no missing information when generating the target reimbursement application form corresponding to the material to be reimbursed. If there is at least one required field that does not match the corresponding reimbursement information, it is considered that there is missing information.

[0063] Step S210, when the comparison result indicates that there is no missing information, fill in the feature information, document classification information, reimbursement text information and reimbursement type into the reimbursement application form to generate a target reimbursement application form corresponding to the material to be reimbursed, wherein the missing information refers to the reimbursement information corresponding to the target field, and the target field refers to the field in the reimbursement application form that has not matched the reimbursement information.

[0064] In this embodiment, if the comparison result obtained in the above steps indicates that there is no missing information, the feature information, document classification information, reimbursement text information and reimbursement type information can be directly filled into the reimbursement application form to generate a target reimbursement application form corresponding to the materials to be reimbursed.

[0065] Step S212, when the comparison result indicates that there is missing information, after receiving the missing information submitted by the target object according to the prompt information, fill in the feature information, document classification information, reimbursement text information, reimbursement type and missing information into the reimbursement application form to generate a target reimbursement application form corresponding to the materials to be reimbursed, wherein the prompt information is used to prompt the target object to submit the prompt information.

[0066] In this embodiment, if the comparison result obtained in the above steps indicates that there is no missing information, the user can be prompted to submit the missing information first. After receiving the missing information submitted by the user, it can be filled in the reimbursement application form together with the information obtained in the above steps to generate a target reimbursement application form corresponding to the materials to be reimbursed.

[0067] According to the above embodiment of the present invention, in the above step S212, when the comparison result indicates that there is missing information, after receiving the missing information submitted by the target object according to the prompt information, the characteristic information, document classification information, reimbursement text information, reimbursement type and missing information are filled in the reimbursement application form to generate a target reimbursement application form corresponding to the material to be reimbursed, including: when the comparison result indicates that there is missing information, determining that the field in the reimbursement application form that does not match the reimbursement information is the target field; determining that the reimbursement information corresponding to the target field is the missing information; generating prompt information according to the missing information to prompt the target object to submit the missing information according to the prompt information; after receiving the missing information submitted by the target object, filling in the characteristic information, document classification information, reimbursement text information, reimbursement type and missing information into the reimbursement application form to generate a target reimbursement application form corresponding to the material to be reimbursed.

[0068] Specifically, you can first determine what information is required for the field that does not match the corresponding reimbursement information. This information is the missing information. Then you can prompt the target object to supplement it through a pop-up window. After the target object is supplemented, you can fill in the missing information, feature information, document classification information, reimbursement text information and reimbursement type together in the reimbursement application form to generate a target reimbursement application form corresponding to the materials to be reimbursed.

[0069] It should be noted that in addition to the above-mentioned method of first matching the acquired information with the reimbursement information corresponding to each field in the reimbursement application form to determine whether there is missing information, and then generating the corresponding target reimbursement application form, you can also directly pre-fill the acquired information into, and then determine whether there is a required field with the corresponding missing information not filled in. If not, directly generate the corresponding target reimbursement application form; if so, first prompt the user to supplement it, and then supplement it in the pre-filled reimbursement application form after the user submits the missing information to generate the corresponding target reimbursement application form.

[0070] Step S214, submit the target reimbursement application form to enter the reimbursement process.

[0071] In this embodiment, after generating a target reimbursement application form corresponding to the materials to be reimbursed, the form can be submitted to an upper management platform or a staff member responsible for this work to facilitate entering the subsequent reimbursement process.

[0072] According to the above embodiment of the present invention, in the above step S214, the target reimbursement application form is submitted to enter the reimbursement process, including: pre-reviewing the target reimbursement application form according to preset rules to obtain a pre-review result, wherein the preset rules are rules pre-set according to the reimbursement requirements; when the pre-review result indicates that the target reimbursement application form has passed the pre-review, the target reimbursement application form is sent to the front end so that the target object can review the target reimbursement application form; when receiving the submission request information, the target reimbursement application form is submitted to enter the reimbursement process, wherein the submission request information refers to the submission request initiated by the target object after the target reimbursement application form is reviewed and confirmed to be correct.

[0073] Specifically, as above Figure 3 As shown, after the target reimbursement application form corresponding to the materials to be reimbursed is initially generated, it must first be pre-reviewed according to preset rules, including: reimbursement time, invoice time, budget limit, expense category, etc.; after passing the pre-review, the generated form is presented to the target object for verification and confirmation. After the verification is correct, the target object can submit the target reimbursement application form by clicking the submit button or by other means to enter the subsequent reimbursement process.

[0074] The preset rules here may include but are not limited to the following rules: 1) Reimbursement time: specifies the time window for submitting reimbursement applications, such as requiring reimbursement applications to be submitted within a certain time after the expense is incurred, or accepting reimbursement applications during a specific time period each month; 2) Invoice time: requires that the invoice issuance date must be within a specific range to ensure the timeliness and rationality of the expenses; 3) Budget limit: sets an upper limit on the reimbursement amount for each department, each project or each employee to avoid exceeding the budget; 4) Expense category: defines the types of reimbursable expenses, such as travel expenses, catering expenses, office supplies expenses, etc., and may have different reimbursement standards and requirements for different types of expenses; 5) Approval process: specifies the approval order and level of reimbursement applications, such as requiring higher-level approval when the expenses reach a certain amount; These preset rules play a key control and guidance role in the reimbursement process, helping to ensure the compliance, rationality and budget control of financial reimbursement. In the intelligent financial reimbursement system, these rules are programmed into the system, and the system automatically performs pre-audit to determine whether the reimbursement application complies with the relevant reimbursement regulations and specifications, thereby improving the efficiency and accuracy of the audit.

[0075] If the pre-review is not passed, the user can be informed through a pop-up window of the parts that do not meet the preset rules, so that the user can make adjustments or apply for reimbursement through other means or not apply for reimbursement.

[0076] According to the above embodiment of the present invention, the reimbursement process optimization method based on the multimodal model also includes: in the process of generating a target reimbursement application form corresponding to the materials to be reimbursed, optimizing the reimbursement classification model and the multimodal image classification model using the materials to be reimbursed.

[0077] Specifically, as above Figure 3 As shown, in the process of continuously generating the target reimbursement application form corresponding to each target object based on the materials to be reimbursed provided by each target object through the above steps, the multimodal image classification model and the reimbursement classification model can be continuously adjusted and optimized according to user feedback and data changes to adapt to the needs of different scenarios.

[0078] As can be seen from the above, through the technical solution provided by the above embodiment of the present invention, it is possible to receive materials to be reimbursed submitted by a target object, wherein the target object is the object that makes a reimbursement request, and the materials to be reimbursed refer to the certification materials required for reimbursement; image recognition and text extraction are performed on the materials to be reimbursed to obtain document classification information and reimbursement text information of the materials to be reimbursed, wherein the document classification information refers to the information on the type of reimbursement voucher in the materials to be reimbursed, and the reimbursement text information refers to the text information extracted from the reimbursement voucher; the document classification information and the reimbursement text information are input into a reimbursement classification model, so as to utilize the reimbursement classification model to obtain the reimbursement voucher based on the document classification information and the reimbursement text information Determine the reimbursement type of the material to be reimbursed, wherein the reimbursement classification model is trained by deep learning using multiple sets of first training data, each of the multiple sets of first training data includes: sample document classification information, sample reimbursement text information, and sample reimbursement types corresponding to the sample document classification information and the sample reimbursement text information; compare the feature information, document classification information, reimbursement text information and reimbursement type of the target object with the fields in the reimbursement application form to obtain a comparison result, wherein the feature information is information found in a database according to the identifier of the target object, the reimbursement application form refers to the reimbursement form template, and the field indicates the target object's The reimbursement information required to be provided when making a reimbursement request; if the comparison result indicates that there is no missing information, fill in the feature information, document classification information, reimbursement text information and reimbursement type into the reimbursement application form to generate a target reimbursement application form corresponding to the material to be reimbursed, where the missing information refers to the reimbursement information corresponding to the target field, and the target field refers to the field in the reimbursement application form that does not match the reimbursement information; if the comparison result indicates that there is missing information, after receiving the missing information submitted by the target object according to the prompt information, fill in the feature information, document classification information, reimbursement text information, reimbursement type and missing information into the reimbursement application form to generate a target reimbursement application form corresponding to the material to be reimbursed. The target reimbursement application form corresponding to the reimbursement materials, wherein the prompt information is used to prompt the target object to submit the prompt information; submit the target reimbursement application form to enter the reimbursement process, thereby achieving the purpose of using the trained model to identify and analyze the materials to be reimbursed submitted by the user from multiple angles, and automatically generating the corresponding reimbursement form based on the materials submitted by the user. This achieves the purpose of only requiring users to submit relevant reimbursement materials, without having to manually fill out the reimbursement form. The model can be used for identification and analysis, and the technical effect of automatically generating the reimbursement form in the background based on the materials submitted by the user and the analysis results, simplifies the reimbursement process, and improves the accuracy and efficiency of filling out the reimbursement form.

[0079] Therefore, the technical solution provided by the above-mentioned embodiments of the present invention solves the technical problem that the traditional financial reimbursement process in the related technology usually relies on manual operation, and the operation process is not only cumbersome and time-consuming, but also prone to errors, resulting in low efficiency and accuracy of the reimbursement process.

[0080] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0081] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0082] According to an embodiment of the present invention, a reimbursement process optimization device based on a multimodal model is also provided for implementing the above-mentioned reimbursement process optimization method based on a multimodal model. Figure 4 is a schematic diagram of a reimbursement process optimization device based on a multimodal model according to an embodiment of the present invention, such as Figure 4 As shown, the device includes: a receiving unit 401, a first acquiring unit 403, a first determining unit 405, a second acquiring unit 407, a first generating unit 409, a second generating unit 411 and a submitting unit 413. The reimbursement process optimization device based on the multimodal model is described in detail below.

[0083] The receiving unit 401 is used to receive materials to be reimbursed submitted by a target object, wherein the target object is a subject who makes a reimbursement request, and the materials to be reimbursed refer to certification materials required for reimbursement.

[0084] The first acquisition unit 403 is used to perform image recognition and text extraction on the materials to be reimbursed, and obtain document classification information and reimbursement text information of the materials to be reimbursed, wherein the document classification information refers to the information of the type of reimbursement voucher in the materials to be reimbursed, and the reimbursement text information refers to the text information extracted from the reimbursement voucher.

[0085] The first determination unit 405 is used to input the document classification information and the reimbursement text information into the reimbursement classification model, so as to use the reimbursement classification model to determine the reimbursement type of the material to be reimbursed based on the document classification information and the reimbursement text information, wherein the reimbursement classification model is trained by deep learning using multiple groups of first training data, and each of the multiple groups of first training data includes: sample document classification information, sample reimbursement text information, and sample reimbursement types corresponding to the sample document classification information and the sample reimbursement text information.

[0086] The second acquisition unit 407 is used to compare the characteristic information, document classification information, reimbursement text information and reimbursement type of the target object with the fields in the reimbursement application form to obtain a comparison result, wherein the characteristic information is information obtained by searching the database according to the identification of the target object, the reimbursement application form refers to the reimbursement form template, and the fields represent the reimbursement information that the target object needs to provide when making a reimbursement request.

[0087] The first generation unit 409 is used to fill in the feature information, document classification information, reimbursement text information and reimbursement type into the reimbursement application form when the comparison result indicates that there is no missing information, so as to generate a target reimbursement application form corresponding to the material to be reimbursed, wherein the missing information refers to the reimbursement information corresponding to the target field, and the target field refers to the field in the reimbursement application form that has not matched the reimbursement information.

[0088] The second generation unit 411 is used to fill in the feature information, document classification information, reimbursement text information, reimbursement type and missing information into the reimbursement application form after receiving the missing information submitted by the target object according to the prompt information, so as to generate a target reimbursement application form corresponding to the material to be reimbursed, wherein the prompt information is used to prompt the target object to submit the prompt information.

[0089] The submitting unit 413 is used to submit the target reimbursement application form to enter the reimbursement process.

[0090] It should be noted here that the above-mentioned receiving unit 401, first acquisition unit 403, first determination unit 405, second acquisition unit 407, first generation unit 409, second generation unit 411 and submission unit 413 correspond to steps S202 to S214 in the above-mentioned embodiments, and the seven units are the same as the instances and application scenarios implemented by the corresponding steps, but are not limited to the contents disclosed in the above-mentioned embodiments.

[0091] As can be seen from the above, in the scheme recorded in the above-mentioned embodiment of the present invention, a receiving unit can be used to receive materials to be reimbursed submitted by a target object, wherein the target object is the object that makes a reimbursement request, and the materials to be reimbursed refer to the certification materials required for reimbursement; then a first acquisition unit is used to perform image recognition and text extraction on the materials to be reimbursed to obtain document classification information and reimbursement text information of the materials to be reimbursed, wherein the document classification information refers to the information on the type of reimbursement voucher in the materials to be reimbursed, and the reimbursement text information refers to the text information extracted from the reimbursement voucher; then a first determination unit is used to input the document classification information and the reimbursement text information into a reimbursement classification model, so as to use the reimbursement classification model to determine the reimbursement text information based on the document classification information. The reimbursement type of the material to be reimbursed is determined by using the information and reimbursement text information of the target object, wherein the reimbursement classification model is obtained by training with multiple sets of first training data by means of deep learning, and each of the multiple sets of first training data includes: sample document classification information, sample reimbursement text information, and sample reimbursement types corresponding to the sample document classification information and the sample reimbursement text information; then, the second acquisition unit is used to compare the feature information, document classification information, reimbursement text information and reimbursement type of the target object with the fields in the reimbursement application form to obtain a comparison result, wherein the feature information is information obtained by searching in a database according to the identifier of the target object, the reimbursement application form refers to the reimbursement form template, and the field represents the target object. The reimbursement information that the object needs to provide when making a reimbursement request; then, using the first generation unit, when the comparison result indicates that there is no missing information, fill in the feature information, document classification information, reimbursement text information and reimbursement type into the reimbursement application form to generate a target reimbursement application form corresponding to the material to be reimbursed, wherein the missing information refers to the reimbursement information corresponding to the target field, and the target field refers to the field in the reimbursement application form that is not matched with the reimbursement information; continue to use the second generation unit, when the comparison result indicates that there is missing information, after receiving the missing information submitted by the target object according to the prompt information, fill in the feature information, document classification information, reimbursement text information, reimbursement type and missing information into the reimbursement application form. In the form, a target reimbursement application form corresponding to the materials to be reimbursed is generated, wherein the prompt information is used to prompt the target object to submit the prompt information; finally, the target reimbursement application form is submitted using the submission unit to enter the reimbursement process, thereby achieving the purpose of using the trained model to identify and analyze the materials to be reimbursed submitted by the user from multiple angles, and automatically generating the corresponding reimbursement form according to the materials submitted by the user, achieving the technical effect that the user only needs to submit the relevant reimbursement materials, and there is no need to manually fill out the reimbursement form. The model can be used for identification and analysis, and the reimbursement form can be automatically generated in the background according to the materials submitted by the user and the analysis results, which simplifies the reimbursement process and improves the accuracy and efficiency of filling out the reimbursement form.

[0092] Therefore, the technical solution provided by the above-mentioned embodiments of the present invention solves the technical problem that the traditional financial reimbursement process in the related technology usually relies on manual operation, and the operation process is not only cumbersome and time-consuming, but also prone to errors, resulting in low efficiency and accuracy of the reimbursement process.

[0093] Optionally, the first acquisition unit includes: a first acquisition module, used to use a multimodal image classification model to identify and process image information in the materials to be reimbursed to obtain document classification information, wherein the image information is information corresponding to the reimbursement voucher, and the multimodal image classification model is obtained by training the multimodal model through deep learning using multiple groups of second training data, and each of the multiple groups of second training data includes: sample image information, sample document classification information corresponding to the sample image information; an extraction module, used to use text recognition technology to identify and extract text information in the image information, and determine that the extracted text information is the reimbursement text information of the materials to be reimbursed.

[0094] Optionally, the reimbursement process optimization device based on the multimodal model also includes: a third acquisition unit, which is used to acquire multiple groups of sample reimbursement information before inputting the document classification information and the reimbursement text information into the reimbursement classification model to determine the reimbursement type of the material to be reimbursed based on the document classification information and the reimbursement text information using the reimbursement classification model, wherein the sample reimbursement information includes: sample document classification information, sample reimbursement text information, and sample reimbursement types corresponding to the sample document classification information and the sample reimbursement text information, and the sample document classification information is information obtained by performing image recognition on the image information in the sample material to be reimbursed by the multimodal image classification model; a fourth acquisition unit, which is used to divide the multiple groups of sample reimbursement information to obtain sample training reimbursement information and sample test reimbursement information, wherein the sample training reimbursement information is used to train the reimbursement classification model, The sample test reimbursement information is used to test the accuracy of the reimbursement classification model; the second determination unit is used to iteratively train the natural language processing model containing the classifier using the sample training reimbursement information until the iteration termination condition is reached, and the model obtained by the current training is determined to be the initial reimbursement classification model, wherein the classifier is located at the top of the natural language processing model, and the iteration termination condition at least includes: the number of iterations reaches a preset number of iterations, and the classification error does not exceed the error threshold; the fifth acquisition unit is used to test the initial reimbursement classification model using the sample test reimbursement information to obtain the test result; the third determination unit is used to determine that the initial reimbursement classification model is the reimbursement classification model when the test result indicates that the initial reimbursement classification model meets the accuracy requirement, wherein meeting the accuracy requirement means that the classification error of the initial reimbursement classification model does not exceed the error threshold.

[0095] Optionally, the second acquisition unit includes: a second acquisition module, used to acquire the reimbursement information corresponding to each field in the reimbursement application form; a first determination module, used to determine the required fields and non-required fields in the reimbursement application form, wherein the required fields refer to the fields that must be filled in to generate the target reimbursement application form, and the non-required fields refer to the fields that are not required to be filled in to generate the target reimbursement application form; a third acquisition module, used to match the characteristic information, document classification information, reimbursement text information and reimbursement type of the target object with the reimbursement information corresponding to each field to obtain a matching result; a second determination module, used to determine that the comparison result is that there is no missing information when the matching result indicates that all required fields in the reimbursement application form are matched to the corresponding reimbursement information; a third determination module, used to determine that the comparison result is that there is missing information when the matching result indicates that at least one required field in the reimbursement application form is not matched to the corresponding reimbursement information.

[0096] Optionally, the second generation unit includes: a fourth determination module, which is used to determine that the field in the reimbursement application form that does not match the reimbursement information is the target field when the comparison result indicates that there is missing information; a fifth determination module, which is used to determine that the reimbursement information corresponding to the target field is the missing information; a first generation module, which is used to generate prompt information based on the missing information to prompt the target object to submit the missing information according to the prompt information; and a second generation module, which is used to fill in the feature information, document classification information, reimbursement text information, reimbursement type and missing information into the reimbursement application form after receiving the missing information submitted by the target object, so as to generate a target reimbursement application form corresponding to the materials to be reimbursed.

[0097] Optionally, the submission unit includes: a fourth acquisition module, which is used to pre-review the target reimbursement application form according to preset rules to obtain a pre-review result, wherein the preset rules are rules pre-set according to the reimbursement requirements; a sending module, which is used to send the target reimbursement application form to the front end when the pre-review result indicates that the target reimbursement application form has passed the pre-review, so that the target object can review the target reimbursement application form; a submission module, which is used to submit the target reimbursement application form when receiving a submission request information to enter the reimbursement process, wherein the submission request information refers to a submission request initiated by the target object after the target reimbursement application form has been reviewed and confirmed to be correct.

[0098] Optionally, the reimbursement process optimization device based on the multimodal model also includes: an optimization unit, which is used to optimize the reimbursement classification model and the multimodal image classification model using the materials to be reimbursed in the process of generating a target reimbursement application form corresponding to the materials to be reimbursed.

[0099] According to another aspect of an embodiment of the present invention, a reimbursement process optimization system based on a multimodal model is further provided. The reimbursement process optimization system based on a multimodal model uses any of the above-mentioned reimbursement process optimization methods based on a multimodal model.

[0100] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is further provided, the computer-readable storage medium comprising a stored program, wherein the program executes any one of the above-mentioned reimbursement process optimization methods based on a multimodal model.

[0101] Optionally, in this embodiment, the computer-readable storage medium may be located in any one of the computer terminals in a computer terminal group in a computer network, or in any one of the communication devices in a communication device group.

[0102] Optionally, in this embodiment, a computer-readable storage medium is configured to store program codes for executing the following steps: receiving materials to be reimbursed submitted by a target object, wherein the target object is the object that makes a reimbursement request, and the materials to be reimbursed refer to the supporting materials required for reimbursement; performing image recognition and text extraction on the materials to be reimbursed to obtain document classification information and reimbursement text information of the materials to be reimbursed, wherein the document classification information refers to information on the type of reimbursement voucher in the materials to be reimbursed, and the reimbursement text information refers to text information extracted from the reimbursement voucher; inputting the document classification information and the reimbursement text information into a reimbursement classification model, so as to determine the reimbursement type of the materials to be reimbursed based on the document classification information and the reimbursement text information using the reimbursement classification model, wherein the reimbursement classification model is trained by deep learning using multiple groups of first training data, and each of the multiple groups of first training data includes: sample document classification information, sample reimbursement text information, and sample reimbursement types corresponding to the sample document classification information and the sample reimbursement text information; inputting the feature information of the target object, the document classification information , the reimbursement text information and the reimbursement type are compared with the fields in the reimbursement application form to obtain a comparison result, wherein the characteristic information is the information found in the database according to the identification of the target object, the reimbursement application form refers to the reimbursement form template, and the fields represent the reimbursement information that the target object needs to provide when making a reimbursement request; when the comparison result indicates that there is no missing information, the characteristic information, document classification information, reimbursement text information and reimbursement type are filled into the reimbursement application form to generate a target reimbursement application form corresponding to the material to be reimbursed, wherein the missing information refers to the reimbursement information corresponding to the target field, and the target field refers to the field in the reimbursement application form that is not matched with the reimbursement information; when the comparison result indicates that there is missing information, after receiving the missing information submitted by the target object according to the prompt information, the characteristic information, document classification information, reimbursement text information, reimbursement type and the missing information are filled into the reimbursement application form to generate a target reimbursement application form corresponding to the material to be reimbursed, wherein the prompt information is used to prompt the target object to submit the prompt information; submit the target reimbursement application form to enter the reimbursement process.

[0103] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: using a multimodal image classification model to identify and process image information in materials to be reimbursed to obtain document classification information, wherein the image information is information corresponding to the reimbursement voucher, and the multimodal image classification model is obtained by training the multimodal model through deep learning using multiple groups of second training data, and each of the multiple groups of second training data includes: sample image information, sample document classification information corresponding to the sample image information; using text recognition technology to identify and extract text information in the image information, and determine that the extracted text information is the reimbursement text information of the materials to be reimbursed.

[0104] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: obtaining multiple sets of sample reimbursement information, wherein the sample reimbursement information includes: sample document classification information, sample reimbursement text information, and sample reimbursement types corresponding to the sample document classification information and the sample reimbursement text information, and the sample document classification information is information obtained by performing image recognition on image information in sample materials to be reimbursed by a multimodal image classification model; dividing the multiple sets of sample reimbursement information to obtain sample training reimbursement information and sample test reimbursement information, wherein the sample training reimbursement information is used to train the reimbursement classification model, and the sample test reimbursement information is used to test the accuracy of the reimbursement classification model. Conduct a test; use sample training reimbursement information to iteratively train a natural language processing model containing a classifier until an iteration termination condition is reached, and determine that the model obtained by the current training is an initial reimbursement classification model, wherein the classifier is located at the top of the natural language processing model, and the iteration termination condition at least includes: the number of iterations reaches a preset number of iterations, and the classification error does not exceed the error threshold; use sample test reimbursement information to test the initial reimbursement classification model to obtain a test result; when the test result indicates that the initial reimbursement classification model meets the accuracy requirements, determine that the initial reimbursement classification model is a reimbursement classification model, wherein meeting the accuracy requirements means that the classification error of the initial reimbursement classification model does not exceed the error threshold.

[0105] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: obtaining reimbursement information corresponding to each field in the reimbursement application form; determining required fields and non-required fields in the reimbursement application form, wherein required fields refer to fields that must be filled in to generate the target reimbursement application form, and non-required fields refer to fields that are not required to be filled in to generate the target reimbursement application form; matching the characteristic information, document classification information, reimbursement text information and reimbursement type of the target object with the reimbursement information corresponding to each field to obtain a matching result; when the matching result indicates that all required fields in the reimbursement application form are matched to the corresponding reimbursement information, determining that the comparison result is that there is no missing information; when the matching result indicates that at least one required field in the reimbursement application form is not matched to the corresponding reimbursement information, determining that the comparison result is that there is missing information.

[0106] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: when the comparison result indicates that there is missing information, determining that the field in the reimbursement application form that does not match the reimbursement information is the target field; determining that the reimbursement information corresponding to the target field is the missing information; generating prompt information based on the missing information to prompt the target object to submit the missing information according to the prompt information; after receiving the missing information submitted by the target object, filling in the feature information, document classification information, reimbursement text information, reimbursement type and missing information into the reimbursement application form to generate a target reimbursement application form corresponding to the materials to be reimbursed.

[0107] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: pre-reviewing the target reimbursement application form according to preset rules to obtain a pre-review result, wherein the preset rules are rules pre-set according to the reimbursement requirements; when the pre-review result indicates that the target reimbursement application form has passed the pre-review, sending the target reimbursement application form to the front end so that the target object can review the target reimbursement application form; when receiving a submission request information, submitting the target reimbursement application form to enter the reimbursement process, wherein the submission request information refers to a submission request initiated by the target object after the target reimbursement application form has been reviewed and confirmed to be correct.

[0108] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: in the process of generating a target reimbursement application form corresponding to the materials to be reimbursed, optimizing the reimbursement classification model and the multimodal image classification model using the materials to be reimbursed.

[0109] According to another aspect of an embodiment of the present invention, a processor is further provided, and the processor is used to run a program, wherein when the program is run, any one of the above-mentioned reimbursement process optimization methods based on a multimodal model is executed.

[0110] According to another aspect of an embodiment of the present invention, a computer program product is also provided, comprising computer instructions, which, when executed by a processor, execute any one of the above-mentioned reimbursement process optimization methods based on a multimodal model.

[0111] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0112] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0113] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0114] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0115] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0116] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0117] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A reimbursement process optimization method based on a multimodal model, characterized in that: include: Receiving materials to be reimbursed submitted by a target object, wherein the target object is the object that makes a reimbursement request, and the materials to be reimbursed refer to the certification materials required for reimbursement; Performing image recognition and text extraction on the materials to be reimbursed to obtain document classification information and reimbursement text information of the materials to be reimbursed, wherein the document classification information refers to information on the type of reimbursement vouchers in the materials to be reimbursed, and the reimbursement text information refers to text information extracted from the reimbursement vouchers; Inputting the document classification information and the reimbursement text information into a reimbursement classification model, so as to use the reimbursement classification model to determine the reimbursement type of the to-be-reimbursed material based on the document classification information and the reimbursement text information, wherein the reimbursement classification model is obtained by training with a plurality of sets of first training data by means of deep learning, and each of the plurality of sets of first training data includes: sample document classification information, sample reimbursement text information, and sample reimbursement types corresponding to the sample document classification information and the sample reimbursement text information; Comparing the characteristic information of the target object, the document classification information, the reimbursement text information and the reimbursement type with the fields in the reimbursement application form to obtain a comparison result, wherein the characteristic information is information obtained by searching in a database according to the identifier of the target object, the reimbursement application form refers to a reimbursement form template, and the fields represent the reimbursement information that the target object needs to provide when making the reimbursement request; In the case where the comparison result indicates that there is no missing information, the feature information, the document classification information, the reimbursement text information and the reimbursement type are filled into the reimbursement application form to generate a target reimbursement application form corresponding to the material to be reimbursed, wherein the missing information refers to the reimbursement information corresponding to the target field, and the target field refers to the field in the reimbursement application form that is not matched to the reimbursement information; In the case where the comparison result indicates that the missing information exists, after receiving the missing information submitted by the target object according to the prompt information, the feature information, the document classification information, the reimbursement text information, the reimbursement type and the missing information are filled into the reimbursement application form to generate the target reimbursement application form corresponding to the material to be reimbursed, wherein the prompt information is used to prompt the target object to submit the prompt information; Submit the target reimbursement application form to enter the reimbursement process.

2. The reimbursement process optimization method based on a multimodal model according to claim 1 is characterized in that: Perform image recognition and text extraction on the materials to be reimbursed to obtain document classification information and reimbursement text information of the materials to be reimbursed, including: Using a multimodal image classification model to identify and process the image information in the material to be reimbursed, to obtain the document classification information, wherein the image information is information corresponding to the reimbursement voucher, and the multimodal image classification model is obtained by training the multimodal model by deep learning using multiple sets of second training data, and each of the multiple sets of second training data includes: sample image information and sample document classification information corresponding to the sample image information; The text information in the image information is identified and extracted using a text recognition technology, and the extracted text information is determined to be the reimbursement text information of the material to be reimbursed.

3. The reimbursement process optimization method based on a multimodal model according to claim 1, characterized in that: Before inputting the document classification information and the reimbursement text information into the reimbursement classification model to determine the reimbursement type of the to-be-reimbursed material based on the document classification information and the reimbursement text information by using the reimbursement classification model, the method further includes: Acquire multiple sets of sample reimbursement information, wherein the sample reimbursement information includes: the sample document classification information, the sample reimbursement text information, and the sample reimbursement type corresponding to the sample document classification information and the sample reimbursement text information, and the sample document classification information is information obtained by performing image recognition on image information in sample materials to be reimbursed by a multimodal image classification model; Dividing the multiple groups of sample reimbursement information to obtain sample training reimbursement information and sample test reimbursement information, wherein the sample training reimbursement information is used to train the reimbursement classification model, and the sample test reimbursement information is used to test the accuracy of the reimbursement classification model; The natural language processing model containing the classifier is iteratively trained using the sample training reimbursement information until an iteration termination condition is reached, and the model obtained by the current training is determined to be an initial reimbursement classification model, wherein the classifier is located at the top of the natural language processing model, and the iteration termination condition at least includes: the number of iterations reaches a preset number of iterations, and the classification error does not exceed an error threshold; Using the sample test reimbursement information to test the initial reimbursement classification model to obtain a test result; When the test result indicates that the initial reimbursement classification model meets the accuracy requirement, the initial reimbursement classification model is determined to be the reimbursement classification model, wherein meeting the accuracy requirement means that the classification error of the initial reimbursement classification model does not exceed the error threshold.

4. The reimbursement process optimization method based on a multimodal model according to claim 1, characterized in that: The characteristic information of the target object, the document classification information, the reimbursement text information and the reimbursement type are compared with the fields in the reimbursement application form to obtain a comparison result, including: Obtaining the reimbursement information corresponding to each of the fields in the reimbursement application form; Determine the required fields and non-required fields in the reimbursement application form, wherein the required fields refer to the fields that must be filled in to generate the target reimbursement application form, and the non-required fields refer to the fields that are not required to be filled in to generate the target reimbursement application form; Matching the characteristic information of the target object, the document classification information, the reimbursement text information and the reimbursement type with the reimbursement information corresponding to each of the fields to obtain a matching result; In the case where the matching result indicates that all the required fields in the reimbursement application form are matched to the corresponding reimbursement information, determining that the comparison result is that the missing information does not exist; In the case where the matching result indicates that at least one of the required fields in the reimbursement application form is not matched to the corresponding reimbursement information, it is determined that the comparison result is that the missing information exists.

5. The reimbursement process optimization method based on a multimodal model according to claim 1, characterized in that: In the case where the comparison result indicates that the missing information exists, after receiving the missing information submitted by the target object according to the prompt information, the feature information, the document classification information, the reimbursement text information, the reimbursement type and the missing information are filled into the reimbursement application form to generate the target reimbursement application form corresponding to the material to be reimbursed, including: In the case where the comparison result indicates that the missing information exists, determining the field in the reimbursement application form that is not matched with the reimbursement information as the target field; Determining that the reimbursement information corresponding to the target field is the missing information; Generate prompt information according to the missing information to prompt the target object to submit the missing information according to the prompt information; After receiving the missing information submitted by the target object, the feature information, the document classification information, the reimbursement text information, the reimbursement type and the missing information are filled into the reimbursement application form to generate the target reimbursement application form corresponding to the material to be reimbursed.

6. The reimbursement process optimization method based on a multimodal model according to claim 1 is characterized in that: Submit the target reimbursement application form to enter the reimbursement process, including: Pre-examination of the target reimbursement application form according to preset rules to obtain a pre-examination result, wherein the preset rules are rules pre-set according to reimbursement requirements; If the pre-examination result indicates that the target reimbursement application form has passed the pre-examination, sending the target reimbursement application form to the front end so that the target object can examine the target reimbursement application form; When receiving the submission request information, the target reimbursement application form is submitted to enter the reimbursement process, wherein the submission request information refers to the submission request initiated by the target object after reviewing the target reimbursement application form.

7. The reimbursement process optimization method based on a multimodal model according to any one of claims 1 to 6, characterized in that: Also includes: In the process of generating the target reimbursement application form corresponding to the material to be reimbursed, the reimbursement classification model and the multimodal image classification model are optimized using the material to be reimbursed.

8. A reimbursement process optimization device based on a multimodal model, characterized in that: include: A receiving unit, configured to receive materials to be reimbursed submitted by a target object, wherein the target object is a subject who makes a reimbursement request, and the materials to be reimbursed refer to certification materials required for reimbursement; A first acquisition unit is used to perform image recognition and text extraction on the material to be reimbursed to obtain document classification information and reimbursement text information of the material to be reimbursed, wherein the document classification information refers to information on the type of reimbursement voucher in the material to be reimbursed, and the reimbursement text information refers to text information extracted from the reimbursement voucher; A first determination unit is used to input the document classification information and the reimbursement text information into a reimbursement classification model, so as to use the reimbursement classification model to determine the reimbursement type of the material to be reimbursed based on the document classification information and the reimbursement text information, wherein the reimbursement classification model is obtained by training with a plurality of sets of first training data by a deep learning method, and each of the plurality of sets of first training data includes: sample document classification information, sample reimbursement text information, and sample reimbursement types corresponding to the sample document classification information and the sample reimbursement text information; A second acquisition unit is used to compare the characteristic information of the target object, the document classification information, the reimbursement text information and the reimbursement type with the fields in the reimbursement application form to obtain a comparison result, wherein the characteristic information is information obtained by searching in a database according to the identifier of the target object, the reimbursement application form refers to a reimbursement form template, and the fields represent the reimbursement information that the target object needs to provide when making the reimbursement request; A first generating unit is used to fill the feature information, the document classification information, the reimbursement text information and the reimbursement type into the reimbursement application form to generate a target reimbursement application form corresponding to the material to be reimbursed, when the comparison result indicates that there is no missing information, wherein the missing information refers to the reimbursement information corresponding to a target field, and the target field refers to the field in the reimbursement application form that is not matched with the reimbursement information; A second generating unit is configured to fill in the feature information, the document classification information, the reimbursement text information, the reimbursement type and the missing information into the reimbursement application form after receiving the missing information submitted by the target object according to the prompt information, so as to generate the target reimbursement application form corresponding to the material to be reimbursed, wherein the prompt information is used to prompt the target object to submit the prompt information, if the comparison result indicates that the missing information exists; The submitting unit is used to submit the target reimbursement application form to enter the reimbursement process.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the reimbursement process optimization method based on a multimodal model as described in any one of claims 1 to 7.

10. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by the processor, the reimbursement process optimization method based on the multimodal model described in any one of claims 1 to 7 is performed.

Images