Combined document information matching method and device, equipment and medium
By constructing the target prompt word structure and generating target prompt words, combined with a pre-trained multimodal model, the problem of incomplete output of large language models when processing complex structures is solved, achieving more efficient and accurate information matching.
Patent Information
- Application Number
- CN202411699400.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-25
- Publication Date
- 2025-03-18
AI Technical Summary
The prior art is difficult to accurately process combined documents with complex structures and large sizes, resulting in large language models being unable to output completely normally when information matching.
By constructing the target prompt word structure, the target prompt word is generated, and the pre-trained multimodal model is retrieved, the target prompt word is input to the multimodal model to obtain the information matching result.
Effectively constrain model output, avoid invalid information output, improve information matching efficiency and accuracy, and reduce the model's adaptation cost to structural data.
Smart Images
Figure CN119646039A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a combined document information matching method, device, equipment and medium. Background Art
[0002] As transactions become more and more complex, the data structures involved in actual scenarios are also becoming more and more complex. In addition to regular tables, wireless tables and tableless forms may also be involved, which cannot be accurately processed by large language models.
[0003] In addition, when using a large language model for document matching, if all the bill information is input into the large language model, since the entire bill contains a lot of information, in addition to the text information of the entire image, it may also include the coordinate information and format information of each field, and the length of the input and output information of a general large language model is limited. Therefore, too much input will affect the output of the model. When the input information is too long, the large language model may not be able to output it completely. For example: assuming that the number of tokens supported by a large language model is 8k, when the input token is 2k, the maximum output is 6k; when the input token is 4k, the maximum output is 4k; when the input token exceeds 8K, the large language model may not be able to output it completely.
[0004] In view of the above problems, how to effectively handle combined documents with complex structures and large volumes has become an issue that needs to be solved urgently. Summary of the invention
[0005] The embodiments of the present application provide a method, device, equipment and medium for combining document information matching, aiming to solve the problem that information matching cannot be accurately performed on combining documents with complex structures and large volumes.
[0006] In a first aspect, an embodiment of the present application provides a combined document information matching method, which includes:
[0007] In response to an information matching instruction for a target combined document and a target invoice, constructing a target prompt word structure according to the information matching instruction;
[0008] Generate a target prompt word according to the target combined document, the target invoice and the target prompt word structure;
[0009] Retrieve pre-trained multimodal models;
[0010] The target prompt word is input into the multimodal model to obtain an information matching result.
[0011] In a second aspect, an embodiment of the present application provides a combined document information matching device, which includes:
[0012] A construction unit, configured to respond to an information matching instruction for a target combined document and a target invoice, and to construct a target prompt word structure according to the information matching instruction;
[0013] A generating unit, configured to generate a target prompt word according to the target combined document, the target invoice and the target prompt word structure;
[0014] A calling unit, used to call a pre-trained multimodal model;
[0015] An input unit is used to input the target prompt word into the multimodal model to obtain an information matching result.
[0016] In a third aspect, an embodiment of the present application further provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the combined document information matching method described in the first aspect when executing the computer program.
[0017] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the combined document information matching method described in the first aspect above.
[0018] The embodiments of the present application provide a combined document information matching method, apparatus, device and medium, which can construct a target prompt word structure according to an information matching instruction, and generate a target prompt word according to a target combined document, a target invoice and a target prompt word structure, so as to restrict the output information of the model in a targeted manner according to the instruction requirements, and avoid incomplete output of valid information due to the output of invalid information; retrieve a pre-trained multimodal model, and input the target prompt word into the multimodal model to obtain an information matching result, thereby combining the multimodal model to achieve accurate understanding and matching of various structural data, improve generalization ability, and reduce the adaptation cost of the model to various structural data. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0020] Figure 1 A schematic diagram of an application scenario of the combined document information matching method provided in an embodiment of the present application;
[0021] Figure 2A schematic diagram of a flow chart of a combined document information matching method provided in an embodiment of the present application;
[0022] Figure 3 A schematic block diagram of a combined document information matching device provided in an embodiment of the present application;
[0023] Figure 4 A schematic block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0024] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0025] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.
[0026] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0027] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0028] See also Figure 1 and Figure 2 , Figure 1 A schematic diagram of an application scenario of the combined document information matching method provided in an embodiment of the present application; Figure 2 A flow chart of a combined document information matching method provided in an embodiment of the present application. The combined document information matching method is applied in a server and is executed by application software installed in the server.
[0029] like Figure 2 As shown, the method includes steps S101 to S104.
[0030] S101. In response to an information matching instruction for a target combined document and a target invoice, a target prompt word structure is constructed according to the information matching instruction.
[0031] In this embodiment, the technical solution is described with the server as the execution subject. The user terminal (such as a smart phone, tablet computer and other intelligent terminals) used by the user can interact with the server for data. Specifically, the server provides an information matching platform, and the user can log in to the information matching platform using the user terminal. The user interaction interface of the information matching platform is displayed on the terminal interface of the user terminal, and there is at least one image upload interface in the user interaction interface. When the user selects an image as the image to be identified and uploads it to the server through the image upload interface, subsequent information matching can be performed in the server.
[0032] In this embodiment, the information matching instruction can be triggered by relevant staff according to actual needs, or it can be automatically triggered when it is detected that an image is uploaded to the server.
[0033] In this embodiment, the target combined document may be data of various structural types such as regular table data, wireless table data, and non-table data in documents such as contract data, outbound orders, and inbound orders.
[0034] In this embodiment, the target invoice may be invoice data generated by both parties to the transaction during the transaction, an associated invoice of contract data, etc.
[0035] It is understandable that when matching combined document information, what needs to be output is often just a list, dictionary or other formatted data, but the output of the model is often mixed with a lot of explanatory information, which will increase the time consumption of model prediction and the difficulty of post-processing. In addition, when resources are limited, the maximum output length may be exceeded, resulting in only explanatory information in the output content, but lacking the required structured comparison results. Therefore, in order to solve this problem, this embodiment optimizes the prompt word structure.
[0036] Specifically, the step of constructing a target prompt word structure according to the information matching instruction includes:
[0037] Get each output item in the universal prompt word structure;
[0038] Obtaining mandatory items from each output item according to the information matching instruction, and determining the information length of the mandatory items;
[0039] Acquiring a model volume of the multimodal model;
[0040] Determining the output information length of the multimodal model according to the model volume;
[0041] Calculate the difference between the output information length of the multimodal model and the information length of the required item to obtain the remaining information length;
[0042] Obtaining other output items in the output items except the required items as each candidate item, and obtaining the priority and information length of each candidate item;
[0043] Selecting target candidate items from the candidate items according to the priority and information length of each candidate item and adding them to the required items until the total information length of the added candidate items is greater than the remaining information length, then stopping adding;
[0044] Combining the currently added candidate items with the required items to obtain the target prompt word structure;
[0045] The sum of the information lengths of the candidate items currently added and the mandatory items is less than or equal to the output information length.
[0046] For example, the general prompt word structure may include: identity specification + matching rule description + output format description + output content description + special attention description + example.
[0047] The identity designation refers to the designation of the model identity, such as designating the model to output data as a senior engineer.
[0048] The matching rule description refers to the description of the matching method, such as matching by row.
[0049] The output format description refers to the specification of the output format, such as outputting in JSON (JavaScript Object Notation) format.
[0050] The output content description refers to the restrictions on the output content, such as outputting product names, product models, etc.
[0051] The special note refers to a special note on the output content, such as: do not output *.
[0052] The instance refers to an output example. The number of the instance may include one or more.
[0053] When the model volume of the multimodal model is 7b, the corresponding output information length is 2500 characters. If the output content description as a required item takes up 2000 characters, the remaining information length is 500 characters. At this time, you can try to add the remaining items in the general prompt word structure one by one. For example, on this basis, the shortest one in the instance will be selected for trial. If no other information can be added after the shortest instance is added, the dynamic end; after adding an instance, the identity designation is added first. If the output information length does not exceed the length after adding the identity designation, other instances are added in sequence until the complete instance cannot be added.
[0054] In the above embodiment, the prompt word structure can be configured in a targeted manner according to the model performance and usage requirements to improve the effectiveness of information matching output and prevent the maximum output length from exceeding under limited resources.
[0055] S102: Generate a target prompt word according to the target combined document, the target invoice and the target prompt word structure.
[0056] In this embodiment, the generating of the target prompt word according to the target combined document, the target invoice and the target prompt word structure includes:
[0057] Obtaining an initial prompt word of the multimodal model;
[0058] The target combined document and the target invoice are added to the initial prompt word according to the target prompt word structure to obtain the target prompt word.
[0059] Through the above embodiments, it is possible to implement effective constraints on the model output according to the constructed target prompt words, which not only enables effective information to be acquired but also reduces the amount of calculation of the model, thereby improving the information matching efficiency of the model.
[0060] S103, retrieve a pre-trained multimodal model.
[0061] In this embodiment, before calling the pre-trained multimodal model, the method further includes:
[0062] Parsing the information matching instruction to obtain model performance parameters;
[0063] Acquire an initial model based on multimodal pre-training according to the model performance parameters;
[0064] Construct a training set;
[0065] The initial model is trained using the training set to obtain the multimodal model.
[0066] For example: when it is determined based on the model performance parameters that only a lightweight model is needed to meet the information matching requirements, a pre-trained lightweight model can be selected as the initial model, which not only reduces the model deployment cost but also ensures the model performance; and retraining based on the pre-trained model can also effectively reduce the training cost and improve the training efficiency.
[0067] In the above embodiment, the initial model can be obtained specifically according to the requirements for model performance in the actual usage scenario, so that the model can be just adapted to meet the usage needs and reduce the cost of use. At the same time, the pre-trained initial model is obtained and retrained on this basis, which not only ensures the stability of the model, but also effectively reduces the training time and the amount of calculation during the training process, thereby further saving costs.
[0068] In this embodiment, constructing a training set includes:
[0069] Get historical training sets;
[0070] Cleaning the data in the historical training set to obtain a training set to be processed;
[0071] Randomly cropping the data in the training set to be processed to obtain a first extended training set;
[0072] Rotating the data in the first extended training set at a random angle to obtain a second extended training set;
[0073] The data in the second extended training set is flipped to obtain the training set.
[0074] In the above embodiment, the training set is constructed based on the historical training set, which can ensure the data quality and availability. At the same time, the data is expanded by trimming and other methods to increase the data volume, so that the model has sufficient data for training, further ensuring the training effect of the model.
[0075] In this embodiment, the using the training set to train the initial model to obtain the multimodal model includes:
[0076] Splitting the training set into a plurality of subsets;
[0077] Get the server corresponding to each subset;
[0078] Performing distributed training on the initial model based on the multiple subsets and the servers corresponding to each subset to obtain a first candidate model;
[0079] Deleting redundant neurons and redundant connections in the first candidate model to obtain a second candidate model;
[0080] Converting the second candidate model into configuration accuracy to obtain the multimodal model;
[0081] The configuration accuracy is lower than the current accuracy of the second candidate model.
[0082] In the above embodiment, distributed training can effectively improve the training efficiency of the model, and de-redundancy operations are performed on the model obtained after training, which can make the model more lightweight, thereby further reducing the deployment cost of the model. At the same time, by reducing the accuracy of the model, the size of the model can be further reduced to reduce the space occupied by the model.
[0083] S104: input the target prompt word into the multimodal model to obtain an information matching result.
[0084] In this embodiment, after obtaining the information matching result, the method further includes:
[0085] When the information matching result is that the information of the target combined document does not match the information of the target invoice, generating a warning prompt according to the information matching result;
[0086] The warning prompt is sent to the designated terminal device.
[0087] The early warning prompt may be sent to the designated terminal device by email or message, so as to assist in timely handling of data anomalies.
[0088] The designated terminal device may be a smart terminal such as a mobile phone or computer of the relevant staff member.
[0089] Among them, the unmatched information in the information matching results can be specially marked to assist relevant staff to quickly locate the anomaly.
[0090] For example, this embodiment can be used to match the commodity information in the invoice with the commodity information in the contract to avoid affecting the security of the transaction due to information mismatch.
[0091] It can be seen from the above technical scheme that the present application can construct a target prompt word structure according to information matching instructions, and generate target prompt words according to the target combination document, target invoice and target prompt word structure, so as to restrict the output information of the model in a targeted manner according to the instruction requirements, and avoid incomplete output of valid information due to the output of invalid information; call the pre-trained multimodal model, and input the target prompt words into the multimodal model to obtain the information matching result, so as to combine the multimodal model to realize accurate understanding and matching of various structural data, improve the generalization ability, and reduce the adaptation cost of the model to various structural data.
[0092] The present application also provides a combined document information matching device, which is used to perform any embodiment of the combined document information matching method described above. Figure 3 , Figure 3 It is a schematic block diagram of the combined document information matching device 100 provided in an embodiment of the present application.
[0093] Among them, Figure 3 As shown, the combined document information matching device 100 includes a construction unit 101, a generation unit 102, a retrieval unit 103, and an input unit 104.
[0094] The construction unit 101 is used to respond to the information matching instruction of the target combined document and the target invoice, and to construct the target prompt word structure according to the information matching instruction.
[0095] In this embodiment, the technical solution is described with the server as the execution subject. The user terminal (such as a smart phone, tablet computer and other intelligent terminals) used by the user can interact with the server for data. Specifically, the server provides an information matching platform, and the user can log in to the information matching platform using the user terminal. The user interaction interface of the information matching platform is displayed on the terminal interface of the user terminal, and there is at least one image upload interface in the user interaction interface. When the user selects an image as the image to be identified and uploads it to the server through the image upload interface, subsequent information matching can be performed in the server.
[0096] In this embodiment, the information matching instruction can be triggered by relevant staff according to actual needs, or it can be automatically triggered when it is detected that an image is uploaded to the server.
[0097] In this embodiment, the target combined document may be data of various structural types such as regular table data, wireless table data, and non-table data in documents such as contract data, outbound orders, and inbound orders.
[0098] In this embodiment, the target invoice may be invoice data generated by both parties to the transaction during the transaction, an associated invoice of contract data, etc.
[0099] It is understandable that when matching combined document information, what needs to be output is often just a list, dictionary or other formatted data, but the output of the model is often mixed with a lot of explanatory information, which will increase the time consumption of model prediction and the difficulty of post-processing. In addition, when resources are limited, the maximum output length may be exceeded, resulting in only explanatory information in the output content, but lacking the required structured comparison results. Therefore, in order to solve this problem, this embodiment optimizes the prompt word structure.
[0100] Specifically, the construction unit 101 constructs a target prompt word structure according to the information matching instruction, including:
[0101] Get each output item in the universal prompt word structure;
[0102] Obtaining mandatory items from each output item according to the information matching instruction, and determining the information length of the mandatory items;
[0103] Acquiring a model volume of the multimodal model;
[0104] Determining the output information length of the multimodal model according to the model volume;
[0105] Calculate the difference between the output information length of the multimodal model and the information length of the required item to obtain the remaining information length;
[0106] Obtaining other output items in the output items except the required items as each candidate item, and obtaining the priority and information length of each candidate item;
[0107] Selecting target candidate items from the candidate items according to the priority and information length of each candidate item and adding them to the required items until the total information length of the added candidate items is greater than the remaining information length, then stopping adding;
[0108] Combining the currently added candidate items with the required items to obtain the target prompt word structure;
[0109] The sum of the information lengths of the candidate items currently added and the mandatory items is less than or equal to the output information length.
[0110] For example, the general prompt word structure may include: identity specification + matching rule description + output format description + output content description + special attention description + example.
[0111] The identity designation refers to the designation of the model identity, such as designating the model to output data as a senior engineer.
[0112] The matching rule description refers to the description of the matching method, such as matching by row.
[0113] The output format description refers to the specification of the output format, such as outputting in JSON (JavaScript Object Notation) format.
[0114] The output content description refers to the restrictions on the output content, such as outputting product names, product models, etc.
[0115] The special note refers to a special note on the output content, such as: do not output *.
[0116] The instance refers to an output example. The number of the instance may include one or more.
[0117] When the model volume of the multimodal model is 7b, the corresponding output information length is 2500 characters. If the output content description as a required item takes up 2000 characters, the remaining information length is 500 characters. At this time, you can try to add the remaining items in the general prompt word structure one by one. For example, on this basis, the shortest one in the instance will be selected for trial. If no other information can be added after the shortest instance is added, the dynamic end; after adding an instance, the identity designation is added first. If the output information length does not exceed the length after adding the identity designation, other instances are added in sequence until the complete instance cannot be added.
[0118] In the above embodiment, the prompt word structure can be configured in a targeted manner according to the model performance and usage requirements to improve the effectiveness of information matching output and prevent the maximum output length from exceeding under limited resources.
[0119] The generating unit 102 is used to generate a target prompt word according to the target combined document, the target invoice and the target prompt word structure.
[0120] In this embodiment, the generating unit 102 generates a target prompt word according to the target combined document, the target invoice and the target prompt word structure, including:
[0121] Obtaining an initial prompt word of the multimodal model;
[0122] The target combined document and the target invoice are added to the initial prompt word according to the target prompt word structure to obtain the target prompt word.
[0123] Through the above embodiments, it is possible to implement effective constraints on the model output according to the constructed target prompt words, which not only enables effective information to be acquired but also reduces the amount of calculation of the model, thereby improving the information matching efficiency of the model.
[0124] The retrieving unit 103 is used to retrieve a pre-trained multimodal model.
[0125] In this embodiment, before calling the pre-trained multimodal model, the information matching instruction is parsed to obtain the model performance parameters;
[0126] Acquire an initial model based on multimodal pre-training according to the model performance parameters;
[0127] Construct a training set;
[0128] The initial model is trained using the training set to obtain the multimodal model.
[0129] For example: when it is determined based on the model performance parameters that only a lightweight model is needed to meet the information matching requirements, a pre-trained lightweight model can be selected as the initial model, which not only reduces the model deployment cost but also ensures the model performance; and retraining based on the pre-trained model can also effectively reduce the training cost and improve the training efficiency.
[0130] In the above embodiment, the initial model can be obtained specifically according to the requirements for model performance in the actual usage scenario, so that the model can be just adapted to meet the usage needs and reduce the cost of use. At the same time, the pre-trained initial model is obtained and retrained on this basis, which not only ensures the stability of the model, but also effectively reduces the training time and the amount of calculation during the training process, thereby further saving costs.
[0131] In this embodiment, constructing a training set includes:
[0132] Get historical training sets;
[0133] Cleaning the data in the historical training set to obtain a training set to be processed;
[0134] Randomly cropping the data in the training set to be processed to obtain a first extended training set;
[0135] Rotating the data in the first extended training set at a random angle to obtain a second extended training set;
[0136] The data in the second extended training set is flipped to obtain the training set.
[0137] In the above embodiment, the training set is constructed based on the historical training set, which can ensure the data quality and availability. At the same time, the data is expanded by trimming and other methods to increase the data volume, so that the model has sufficient data for training, further ensuring the training effect of the model.
[0138] In this embodiment, the using the training set to train the initial model to obtain the multimodal model includes:
[0139] Splitting the training set into a plurality of subsets;
[0140] Get the server corresponding to each subset;
[0141] Performing distributed training on the initial model based on the multiple subsets and the servers corresponding to each subset to obtain a first candidate model;
[0142] Deleting redundant neurons and redundant connections in the first candidate model to obtain a second candidate model;
[0143] Converting the second candidate model into configuration accuracy to obtain the multimodal model;
[0144] The configuration accuracy is lower than the current accuracy of the second candidate model.
[0145] In the above embodiment, distributed training can effectively improve the training efficiency of the model, and de-redundancy operations are performed on the model obtained after training, which can make the model more lightweight, thereby further reducing the deployment cost of the model. At the same time, by reducing the accuracy of the model, the size of the model can be further reduced to reduce the space occupied by the model.
[0146] The input unit 104 is used to input the target prompt word into the multimodal model to obtain an information matching result.
[0147] In this embodiment, after obtaining the information matching result, when the information matching result is that the information of the target combined document does not match the information of the target invoice, a warning prompt is generated according to the information matching result;
[0148] The warning prompt is sent to the designated terminal device.
[0149] The early warning prompt may be sent to the designated terminal device by email or message, so as to assist in timely handling of data anomalies.
[0150] The designated terminal device may be a smart terminal such as a mobile phone or computer of the relevant staff member.
[0151] Among them, the unmatched information in the information matching results can be specially marked to assist relevant staff to quickly locate the anomaly.
[0152] For example, this embodiment can be used to match the commodity information in the invoice with the commodity information in the contract to avoid affecting the security of the transaction due to information mismatch.
[0153] It can be seen from the above technical scheme that the present application can construct a target prompt word structure according to information matching instructions, and generate target prompt words according to the target combination document, target invoice and target prompt word structure, so as to restrict the output information of the model in a targeted manner according to the instruction requirements, and avoid incomplete output of valid information due to the output of invalid information; call the pre-trained multimodal model, and input the target prompt words into the multimodal model to obtain the information matching result, so as to combine the multimodal model to realize accurate understanding and matching of various structural data, improve the generalization ability, and reduce the adaptation cost of the model to various structural data.
[0154] The combined document information matching device can be implemented in the form of a computer program. The computer program can be implemented in the form of a computer program. Figure 4 Runs on the computer device shown.
[0155] See also Figure 4 , Figure 4 4 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 400 is a server, or a server cluster. The server can be an independent server, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.
[0156] See also Figure 4 The computer device 400 includes a processor 402 , a memory and a network interface 405 connected via a device bus 401 , wherein the memory may include a storage medium 403 and an internal memory 404 .
[0157] The storage medium 403 may store an operating system 4031 and a computer program 4032. When the computer program 4032 is executed, the processor 402 may execute the combined document information matching method.
[0158] The processor 402 is used to provide computing and control capabilities to support the operation of the entire computer device 400 .
[0159] The internal memory 404 provides an environment for the operation of the computer program 4032 in the storage medium 403. When the computer program 4032 is executed by the processor 402, the processor 402 can execute the combined document information matching method.
[0160] The network interface 405 is used for network communication, such as providing data information transmission, etc. Those skilled in the art will understand that Figure 4 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device 400 to which the solution of the present application is applied. The specific computer device 400 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0161] The processor 402 is used to run a computer program 4032 stored in the memory to implement the combined document information matching method disclosed in the embodiment of the present application.
[0162] Those skilled in the art will understand that Figure 4The embodiments of the computer device shown in the figure do not constitute a limitation on the specific composition of the computer device. In other embodiments, the computer device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, in some embodiments, the computer device may only include a memory and a processor. In such embodiments, the structure and function of the memory and the processor are the same as those of the embodiment of the present invention. Figure 4 The embodiments shown are consistent and will not be described again here.
[0163] It should be understood that in the embodiment of the present application, the processor 402 may be a central processing unit (CPU), and the processor 402 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0164] In another embodiment of the present application, a computer-readable storage medium is provided. The computer-readable storage medium may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the combined document information matching method disclosed in the embodiment of the present application is implemented.
[0165] It should be noted that the data involved in this case were all obtained legally.
[0166] It will be clear to those skilled in the art that, for the convenience and simplicity of description, the specific working process of the above-described equipment, devices and units can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. It will be appreciated by those of ordinary skill in the art that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.
[0167] In the several embodiments provided in the present application, it should be understood that the disclosed equipment, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, and units with the same function may be combined into one unit. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices or units, or may be electrical, mechanical or other forms of connection.
[0168] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present application.
[0169] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0170] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a background server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a disk or an optical disk.
[0171] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A combined document information matching method, characterized in that: include: In response to an information matching instruction for a target combined document and a target invoice, constructing a target prompt word structure according to the information matching instruction; Generate a target prompt word according to the target combined document, the target invoice and the target prompt word structure; Retrieve pre-trained multimodal models; The target prompt word is input into the multimodal model to obtain an information matching result.
2. The combined document information matching method according to claim 1, characterized in that: The step of constructing a target prompt word structure according to the information matching instruction includes: Get each output item in the universal prompt word structure; Obtaining mandatory items from each output item according to the information matching instruction, and determining the information length of the mandatory items; Acquiring a model volume of the multimodal model; Determining the output information length of the multimodal model according to the model volume; Calculate the difference between the output information length of the multimodal model and the information length of the required item to obtain the remaining information length; Obtaining other output items in the output items except the required items as each candidate item, and obtaining the priority and information length of each candidate item; Selecting target candidate items from the candidate items according to the priority and information length of each candidate item and adding them to the required items until the total information length of the added candidate items is greater than the remaining information length, then stopping adding; Combining the currently added candidate items with the required items to obtain the target prompt word structure; The sum of the information lengths of the candidate items currently added and the mandatory items is less than or equal to the output information length.
3. The combined document information matching method according to claim 1, characterized in that: The generating of the target prompt word according to the target combined document, the target invoice and the target prompt word structure includes: Obtaining an initial prompt word of the multimodal model; The target combined document and the target invoice are added to the initial prompt word according to the target prompt word structure to obtain the target prompt word.
4. The combined document information matching method according to claim 1, characterized in that: Before calling the pre-trained multimodal model, the method further includes: Parsing the information matching instruction to obtain model performance parameters; Acquire an initial model based on multimodal pre-training according to the model performance parameters; Construct a training set; The initial model is trained using the training set to obtain the multimodal model.
5. The combined document information matching method according to claim 4, characterized in that: The constructing of the training set comprises: Get historical training sets; Cleaning the data in the historical training set to obtain a training set to be processed; Randomly cropping the data in the training set to be processed to obtain a first extended training set; Rotating the data in the first extended training set at a random angle to obtain a second extended training set; The data in the second extended training set is flipped to obtain the training set.
6. The combined document information matching method according to claim 4, characterized in that: The using the training set to train the initial model to obtain the multimodal model includes: Splitting the training set into a plurality of subsets; Get the server corresponding to each subset; Performing distributed training on the initial model based on the multiple subsets and the servers corresponding to each subset to obtain a first candidate model; Deleting redundant neurons and redundant connections in the first candidate model to obtain a second candidate model; Converting the second candidate model into configuration accuracy to obtain the multimodal model; The configuration accuracy is lower than the current accuracy of the second candidate model.
7. The combined document information matching method according to claim 1, characterized in that: After obtaining the information matching result, the method further includes: When the information matching result is that the information of the target combined document does not match the information of the target invoice, generating a warning prompt according to the information matching result; The warning prompt is sent to the designated terminal device.
8. A combined document information matching device, characterized in that: include: A construction unit, configured to respond to an information matching instruction for a target combined document and a target invoice, and to construct a target prompt word structure according to the information matching instruction; A generating unit, configured to generate a target prompt word according to the target combined document, the target invoice and the target prompt word structure; A calling unit, used to call a pre-trained multimodal model; An input unit is used to input the target prompt word into the multimodal model to obtain an information matching result.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the combined document information matching method according to any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the combined document information matching method according to any one of claims 1 to 7.