Audit method, device and equipment based on visual language model and large language model
By combining visual language models with large language models, the application materials are processed automatically to generate review conclusions, solving the problem of low efficiency in manual review in existing technologies and realizing an efficient and reliable review process.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHENZHEN TAIJI SOFTWARE CO LTD
- Filing Date
- 2025-12-29
- Publication Date
- 2026-04-24
AI Technical Summary
The process of obtaining the review results of existing application materials is cumbersome, consumes a lot of human resources and time, and is easily affected by human intervention, resulting in low efficiency.
The review method adopts a visual language model and a large language model. It receives application materials through a data transmission channel, performs image processing, feature extraction and fusion, and uses the trained large language model to generate review conclusions, thus avoiding human intervention.
This reduced the time required to obtain audit conclusions, improved the efficiency and reliability of obtaining audit conclusions, and ensured the consistency and reliability of audit conclusions.
Smart Images
Figure CN121921607A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of artificial intelligence technology and review technology, and in particular to review methods, apparatus and equipment based on visual language models and large language models. Background Technology
[0002] Reviewing the application materials is a core step in ensuring the standardization, legality, and effectiveness of relevant business processes. Its main purpose is to control the authenticity, completeness, and compliance of the application information. Through review, it is possible to verify whether the applicant's qualifications meet the requirements, whether the application content is true and accurate, whether the required materials are complete, and whether the information filled in complies with relevant policy standards and procedural regulations, thereby preventing problems such as false applications, missing materials, and inconsistent information.
[0003] However, the current process for obtaining the review conclusions of submitted materials is cumbersome, which hinders the efficiency of obtaining these conclusions. This is because existing technologies primarily rely on manual review to obtain these conclusions. Manual review consumes significant human and time resources, increasing the time required to obtain the review conclusions. Furthermore, it is susceptible to human intervention, thus hindering the improvement of efficiency in obtaining review conclusions. Summary of the Invention
[0004] This application provides a method, apparatus, and device for reviewing application materials based on visual language models and large language models, in order to solve the technical problem that the process of obtaining review conclusions for existing application materials is cumbersome and not conducive to improving the efficiency of obtaining review conclusions.
[0005] In a first aspect, embodiments of this application provide an auditing method based on a visual language model and a large language model, applied to a server-side device, the auditing method comprising: Establish a data transmission channel between the server-side device and the client-side device. Receive the currently submitted application materials uploaded by the client-side device through the data transmission channel. The currently submitted application materials include the image of the business license to be reviewed and the corresponding application content of the image of the business license to be reviewed. For the application content corresponding to the business license image to be reviewed, typos are removed to generate the processed application content. For the business license image to be reviewed, noise removal and angle correction are performed to generate the processed business license image. The processed business license image is input into the visual language model. The visual encoder in the visual language model generates the visual features of the processed business license image. The optical character recognition module of the visual language model generates the text content of the processed business license image. A text feature extraction module is used to extract features from the processed application content and the processed business license image, respectively, to obtain the text features of the processed application content and the text features of the business license image. Through a multimodal fusion model, the visual features of the processed business license image, the text features of the processed application content, and the text features of the business license image are fused to generate the fused features of the current application material. The fusion features of the current application materials are submitted to the query interface of the knowledge base. The fusion features of the current application materials are queried through the query interface to obtain the review rules of the current application materials. Features are extracted from the review rules of the current application materials to generate the features of the review rules of the current application materials. The fusion features and the features of the review rules of the current application materials are input into the trained large language model. The review conclusion of the current application materials is generated through the trained large language model.
[0006] In one possible implementation of the first aspect, before establishing a data transmission channel between the server device and the client device, and receiving the currently submitted application materials uploaded by the client device through the data transmission channel, wherein the currently submitted application materials include an image of a business license to be reviewed and the application content corresponding to the image of the business license to be reviewed, the review method includes: Obtain the preset application materials and the review conclusions of the preset application materials, combine the preset application materials and the review conclusions of the preset application materials into an review sample, and combine the different review samples into a training set; The large language model is trained using a training set. The loss value of the large language model on the training set is obtained through a classification loss function. When the loss value of the large language model on the training set is less than a preset value, the trained large language model is saved.
[0007] In one possible implementation of the first aspect, the establishment of a data transmission channel between the server device and the client device, through which the current application materials uploaded by the client device are received, includes an image of the business license to be reviewed and the corresponding application content of the image of the business license to be reviewed, including: Obtain the connection request uploaded by the client device. The connection request carries the device identifier of the client device. When the device identifier is in the preset identifier list, a data transmission channel is established between the server device and the client device. The current application materials uploaded by the client device are received through the data transmission channel. The current application materials include the business license image to be reviewed and the application content corresponding to the business license image to be reviewed.
[0008] In one possible implementation of the first aspect, the declaration content corresponding to the business license image to be reviewed is subjected to typo removal processing to generate processed declaration content; the business license image to be reviewed is subjected to noise removal processing and angle correction processing to generate a processed business license image, including: The application content corresponding to the business license image to be reviewed is transmitted to the error correction interface. The error correction interface performs typo removal processing on the application content corresponding to the business license image to be reviewed and generates the processed application content. The image of the business license to be reviewed is transmitted to the image processing interface. The image processing interface performs noise removal and angle correction on the image of the business license to be reviewed, and generates the processed business license image.
[0009] In one possible implementation of the first aspect, the step of submitting the fusion features of the current application materials to a query interface of the knowledge base, querying the fusion features of the current application materials through the query interface to obtain the review rules of the current application materials, extracting features from the review rules of the current application materials to generate features of the review rules of the current application materials, inputting the fusion features and the features of the review rules of the current application materials into a trained large language model, and generating a review conclusion of the current application materials through the trained large language model, including: Using the encapsulation format corresponding to the knowledge base, the fusion features and query identifiers of the application materials are encapsulated, a query statement of the fusion features of the application materials is generated, the fusion features of the current application materials are submitted to the query engine of the knowledge base, and the review rules of the current application materials are obtained by querying the fusion features of the current application materials through the query engine. A text feature extraction module is used to extract features from the review rules of the current application materials, generating features of the review rules of the current application materials. The fusion features of the current application materials and the features of the review rules of the current application materials are input into the trained large language model, and the review conclusion of the current application materials is generated through the trained large language model.
[0010] In one possible implementation of the first aspect, after submitting the fusion features of the current application materials to the query interface of the knowledge base, querying the fusion features of the current application materials through the query interface to obtain the review rules of the current application materials, extracting features from the review rules of the current application materials to generate features of the review rules of the current application materials, inputting the fusion features of the current application materials and the features of the review rules of the current application materials into the trained large language model, and generating the review conclusion of the current application materials through the trained large language model, the review method includes: The review results of the current application materials are sent to the client device through the data transmission channel.
[0011] In one possible implementation of the first aspect, the audit conclusion is either pass or fail.
[0012] In one possible implementation of the first aspect, the encapsulation format is JSON or XML.
[0013] Secondly, embodiments of this application provide an auditing device based on a visual language model and a large language model, applied to a server-side device, comprising: The receiving module is used to establish a data transmission channel between the server device and the client device. It receives the current application materials uploaded by the client device through the data transmission channel. The current application materials include the business license image to be reviewed and the application content corresponding to the business license image to be reviewed. The processing module is used to perform typo removal processing on the application content corresponding to the business license image to be reviewed, and generate the processed application content. It also performs noise removal and angle correction processing on the business license image to be reviewed, and generates the processed business license image. The first generation module is used to input the processed business license image into the visual language model, generate the visual features of the processed business license image through the visual encoder in the visual language model, and generate the text content of the processed business license image through the optical character recognition module of the visual language model. The extraction module is used to extract features from the processed application content and the processed business license image using the text feature extraction module, respectively, to obtain the text features of the processed application content and the text features of the business license image. Through a multimodal fusion model, the visual features of the processed business license image, the text features of the processed application content, and the text features of the business license image are fused to generate the fused features of the current application material. The review module is used to submit the fusion features of the current application materials to the query interface of the knowledge base. The query interface queries the fusion features of the current application materials to obtain the review rules of the current application materials. Features are extracted from the review rules of the current application materials to generate the features of the review rules of the current application materials. The fusion features and the features of the review rules of the current application materials are input into the trained large language model. The trained large language model generates the review conclusion of the current application materials.
[0014] Thirdly, embodiments of this application provide a server device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the auditing method described in the first aspect above.
[0015] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the auditing method described in the first aspect above.
[0016] Fifthly, embodiments of this application provide a computer program product that, when run on a server device, causes the server device to execute the auditing method described in the first aspect.
[0017] The beneficial effects of the embodiments of this application are as follows: Firstly, the fusion features of the current application materials are submitted to the query interface of the knowledge base. The fusion features of the current application materials are queried through the query interface to obtain the review rules of the current application materials. Features of the review rules of the current application materials are extracted to generate the features of the review rules of the current application materials. The fusion features and the features of the review rules of the current application materials are input into the trained large language model. The review conclusion of the current application materials is generated through the trained large language model. Since the current application materials do not require manual review, the time for obtaining the review conclusion of the current application materials is reduced, which is conducive to improving the efficiency of obtaining the review conclusion of the current application materials. Secondly, the review conclusions of the current application materials are generated by the trained large language model, which is not affected by human intervention, thus improving the reliability of the review conclusions of the current application materials. Attached Figure Description
[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 An application scenario diagram of the review method provided in the embodiments of this application; Figure 2 This is a flowchart illustrating the review method provided in an embodiment of this application; Figure 3 A flowchart illustrating the implementation of S205 provided in this application embodiment; Figure 4 A schematic block diagram of the review device provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of the server device provided in an embodiment of this application. Detailed Implementation
[0020] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.
[0021] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.
[0022] It should be understood that in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance. The terms "comprising," "including," "having," and their variations all mean "including but not limited to," unless otherwise specifically emphasized.
[0023] Furthermore, the technical solutions of the various embodiments can be combined with each other, but only if they are based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed in this application.
[0024] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.
[0025] The auditing method provided in this application can be applied to server-side devices, including but not limited to cloud servers, tower servers, rack servers, and blade servers. This application does not impose any restrictions on the specific type of server-side device.
[0026] Please see Figure 1 , Figure 1 The application scenario diagram of the review method provided in the embodiments of this application is described in detail below: The server device receives the connection request uploaded by the client device. The connection request carries the device identifier of the client device. When the device identifier is in the preset identifier list, a data transmission channel is established between the server device and the client device. The server device receives the current application materials uploaded by the client device through the data transmission channel. The current application materials include the business license image to be reviewed and the application content corresponding to the business license image to be reviewed.
[0027] Client devices include, but are not limited to, mobile phones, tablets, wearable devices, in-vehicle devices, and laptops.
[0028] In this embodiment of the application, the server device receives the current application materials uploaded by the client device through a data transmission channel. The data transmission channel has a data verification and retransmission mechanism. When a data packet is detected to be lost, the server will promptly request the client to resend the lost data packet, thereby ensuring that the current application materials received by the server are complete.
[0029] Please see Figure 2 , Figure 2 This is a flowchart illustrating the review method provided in this application embodiment, which can be applied to server-side devices.
[0030] like Figure 2 As shown, the review method provided in this application includes the following steps, detailed below: S201, Establish a data transmission channel between the server device and the client device, and receive the current application materials uploaded by the client device through the data transmission channel. The current application materials include the business license image to be reviewed and the application content corresponding to the business license image to be reviewed. Specifically, before establishing a data transmission channel between the server-side device and the client-side device, and receiving the currently submitted application materials uploaded by the client-side device through the data transmission channel (the currently submitted application materials include an image of the business license to be reviewed and the corresponding application content), the review method includes: Obtain the preset application materials and the review conclusions of the preset application materials, combine the preset application materials and the review conclusions of the preset application materials into an review sample, and combine the different review samples into a training set; The large language model is trained using a training set. The loss value of the large language model on the training set is obtained through a classification loss function. When the loss value of the large language model on the training set is less than a preset value, the trained large language model is saved.
[0031] S202, For the application content corresponding to the business license image to be reviewed, perform typo removal processing to generate the processed application content; For the business license image to be reviewed, perform noise removal processing and angle correction processing to generate the processed business license image. The process involves: performing typo removal on the application content corresponding to the business license image to be reviewed, generating processed application content; and performing noise removal and angle correction on the business license image to be reviewed, generating a processed business license image, including: The application content corresponding to the business license image to be reviewed is transmitted to the error correction interface. The error correction interface performs typo removal processing on the application content corresponding to the business license image to be reviewed and generates the processed application content. The image of the business license to be reviewed is transmitted to the image processing interface. The image processing interface performs noise removal and angle correction on the image of the business license to be reviewed, and generates the processed business license image.
[0032] For example, the image of the business license to be reviewed is transmitted to an image processing interface. The image processing interface performs noise removal and angle correction processing on the image of the business license to be reviewed, generating a processed business license image, including: The image of the business license to be reviewed is transmitted to the image processing interface. After receiving the image of the business license to be reviewed, the image processing interface automatically starts the noise removal module and the angle correction module. The noise removal module is based on a preset image filtering algorithm to identify and filter interference information such as speckle noise, stripe noise, and background noise in the business license image to be reviewed, while retaining the text, borders, and seals in the business license image to be reviewed, thus obtaining a noise-removed business license image. The noise-removed business license image is transmitted to the angle correction module. The angle correction module extracts the rectangular border features of the noise-removed business license image through an edge detection algorithm, calculates the current tilt angle of the noise-removed business license image, and performs rotation correction on the noise-removed business license image when the current tilt angle is greater than the preset angle, so that the noise-removed business license image is restored to a horizontal and upright state, generating a processed business license image. The processed business license image can ensure that the text, patterns and other content are arranged neatly.
[0033] S203, input the processed business license image into the visual language model, generate the visual features of the processed business license image through the visual encoder in the visual language model, and generate the text content of the processed business license image through the optical character recognition module of the visual language model. The processed business license image is input into a visual language model. The visual encoder in the visual language model extracts the license border, text stroke edges, and seal outline of the business license image through the bottom convolutional layer. The license border, text stroke edges, and seal outline are then fused to generate visual features containing the processed business license image. The visual features of the business license image have a dimension of 128.
[0034] S204. The text feature extraction module is used to extract features from the processed application content and the processed business license image respectively, and the text features of the processed application content and the business license image are obtained respectively. Through the multimodal fusion model, the visual features of the processed business license image, the text features of the processed application content, and the text features of the business license image are fused to generate the fused features of the current application material. S205, submit the fusion features of the current application materials to the query interface of the knowledge base, query the fusion features of the current application materials through the query interface to obtain the review rules of the current application materials, extract features from the review rules of the current application materials to generate the features of the review rules of the current application materials, input the fusion features of the current application materials and the features of the review rules of the current application materials into the trained large language model, and generate the review conclusion of the current application materials through the trained large language model.
[0035] Among them, the trained large language model has powerful contextual understanding and detail recognition capabilities, which can accurately capture subtle problems in the materials, such as incomplete information, contradictory data, missing attachments, and non-standard formatting, which are easily overlooked by humans. At the same time, by continuously learning the review rules of the current application materials, the judgment ability of the trained large language model can be continuously iterated and optimized, further reducing the probability of missed or incorrect reviews and ensuring the reliability of the review conclusions of the current application materials.
[0036] The trained large language model follows the current review rules for application materials, which can completely avoid the judgment bias caused by subjective experience differences and emotional fluctuations in manual review, and ensure that the review standards are consistent for different applicants and at different review periods.
[0037] For ease of explanation, the following example is provided: For example, in the qualification application and review process of a certain industry, there are several vague judgment clauses involving enterprises, such as having no major illegal records in the past three years and meeting the requirements for the number of technical personnel.
[0038] During manual review, different reviewers may have different definitions of major violations, leading to inconsistent judgments for similar cases involving the same company. The trained big data model uses the review rules of the current application materials to conduct a unified check on the violations and personnel qualifications of the current application materials. This helps to improve the consistency of the review conclusions of the current application materials and effectively ensures the fairness of the review process.
[0039] The review result is either "pass" or "fail".
[0040] The review method, which involves submitting the fusion features of the current application materials to the knowledge base query interface, querying the fusion features of the current application materials through the query interface to obtain the review rules of the current application materials, extracting features from the review rules of the current application materials to generate the features of the review rules of the current application materials, inputting the fusion features and the features of the review rules of the current application materials into the trained large language model, and generating the review conclusion of the current application materials through the trained large language model, includes: The review results of the current application materials are sent to the client device through the data transmission channel.
[0041] The beneficial effects of the embodiments of this application are as follows: Firstly, the fusion features of the current application materials are submitted to the query interface of the knowledge base. The fusion features of the current application materials are queried through the query interface to obtain the review rules of the current application materials. Features of the review rules of the current application materials are extracted to generate the features of the review rules of the current application materials. The fusion features and the features of the review rules of the current application materials are input into the trained large language model. The review conclusion of the current application materials is generated through the trained large language model. Since the current application materials do not require manual review, the time for obtaining the review conclusion of the current application materials is reduced, which is conducive to improving the efficiency of obtaining the review conclusion of the current application materials. Secondly, the review conclusions of the current application materials are generated by the trained large language model, which is not affected by human intervention, thus improving the reliability of the review conclusions of the current application materials.
[0042] Please see Figure 3 , Figure 3 The implementation flowchart of S205 provided in the embodiments of this application is described in detail below: S301 uses the encapsulation format corresponding to the knowledge base to encapsulate the fusion features and query identifiers of the application materials, generates a query statement for the fusion features of the application materials, submits the fusion features of the current application materials to the query engine of the knowledge base, and queries the fusion features of the current application materials through the query engine to obtain the review rules for the current application materials. Among them, the encapsulation format is JSON format or XML format.
[0043] The Chinese full name of JSON: JavaScript Object Notation, and the English full name of JSON: JavaScript Object Notation.
[0044] The Chinese full name of XML: Extensible Markup Language, and the English full name of XML: Extensible Markup Language.
[0045] Both JSON format and XML format are structured data formats, which encapsulate the fusion features and query identifiers of the declaration materials, generate query statements for the fusion features of the declaration materials, and can reduce the error probability in the parsing of query statements.
[0046] S302. Use a text feature extraction module to extract features from the review rules of the current declaration materials, generate features of the review rules of the current declaration materials, input the fusion features of the current declaration materials and the features of the review rules of the current declaration materials into the trained large language model, and generate the review conclusion of the current declaration materials through the trained large language model.
[0047] In the embodiment of the present application, inputting the fusion features of the current declaration materials and the features of the review rules of the current declaration materials into the trained large language model, and generating the review conclusion of the current declaration materials through the trained large language model can avoid the conclusion deviation caused by subjective judgment and experience differences in the manual review process, ensure the unity of the judgment criteria for different declaration entities, and thus help improve the stability of the review conclusion of the current declaration materials.
[0048] Corresponding to the review method described in the above embodiment, please refer to Figure 4 , Figure 4 which is a schematic block diagram of the review device provided by the embodiment of the present application. Figure 4 The review device 400 shown can be applied to the server device in the application scenario diagram shown in Figure 1 . Taking the server device as an example, the review device 400 shown in Figure 4 will be elaborated in detail. The review device 400 may include a receiving module 401, a processing module 402, a first generating module 403, an extracting module 404, and a reviewing module 405.
[0049] The receiving module 401 is used to establish a data transmission channel between the server device and the client device, and receive the current declaration materials uploaded by the client device through the data transmission channel. The current declaration materials include the business license image to be reviewed and the declaration content corresponding to the business license image to be reviewed. The processing module 402 is used to perform typo removal processing on the declaration content corresponding to the business license image to be reviewed, and generate the processed declaration content. It also performs noise removal and angle correction processing on the business license image to be reviewed, and generates the processed business license image. The first generation module 403 is used to input the processed business license image into the visual language model, generate the visual features of the processed business license image through the visual encoder in the visual language model, and generate the text content of the processed business license image through the optical character recognition module of the visual language model. The extraction module 404 is used to extract features from the processed application content and the processed business license image using the text feature extraction module, respectively, to obtain the text features of the processed application content and the text features of the business license image. Through a multimodal fusion model, the visual features of the processed business license image, the text features of the processed application content, and the text features of the business license image are fused to generate the fused features of the current application material. The review module 405 is used to submit the fusion features of the current application materials to the query interface of the knowledge base. The fusion features of the current application materials are queried through the query interface to obtain the review rules of the current application materials. Features are extracted from the review rules of the current application materials to generate the features of the review rules of the current application materials. The fusion features and the features of the review rules of the current application materials are input into the trained large language model, and the review conclusion of the current application materials is generated through the trained large language model.
[0050] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0051] The beneficial effects of the embodiments of this application are as follows: Firstly, the fusion features of the current application materials are submitted to the query interface of the knowledge base. The fusion features of the current application materials are queried through the query interface to obtain the review rules of the current application materials. Features of the review rules of the current application materials are extracted to generate the features of the review rules of the current application materials. The fusion features and the features of the review rules of the current application materials are input into the trained large language model. The review conclusion of the current application materials is generated through the trained large language model. Since the current application materials do not require manual review, the time for obtaining the review conclusion of the current application materials is reduced, which is conducive to improving the efficiency of obtaining the review conclusion of the current application materials. Secondly, the review conclusions of the current application materials are generated by the trained large language model, which is not affected by human intervention, thus improving the reliability of the review conclusions of the current application materials.
[0052] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of the server device provided in an embodiment of this application.
[0053] like Figure 5 As shown, Figure 5 The server device 2 includes: at least one processor 20, a memory 21, and a computer program 22 stored in the memory 21 and executable on the at least one processor 20, wherein the processor 20 executes the computer program 22 to implement the steps in any of the above method embodiments.
[0054] The server-side device 2 may include, but is not limited to, a processor 20 and a memory 21. Those skilled in the art will understand that... Figure 5 This is merely an example of server device 2 and does not constitute a limitation on server device 2. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, etc.
[0055] The processor 20 is used to run a computer program 22 stored in the memory 21, and performs the following steps when executing the computer program 22: Establish a data transmission channel between the server-side device and the client-side device. Receive the currently submitted application materials uploaded by the client-side device through the data transmission channel. The currently submitted application materials include the image of the business license to be reviewed and the corresponding application content of the image of the business license to be reviewed. For the application content corresponding to the business license image to be reviewed, typos are removed to generate the processed application content. For the business license image to be reviewed, noise removal and angle correction are performed to generate the processed business license image. The processed business license image is input into the visual language model. The visual encoder in the visual language model generates the visual features of the processed business license image. The optical character recognition module of the visual language model generates the text content of the processed business license image. A text feature extraction module is used to extract features from the processed application content and the processed business license image, respectively, to obtain the text features of the processed application content and the text features of the business license image. Through a multimodal fusion model, the visual features of the processed business license image, the text features of the processed application content, and the text features of the business license image are fused to generate the fused features of the current application material. The fusion features of the current application materials are submitted to the query interface of the knowledge base. The fusion features of the current application materials are queried through the query interface to obtain the review rules of the current application materials. Features are extracted from the review rules of the current application materials to generate the features of the review rules of the current application materials. The fusion features and the features of the review rules of the current application materials are input into the trained large language model. The review conclusion of the current application materials is generated through the trained large language model.
[0056] The processor 20 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors, field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.
[0057] In some embodiments, the memory 21 may be an internal storage unit of the server device 2, such as a hard disk or memory of the server device 2. In other embodiments, the memory 21 may be an external storage device of the server device 2, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on the server device 2.
[0058] Furthermore, the memory 21 may include both internal storage units of the server device 2 and external storage devices. The memory 21 is used to store the operating system, applications, boot loader, data, and other programs, such as the program code of the computer program. The memory 21 can also be used to temporarily store data that has been output or will be output.
[0059] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.
[0060] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.
[0061] The computer-readable storage medium may also be an external storage device of the auditing device or server equipment, such as a plug-in hard drive, smart media card (SMC), secure digital (SD) card, flash card, or non-transitory computer-readable storage medium equipped on the auditing device or server equipment.
[0062] Since the computer program stored in the computer-readable storage medium can execute any of the auditing methods based on visual language models and large language models provided in the embodiments of this application, the computer-readable storage medium can achieve the beneficial effects that any of the auditing methods based on visual language models and large language models provided in the embodiments of this application can achieve, as detailed in the preceding embodiments, and will not be repeated here.
[0063] This application provides a computer program product that, when run on a server device, causes the server device to perform the aforementioned auditing method.
[0064] Once a computer program product is loaded by a server-side device, it can perform the following steps: Establish a data transmission channel between the server-side device and the client-side device. Receive the currently submitted application materials uploaded by the client-side device through the data transmission channel. The currently submitted application materials include the image of the business license to be reviewed and the corresponding application content of the image of the business license to be reviewed. For the application content corresponding to the business license image to be reviewed, typos are removed to generate the processed application content. For the business license image to be reviewed, noise removal and angle correction are performed to generate the processed business license image. The processed business license image is input into the visual language model. The visual encoder in the visual language model generates the visual features of the processed business license image. The optical character recognition module of the visual language model generates the text content of the processed business license image. A text feature extraction module is used to extract features from the processed application content and the processed business license image, respectively, to obtain the text features of the processed application content and the text features of the business license image. Through a multimodal fusion model, the visual features of the processed business license image, the text features of the processed application content, and the text features of the business license image are fused to generate the fused features of the current application material. The fusion features of the current application materials are submitted to the query interface of the knowledge base. The fusion features of the current application materials are queried through the query interface to obtain the review rules of the current application materials. Features are extracted from the review rules of the current application materials to generate the features of the review rules of the current application materials. The fusion features and the features of the review rules of the current application materials are input into the trained large language model. The review conclusion of the current application materials is generated through the trained large language model.
[0065] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.
[0066] Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium includes: an entity or device for carrying computer program code to a server device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium.
[0067] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0068] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.
Claims
1. A review method based on visual language models and large language models, characterized in that, The auditing method, applied to server-side devices, includes: Establish a data transmission channel between the server-side device and the client-side device. Receive the currently submitted application materials uploaded by the client-side device through the data transmission channel. The currently submitted application materials include the image of the business license to be reviewed and the corresponding application content of the image of the business license to be reviewed. For the application content corresponding to the business license image to be reviewed, typos are removed to generate the processed application content. For the business license image to be reviewed, noise removal and angle correction are performed to generate the processed business license image. The processed business license image is input into the visual language model. The visual encoder in the visual language model generates the visual features of the processed business license image. The optical character recognition module of the visual language model generates the text content of the processed business license image. A text feature extraction module is used to extract features from the processed application content and the processed business license image, respectively, to obtain the text features of the processed application content and the text features of the business license image. Through a multimodal fusion model, the visual features of the processed business license image, the text features of the processed application content, and the text features of the business license image are fused to generate the fused features of the current application material. The fusion features of the current application materials are submitted to the query interface of the knowledge base. The fusion features of the current application materials are queried through the query interface to obtain the review rules of the current application materials. Features are extracted from the review rules of the current application materials to generate the features of the review rules of the current application materials. The fusion features and the features of the review rules of the current application materials are input into the trained large language model. The review conclusion of the current application materials is generated through the trained large language model.
2. The auditing method according to claim 1, characterized in that, Before establishing a data transmission channel between the server and client devices, and receiving the currently submitted application materials uploaded by the client devices through the data transmission channel (the currently submitted application materials include an image of the business license to be reviewed and the corresponding application content), the review method includes: Obtain the pre-defined application materials and the review conclusions of the pre-defined application materials, combine the pre-defined application materials and the review conclusions of the pre-defined application materials into an review sample, and combine the different review samples into a training set; The large language model is trained using a training set. The loss value of the large language model on the training set is obtained through a classification loss function. When the loss value of the large language model on the training set is less than a preset value, the trained large language model is saved.
3. The auditing method according to claim 1, characterized in that, The process involves establishing a data transmission channel between the server and client devices. This channel is used to receive the currently submitted application materials uploaded by the client devices. The materials include an image of the business license awaiting review and the corresponding application content, including: Obtain the connection request uploaded by the client device. The connection request carries the device identifier of the client device. When the device identifier is in the preset identifier list, a data transmission channel is established between the server device and the client device. The current application materials uploaded by the client device are received through the data transmission channel. The current application materials include the business license image to be reviewed and the application content corresponding to the business license image to be reviewed.
4. The auditing method according to claim 1, characterized in that, The application content corresponding to the business license image to be reviewed undergoes typo removal processing to generate processed application content. The business license image to be reviewed undergoes noise removal and angle correction processing to generate a processed business license image, including: The application content corresponding to the business license image to be reviewed is transmitted to the error correction interface. The error correction interface performs typo removal processing on the application content corresponding to the business license image to be reviewed and generates the processed application content. The image of the business license to be reviewed is transmitted to the image processing interface. The image processing interface performs noise removal and angle correction on the image of the business license to be reviewed, and generates the processed business license image.
5. The auditing method according to claim 1, characterized in that, The process involves submitting the fusion features of the current application materials to a query interface in the knowledge base. The query interface is used to retrieve the fusion features of the current application materials, obtaining the review rules for the current application materials. Feature extraction is performed on the review rules to generate the features of the review rules. The fusion features and the features of the review rules are then input into a trained large language model. The trained large language model then generates the review conclusion for the current application materials, including: Using the encapsulation format corresponding to the knowledge base, the fusion features and query identifiers of the application materials are encapsulated, a query statement of the fusion features of the application materials is generated, the fusion features of the current application materials are submitted to the query engine of the knowledge base, and the review rules of the current application materials are obtained by querying the fusion features of the current application materials through the query engine. A text feature extraction module is used to extract features from the review rules of the current application materials, generating features of the review rules of the current application materials. The fusion features of the current application materials and the features of the review rules of the current application materials are input into the trained large language model, and the review conclusion of the current application materials is generated through the trained large language model.
6. The auditing method according to claim 1, characterized in that, The review method, after submitting the fusion features of the current application materials to the knowledge base query interface, queries the fusion features of the current application materials to obtain the review rules of the current application materials, extracts features from the review rules of the current application materials to generate the features of the review rules of the current application materials, inputs the fusion features and the features of the review rules of the current application materials into the trained large language model, and generates the review conclusion of the current application materials through the trained large language model, includes: The review results of the current application materials are sent to the client device through the data transmission channel.
7. The auditing method according to claim 1, characterized in that, The review result is either "pass" or "fail".
8. The auditing method according to claim 5, characterized in that, The encapsulation format is JSON or XML.
9. A verification device based on a visual language model and a large language model, characterized in that, Applied to server-side devices, including: The receiving module is used to establish a data transmission channel between the server device and the client device. It receives the current application materials uploaded by the client device through the data transmission channel. The current application materials include the image of the business license to be reviewed and the application content corresponding to the image of the business license to be reviewed. The processing module is used to perform typo removal processing on the application content corresponding to the business license image to be reviewed, and generate the processed application content. It also performs noise removal and angle correction processing on the business license image to be reviewed, and generates the processed business license image. The first generation module is used to input the processed business license image into the visual language model, generate the visual features of the processed business license image through the visual encoder in the visual language model, and generate the text content of the processed business license image through the optical character recognition module of the visual language model. The extraction module is used to extract features from the processed application content and the processed business license image using the text feature extraction module, respectively, to obtain the text features of the processed application content and the text features of the business license image. Through a multimodal fusion model, the visual features of the processed business license image, the text features of the processed application content, and the text features of the business license image are fused to generate the fused features of the current application material. The review module is used to submit the fusion features of the current application materials to the query interface of the knowledge base. The query interface queries the fusion features of the current application materials to obtain the review rules of the current application materials. Features are extracted from the review rules of the current application materials to generate the features of the review rules of the current application materials. The fusion features and the features of the review rules of the current application materials are input into the trained large language model. The trained large language model generates the review conclusion of the current application materials.
10. A server-side device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the auditing method as described in any one of claims 1 to 7.