Audit service data identification method and system based on multi-modal model
The multi-modal model-based auditing business data recognition method effectively addresses the limitations of traditional auditing methods by processing text, image, and audio data through intent recognition and semantic space models, resulting in enhanced auditing efficiency and accuracy.
Patent Information
- Application Number
- CN202411790214.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-06
- Publication Date
- 2025-03-28
AI Technical Summary
Traditional auditing methods, such as sampling and statistical inference, are inadequate for efficiently and accurately processing the diverse and voluminous data types generated in the big data era, including text, images, and audio, which are essential for modern auditing needs.
A multi-modal model-based auditing business data recognition method and system that involves processing text, image, and audio data through an intent recognition model, followed by a semantic space model to generate fusion feature vectors, which are then input into a multi-modal information association model for enhanced recognition results.
This approach significantly improves auditing efficiency and accuracy by integrating diverse data types, providing a robust and precise identification of auditing business data.
Smart Images
Figure CN119719904A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of audit business data identification, and in particular to an audit business data identification method and system based on a multimodal model. Background Art
[0002] With the advent of the big data era, the amount of data generated by enterprises has exploded, and the data types are becoming more diverse, including text, images, audio, video and other modalities. Traditional audit methods, such as those based on sampling and statistical inference, can no longer meet the needs of modern audits. It is crucial to process audit business quickly and accurately. Summary of the invention
[0003] The purpose of the embodiments of the present invention is to provide an audit business data identification method and system based on a multimodal model, which data identification method and system can significantly improve the efficiency and accuracy of the audit.
[0004] In order to achieve the above object, an embodiment of the present invention provides an audit business data identification method based on a multimodal model, and the data identification method includes:
[0005] Acquire audit business data to be identified, wherein the audit business data to be identified includes at least two of text to be identified, image to be identified, and audio to be identified;
[0006] Input the audit business data to be identified into an intention recognition model, and obtain an intention recognition result corresponding to the audit business data to be identified output by the intention recognition model;
[0007] Inputting the audit business data to be identified and the intention recognition result into a semantic space model to obtain a fusion feature vector of the audit business data to be identified output by the semantic space model;
[0008] The fused feature vector is input into a multimodal information association model to obtain an association identification result of the audit business data to be identified output by the multimodal information association model.
[0009] Optionally, the semantic space model includes a first neural network group, a second neural network group and a third neural network group.
[0010] Optionally, the audit business data to be identified and the intention recognition result are input into a semantic space model to obtain a fusion feature vector of the audit business data to be identified output by the semantic space model, including:
[0011] Inputting the audit business data to be identified and the intention recognition result into the first neural network group, and obtaining a multimodal feature vector corresponding to the audit business data to be identified output by the first neural network group;
[0012] Inputting the multimodal feature vector into the second neural network group, obtaining similarity measurement results and correlation measurement results of the audit business data to be identified output by the second neural network group;
[0013] The multimodal feature vector, the similarity measurement result and the correlation measurement result are input into the third neural network group to obtain the fused feature vector of the audit business data to be identified output by the third neural network group.
[0014] Optionally, the data identification method further includes:
[0015] Obtaining sample audit business data, an intent recognition result label corresponding to the sample audit business data, and a fusion feature vector corresponding to the sample audit business data;
[0016] The intention recognition model is trained to obtain the semantic space model.
[0017] Optionally, the semantic space model includes a feature extraction network and a classification network.
[0018] Optionally, inputting the audit business data to be identified into an intention recognition model to obtain an intention recognition result corresponding to the audit business data to be identified output by the intention recognition model includes:
[0019] Inputting the audit business data to be identified into the feature extraction network, and obtaining the intention feature vector corresponding to the audit business data to be identified output by the feature extraction network;
[0020] The intention feature vector is input into the classification network to obtain the intention recognition result corresponding to the audit business data to be identified output by the classification network.
[0021] Optionally, the audit business data to be identified is obtained, including:
[0022] Obtain original audit business data;
[0023] Performing data cleaning processing on the original audit business data to obtain first intermediate data;
[0024] Performing feature extraction processing on the first intermediate data to obtain second intermediate data;
[0025] The second intermediate data is subjected to noise reduction processing to obtain the audit business data to be identified.
[0026] On the other hand, the present invention also provides an audit business data identification system based on a multimodal model, wherein the data identification system includes a processor for executing any of the data identification methods described above.
[0027] Through the above technical solution, the embodiment of the present invention provides an audit business data identification method and system based on a multimodal model, by obtaining audit business data to be identified, the audit business data to be identified includes at least two of text to be identified, image to be identified, and audio to be identified; inputting the audit business data to be identified into an intention recognition model, obtaining the intention recognition result corresponding to the audit business data to be identified output by the intention recognition model; inputting the audit business data to be identified and the intention recognition result into a semantic space model, obtaining the fused feature vector of the audit business data to be identified output by the semantic space model; inputting the fused feature vector into a multimodal information association model, obtaining the association recognition result of the audit business data to be identified output by the multimodal information association model. The data identification method and system can significantly improve the efficiency and accuracy of auditing.
[0028] Other features and advantages of the embodiments of the present invention will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the accompanying drawings:
[0030] Figure 1 is a flow chart of a data identification method according to an embodiment of the present invention;
[0031] Figure 2 is a flow chart of obtaining an output fusion feature vector according to an embodiment of the present invention;
[0032] Figure 3 is a flow chart of obtaining a semantic space model according to an embodiment of the present invention;
[0033] Figure 4 is a flow chart of obtaining intention recognition results according to an embodiment of the present invention;
[0034] Figure 5 It is a flow chart of obtaining audit business data to be identified according to an embodiment of the present invention. DETAILED DESCRIPTION
[0035] The specific implementation of the embodiment of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the embodiment of the present invention, and is not used to limit the embodiment of the present invention.
[0036] It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of this application are in compliance with the relevant provisions of national laws and regulations. In the embodiments of this application, some existing solutions in the industry such as certain software, components, and models may be mentioned, which should be considered as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of this application, but it does not mean that the applicant has or will necessarily use the solution.
[0037] like Figure 1 The data identification method flow chart of one embodiment of the present invention is shown. Figure 1 In the data identification method, the data identification method may include:
[0038] In step S10, the audit business data to be identified is obtained, and the audit business data to be identified includes at least two of text to be identified, image to be identified, and audio to be identified;
[0039] In step S20, the audit business data to be identified is input into the intention recognition model to obtain the intention recognition result corresponding to the audit business data to be identified output by the intention recognition model;
[0040] In step S30, the audit business data to be identified and the intention recognition result are input into the semantic space model to obtain the fusion feature vector of the audit business data to be identified output by the semantic space model;
[0041] In step S40, the fused feature vector is input into the multimodal information association model to obtain the association identification result of the audit business data to be identified output by the multimodal information association model.
[0042] In this Figure 1 In the method shown, step S10 can be used to obtain audit business data to be identified, and the audit business data to be identified includes at least two of text to be identified, images to be identified, and audio to be identified. Step S20 can be used to input the audit business data to be identified into the intention recognition model to obtain the intention recognition result corresponding to the audit business data to be identified output by the intention recognition model. The specific method for obtaining the intention recognition result can be in various forms known to those skilled in the art. In one example of the present invention, it can include the following: Figure 4 The steps shown include:
[0043] In step S41, the audit business data to be identified is input into the feature extraction network to obtain the intention feature vector corresponding to the audit business data to be identified output by the feature extraction network;
[0044] In step S42, the intention feature vector is input into the classification network to obtain the intention recognition result corresponding to the audit business data to be identified output by the classification network.
[0045] Step S30 can be used to input the audit business data to be identified and the intention recognition result into the semantic space model to obtain the fused feature vector of the audit business data to be identified output by the semantic space model. The specific form of obtaining the fused feature vector can be various known to those skilled in the art. In one example of the present invention, it can include the following: Figure 2 The steps shown include:
[0046] In step S21, the audit business data to be identified and the intention recognition result are input into the first neural network group to obtain a multimodal feature vector corresponding to the audit business data to be identified output by the first neural network group;
[0047] In step S22, the multimodal feature vector is input into the second neural network group to obtain the similarity measurement result and the correlation measurement result of the audit business data to be identified output by the second neural network group;
[0048] In step S23, the multimodal feature vector, the similarity measurement result and the correlation measurement result are input into the third neural network group to obtain the fused feature vector of the audit business data to be identified output by the third neural network group.
[0049] In this embodiment, the semantic space model includes a first neural network group, a second neural network group and a third neural network group. Step S40 can be used to input the fused feature vector into the multimodal information association model to obtain the association identification result of the audit business data to be identified output by the multimodal information association model.
[0050] In this embodiment, the specific method of obtaining the semantic space model can be various forms known to those skilled in the art. In one example of the present invention, it can include: Figure 3 The steps shown include:
[0051] In step S31, sample audit business data, intention recognition result labels corresponding to the sample audit business data, and fusion feature vectors corresponding to the sample audit business data are obtained;
[0052] In step S32, the intent recognition model is trained to obtain a semantic space model.
[0053] In this embodiment, the semantic space model includes a feature extraction network and a classification network.
[0054] The specific method for obtaining the audit business data to be identified may be in various forms known to those skilled in the art. In one example of the present invention, the method may include: Figure 5 The steps shown include:
[0055] In step S51, original audit business data is obtained;
[0056] In step S52, the original audit business data is cleansed to obtain first intermediate data;
[0057] In step S53, feature extraction is performed on the first intermediate data to obtain second intermediate data;
[0058] In step S54, the second intermediate data is subjected to noise reduction processing to obtain the audit business data to be identified.
[0059] On the other hand, the present invention also provides an audit business data identification system based on a multimodal model, wherein the data identification system includes a processor for executing any of the data identification methods described above. The data identification method may include: Figures 1 to 5 The steps shown. Figure 1 In the data identification method, the data identification method may include:
[0060] In step S10, the audit business data to be identified is obtained, and the audit business data to be identified includes at least two of text to be identified, image to be identified, and audio to be identified;
[0061] In step S20, the audit business data to be identified is input into the intention recognition model to obtain the intention recognition result corresponding to the audit business data to be identified output by the intention recognition model;
[0062] In step S30, the audit business data to be identified and the intention recognition result are input into the semantic space model to obtain the fusion feature vector of the audit business data to be identified output by the semantic space model;
[0063] In step S40, the fused feature vector is input into the multimodal information association model to obtain the association identification result of the audit business data to be identified output by the multimodal information association model.
[0064] In this Figure 1 In the method shown, step S10 can be used to obtain audit business data to be identified, and the audit business data to be identified includes at least two of text to be identified, images to be identified, and audio to be identified. Step S20 can be used to input the audit business data to be identified into the intention recognition model to obtain the intention recognition result corresponding to the audit business data to be identified output by the intention recognition model. The specific method for obtaining the intention recognition result can be in various forms known to those skilled in the art. In one example of the present invention, it can include the following: Figure 4 The steps shown include:
[0065] In step S41, the audit business data to be identified is input into the feature extraction network to obtain the intention feature vector corresponding to the audit business data to be identified output by the feature extraction network;
[0066] In step S42, the intention feature vector is input into the classification network to obtain the intention recognition result corresponding to the audit business data to be identified output by the classification network.
[0067] Step S30 can be used to input the audit business data to be identified and the intention recognition result into the semantic space model to obtain the fused feature vector of the audit business data to be identified output by the semantic space model. The specific form of obtaining the fused feature vector can be various known to those skilled in the art. In one example of the present invention, it can include the following: Figure 2 The steps shown include:
[0068] In step S21, the audit business data to be identified and the intention recognition result are input into the first neural network group to obtain a multimodal feature vector corresponding to the audit business data to be identified output by the first neural network group;
[0069] In step S22, the multimodal feature vector is input into the second neural network group to obtain the similarity measurement result and the correlation measurement result of the audit business data to be identified output by the second neural network group;
[0070] In step S23, the multimodal feature vector, the similarity measurement result and the correlation measurement result are input into the third neural network group to obtain the fused feature vector of the audit business data to be identified output by the third neural network group.
[0071] In this embodiment, the semantic space model includes a first neural network group, a second neural network group and a third neural network group. Step S40 can be used to input the fused feature vector into the multimodal information association model to obtain the association identification result of the audit business data to be identified output by the multimodal information association model.
[0072] In this embodiment, the specific method of obtaining the semantic space model can be various forms known to those skilled in the art. In one example of the present invention, it can include: Figure 3 The steps shown include:
[0073] In step S31, sample audit business data, intention recognition result labels corresponding to the sample audit business data, and fusion feature vectors corresponding to the sample audit business data are obtained;
[0074] In step S32, the intent recognition model is trained to obtain a semantic space model.
[0075] In this embodiment, the semantic space model includes a feature extraction network and a classification network.
[0076] The specific method for obtaining the audit business data to be identified may be in various forms known to those skilled in the art. In one example of the present invention, the method may include: Figure 5 The steps shown include:
[0077] In step S51, original audit business data is obtained;
[0078] In step S52, the original audit business data is cleansed to obtain first intermediate data;
[0079] In step S53, feature extraction is performed on the first intermediate data to obtain second intermediate data;
[0080] In step S54, the second intermediate data is subjected to noise reduction processing to obtain the audit business data to be identified.
[0081] Through the above technical solution, the embodiment of the present invention provides an audit business data identification method and system based on a multimodal model, by obtaining audit business data to be identified, the audit business data to be identified includes at least two of text to be identified, image to be identified, and audio to be identified; inputting the audit business data to be identified into an intention recognition model, obtaining the intention recognition result corresponding to the audit business data to be identified output by the intention recognition model; inputting the audit business data to be identified and the intention recognition result into a semantic space model, obtaining the fused feature vector of the audit business data to be identified output by the semantic space model; inputting the fused feature vector into a multimodal information association model, obtaining the association recognition result of the audit business data to be identified output by the multimodal information association model. The data identification method and system can significantly improve the efficiency and accuracy of auditing.
[0082] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0083] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0084] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0085] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0086] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0087] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0088] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0089] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0090] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.
Claims
1. A method for identifying audit business data based on a multimodal model, characterized in that: The data identification method comprises: Acquire audit business data to be identified, wherein the audit business data to be identified includes at least two of text to be identified, image to be identified, and audio to be identified; Input the audit business data to be identified into an intention recognition model, and obtain an intention recognition result corresponding to the audit business data to be identified output by the intention recognition model; Inputting the audit business data to be identified and the intention recognition result into a semantic space model to obtain a fusion feature vector of the audit business data to be identified output by the semantic space model; The fused feature vector is input into a multimodal information association model to obtain an association identification result of the audit business data to be identified output by the multimodal information association model.
2. The data identification method according to claim 1, characterized in that: The semantic space model includes a first neural network group, a second neural network group and a third neural network group.
3. The data identification method according to claim 2, characterized in that: Inputting the audit business data to be identified and the intention recognition result into a semantic space model to obtain a fusion feature vector of the audit business data to be identified output by the semantic space model, including: Inputting the audit business data to be identified and the intention recognition result into the first neural network group, and obtaining a multimodal feature vector corresponding to the audit business data to be identified output by the first neural network group; Inputting the multimodal feature vector into the second neural network group, obtaining similarity measurement results and correlation measurement results of the audit business data to be identified output by the second neural network group; The multimodal feature vector, the similarity measurement result and the correlation measurement result are input into the third neural network group to obtain the fused feature vector of the audit business data to be identified output by the third neural network group.
4. The data identification method according to claim 1, characterized in that: The data identification method further includes: Obtaining sample audit business data, an intent recognition result label corresponding to the sample audit business data, and a fusion feature vector corresponding to the sample audit business data; The intention recognition model is trained to obtain the semantic space model.
5. The data identification method according to claim 1, characterized in that: The semantic space model includes a feature extraction network and a classification network.
6. The data identification method according to claim 5, characterized in that: Inputting the audit business data to be identified into an intention recognition model, and obtaining an intention recognition result corresponding to the audit business data to be identified output by the intention recognition model, including: Inputting the audit business data to be identified into the feature extraction network, and obtaining the intention feature vector corresponding to the audit business data to be identified output by the feature extraction network; The intention feature vector is input into the classification network to obtain the intention recognition result corresponding to the audit business data to be identified output by the classification network.
7. The data identification method according to claim 1, characterized in that: Obtain audit business data to be identified, including: Obtain original audit business data; Performing data cleaning processing on the original audit business data to obtain first intermediate data; Performing feature extraction processing on the first intermediate data to obtain second intermediate data; The second intermediate data is subjected to noise reduction processing to obtain the audit business data to be identified.
8. An audit business data identification system based on a multimodal model, characterized in that: The data identification system comprises a processor for executing the data identification method according to any one of claims 1 to 7.