Multi-modal based agent collaborative auditing method and system

Abstract

The application relates to the technical field of process automation and artificial intelligence technology, in particular to a multi-modal-based agent collaborative auditing method and system, which comprises the following steps: obtaining case file information, performing similarity analysis on a characteristic vector set and a decision vector set, and obtaining a weight set corresponding to a historical case file with the highest similarity from a database; based on the similarity result, setting the weight set as an initial weight set or an optimal weight set; based on the decision vector set, performing decision conflict analysis, and optimizing the initial weight set based on the analysis result to obtain the optimal weight set; and based on the optimal weight set and the decision vector set, performing weighted fusion to obtain a fusion decision, so that the application has the advantages of effectively solving decision conflicts in a multi-modal data fusion process, accurately locating problem roots and improving human-machine collaborative efficiency.

Description

Technical Field

[0001] This application relates to the technical fields of process automation and artificial intelligence technologies, and in particular to a multimodal intelligent agent collaborative auditing method and system. Background Technology

[0002] In legal review work, with the rapid development of information technology, the evidence materials involved in various cases are becoming increasingly rich and diverse. Intelligent review technology of multimodal case files has become the key to improving review efficiency and decision-making accuracy.

[0003] These multimodal case files cover various forms of data, including audiovisual materials, written materials, and image evidence. However, achieving intelligent review faces many severe challenges, among which the core problem is the inconsistency in decision-making during the multimodal data fusion process.

[0004] Traditional auditing systems typically employ single-model processing or multi-agent independent decision-making mechanisms when dealing with multimodal data.

[0005] Single-model fusion methods simply concatenate data from different modalities and directly input them into a single model for processing. While this approach is convenient, it severely overlooks the semantic gaps that exist between different modalities. Different types of data, such as the intuitive information contained in audiovisual materials and the standardized expressions carried in written documents, have vastly different semantic connotations and modes of expression. Simple concatenation averages out key contradictions during the fusion process, masking them and preventing them from being effectively highlighted during review, ultimately significantly reducing the accuracy of the review results.

[0006] While multi-agent voting mechanisms consider the independence of data from different modalities to some extent, they also have significant drawbacks. In this mechanism, each modality of agent independently generates decisions, which are then merged through weighted averaging or majority voting. However, the static weights used in this mechanism cannot effectively respond to dynamic decision conflicts that arise during the actual review process. For example, when the facts presented by audiovisual evidence contradict the content of written evidence, a fixed weight allocation not only fails to resolve the conflict but may also amplify the impact of erroneous decisions, causing the review results to deviate from the truth.

[0007] More importantly, neither single-model processing nor multi-agent voting mechanisms can accurately pinpoint the root cause of decision-making conflicts. Once a questionable review result arises, manual review must sift through all case files, which undoubtedly greatly reduces the efficiency of human-machine collaboration.

[0008] Furthermore, because these traditional methods rely on static rules, they are difficult to adapt to the complex and ever-changing circumstances of cases, resulting in frequent manual parameter adjustments in practical applications, which further increases the burden and cost of the review work.

[0009] In summary, existing legal review technologies, when processing multimodal case files, suffer from issues such as inconsistent decision-making, inability to accurately pinpoint the root causes of conflicts, and difficulty in adapting to changes, thus failing to meet the increasingly complex demands of legal review work. Therefore, there is an urgent need for an innovative intelligent review technology that can effectively resolve decision-making conflicts during multimodal data fusion, accurately locate the root causes of problems, and improve human-machine collaboration efficiency to better serve legal review work. Summary of the Invention

[0010] To effectively resolve decision conflicts in the process of multimodal data fusion, accurately locate the root cause of the problem, and improve the efficiency of human-machine collaboration, this application provides a multimodal intelligent agent collaborative auditing method and system.

[0011] Firstly, this application provides a multimodal intelligent agent collaborative auditing method, which adopts the following technical solution:

[0012] A multimodal agent-based collaborative auditing method includes the following steps:

[0013] Obtain case file information, which includes video information, audio information, text information, and image information;

[0014] The case file information is preprocessed to output a set of feature vectors, which includes video feature vectors, audio feature vectors, semantic feature vectors, and image feature vectors.

[0015] Based on a pre-defined intelligent agent, decisions are generated for each type of feature in the feature vector set, and a decision vector set is output, which includes video decision vectors, audio decision vectors, semantic decision vectors, and image decision vectors;

[0016] By performing similarity analysis on the feature vector set and decision vector set, the weight set corresponding to the historical case file with the highest similarity is obtained from the database;

[0017] Based on the similarity results, the weight set is set as either the initial weight set or the final weight set;

[0018] Based on the decision vector set, decision conflict analysis is performed, and the initial weight set is optimized based on the analysis results to obtain the final weight set;

[0019] The final weight set and decision vector set are weighted and fused to obtain the fused decision.

[0020] In one embodiment, the similarity analysis method is to calculate cosine similarity after merging the feature vector set and the decision vector set, or to calculate cosine similarity separately for the feature vector set and the decision vector set and then perform weighted fusion.

[0021] In one embodiment: the step of performing decision conflict analysis based on the decision vector set, and optimizing the initial weight set based on the analysis results to obtain the final weight set specifically includes:

[0022] Based on the decision vector set, decision conflict analysis is performed to obtain global conflict information;

[0023] By obtaining the accuracy of historical decisions and combining it with global conflict information, corresponding reward information is generated.

[0024] The initial weight set is iteratively optimized based on reward information using the policy gradient algorithm to obtain the final weight set.

[0025] In one embodiment: the step of performing decision conflict analysis based on the decision vector set to obtain global conflict information specifically includes:

[0026] Based on the distance algorithm, the distance between all decision vectors is calculated to obtain local conflict information;

[0027] Based on the averaging algorithm, local conflict information is calculated to obtain global conflict information.

[0028] In one embodiment: the step of obtaining historical decision accuracy and combining it with global conflict information to generate corresponding reward information specifically includes:

[0029] All historical case files with a similarity greater than the similarity threshold are identified as similar case files.

[0030] Based on the decision-making accuracy corresponding to all similar case files, obtain the reference accuracy.

[0031] Reward information is generated based on global conflict information and reference accuracy.

[0032] In one embodiment: the step of obtaining reference accuracy based on the decision accuracy corresponding to all similar case files specifically includes:

[0033] Based on similarity, calculate the decision accuracy weight corresponding to each similar case file;

[0034] The reference accuracy is obtained by weighting the decision accuracy with the decision accuracy weight.

[0035] In one embodiment: the step of generating reward information based on global conflict information and reference accuracy specifically includes:

[0036] Calculate the average similarity of all similar case files;

[0037] The average similarity is used as a balance coefficient to calculate the accuracy.

[0038] The difference between the weighted calculation result and the global conflict information is used as the reward information.

[0039] In one embodiment, the method further includes updating the final weight set and fusion decision for case file information below an accuracy threshold based on decision accuracy from human feedback.

[0040] In one embodiment: the step of updating the final weight set and fusion decision for case file information below the accuracy threshold based on human feedback specifically includes:

[0041] The results of human feedback are obtained and compared with the fusion decision results to obtain the accuracy of the human feedback decision.

[0042] If the accuracy of the decision based on human feedback is lower than the accuracy threshold, the reward information is regenerated based on the accuracy of the decision based on human feedback and the global conflict information, and the final weight set is optimized again based on the policy gradient algorithm.

[0043] Based on the optimized final weight set, the fusion decision is recalculated;

[0044] Based on the new integrated decision-making, the accuracy of updated case file information is improved.

[0045] Secondly, this application provides a multimodal intelligent agent collaborative auditing method and system, which adopts the following technical solution:

[0046] A multimodal intelligent agent collaborative auditing system includes:

[0047] Information acquisition module: used to acquire case file information, which includes video information, audio information, text information and image information;

[0048] Preprocessing module: preprocesses the case file information acquired by the information acquisition module and outputs a set of feature vectors, which includes video feature vectors, audio feature vectors, semantic feature vectors and image feature vectors;

[0049] Decision generation module: Based on a preset intelligent agent, it generates decisions on various features in the feature vector set output by the preprocessing module and outputs a decision vector set, which includes video decision vectors, audio decision vectors, semantic decision vectors and image decision vectors;

[0050] Similarity analysis module: By performing similarity analysis on the feature vector set and decision vector set, the weight set corresponding to the historical case file with the highest similarity is obtained from the database; based on the similarity results of the similarity analysis module, the weight set is set as the initial weight set or the final weight set;

[0051] Weight optimization module: Based on the set of decision vectors output by the decision generation module, it performs decision conflict analysis and optimizes the initial weight set set by the similarity analysis module based on the analysis results to obtain the final weight set;

[0052] Fusion Decision Module: Based on the final weight set and the decision vector set output by the decision generation module, a weighted fusion is performed to obtain the fusion decision.

[0053] In summary, this application has the following beneficial effects:

[0054] 1. Based on similarity analysis, weights are dynamically obtained, and the weights can be dynamically optimized based on the similarity results;

[0055] 2. By performing decision conflict analysis on the decision vector set, the location and cause of the conflict can be identified, the root cause of the problem can be accurately located, and manual review can be facilitated.

[0056] 3. The system adopts a human-machine collaborative approach to automatically optimize and iterate updates. After each manual review, the weight and fusion decision of the case file are further optimized, which effectively improves the accuracy of decision-making for subsequent similar cases. Attached Figure Description

[0057] Figure 1 This is a framework diagram of the multimodal intelligent agent collaborative review system in this embodiment;

[0058] Figure 2 This is a flowchart of the multimodal intelligent agent collaborative auditing method in this embodiment.

[0059] In the diagram, 10 is the information acquisition module; 20 is the preprocessing module; 30 is the decision generation module; 40 is the similarity analysis module; 50 is the weight optimization module; and 60 is the fusion decision module. Detailed Implementation

[0060] The present application will be further described in detail below with reference to the accompanying drawings.

[0061] To better understand the purpose, technical solutions, and advantages of this application, it has been described and illustrated below with reference to the accompanying drawings and embodiments. However, those skilled in the art should understand that this application can be implemented without these details. In some cases, to avoid obscuring various aspects of this application due to unnecessary description, well-known methods, processes, systems, components, and / or circuits already described at a higher level will not be elaborated upon. It will be apparent to those skilled in the art that various modifications can be made to the embodiments disclosed in this application, and the general principles defined in this application can be applied to other embodiments and application scenarios without departing from the principles and scope of this application. Therefore, this application is not limited to the illustrated embodiments, but conforms to the broadest scope consistent with the scope of protection claimed in this application.

[0062] It should be noted that the descriptions of these embodiments are for the purpose of aiding understanding the present invention, but do not constitute a limitation thereof. Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0063] In the description of this application, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0064] In the description of this application, the terms "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples.

[0065] like Figure 1 As shown, this embodiment discloses a multimodal intelligent agent collaborative review system. This system continuously optimizes and improves the accuracy of intelligent review through human-machine collaboration. It includes an information acquisition module 10, a preprocessing module 20, a decision generation module 30, a similarity analysis module 40, a weight optimization module 50, and a fusion decision module 60.

[0066] The information acquisition module 10 can be a set of data interface programs responsible for connecting to different data storage locations within the legal system, such as database servers and file storage systems. It periodically or in real-time queries and acquires the required case file information, including video, audio, text, and image information, according to pre-defined rules, and then transmits this information to the pre-processing module 20.

[0067] The preprocessing module 20 consists of multiple sub-modules, each corresponding to different types of information preprocessing. The sub-modules obtain video feature vectors, audio feature vectors, semantic feature vectors, and image feature vectors respectively.

[0068] Among them, video feature vectors can be obtained through pre-trained C3D or I3D models, audio feature vectors can be obtained through VGGish pre-trained networks, semantic feature vectors can be obtained through BERT-LEAL models, and image feature vectors can be obtained through YOLO algorithms based on EfficientNet. After acquisition, the preprocessing module 20 integrates the processed feature vectors into a feature vector set and passes it to the decision generation module 30.

[0069] The decision generation module 30 constructs and trains corresponding agent models for feature vectors of different modalities. For example, the agent can be a deep learning model based on RNN or LSTM. Taking the video agent as an example, during the training phase, a large amount of labeled video data is used to train the model, enabling it to accurately generate reasonable video decision vectors from video feature vectors. Audio agents, semantic agents, and image agents are constructed and trained in a similar manner.

[0070] Upon receiving the feature vector set from the preprocessing module 20, each agent processes its respective feature vector to obtain video decision vectors, audio decision vectors, semantic decision vectors, and image decision vectors. These are then integrated to generate a decision vector set, which is then passed to the similarity analysis module 40 and the weight optimization module 50.

[0071] The similarity analysis module 40 comprises two functional components: similarity calculation and weight acquisition. The similarity calculation component performs calculations on the feature vector set according to the selected similarity analysis method, such as cosine similarity calculation. The weight acquisition component, based on the calculated similarity results, retrieves the weight set corresponding to the historical case files with the highest similarity from the database, and sets it as the initial or final weight set according to the similarity situation. Simultaneously, it passes the relevant information to the weight optimization module 50 and the fusion decision module 60.

[0072] The weight optimization module 50 integrates functions such as conflict analysis, reward generation, and weight iterative optimization. The conflict analysis component calculates global conflict information for the decision vector set based on distance and averaging algorithms. The reward generation component generates reward information according to specific calculation steps based on historical decision accuracy and global conflict information. The weight iterative optimization component uses a policy gradient algorithm to iteratively optimize the initial weight set based on the reward information, generating the final weight set, which is then passed to the fusion decision module 60.

[0073] The fusion decision module 60 receives the final weight set from the similarity analysis module 40 or the weight optimization module 50, and the decision vector set from the decision generation module 30. It then performs calculations on the weights and decision vectors using a weighted fusion method to obtain the fusion decision result. This module is also responsible for outputting the fusion decision result and updating and recording the result based on human feedback when necessary.

[0074] like Figure 2 As shown, to better illustrate the technical solution of this application, this embodiment also discloses a multimodal intelligent agent collaborative review method, which specifically includes the following steps:

[0075] S100. Obtain case file information, including video information, audio information, text information, and image information.

[0076] In this step, case file information of various types of cases is obtained by interacting with the data interface or storage device inside the legal system. This information includes video information such as on-site surveillance video and law enforcement recorder video; audio information such as interrogation recordings and witness testimony audio; text information such as case documents and transcripts; and image information such as on-site photos and physical evidence pictures.

[0077] S200. Preprocess the case file information and output a set of feature vectors, which includes video feature vectors, audio feature vectors, semantic feature vectors and image feature vectors.

[0078] S300, based on a preset intelligent agent, generates decision vectors for each type of feature in the feature vector set, and outputs a decision vector set, which includes video decision vectors, audio decision vectors, semantic decision vectors and image decision vectors.

[0079] The decision vector in this step is expressed in probability using confidence levels, such as a binary probability distribution: [P(suspicion established), P(material contradiction)].

[0080] S400. By performing similarity analysis on the feature vector set and decision vector set, the weight set corresponding to the historical case file with the highest similarity is obtained from the database.

[0081] In this step, the above similarity analysis method is to calculate cosine similarity after merging the feature vector set and the decision vector set, or to calculate cosine similarity separately for the feature vector set and the decision vector set and then perform weighted fusion.

[0082] S500. Based on the similarity results, set the weight set as the initial weight set or the final weight set.

[0083] In this step, a preset threshold can be used for judgment. If the similarity result is greater than the threshold, the weight set is directly used as the final weight set; otherwise, the weight set is set as the initial weight set.

[0084] S600. Based on the decision vector set, perform decision conflict analysis, and optimize the initial weight set based on the analysis results to obtain the final weight set.

[0085] In this step, decision conflict analysis is performed using a set of decision vectors to dynamically detect conflicts between decisions made by various agents and quantify the degree of inconsistency. On the one hand, this locates the position and cause of the conflict, accurately pinpointing the root cause of the problem and facilitating manual review. On the other hand, iterative optimization of the initial weight set based on the analysis results can improve the accuracy of its decisions.

[0086] S700: Based on the final weight set and decision vector set, a weighted fusion is performed to obtain the fused decision.

[0087] In one embodiment, step S600 above, which involves performing decision conflict analysis based on the decision vector set and optimizing the initial weight set based on the analysis results to obtain the final weight set, specifically includes:

[0088] Based on the decision vector set, decision conflict analysis is performed to obtain global conflict information;

[0089] By obtaining the accuracy of historical decisions and combining it with global conflict information, corresponding reward information is generated.

[0090] The initial weight set is iteratively optimized based on reward information using the policy gradient algorithm to obtain the final weight set.

[0091] In this embodiment, global conflict information is obtained in the following way:

[0092] Based on the distance algorithm, the distance between all decision vectors is calculated to obtain local conflict information.

[0093] Based on the averaging algorithm, local conflict information is calculated to obtain global conflict information.

[0094] In this embodiment, the Euclidean distance algorithm is preferred. This can intuitively reflect the difference between any two decision vectors. Finally, global conflict information is obtained by averaging the local conflict information to reflect the conflict situation.

[0095] In one embodiment, the steps of obtaining historical decision accuracy, combining it with global conflict information, and generating corresponding reward information specifically include:

[0096] All historical case files with a similarity greater than the similarity threshold are identified as similar case files.

[0097] Based on the decision-making accuracy corresponding to all similar case files, obtain the reference accuracy.

[0098] Reward information is generated based on global conflict information and reference accuracy.

[0099] In this embodiment, the similarity threshold used for judging similar cases is lower than the threshold used when judging the initial weight set or the final weight set. This method allows for the selection of multiple historical case files with high similarity.

[0100] In addition, the similarity threshold is preferably set in a stepped manner, and the judgment is made from high to low. In this way, when judging similar cases, if there are no historical case files with similarity greater than the similarity threshold, the similarity threshold can be gradually reduced until the number of historical case files obtained is greater than the preset value, ensuring that there are enough similar case files for reference.

[0101] The similar case files obtained are all bound to the results of human feedback and decision-making. The results of human feedback are used as the true probability, and the decision-making results are used as the predicted probability. The true probability is 0 or 1. Finally, the decision accuracy can be calculated as 1 - |predicted probability - true probability|.

[0102] In this embodiment, the step of obtaining the reference accuracy based on the decision accuracy corresponding to all similar case files includes:

[0103] Based on similarity, calculate the decision accuracy weight corresponding to each similar case file;

[0104] The reference accuracy is obtained by weighting the decision accuracy with the decision accuracy weight.

[0105] In the above steps, the decision accuracy weight corresponding to each similar case file is calculated based on the ratio of the similarity of that similar case file to the sum of the similarities of all similar case files. Then, the decision accuracy weight corresponding to each similar case file is multiplied by the decision accuracy, and the sum of the products of all similar case files is used to obtain the reference accuracy.

[0106] Furthermore, in this embodiment, the step of generating reward information based on global conflict information and reference accuracy specifically includes:

[0107] Calculate the average similarity of all similar case files;

[0108] The average similarity is used as a balance coefficient to calculate the accuracy.

[0109] The difference between the weighted calculation result and the global conflict information is used as the reward information.

[0110] In the above steps, by adding the average similarity as a balancing coefficient for reference accuracy, the accuracy and similarity of similar cases can be further combined. This ensures that the reward information for cases with high similarity has a stronger foundation, and correspondingly, the impact of global conflict information is reduced. Conversely, the reward information for cases with low similarity has a weaker foundation, and correspondingly, the impact of global conflict information is increased. This allows for dynamic adjustment of the reward information.

[0111] In one embodiment, the multimodal agent-based collaborative review method further includes the step of: based on the decision accuracy of human feedback, performing final weight set and fusion decision update for case file information below the accuracy threshold.

[0112] Specifically, the above steps include:

[0113] The results of human feedback are obtained and compared with the fusion decision results to obtain the accuracy of the human feedback decision.

[0114] If the accuracy of the decision based on human feedback is lower than the accuracy threshold, the reward information is regenerated based on the accuracy of the decision based on human feedback and the global conflict information, and the final weight set is optimized again based on the policy gradient algorithm.

[0115] Based on the optimized final weight set, the fusion decision is recalculated;

[0116] Based on the new integrated decision-making, the accuracy of updated case file information is improved.

[0117] In this embodiment, the decision accuracy is calculated in the same way as in the embodiments described above. Then, based on a comparison and analysis of the decision accuracy results from manual feedback with the accuracy threshold, it is determined whether the result of the fused decision meets the requirements.

[0118] When the probability of the fusion decision is higher than the accuracy threshold, it indicates that it is relatively accurate and can be directly used as a standard for evaluating similar cases in the future. Otherwise, it needs to be optimized and adjusted.

[0119] In this embodiment, on the one hand, the accuracy of the prediction decision is checked by combining manual review, and on the other hand, the data that does not meet the requirements of the check results is further optimized and adjusted to achieve data iteration and improve the accuracy of subsequent predictions.

[0120] The embodiments described in this specific implementation are preferred embodiments of this application and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A multi-modal based agent collaborative review method, characterized in that, Includes the following steps: Obtain case file information, which includes video information, audio information, text information, and image information; The case file information is preprocessed to output a set of feature vectors, which includes video feature vectors, audio feature vectors, semantic feature vectors, and image feature vectors. Based on a pre-defined intelligent agent, decision-making is generated for each type of feature in the feature vector set, and a decision vector set is output. The decision vector set includes video decision vectors, audio decision vectors, semantic decision vectors, and image decision vectors; among them, the decision vectors are expressed in probabilistic form using confidence. By performing similarity analysis on the feature vector set and the decision vector set, the weight set corresponding to the historical case file with the highest similarity is obtained from the database; the similarity analysis method is to calculate the cosine similarity after merging the feature vector set and the decision vector set, or to calculate the cosine similarity of the feature vector set and the decision vector set separately and then perform weighted fusion. Based on the similarity analysis results, the obtained weight set is set as the initial weight set or the final weight set of the decision vector set. This setting process is determined by a preset threshold. When the similarity is greater than the preset threshold, the obtained weight set is set as the final weight set of the decision vector set; otherwise, the obtained weight set is set as the initial weight set of the decision vector set. If the obtained weight set is set as the initial weight set, decision conflict analysis is performed based on the decision vector set, and the initial weight set is optimized based on the analysis results to obtain the final weight set; The final weight set and decision vector set are weighted and fused to obtain the fused decision. The steps of performing decision conflict analysis based on the decision vector set and optimizing the initial weight set based on the analysis results to obtain the final weight set specifically include: Based on the set of decision vectors, decision conflict analysis is performed to obtain global conflict information. The method for obtaining global conflict information is as follows: based on the distance algorithm, the distance between all decision vectors is calculated to obtain local conflict information; based on the averaging algorithm, the local conflict information is calculated to obtain global conflict information. The accuracy of historical decisions is obtained and combined with global conflict information to generate corresponding reward information. The specific steps for generating the reward information are as follows: all historical case files with a similarity greater than a similarity threshold are obtained as similar case files; a reference accuracy is obtained based on the decision accuracy corresponding to all similar case files; and reward information is generated based on global conflict information and reference accuracy. The initial weight set is iteratively optimized based on reward information using the policy gradient algorithm to obtain the final weight set.

2. The multi-modal based agent collaborative review method of claim 1, wherein, The step of obtaining the reference accuracy based on the decision accuracy corresponding to all similar case files specifically includes: Based on similarity, calculate the decision accuracy weight corresponding to each similar case file; The reference accuracy is obtained by weighting the decision accuracy with the decision accuracy weight.

3. The multi-modal based agent collaborative review method of claim 1, wherein, The step of generating reward information based on global conflict information and reference accuracy specifically includes: Calculate the average similarity of all similar case files; The average similarity is used as a balance coefficient to calculate the accuracy. The difference between the weighted calculation result and the global conflict information is used as the reward information.

4. The multimodal agent-based collaborative auditing method according to claim 1, characterized in that: The method also includes decision accuracy based on human feedback, and final weighting and fusion decision updates for case file information below the accuracy threshold.

5. The multimodal agent-based collaborative auditing method according to claim 4, characterized in that, The steps for updating the final weight set and fusion decision based on the accuracy of decision-making according to human feedback, and for case file information below the accuracy threshold, specifically include: The results of human feedback are obtained and compared with the fusion decision results to obtain the accuracy of the human feedback decision. If the accuracy of the decision based on human feedback is lower than the accuracy threshold, the reward information is regenerated based on the accuracy of the decision based on human feedback and the global conflict information, and the final weight set is optimized again based on the policy gradient algorithm. Based on the optimized final weight set, the fusion decision is recalculated; Based on the new integrated decision-making, the accuracy of updated case file information is improved.

6. A multimodal agent-based collaborative auditing system, applied to the multimodal agent-based collaborative auditing method according to any one of claims 1-5, characterized in that, include: Information acquisition module (10): used to acquire case file information, which includes video information, audio information, text information and image information; Preprocessing module (20): preprocesses the case file information acquired by the information acquisition module (10) and outputs a set of feature vectors, which includes video feature vectors, audio feature vectors, semantic feature vectors and image feature vectors; Decision generation module (30): Based on the preset intelligent agent, it generates decisions on various features in the feature vector set output by the preprocessing module (20) and outputs a decision vector set, which includes video decision vectors, audio decision vectors, semantic decision vectors and image decision vectors; Similarity analysis module (40): By performing similarity analysis on the feature vector set and decision vector set, the weight set corresponding to the historical case file with the highest similarity is obtained from the database; based on the similarity results of the similarity analysis module (40), the weight set is set as the initial weight set or the final weight set; Weight optimization module (50): Based on the decision vector set output by the decision generation module (30), it performs decision conflict analysis and optimizes the initial weight set set set by the similarity analysis module (40) based on the analysis results to obtain the final weight set; Fusion Decision Module (60): Based on the final weight set and the decision vector set output by the decision generation module (30), a weighted fusion is performed to obtain the fusion decision.

Images