An ai system multi-modal object conservation and conflict auditing method and system

Abstract

The application provides an AI system multi-modal object conservation and conflict auditing method and system, relating to the technical field of computer-aided decision-making, which comprises the following steps: receiving multi-modal input and generating modal analysis results; identifying each modal object, generating modal object identification, and establishing candidate mapping according to object name, spatial position, image area, video track, table row and column, sensor channel, timestamp, version information or semantic features; mapping the modal objects meeting the object identity consistency condition to a unified object identification or a main object identification, and performing object conservation, time version, state observation and evidence role auditing; when object mismatch, time mismatch, version conflict, state conflict, evidence role conflict or cross-modal uncertainty are detected, outputting fusion results, degradation results, frozen states, review requests or write-back records, and generating a cross-modal auditing report for object conservation, conflict auditing and risk disposal.

Description

Technical Field

[0001] This invention relates to the field of computer-aided decision-making technology, and in particular to a method and system for multimodal object conservation and conflict auditing in AI systems. Background Technology

[0002] In recent years, artificial intelligence systems have gradually evolved from single-text processing to multimodal processing, enabling them to identify, analyze, summarize, answer questions, and generate data from various types of information, including text, images, videos, audio, tables, sensor data, and tool logs. As multimodal AI systems are increasingly applied in scenarios such as engineering review, medical assistance, industrial inspection, video evidence analysis, enterprise knowledge management, and scientific research analysis, these systems not only need to identify the content contained in each modality but also need to determine whether the objects, time, version, state, and evidentiary relationships in different modalities remain consistent to ensure the accuracy, reliability, and traceability of the comprehensive analysis results.

[0003] Existing multimodal AI systems typically perform correlation processing on data from different modalities through methods such as feature extraction, semantic matching, vector retrieval, or cross-modal fusion. For example, the system can match text descriptions with targets in images, associate video events with text records, jointly analyze tabular data with sensor data, and generate comprehensive conclusions based on information from multiple modalities. Some existing systems can also perform preliminary alignment of information from different modalities based on object names, semantic similarity, time tags, or spatial locations, and identify relatively obvious data differences or content conflicts.

[0004] However, existing technologies lack a robust cross-modal object conservation mechanism. They typically rely on object names, semantic similarity, or local features to associate objects across different modalities. This makes it difficult to comprehensively determine whether objects consistently correspond in terms of identity, boundaries, spatial location, time, and version. This can easily lead to incorrect merging of different objects or splitting the same object into multiple objects, thus affecting the accuracy of subsequent state assessments and evidence tracking. Furthermore, existing technologies lack a comprehensive conflict audit and risk management mechanism. When inconsistencies in time or version, contradictory state observations, or differences in evidence reliability exist between different modalities, existing systems often directly fuse relevant information and output conclusions. It is difficult to perform downgrading, freezing, review, rollback, or write-back operations based on the conflict type and risk level, and it is also difficult to generate structured audit results that include object mapping relationships, conflict sources, evidence roles, and responsibility records. Summary of the Invention

[0005] To address the lack of a robust cross-modal object conservation mechanism in existing technologies, which typically rely on object names, semantic similarity, or local features to associate objects across different modalities, it is difficult to comprehensively determine whether objects consistently correspond in terms of identity, boundaries, spatial location, time, and version. This can easily lead to erroneous merging of different objects or splitting the same object into multiple objects, thus affecting the accuracy of subsequent state assessments and evidence tracking. Furthermore, existing technologies lack a comprehensive conflict audit and risk management mechanism. When inconsistencies in time or version, contradictory state observations, or differences in evidence reliability exist between different modalities, existing systems often directly fuse relevant information and output conclusions. This makes it difficult to perform downgrading, freezing, review, rollback, or write-back operations based on conflict type and risk level, and it is also difficult to generate structured audit results that include object mapping relationships, conflict sources, evidence roles, and responsibility records. This invention provides a method and system for multimodal object conservation and conflict auditing in AI systems.

[0006] The technical solutions provided by the embodiments of the present invention are as follows: First aspect: This invention provides a method for multimodal object conservation and conflict auditing in an AI system, the method being executed by at least one processor, and comprising: Receive multimodal input data and generate modal analysis results based on the modal types of the multimodal input data; Identify modal objects in different modes within the modality parsing results and generate modality object identifiers; Extract object feature information associated with the modal object identifier from the modality parsing results; Based on the modal object identifier and the associated object feature information, a candidate object mapping is generated; Based on the candidate object mapping, it is determined whether different modal objects meet the object identity consistency condition; if they do, the different modal objects are mapped to a unified object identifier or a main object identifier; if they do not meet the condition, or the object identity is uncertain or the object boundary conflicts, the corresponding candidate object mapping is marked as a mapping to be reviewed or a conflict mapping. Perform object conservation auditing on modal objects mapped to the same unified object identifier or master object identifier, and generate object conservation audit results; Perform time-version alignment on the time state and version state corresponding to the modal object, and generate a time-version alignment result; Perform state observation alignment on the state observations corresponding to the modal object, and generate state observation alignment results; Based on the source, data type, collection method, timestamp, version status, trust level, object correspondence or risk level of the modal object, the role of evidence is determined; Based on the object conservation audit results, the time version alignment results, the state observation alignment results, and the evidence roles, a cross-modal conflict audit result is generated. Based on the cross-modal conflict audit results, output the fusion results, downgrade results, freeze status, review request, risk warning, rollback target or write-back record, and generate a cross-modal audit report.

[0007] The second aspect: This invention provides an AI system multimodal object conservation and conflict auditing system, comprising: The multimodal object parsing module is used to receive multimodal input data, generate modality parsing results according to the modality type of the multimodal input data, identify objects in different modalities based on the modality parsing results, and generate modality object identifiers. The object mapping and conservation audit module is used to extract object feature information associated with the modal object identifier from the modal parsing result, and generate candidate object mappings based on the modal object identifier and its associated object feature information. The object feature information includes at least one of the following: object name, object alias, spatial location, image region, video trajectory, audio speaker, table rows and columns, sensor channels, tool log objects, timestamp, version information, contextual relationship, or semantic features. Modal objects that meet the object identity consistency condition are mapped to a unified object identifier or a master object identifier. Candidate object mappings that do not meet the object identity consistency condition, have uncertain object identities, or have conflicting object boundaries are marked as mappings to be reviewed or conflicting mappings. Object conservation audits are performed on modal objects mapped to the same unified object identifier or master object identifier. The alignment and evidence judgment module is used to perform time and version alignment on the time state and version state corresponding to the modal object, perform state observation alignment on the state observation corresponding to the modal object, and determine the evidence role based on the source, data type, collection method, timestamp, version state, trust level, object correspondence or risk level corresponding to the modal object. The conflict audit and handling module is used to generate cross-modal conflict audit results based on object conservation audit results, time version alignment results, state observation alignment results, and evidence roles, and output fusion results, downgrade results, freeze status, review requests, risk warnings, rollback targets, or write-back records according to the cross-modal conflict audit results; The reporting and write-back module is used to generate cross-modal audit reports and write modal object identifiers, uniform object identifiers, master object identifiers, object mapping status, time version alignment results, status observation alignment results, evidence roles, conflict types, action actions, write-back targets or responsibility logs to the corresponding data storage.

[0008] Third aspect: This invention provides an AI system multimodal object conservation and conflict auditing device, comprising: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores a computer program that can be executed by the at least one processor. When the computer program is executed by the at least one processor, it causes the at least one processor to perform the AI ​​system multimodal object conservation and conflict auditing method as described in the first aspect.

[0009] Fourth aspect: The present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the AI ​​system multimodal object conservation and conflict auditing method as described in the first aspect.

[0010] Fifth aspect: The present invention provides a computer program product, characterized in that: The computer program product includes a computer program that, when executed by a processor, implements the AI ​​system multimodal object conservation and conflict auditing method as described in the first aspect.

[0011] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: (1) By parsing multimodal input data, objects in different modalities are identified, and candidate object mappings are generated by combining object name, alias, spatial location, image region, video trajectory, audio speaker, table rows and columns, sensor channel, tool log object, timestamp, version information, context relationship and semantic features. Then, based on the object identity consistency condition, objects in different modalities are mapped to a unified object identifier or master object identifier. This can avoid the erroneous merging or splitting of objects caused by associating only based on object name, semantic similarity or local features, improve the accuracy of continuous correspondence of cross-modal objects in terms of identity, boundary, space, time and version, and realize the conservation audit of multimodal objects.

[0012] (2) By aligning and judging time versions, status observations and evidence roles, and combining the audit results of object conservation, cross-modal conflict audit results can be generated in a timely manner. Inconsistencies in time versions, contradictory states and differences in evidence reliability between different modalities can be identified in a timely manner. The results of fusion, downgrading, freezing, review, risk warning, rollback or write-back can be output according to the conflict type and risk level. This reduces the risk of erroneous conclusions caused by the direct fusion of multimodal information, improves the reliability, traceability and risk handling capabilities of the multimodal AI system analysis results, and forms a structured cross-modal audit report containing object mapping relationships, conflict sources, evidence roles and responsibility records. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a flowchart illustrating a method for multimodal object conservation and conflict auditing in an AI system, provided as an embodiment of the present invention.

[0015] Figure 2 This is a mapping diagram between modal object identifiers and unified object identifiers provided in an embodiment of the present invention.

[0016] Figure 3 This is a flowchart of a multimodal object conservation audit provided in an embodiment of the present invention.

[0017] Figure 4 This is a time alignment and version alignment relationship diagram provided for an embodiment of the present invention.

[0018] Figure 5 This invention provides a state observation alignment and cross-modal state conflict identification map.

[0019] Figure 6 This invention provides an evidence role determination and multimodal evidence audit diagram.

[0020] Figure 7 This is a flowchart of cross-modal conflict auditing and risk management provided in an embodiment of the present invention.

[0021] Figure 8 This is a diagram illustrating the relationship between cross-modal audit report output and write-back, provided as an embodiment of the present invention.

[0022] Figure 9 This is a diagram illustrating the overall architecture of an AI system for multimodal object conservation and conflict auditing, provided as an embodiment of the present invention. Detailed Implementation

[0023] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0024] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0025] Example 1 Reference manual attached Figure 1The diagram illustrates a flowchart of a multimodal object conservation and conflict auditing method for an AI system provided by an embodiment of the present invention.

[0026] This invention provides a method for multimodal object conservation and conflict auditing in AI systems. This method can be implemented by a device for multimodal object conservation and conflict auditing in AI systems, which can be a terminal or a server. The processing flow of the method may include the following steps: It receives multimodal input data and generates modal analysis results based on the modal types of the multimodal input data.

[0027] In some embodiments, the modal parsing result includes at least one of the following: text entity result obtained by entity extraction of text; image object result obtained by target recognition or region recognition of image; video object or video event result obtained by object tracking or event recognition of video; audio object or sound event result obtained by speaker recognition or sound event recognition of audio; table object or table field result obtained by row and column parsing of table; sensor channel or state sequence result obtained by channel parsing or time series parsing of sensor data; log object or call state result obtained by object parsing or call state parsing of tool log.

[0028] In some embodiments, generating modal analysis results includes: Perform entity extraction on text, target or region recognition on images, object tracking or event recognition on videos, speaker or sound event recognition on audio, row and column parsing on tables, channel or time series parsing on sensor data, and object or call status parsing on tool logs.

[0029] Specifically, the system performs text entity extraction, image target recognition, video object tracking, audio speaker or event recognition, table row and column parsing, sensor channel parsing, tool log object parsing, or structured data field parsing according to the modality type of the input data, and generates modality parsing results.

[0030] Reference manual attached Figure 2 This diagram illustrates a mapping relationship between modal object identifiers and unified object identifiers provided in an embodiment of the present invention.

[0031] Identify modal objects in different modalities in the modal parsing results and generate modal object identifiers.

[0032] In some embodiments, a modal object identifier is used to identify text objects, image objects, video objects, audio objects, table objects, sensor objects, tool log objects, or structured data objects, and to associate them with the corresponding modal type, source, timestamp, version status, original material location, or responsibility log.

[0033] Extract object feature information associated with the modal object identifier from the modal parsing results.

[0034] Optionally, the object feature information includes at least one of the following: object name, object alias, spatial location, image region, video trajectory, audio speaker, table rows and columns, sensor channels, tool log object, timestamp, version information, contextual relationship, or semantic features.

[0035] It should be noted that the object name, object alias, spatial location, image region, video trajectory, audio speaker, table rows and columns, sensor channels, tool log objects, timestamp, version information, contextual relationships, or semantic features are not directly generated from the modal object identifier itself. Instead, they are object feature information extracted by the system from the modal parsing results and associated with the corresponding modal object identifier. The system generates candidate object mappings based on the modal object identifier and its associated object feature information.

[0036] Candidate object mappings are generated based on modal object identifiers and associated object feature information.

[0037] In some embodiments, the candidate object mapping includes modal object identifier, candidate unified object identifier, candidate master object identifier, mapping basis, mapping confidence, object mapping status, or a flag to be reviewed.

[0038] In this embodiment of the invention, by identifying modal objects in different modal parsing results and generating unique or traceable modal object identifiers, each modal object can be associated with its modal type, data source, timestamp, version status, original material location, and responsibility log. Furthermore, by combining multi-dimensional information such as object name, object alias, spatial location, image region, video trajectory, audio speaker, table rows and columns, sensor channels, tool log objects, timestamp, version information, contextual relationships, and semantic features, candidate object mappings are generated, and the mapping basis, mapping confidence, mapping status, and verification mark are recorded. This improves the accuracy and interpretability of associating different modal objects, reduces the erroneous merging or splitting of objects caused by associating solely based on name or semantic similarity, and provides a reliable data foundation for subsequent object identity consistency judgment, object conservation auditing, and cross-modal evidence tracking.

[0039] Based on the candidate object mapping, it is determined whether different modal objects meet the condition of consistent object identity. If they do, different modal objects are mapped to a unified object identifier or a main object identifier. If they do not meet the condition, or if the object identity is uncertain or the object boundaries conflict, the corresponding candidate object mapping is marked as a mapping to be reviewed or a conflicting mapping.

[0040] In some embodiments, the object identity consistency condition includes at least one of the following: consistent object name, corresponding object alias, corresponding spatial location, corresponding image region, continuous video trajectory, corresponding audio speaker, corresponding table row and column, corresponding sensor channel, corresponding tool log object, corresponding time window, corresponding version relationship, or corresponding context relationship.

[0041] In this embodiment of the invention, based on candidate object mapping, information such as object name, object alias, spatial location, image region, video trajectory, audio speaker, table rows and columns, sensor channels, tool log objects, time window, version relationship and context relationship are integrated to determine the identity consistency of different modal objects, and different modal objects that meet the object identity consistency conditions are mapped to a unified object identifier or master object identifier. This can improve the accuracy and consistency of cross-modal object association and reduce the situation where objects with the same name but different objects are mistakenly merged or objects with different names but the same object are mistakenly split.

[0042] Reference manual attached Figure 3 The diagram illustrates a multimodal object conservation audit flowchart provided by an embodiment of the present invention.

[0043] Perform object conservation auditing on modal objects mapped to the same unified object identifier or master object identifier, and generate object conservation audit results.

[0044] In some embodiments, object conservation auditing is used to determine whether multiple modal objects mapped to the same unified object identifier or master object identifier are consistently consistent in terms of object identity, object boundaries, time state, version state, and contextual relationship.

[0045] When different objects with the same name, different objects with the same name, inconsistent object boundaries, inconsistent object versions, inconsistent object contexts, or uncertain cross-modal objects are detected, an object conservation exception record is generated.

[0046] In this embodiment of the invention, by performing object conservation audit on multiple modal objects mapped to the same unified object identifier or master object identifier, it is possible to further determine whether each modal object is consistently consistent in terms of identity, boundaries, time status, version status, and contextual relationships, thereby avoiding the problem of only completing the initial object mapping without discovering subsequent abnormal correspondences. When situations such as objects with the same name but different names, objects with different names but the same name, inconsistent object boundaries, inconsistent object versions, inconsistent object contexts, or uncertain cross-modal objects are detected, an object conservation anomaly record is generated, which can promptly mark potential object mis-merging, mis-splitting, or distorted correspondences, improving the reliability, traceability, and accuracy of subsequent conflict audits of cross-modal object association results.

[0047] Reference manual attached Figure 4 The diagram illustrates a time alignment and version alignment relationship provided by an embodiment of the present invention.

[0048] Perform time-version alignment on the time state and version state corresponding to the modal object, and generate the time-version alignment result.

[0049] In some embodiments, time version alignment includes: Based on the collection time, generation time, publication time, revision time, timestamp, version number, source status, publication status, revision status, or validity period, determine whether different modal materials are within the same time window or under the same version condition. When inconsistencies are detected between historical materials and current materials, conflicts are detected between old version materials and current status, inconsistencies are detected in the time sequence of different modal materials, or the time window cannot be confirmed, generate a time mismatch record or version conflict record.

[0050] Reference manual attached Figure 5 This illustration shows a state observation alignment and cross-modal state conflict identification diagram provided by an embodiment of the present invention.

[0051] Perform state observation alignment on the state observations corresponding to the modal object and generate state observation alignment results.

[0052] In some embodiments, state observations include text state descriptions, image state observations, video event states, audio event states, tabular numerical states, sensor data states, tool log states, or manually recorded states. When inconsistencies exist between state observations in different modalities, a cross-modal state conflict record is generated. The cross-modal state conflict record includes the conflicting object, conflicting modality, conflicting field, conflicting state, conflicting time, conflicting version, or conflicting description.

[0053] Reference manual attached Figure 6 This diagram illustrates an evidence role determination and multimodal evidence auditing method provided by an embodiment of the present invention.

[0054] The role of evidence is determined based on the source, data type, collection method, timestamp, version status, trust level, object correspondence, or risk level of the modal object.

[0055] In some embodiments, the evidence role includes at least one of direct evidence, indirect evidence, continuous monitoring evidence, structured report evidence, manually recorded evidence, tool execution evidence, background-related information, counter-evidence, or material to be reviewed; the evidence role is used to identify the evidentiary role of the corresponding modality material in the target judgment, and the evidentiary role includes at least one of direct support, indirect support, providing background relationship, forming counter-evidence, requiring downgraded use, or requiring manual review.

[0056] Reference manual attached Figure 7 The diagram illustrates a cross-modal conflict audit and risk management flowchart provided by an embodiment of the present invention.

[0057] Based on object conservation audit results, time version alignment results, state observation alignment results, and evidence roles, generate cross-modal conflict audit results.

[0058] In some embodiments, the cross-modal conflict audit result includes at least one of object mismatch, time mismatch, version conflict, state conflict, evidence role conflict, insufficient evidence support, cross-modal uncertainty, or no conflict; when the cross-modal conflict audit result is no conflict, a fusion result is output; when the cross-modal conflict audit result is cross-modal uncertainty, insufficient evidence support, time mismatch, or version conflict and does not meet the high-risk condition, a downgrade result, a review request, or a write-back record is output; when the cross-modal conflict audit result is object mismatch, state conflict, or evidence role conflict and affects the high-risk judgment, a freeze status, a review request, a risk warning, or a rollback target is output.

[0059] Reference manual attached Figure 8 The diagram illustrates a cross-modal audit report output and write-back relationship provided by an embodiment of the present invention.

[0060] Based on the cross-modal conflict audit results, output the fusion results, downgrade results, freeze status, review requests, risk warnings, rollback targets or write-back records, and generate a cross-modal audit report.

[0061] In some embodiments, the write-back record includes a modal object identifier, a uniform object identifier, a master object identifier, an object mapping status, a time version alignment result, a status observation alignment result, an evidence role, a conflict type, a disposition action, a review status, and a write-back target or responsibility log. The write-back record is written to at least one of the following: an object repository, a modal mapping repository, an evidence repository, a risk repository, a structured event metadata repository, or a responsibility log repository.

[0062] Optionally, cross-modal audit reports include modality type, modality object identifier, uniform object identifier, master object identifier, object mapping status, mapping confidence, time alignment status, version alignment status, status observation, evidence role, conflict type, fusion result, degradation result, freeze status, review requirements, and write-back to target or responsibility log.

[0063] Through the above technical solutions, the system can be upgraded from simple content recognition to object conservation auditing in the multimodal AI processing process, so that objects, time, version, state and evidence roles in different modalities can form a structured relationship that is traceable, auditable, verifiable and rewritable.

[0064] Example 2 Multimodal review scenarios for industrial equipment.

[0065] This embodiment illustrates how the present invention can be applied in industrial inspection, equipment maintenance, engineering safety, product testing, or accident debriefing. The embodiment focuses on how the system maps objects from drawings, photographs, maintenance records, inspection forms, sensor data, and tool logs to the same main device object, and identifies cross-modal state conflicts.

[0066] In this embodiment, the system receives multimodal data from a certain engineering equipment. The multimodal data includes equipment design drawings, on-site photos, maintenance record text, test report forms, sensor data, tool call logs, and engineer remarks.

[0067] The system first performs modal analysis on the input data. For design drawings, it identifies the drawing number, equipment name, component area, annotation location, and drawing version. For on-site photos, it identifies the equipment appearance, component location, damaged area, shooting time, and source of the image. For maintenance records, it extracts the equipment name, fault description, maintenance time, maintenance personnel, and handling result. For inspection forms, it parses the inspection items, inspection values, units, thresholds, conclusions, and report versions. For sensor data, it extracts the sensor channels, timestamps, measured values, abnormal fluctuations, and corresponding equipment. For tool logs, the system parses the tool call object, call time, return status, and exception information.

[0068] Each modal object is assigned a modal object identifier. For example, equipment objects in drawings are assigned drawing modal object identifiers, equipment objects in field photos are assigned image modal object identifiers, equipment objects in maintenance records are assigned text modal object identifiers, equipment objects in inspection tables are assigned table modal object identifiers, equipment corresponding to sensor channels are assigned sensor modal object identifiers, and equipment corresponding to tool logs are assigned log modal object identifiers.

[0069] Based on the device name, serial number, spatial location, drawing area, photo area, maintenance record description, inspection table rows and columns, sensor channels, tool log objects, timestamp, and version information, a candidate object mapping is generated. If the system determines that the above modal objects all point to the same device, it generates or calls the device master object identifier (Master_Object_ID_Device_001) and establishes a multimodal object mapping (Modality_Object_Map).

[0070] The multimodal object mapping record includes modality type, modality object identifier, main object identifier, alias or region, time alignment status, version alignment status, status observation, evidence role, mapping confidence, and accountability log.

[0071] Next, time alignment is performed. The above materials are arranged chronologically, and it is determined whether they describe the state of the same equipment at the same or different stages. Version alignment is performed. If the design drawings are an older version, but the field photos and maintenance records show that the equipment has been modified, the system marks the drawing version status as historical reference or expired reference, and does not directly use the old drawing status as the basis for the current equipment status. Status observation alignment is performed. If the inspection table shows that a certain inspection item is qualified, but the field photos show that the corresponding component has damage, corrosion, leakage, deformation, or abnormal markings, the system marks the table status observation as normal, marks the image status observation as abnormal, and generates a cross-modal status conflict record. If the sensor data also shows abnormal fluctuations, the system can further increase the conflict level or risk level. Evidence role judgment is performed. Field photos with clear timestamps and equipment identification can be marked as direct visual evidence; inspection tables can be marked as structured report evidence; sensor data can be marked as continuous monitoring evidence; maintenance records can be marked as manual record evidence; tool logs can be marked as tool execution evidence. The system determines whether a fusion conclusion can be output based on the evidence role and risk level.

[0072] When the system detects a conflict between the table conclusion and the image status, it does not directly output a definitive conclusion of "equipment status is qualified," but instead generates a cross-modal conflict audit result. This audit result includes the equipment master object identifier, the identifiers of each modal object, the time version alignment result, the status observation difference, the evidence role, the conflict type, the risk level, the freeze status, the review requirements, and the responsibility log.

[0073] If the device is a safety-critical device, the system will set the freeze status to pending review freeze, set the review request to true, and output a downgrade result or review request. The system can write relevant audit results to the object library, modal mapping library, evidence library, risk library, structured event metadata library, or responsibility log library.

[0074] Through this embodiment, the present invention can map multimodal materials related to industrial equipment to the same master object and identify state conflicts between different modes, thereby avoiding the system from generating incorrect judgments based on a single table conclusion or a single text description.

[0075] Example 3 Scenario for assisting in the review of medical images.

[0076] This embodiment illustrates the application of the present invention in medical image assistance, medical document verification, examination report review, or multi-source medical record material review. This embodiment focuses on how the system performs multimodal object conservation and evidence role auditing on objects in medical images, examination reports, medical record texts, table indicators, and doctor's records.

[0077] In this embodiment, the system receives multimodal materials related to a patient's examination. These materials include images, image report text, laboratory indicator tables, medical records, doctor's notes, and historical examination data. Instead of directly providing a medical diagnosis, the system performs a structured audit of the objects, time, version, status observations, and evidentiary roles within the different modalities.

[0078] First, the system analyzes the images, identifying image regions, organs or parts of the body, suspected abnormal areas, image acquisition time, and image source, and generates image modal object identifiers. Next, the system extracts entities from the image report text, identifying object names, site descriptions, abnormality descriptions, examination time, and report status, and generates text modal object identifiers. Then, the system parses the laboratory indicator tables, identifying indicator names, values, units, reference ranges, sampling time, and report versions, and generates table modal object identifiers. Finally, the system parses medical records and physician notes, generating record modal object identifiers.

[0079] Next, candidate mappings are generated based on the object name, body part, examination item, acquisition time, report number, image region, table indicators, and context. If the target region in the image, the described object in the image report, and the description of the same part in the medical record text can be aligned, the system maps it to a unified object identifier or a primary object identifier. If there are cases where the part names are similar but the ranges are different, historical data and current data are mixed, or the left and right or upper and lower objects are inconsistent, the system generates an object conservation anomaly record.

[0080] Next, the system performs time version alignment. If the image is the current examination material, but the medical record references historical examination results, the system marks the historical material as historical reference and does not use it as direct evidence of the current state. If the sampling time of the test indicator is later than the image acquisition time, the system records the time difference and highlights the difference in the audit report. It then performs state observation alignment. If the image report text states "no obvious abnormalities found," but the image recognition module marks a suspected abnormal area, the system does not directly use either as the final conclusion, but records the inconsistency between the image state observation and the text state observation. If the test indicator shows an abnormality, but the report text does not mention the corresponding state, the system can generate a cross-modal state inconsistency record. Finally, it performs evidence role judgment. Images can be marked as original image evidence, image reports can be marked as professional report evidence, test indicator tables can be marked as structured test evidence, medical records can be marked as historical clinical record evidence, and physician remarks can be marked as manually recorded evidence. The system determines which materials can be used as direct evidence and which materials can only be used as background or materials awaiting review based on the evidence role.

[0081] When the system detects object uncertainty, time mismatch, state conflict, or inconsistency in evidence roles among image objects, report objects, or table indicators, the system outputs cross-modal audit results instead of a definitive medical judgment. These audit results include modal object identifiers, uniform object identifiers, time version alignment results, state observation differences, evidence roles, conflict types, review requirements, and accountability logs.

[0082] If the conflict involves a high-risk medical scenario, the system will freeze the system and set it to pending professional review, prompting that a professional should make a judgment based on the original images, reports, and clinical data. The system will then write the audit result to the modality mapping library, evidence library, risk library, or responsibility log library.

[0083] Through this embodiment, the present invention can reduce the risk of object mismatch and state conflict between images, reports, tables and medical records in medical imaging-assisted scenarios, enabling the multimodal AI system to output auditable material consistency results instead of generating deterministic diagnostic conclusions beyond its authority.

[0084] Example 4 Video evidence review scenario.

[0085] This embodiment illustrates how the present invention can be applied in video evidence review, security incident analysis, traffic accident debriefing, public place incident verification, or enterprise security auditing. This embodiment focuses on how the system performs object conservation, trajectory alignment, and conflict auditing on video objects, audio objects, text records, and time logs.

[0086] In this embodiment, the system receives video clips, audio materials, event record text, time logs, personnel registration forms, or equipment logs related to a specific event. The system parses the video frames to identify people, vehicles, equipment, areas, or action events, and generates video modal object identifiers. The system parses the audio materials to identify speakers, voiceprint candidates, speech content, abnormal sounds, and timestamps, and generates audio modal object identifiers. The system parses the text records to identify event objects, person names, locations, times, behavior descriptions, and responsibility records. The system parses the time logs and equipment logs to identify access control records, equipment on / off records, tool call records, or system timestamps.

[0087] Candidate object mappings are generated based on the person's name, physical characteristics, location region, trajectory continuity, audio speaker, text record object, timestamp, and log records. If the person object in the video, the speaker object in the audio, and the person object in the text record can be aligned in time, location, and context, the system maps them to the same master object identifier. If the identity of the video object and the text record object is uncertain, the system generates a mapping to be reviewed.

[0088] Next, the system aligns the trajectory and time. If the video shows an object entering an area at a specific time, but the access control log shows no corresponding record for that time, the system generates a time log conflict. If the timestamp of a video clip is inconsistent with the time of the text record, the system generates a time mismatch record. If the audio content occurs after the video event, but the text record describes it as a state before the event, the system generates a time sequence conflict. Next, the system aligns the state observation. If the text record states that a device was not started during the event, but the video shows the device indicator light is on or the device is running, the system generates a video-text state conflict. If an abnormal sound is heard in the audio but not mentioned in the text record, the system can record the audio anomaly as supplementary evidence. Finally, the system determines the evidence role. Video clips can be used as visual event evidence, audio materials can be used as sound event evidence, text records can be used as evidence of human recording, and access control logs or device logs can be used as system log evidence. The system determines the credibility level and evidence role of each modality of evidence based on the source of the material, the integrity of the timestamp, whether it has been edited, whether it has continuity, and the risk level.

[0089] When object mismatches, trajectory inconsistencies, time mismatches, or state conflicts are detected among video, audio, text records, and logs, the system does not directly output a definitive liability conclusion. Instead, it generates a cross-modal audit report. This report includes the main object identifier, video object identifier, audio object identifier, text object identifier, log object identifier, time alignment, trajectory changes, state conflicts, evidence roles, review requirements, and liability logs.

[0090] If the review is used for high-risk decision-making, the system can mark the relevant conclusions as pending review or frozen, and write the conflict record to the risk database and responsibility log database.

[0091] Through this embodiment, the present invention can prevent the system from generating overly certain conclusions based solely on a single video clip or text record in video evidence review scenarios, thereby improving the audit reliability among video, audio, logs, and text.

[0092] Example 5 Document and chart review scenario.

[0093] This embodiment illustrates how the present invention can be applied to the review of enterprise knowledge bases, audit reports, research papers, policy documents, financial materials, engineering reports, or intelligent office documents. This embodiment focuses on how the system performs object conservation and state conflict auditing on text descriptions, figure and table titles, table values, footnotes, image screenshots, and cited materials.

[0094] In this embodiment, the system receives a document containing main text, charts, tables, screenshots, footnotes, and cited materials. The system performs text parsing on the main text, extracting the subject, claims, conclusions, and time range. The system performs row and column parsing on the tables, extracting indicator names, values, units, time intervals, and conclusions. The system identifies images or screenshots, extracting objects, labels, curves, coordinate axes, and labeled areas. The system identifies the sources of footnotes and cited materials, extracting the source name, version, citation location, and publication time.

[0095] Next, the system generates modal object identifiers for text objects, table objects, image objects, footnote objects, and referenced objects. Based on the object name, indicator name, chart title, table rows and columns, image labels, source of reference, time range, and semantic features, the system generates candidate object mappings. If "sales volume of a certain product" in the text, "sales data of product A" in the table, and "curve of product A" in the chart all point to the same object, the system maps them to a unified object identifier. If the time range described in the text is the entire year of 2025, while the table data only covers the first quarter of 2025, the system generates records with inconsistent time ranges.

[0096] Then, the system performs status observation alignment. If the main text states "sales continued to grow," but the chart curve shows a later decline, or the table values ​​show a year-on-year decline, the system generates a text-chart status conflict. If the main text conclusion cites a source, but the footnote source version is outdated, the system generates a version conflict or evidence role downgrade. The system then performs evidence role assessment. The main text conclusion can be marked as assertive text, table values ​​can be marked as structured data evidence, charts can be marked as visual data representations, footnote sources can be marked as source evidence, and screenshots can be marked as image evidence or presentation materials. The system determines whether different evidence roles are sufficient to support the main text conclusion.

[0097] When the system detects conflicts between text, tables, images, and cited materials, it outputs a downgrade result or a review request. For example, the system may indicate "The conclusion in the text is inconsistent with the data in the table," "The time range of the chart is inconsistent with the time range in the text," "The version of the cited source is outdated," or "The screenshot cannot directly support this statistical conclusion." The system writes the conflict results back to the document audit report, evidence library, or accountability log.

[0098] Through this embodiment, the present invention can identify object mismatches, time mismatches, state conflicts, and inconsistent evidence roles among text, tables, images, and references in document chart review, thereby improving the credibility of document generation and knowledge base review.

[0099] Example 6 Audit scenarios involving conflicts between sensor data and manual records.

[0100] This embodiment illustrates how the present invention can be applied in Internet of Things (IoT) systems, industrial equipment monitoring, environmental monitoring, logistics and transportation, energy management, or laboratory monitoring. This embodiment focuses on how the system performs object conservation and conflict auditing on sensor data, manual inspection records, equipment logs, and text reports.

[0101] In this embodiment, the system receives sensor data streams, manual inspection records, equipment logs, and maintenance reports. Sensor data includes sensor channels, timestamps, numerical values, units, anomaly thresholds, and the data acquisition device. Manual inspection records include equipment name, inspection time, inspector, status description, and processing results. Equipment logs include equipment operating status, error codes, restart records, and tool call records. The maintenance report includes conclusions, risk descriptions, and recommended measures.

[0102] The system parses sensor channels into sensor modal objects, device names in manual inspection records into text modal objects, device identifiers in device logs into log modal objects, and object descriptions in maintenance reports into report modal objects. The system generates candidate mappings based on device number, channel number, location, timestamp, log object, and semantic description, and maps the corresponding objects to the primary object identifier.

[0103] The system aligns execution time and status observation. If a manual inspection record shows "equipment is operating normally," but sensor data continuously exceeds the threshold within the same time window, the system generates a sensor-manual record status conflict. If the equipment log records multiple abnormal restarts, but the maintenance report does not mention the abnormality, the system generates a log-report status conflict. The system also determines the role of evidence. Sensor data can be marked as continuous monitoring evidence, equipment logs can be marked as system log evidence, manual inspection records can be marked as manual record evidence, and maintenance reports can be marked as summary report evidence. If continuous monitoring evidence conflicts with summary reports, the system can output a downgrade result or trigger a review based on the risk level.

[0104] The system ultimately generates a cross-modal audit report, recording the main object identifier, sensor channels, manual records, device logs, time window, state conflict type, evidence role, risk level, and review requirements. If the risk level is high, the system freezes the relevant conclusions and requires manual review or on-site inspection.

[0105] Through this embodiment, the present invention can avoid AI systems simply relying on manual records or summary reports while ignoring anomalies in sensor data and device logs, thereby improving the reliability of multimodal auditing in IoT and industrial monitoring scenarios.

[0106] Example 7 Cross-modal conflict handling and write-back scenarios.

[0107] This embodiment illustrates how the present invention performs fusion, degradation, freezing, verification, rollback, and write-back processing after detecting cross-modal conflicts.

[0108] In this embodiment, after receiving multimodal materials and establishing a modal object mapping, the system detects a cross-modal conflict in a certain master object. Conflicts may include object mismatch, time mismatch, version conflict, state conflict, evidence role conflict, insufficient evidence support, or cross-modal uncertainty.

[0109] The system first generates conflict type identifiers based on the conflict type. For example, if objects have the same name but different spatial locations, it is marked as an object mismatch; if text records and image states are inconsistent, it is marked as a state conflict; if old version reports are inconsistent with current sensor data, it is marked as a version conflict; if tabular data is used for summary statistics but the main text treats it as direct evidence, it is marked as an evidence role conflict.

[0110] The system then determines the appropriate course of action based on the risk level. If the conflict is minor and does not affect the main conclusions, the system can output a fusion result along with an explanation of uncertainty. If the conflict affects the strength of the conclusions but does not reach a high-risk level, the system outputs a downgrade result, downgrading the definitive conclusion to a conditional judgment or an explanation pending review. If the conflict affects high-risk decisions, the system is frozen, requiring manual or expert review.

[0111] Next, review requirements are generated. Review requirements may include supplementing original images, supplementing original sensor data, verifying drawing versions, verifying form sources, confirming timestamps, confirming object identity, or conducting on-site reviews, etc.

[0112] Finally, based on the write-back objective, the conflict audit results are written to the object library, modal mapping library, evidence library, risk library, responsibility log library, or structured event metadata library. The write-back records include the conflict trigger time, relevant modal object identifiers, master object identifiers, conflict type, state observation differences, evidence role judgments, action actions, review requirements, and responsibility logs.

[0113] If the system is connected to a continuous information structure field platform, the write-back records can serve as the basis for subsequent AI system retrieval, generation, risk auditing, task execution, or manual review. If the system is connected to an independent structured event meta-record system, cross-modal conflicts can be registered as modal structured event meta-records; if the system is not connected to an external event system, the conflict results can be saved only as multimodal audit records.

[0114] Through this embodiment, the present invention can transform cross-modal conflict from a one-time identification result into an audit state that is rewritable, verifiable, traceable, and manageable.

[0115] Example 8 System deployment and data storage methods.

[0116] This embodiment illustrates one deployment method of the present invention in a computer system.

[0117] In one implementation, the system of the present invention includes a multimodal input layer, a modality parsing layer, an object mapping layer, a conservation audit layer, a conflict resolution layer, a report output layer, and a data storage layer.

[0118] The multimodal input layer receives text, images, video, audio, tables, sensor data, tool logs, or structured database records. The modal parsing layer invokes text extraction models, image recognition models, video tracking models, audio recognition models, table parsing modules, sensor data analysis modules, or tool log parsing modules. The object mapping layer generates modal object identifiers, candidate mapping relationships, unified object identifiers, or master object identifiers. The conservation audit layer performs object conservation, time alignment, version alignment, status observation alignment, and evidence role determination. The conflict resolution layer identifies conflict types, generates fusion results, downgrade results, freeze status, review requirements, rollback targets, or risk warnings. The report output layer generates cross-modal audit reports. The data storage layer stores object tables, modal object tables, object mapping tables, time / version alignment tables, status observation tables, evidence role tables, conflict audit tables, risk resolution tables, and responsibility log tables.

[0119] In a data structure, the object table should at least include a unified object identifier, a primary object identifier, an object name, an object alias, object boundaries, object type, and a responsibility log. The modal object table should at least include a modal object identifier, modal type, source identifier, timestamp, version status, and original material location information. The object mapping table should at least include a modal object identifier, a unified object identifier, a primary object identifier, object mapping status, and mapping confidence level. The status observation table should at least include a status observation, observation source, observation time, and observation object. The evidence role table should at least include an evidence role, credibility level, source type, and scope of application. The conflict audit table should at least include a conflict type, modal conflict status, conflict description, risk level, and handling result. The responsibility log table should at least include a processing time, triggering source, processing module, handling action, review status, and write-back target.

[0120] The data described above can be implemented using relational databases, graph databases, document databases, vector databases, object storage, log databases, or a hybrid storage approach. Fields can be linked using primary keys, foreign keys, graph relationships, indexes, vector similarity, timestamps, or accountability logs.

[0121] Through this embodiment, the present invention can be deployed as an independent multimodal audit system, middleware, enterprise private platform, cloud service, API service, plug-in system or local software module, and can be integrated with existing multimodal AI models, knowledge bases, business systems and risk governance systems.

[0122] Reference manual attached Figure 9 The diagram shows the overall architecture of an AI system multimodal object conservation and conflict auditing system provided by the present invention.

[0123] This invention also provides an AI system multimodal object conservation and conflict auditing system, comprising: The multimodal object parsing module is used to receive multimodal input data, generate modality parsing results according to the modality type of the multimodal input data, identify objects in different modalities based on the modality parsing results, and generate modality object identifiers. The object mapping and conservation audit module is used to extract object feature information associated with the modal object identifier from the modal parsing result, and generate candidate object mappings based on the modal object identifier and its associated object feature information. The object feature information includes at least one of the following: object name, object alias, spatial location, image region, video trajectory, audio speaker, table rows and columns, sensor channels, tool log objects, timestamp, version information, contextual relationship, or semantic features. Modal objects that meet the object identity consistency condition are mapped to a unified object identifier or a master object identifier. Candidate object mappings that do not meet the object identity consistency condition, have uncertain object identities, or have conflicting object boundaries are marked as mappings to be reviewed or conflicting mappings. Object conservation audits are performed on modal objects mapped to the same unified object identifier or master object identifier. The alignment and evidence judgment module is used to perform time and version alignment on the time state and version state corresponding to the modal object, perform state observation alignment on the state observation corresponding to the modal object, and determine the evidence role based on the source, data type, collection method, timestamp, version state, trust level, object correspondence or risk level corresponding to the modal object. The conflict audit and handling module is used to generate cross-modal conflict audit results based on object conservation audit results, time version alignment results, state observation alignment results, and evidence roles, and output fusion results, downgrade results, freeze status, review requests, risk warnings, rollback targets, or write-back records according to the cross-modal conflict audit results; The reporting and write-back module is used to generate cross-modal audit reports and write modal object identifiers, uniform object identifiers, master object identifiers, object mapping status, time version alignment results, status observation alignment results, evidence roles, conflict types, action actions, write-back targets or responsibility logs to the corresponding data storage.

[0124] The AI ​​system multimodal object conservation and conflict auditing system provided by the present invention can execute the above-mentioned AI system multimodal object conservation and conflict auditing method and achieve the same or similar technical effects. To avoid duplication, the present invention will not elaborate further.

[0125] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. The scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for multimodal object conservation and conflict auditing in an AI system, characterized in that, The method is executed by at least one processor and includes: Receive multimodal input data and generate modal analysis results based on the modal types of the multimodal input data; Identify modal objects in different modes within the modality parsing results and generate modality object identifiers; Extract object feature information associated with the modal object identifier from the modality parsing results; Based on the modal object identifier and the associated object feature information, a candidate object mapping is generated; Based on the candidate object mapping, it is determined whether different modal objects meet the object identity consistency condition; if they do, the different modal objects are mapped to a unified object identifier or a main object identifier; if they do not meet the condition, or the object identity is uncertain or the object boundary conflicts, the corresponding candidate object mapping is marked as a mapping to be reviewed or a conflict mapping. Perform object conservation auditing on modal objects mapped to the same unified object identifier or master object identifier, and generate object conservation audit results; Perform time-version alignment on the time state and version state corresponding to the modal object, and generate a time-version alignment result; Perform state observation alignment on the state observations corresponding to the modal object, and generate state observation alignment results; Based on the source, data type, collection method, timestamp, version status, trust level, object correspondence or risk level of the modal object, the role of evidence is determined; Based on the object conservation audit results, the time version alignment results, the state observation alignment results, and the evidence roles, a cross-modal conflict audit result is generated. Based on the cross-modal conflict audit results, output the fusion results, downgrade results, freeze status, review request, risk warning, rollback target or write-back record, and generate a cross-modal audit report.

2. The AI ​​system multimodal object conservation and conflict auditing method according to claim 1, characterized in that, The multimodal input data includes text, images, videos, audio, tables, sensor data, tool logs, and / or structured database records; the modality types include text modality, image modality, video modality, audio modality, table modality, sensor data modality, tool log modality, or structured data modality.

3. The AI ​​system multimodal object conservation and conflict auditing method according to claim 1, characterized in that, The modality parsing result includes at least one of the following: text entity results obtained by performing entity extraction on text; image object results obtained by performing target recognition or region recognition on images; The video object or video event results obtained by performing object tracking or event recognition on the video; The audio object or sound event result obtained by performing speaker recognition or sound event recognition on the audio; The table object or table field result obtained by performing row and column parsing on the table; Sensor channel or state sequence results obtained by performing channel parsing or time series parsing on sensor data; log object or call state results obtained by performing object parsing or call state parsing on tool logs.

4. The AI ​​system multimodal object conservation and conflict auditing method according to claim 1, characterized in that, The modal object identifier is used to identify text objects, image objects, video objects, audio objects, table objects, sensor objects, tool log objects, or structured data objects, and to establish an association with the corresponding modal type, source, timestamp, version status, original material location, or responsibility log.

5. The AI ​​system multimodal object conservation and conflict auditing method according to claim 1, characterized in that, The candidate object mapping includes modal object identifier, candidate unified object identifier, candidate main object identifier, mapping basis, mapping confidence, object mapping status or pending review mark.

6. The AI ​​system multimodal object conservation and conflict auditing method according to claim 1, characterized in that, The conditions for consistent object identity include at least one of the following: consistent object name, corresponding object alias, corresponding spatial location, corresponding image region, continuous video trajectory, corresponding audio speaker, corresponding table row and column, corresponding sensor channel, corresponding tool log object, corresponding time window, corresponding version relationship, or corresponding context relationship.

7. The AI ​​system multimodal object conservation and conflict auditing method according to claim 1, characterized in that, The object conservation audit is used to determine whether multiple modal objects mapped to the same unified object identifier or master object identifier are consistently consistent in terms of object identity, object boundaries, time status, version status, and contextual relationships. When different objects with the same name, different objects with the same name, inconsistent object boundaries, inconsistent object versions, inconsistent object contexts, or uncertain cross-modal objects are detected, an object conservation exception record is generated.

8. The AI ​​system multimodal object conservation and conflict auditing method according to claim 1, characterized in that, The time version alignment includes: Based on the collection time, generation time, release time, revision time, timestamp, version number, source status, release status, revision status, or validity period, determine whether different modal materials are in the same time window or under the same version conditions; when inconsistencies are detected between historical materials and current materials, conflicts are detected between old version materials and current status, inconsistencies are detected in the time sequence of different modal materials, or the time window cannot be confirmed, generate time mismatch records or version conflict records.

9. The AI ​​system multimodal object conservation and conflict auditing method according to claim 1, characterized in that, The state observations include text state descriptions, image state observations, video event states, audio event states, table value states, sensor data states, tool log states, or manually recorded states. When there are inconsistencies between state observations in different modalities, a cross-modal state conflict record is generated. The cross-modal state conflict record includes the conflict object, conflict modality, conflict field, conflict state, conflict time, conflict version, or conflict description.

10. The AI ​​system multimodal object conservation and conflict auditing method according to claim 1, characterized in that, The evidence roles include at least one of the following: direct evidence, indirect evidence, continuous monitoring evidence, structured report evidence, manually recorded evidence, tool execution evidence, background-related information, counter-evidence, or materials to be reviewed; the evidence roles are used to identify the evidentiary role of the corresponding modality material in the target judgment, and the evidentiary role includes at least one of the following: directly supporting, indirectly supporting, providing background relationships, forming counter-evidence, requiring downgraded use, or requiring manual review.

11. The AI ​​system multimodal object conservation and conflict auditing method according to claim 1, characterized in that, The cross-modal conflict audit result includes at least one of the following: object mismatch, time mismatch, version conflict, state conflict, evidence role conflict, insufficient evidence support, cross-modal uncertainty, or no conflict; when the cross-modal conflict audit result is no conflict, the fusion result is output. When the cross-modal conflict audit result is cross-modal uncertainty, insufficient evidence support, time mismatch or version conflict and does not meet the high-risk conditions, the output is a downgrade result, a review request or a write-back record; when the cross-modal conflict audit result is object mismatch, state conflict or evidence role conflict and affects the high-risk judgment, the output is a freeze status, a review request, a risk warning or a rollback target.

12. The AI ​​system multimodal object conservation and conflict auditing method according to claim 11, characterized in that, The write-back record includes modal object identifier, unified object identifier, master object identifier, object mapping status, time version alignment result, status observation alignment result, evidence role, conflict type, handling action, review status, write-back target or responsibility log; the write-back record is written to at least one of the object library, modal mapping library, evidence library, risk library, structured event meta-record library or responsibility log library.

13. The AI ​​system multimodal object conservation and conflict auditing method according to claim 1, characterized in that, The cross-modal audit report includes modality type, modality object identifier, unified object identifier, master object identifier, object mapping status, mapping confidence, time alignment status, version alignment status, status observation, evidence role, conflict type, fusion result, downgrade result, freeze status, review requirements, and write-back to target or responsibility log.

14. A multimodal object conservation and conflict auditing system for AI systems, characterized in that, include: The multimodal object parsing module is used to receive multimodal input data, generate modality parsing results according to the modality type of the multimodal input data, identify objects in different modalities based on the modality parsing results, and generate modality object identifiers. The object mapping and conservation audit module is used to extract object feature information associated with the modal object identifier from the modal parsing result, and generate candidate object mappings based on the modal object identifier and its associated object feature information. The object feature information includes at least one of the following: object name, object alias, spatial location, image region, video trajectory, audio speaker, table rows and columns, sensor channels, tool log objects, timestamp, version information, contextual relationship, or semantic features. Modal objects that meet the object identity consistency condition are mapped to a unified object identifier or a master object identifier. Candidate object mappings that do not meet the object identity consistency condition, have uncertain object identities, or have conflicting object boundaries are marked as mappings to be reviewed or conflicting mappings. Object conservation audits are performed on modal objects mapped to the same unified object identifier or master object identifier. The alignment and evidence judgment module is used to perform time and version alignment on the time state and version state corresponding to the modal object, perform state observation alignment on the state observation corresponding to the modal object, and determine the evidence role based on the source, data type, collection method, timestamp, version state, trust level, object correspondence or risk level corresponding to the modal object. The conflict audit and handling module is used to generate cross-modal conflict audit results based on object conservation audit results, time version alignment results, state observation alignment results, and evidence roles, and output fusion results, downgrade results, freeze status, review requests, risk warnings, rollback targets, or write-back records according to the cross-modal conflict audit results; The reporting and write-back module is used to generate cross-modal audit reports and write modal object identifiers, uniform object identifiers, master object identifiers, object mapping status, time version alignment results, status observation alignment results, evidence roles, conflict types, action actions, write-back targets or responsibility logs to the corresponding data storage.

15. The AI ​​system multimodal object conservation and conflict auditing system according to claim 14, characterized in that, The corresponding data storage includes at least one of the following: relational database, graph database, document database, vector database, object storage, log library, object library, modality mapping library, evidence library, risk library, structured event metadata library, or responsibility log library.

16. A device for multimodal object conservation and conflict auditing in an AI system, characterized in that, include: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores a computer program that can be executed by the at least one processor. When the computer program is executed by the at least one processor, it causes the at least one processor to perform the AI ​​system multimodal object conservation and conflict auditing method as described in any one of claims 1 to 13.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program that, when executed by a processor, implements the AI ​​system multimodal object conservation and conflict auditing method as described in any one of claims 1 to 13.

18. A computer program product, characterized in that: The computer program product includes a computer program that, when executed by a processor, implements the AI ​​system multimodal object conservation and conflict auditing method as described in any one of claims 1 to 13.

Images