A service anomaly detection method based on multi-modal data fusion

Abstract

This invention relates to the field of anomaly detection technology, specifically a business anomaly detection method based on multimodal data fusion. The invention extracts connection status and power supply status from on-site inspection images to form first modality data; determines the associated time window based on image time reference, standard time source deviation, and logical layer time synchronization records, retrieves logical activity data to form second modality data, and associates them as state data pairs according to object identifiers; judges based on the causal relationship between connection status, power supply status, and logical activities, observes state data pairs that do not satisfy the relationship within a bounded recovery time range, classifies those that recover to consistency as transient deviations, and classifies those that are inconsistent beyond the time limit as persistent contradictions; clusters persistent contradictions based on topological connection relationships, generates a review task, and uses re-acquired data to review the status or update reference features. This invention can reduce misjudgments caused by time offset and state propagation lag.

Description

Technical Field

[0001] This invention relates to the field of anomaly detection technology, specifically to a business anomaly detection method based on multimodal data fusion. Background Technology

[0002] Existing business anomaly detection methods based on multimodal data fusion typically rely on continuous, synchronously collected, and automated multi-mode monitoring data for feature extraction and fusion processing. A baseline for normal behavior is then constructed based on historical statistical patterns, and anomaly identification is achieved by judging the degree of deviation of the data to be detected relative to this baseline. However, these methods all assume that the data from each modality are continuously and frequently collected. They do not provide technical means to introduce discrete, low-frequency, but professionally verified, high-confidence verification data generated during the detection process as an independent modality into the fusion processing and baseline construction process. Because this type of verification data differs by orders of magnitude from continuous multimodal monitoring data in terms of collection frequency and confidence level, if existing feature splicing or fixed-time-window alignment and weighting methods are used for fusion, the information carried by this data may be diluted by the high-frequency continuous data, making it difficult to produce an effective correction effect. Alternatively, manual interpolation may distort the time-varying characteristics of its confidence level, resulting in cross-modal fusion feature distortion. Therefore, when the actual operating state of the information system being tested deviates due to factors such as configuration changes, existing methods are unable to correct the detection baseline in a timely manner, resulting in a decrease in the accuracy and timeliness of business anomaly detection.

[0003] To address this, a business anomaly detection method based on multimodal data fusion is proposed. Summary of the Invention

[0004] The purpose of this invention is to provide a business anomaly detection method based on multimodal data fusion. By associating connection and power supply status with time-aligned logical activity data in on-site images, transient deviations and persistent contradictions are distinguished based on causal relationships and bounded recovery processes. The method also verifies and updates reference features by clustering according to topological relationships.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A business anomaly detection method based on multimodal data fusion includes: Connection status data and power supply status data are extracted from the collected on-site verification image data to generate the first modality data; The first time reference and the deviation relative to the standard time source of the on-site verification image data are obtained. The time reference offset range is obtained by combining the time synchronization records in the logic layer operation data, and the associated time window is determined. Within the associated time window, retrieve logical activity data from the logical layer's operational data to generate second modal data, and associate the first modal data and the second modal data as a state data pair; Based on the causal correspondence between connection status data, power supply status data, and logical activity data, determine whether the status data pairs satisfy the causal correspondence. If the status data pairs do not satisfy the causal correspondence, observe within the standard-defined bounded recovery time range corresponding to the connection status data and power supply status data. If the recovery is consistent within the range, it is determined as a transient deviation. If the condition is still not satisfied outside the range, it is determined as a persistent contradiction, and the business anomaly judgment result is output. Based on the topology connection relationship data, cluster persistent contradictions to generate review task data and obtain review data.

[0006] Preferably, the first modal data acquisition process includes: if the state presentation mode of the object identifier corresponding to the on-site verification image data is unchangeable and has a physical connection form, it is determined as image data to be extracted; in the image data to be extracted, a region with a fixed positional correspondence with the object identifier is identified, and the color presentation features, connection entity presentation features, and corresponding shooting condition information of the region are extracted; based on the difference between the shooting condition information and the shooting condition information corresponding to the reference features of the same region in a known state, the color presentation features are adjusted; the adjusted color presentation features and connection entity presentation features are compared with the reference features respectively, and the connection status data and power supply status data are determined based on the comparison results to generate the first modal data.

[0007] Preferably, the process of determining the associated time window includes: reading the number of time synchronization records in the logical layer running data; if the number of time synchronization records is more than two, and the time interval between the first time reference and the calibration time corresponding to the most recent time synchronization record does not exceed the effective time range of the time synchronization mechanism corresponding to the data source, then the trend of deviation over time is determined based on the change in the deviation value between two adjacent time synchronization records; based on the trend and the time interval between the first time reference and the most recent time synchronization record, the expected deviation of the corresponding time of the on-site verification image data is calculated; and the time reference offset range is obtained based on the expected deviation and the deviation relative to the standard time source; if the number of time synchronization records is less than two, or the aforementioned time interval exceeds the effective time range, then the deviation relative to the standard time source is used as the time reference offset range; and the associated time window is determined by expanding forward and backward in both directions according to the time reference offset range, with the first time reference as the center.

[0008] Preferably, associating the first modal data and the second modal data as a state data pair includes: extracting the identifier presentation features corresponding to the object identifier from the on-site verification image data as the image-side identifier; extracting the identifier information of the record annotation from the logical activity data as the record-side identifier; determining whether the image-side identifier and the record-side identifier point to the same object identifier; if they point to the same object identifier, associating the first modal data with the second modal data generated from the logical activity data retrieved within the association time window as a state data pair; if they point to different object identifiers, no state data pair is generated, and it is recorded as an anomaly to be associated.

[0009] Preferably, determining whether a pair of status data satisfies a causal relationship includes: determining whether the connection status data indicates that the corresponding object identifier is in a connected state, and whether the power supply status data indicates that it is in a powered-on state; if both the connection status data and the power supply status data indicate that the preconditions are met, then the status data pair is determined to satisfy the causal correspondence; if either the connection status data or the power supply status data indicates that the preconditions are not met, then it is determined whether there is an activity record in the logical activity data after the end time of the associated time window; if there is an activity record, then the status data pair is determined not to satisfy the causal correspondence; if there is no activity record, then the status data pair is determined to satisfy the causal correspondence.

[0010] Preferably, the process for outputting transient deviation and service anomaly determination results includes: acquiring logical activity data multiple times at preset intervals within a standard-defined bounded recovery time range to obtain a sequence of changes in activity records over time; determining whether the change sequence shows a process of gradual change in activity records that matches connection status data and power supply status data; if the change sequence shows the change process and matches within the standard-defined bounded recovery time range, it is determined to be a transient deviation; if the change sequence does not show a change process, or does not match even after exceeding the standard-defined bounded recovery time range, it is determined to be a persistent contradiction, and the service anomaly determination result is output, wherein the service anomaly determination result includes object identifier, connection status data, power supply status data, and change sequence.

[0011] Preferably, the data acquisition process for verification includes: based on the verification task data, re-collecting on-site verification image data and corresponding logical activity data for the object identifier; extracting original features from the re-collected on-site verification image data that are at the same representation level as the reference features on which the first modality data is based; comparing the original features with the reference features to determine whether the reference features have changed; if the reference features have not changed, extracting connection status data and power supply status data based on the original features; determining the difference between the obtained data and the status data pair as a change in the status corresponding to the object identifier; if the reference features have changed, updating the reference features with the original features; and re-extracting connection status data and power supply status data based on the updated reference features to obtain the verification data.

[0012] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention extracts connection status data and power supply status data from on-site inspection images, and uses object identifiers to associate physical-side status with logical-layer activities as status data pairs. Consistency judgment is then performed based on the causal correspondence between connection status, power supply status, and logical activities. Simultaneously, color rendering features are adjusted according to differences in shooting conditions, and the adjusted features and connection entity rendering features are compared with reference features. Different modal data can corroborate each other, reducing the interference of changes in the shooting environment, image recognition bias, object mismatch, and isolated log records on the detection results, thus improving the accuracy and completeness of evidence in identifying business anomalies.

[0013] 2. This invention integrates the first time reference of the integrated image data, the deviation from the standard time source, and the time synchronization records of the logical layer to determine the time reference offset range and construct an associated time window. For state data pairs that do not satisfy the causal correspondence, logical activity data is continuously acquired within the standard-defined bounded recovery time range. The change sequence of the activity records is used to determine whether the physical state and the logical state have recovered to be consistent. This can compensate for clock drift and acquisition time difference between different data sources, and identify situations where consistency can be restored on schedule during state propagation as transient deviations, and situations where no recovery process has been formed or where inconsistency has exceeded the deadline as persistent contradictions, reducing false alarms caused by state delays.

[0014] 3. This invention clusters persistent contradictions based on topological connection data, generating review task data oriented towards related objects, avoiding the need for item-by-item, unordered verification of discrete anomalies. During the review process, image data and logical activity data are re-acquired, and the original features are compared with reference features at the same representation level. This distinguishes between actual changes in object status and changes in reference features. When reference features change, updates are completed and status data is re-extracted. This invention forms a closed loop of anomaly detection, topological clustering, on-site review, and reference update, while retaining object identifiers, physical states, and change sequences, improving the executability of anomaly location, tracing, and review handling. Attached Figure Description

[0015] Figure 1 This invention provides a schematic diagram of a business anomaly detection method based on multimodal data fusion. Figure 2 This is a schematic diagram of the extraction and generation process of the first modal data provided by the present invention; Figure 3 A schematic diagram illustrating the process for determining the associated time window provided by this invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention.

[0017] Example 1: This invention provides a service anomaly detection method based on multimodal data fusion. It is used to perform object-level and time-level correlation between the connection status and power supply status represented by on-site inspection images and real-time logical activity records of the same object, identify state data pairs that do not meet the necessary consistency conditions, and distinguish between transient deviations and persistent contradictions through bounded observation and verification processes. The specific implementation methods of each step are described in detail below.

[0018] Please see Figure 1 This invention provides a service anomaly detection method based on multimodal data fusion. The technical solution is as follows: Connection status data and power supply status data are extracted from collected on-site inspection image data to generate first modal data; the first time reference and deviation relative to the standard time source of the on-site inspection image data are obtained, and the time reference offset range is determined by combining the time synchronization records in the logic layer operation data, thus determining the associated time window; logical activity data in the logic layer operation data is retrieved within the associated time window to generate second modal data, and the first modal data and second modal data are associated as state data pairs; the causal correspondence between connection status data, power supply status data, and logical activity data is used to determine whether the state data pairs satisfy the causal correspondence; if the state data pairs do not satisfy the causal correspondence, they are observed within the standard-defined bounded recovery time range corresponding to the connection status data and power supply status data. If the recovery is consistent within the range, it is determined as a transient deviation; if it still does not satisfy the causal correspondence outside the range, it is determined as a persistent contradiction, and the service anomaly determination result is output; persistent contradictions are clustered based on topology connection relationship data to generate review task data, and review data is obtained.

[0019] Furthermore, referring to Figure 2 The first modality data acquisition process includes: if the state presentation mode of the object identifier corresponding to the on-site verification image data is unchangeable and has a physical connection form, it is determined as the image data to be extracted; in the image data to be extracted, the region with a fixed positional correspondence with the object identifier is determined, and the color presentation features, connection entity presentation features and corresponding shooting condition information of the region are extracted; based on the difference between the shooting condition information and the shooting condition information corresponding to the reference features of the same region in a known state, the color presentation features are adjusted; the adjusted color presentation features and connection entity presentation features are compared with the reference features respectively, and the connection status data and power supply status data are determined based on the comparison results to generate the first modality data.

[0020] Specifically, on-site verification image data refers to image data collected during on-site verification of specific object identifiers. One or more images can be collected for the same object identifier during the same on-site verification to cover the connection interface area and the identifier display area on the physical carrier where the object identifier is located, which are located in different orientations or positions. Among the multiple images collected for the same object identifier during the same on-site verification, the local time record corresponding to the first image collected is used as the first time reference for that object identifier, and the remaining images are only used for feature extraction and the time reference is not determined repeatedly.

[0021] Before generating the first modality data, the status presentation method of the object identifiers pointed to by the on-site verification image data is first confirmed: the status presentation method refers to the specific form in which the connection and power supply status corresponding to the object identifier are presented as observable visual features at the physical level, including whether the visual features can be changed through configuration rather than being directly determined by the physical connection and power-on status, and whether the connection form corresponding to the visual features is a physical connection. Only when the status presentation method is confirmed to be unchangeable through configuration and has a physical connection form, the corresponding on-site verification image data is determined to be the image data to be extracted; on-site verification image data that does not meet the above conditions are not used to generate the first modality data.

[0022] In the image data to be extracted, regions with a fixed positional correspondence with the object identifier are determined based on the appearance characteristics of the object identifier in the image. A fixed positional correspondence means that the relative position of this region with respect to the object identifier in the image remains consistent across multiple images acquired for the same object identifier. Color rendering features, connected entity rendering features, and corresponding shooting condition information are extracted from this region. Shooting condition information refers to information characterizing the imaging environment of this region, including the category of illumination intensity, the category of light source color temperature, and the category of shooting angle.

[0023] Reference features are color rendering features, connection entity rendering features, and corresponding shooting condition information that have been independently confirmed in advance for the same area, with corresponding connection and power supply status beyond on-site verification. These serve as comparison benchmarks under known conditions. For object identifiers in the initial stage of method operation, where no historical reference features exist, their reference features are established through initial on-site verification. Specifically, during this on-site verification process, in addition to collecting on-site verification image data, the corresponding connection and power supply status are obtained through direct on-site verification. The color rendering features, connection entity rendering features, and shooting condition information collected in this instance are determined as the initial reference features corresponding to that object identifier. The extracted color rendering features are adjusted based on the differences between the current shooting condition information and the corresponding shooting condition information of the reference features. The adjusted color rendering features and connection entity rendering features are then compared with the reference features, and the connection status data and power supply status data are determined based on the comparison results. These two together constitute the first modality data.

[0024] The adjustment method for color presentation features refers to a pre-established reference table that records the correspondence between the degree of difference in shooting conditions and the adjustment amount of color presentation features for different region types. This reference table is derived from image processing-related technical specifications or existing practical experience. After determining the direction and degree of difference between the current shooting conditions and the shooting conditions corresponding to the reference features, based on the region type of the extracted area, a record item matching the direction and degree of difference is found in the reference table. The adjustment amount corresponding to the record item is used as the basis for this adjustment to adjust the extracted color presentation features, resulting in the adjusted color presentation features, which are then compared with the reference features. If no matching record item is found in the reference table for the direction and degree of difference, the on-site verification image data is deemed unreliable, and connection status data and power supply status data are not determined based on them. Image re-acquisition task data is generated according to the process described in Example 2.

[0025] Furthermore, referring to Figure 3The process of determining the associated time window includes: reading the number of time synchronization records in the logical layer running data; if the number of time synchronization records is more than two, and the time interval between the first time reference and the calibration time corresponding to the most recent time synchronization record does not exceed the effective time range of the time synchronization mechanism corresponding to the data source, then the trend of deviation over time is determined based on the change in the deviation value between two adjacent time synchronization records; based on the trend and the time interval between the first time reference and the most recent time synchronization record, the expected deviation of the corresponding time of the on-site verification image data is calculated; and the time reference offset range is obtained based on the expected deviation and the deviation relative to the standard time source; if the number of time synchronization records is less than two, or the aforementioned time interval exceeds the effective time range, then the deviation relative to the standard time source is used as the time reference offset range; and the associated time window is determined by expanding forward and backward in both directions according to the time reference offset range, with the first time reference as the center.

[0026] Specifically, the primary time reference for on-site verification image data refers to the local time record used when acquiring the on-site verification image data; the deviation relative to the standard time source refers to the known difference between the local time record and the unified standard time source. The time synchronization record in the logical layer operational data refers to the record retained when the data source and the unified standard time source complete time calibration. Each time synchronization record corresponds to a calibration time and the deviation value determined at that time. Both types of deviations use the same time unit and direction rules, uniformly recording local time advance as positive and lag as negative, and recording the allowable error range for each.

[0027] The system reads the number of time synchronization records in the logical layer's operational data and the time interval between the first time reference and the calibration time corresponding to the most recent time synchronization record. If there are two or more time synchronization records, and the time interval does not exceed the effective time range of the time synchronization mechanism corresponding to the data source (this range, along with the standard-specified bounded recovery time range, is obtained from a pre-established reference table based on the data source type), the system determines the trend of deviation over time based on the difference between the deviation values ​​corresponding to two adjacent time synchronization records and the time interval between the calibration times corresponding to the two records. Based on this trend and the aforementioned time interval, the system calculates the expected deviation of the first time reference at the corresponding time, starting from the deviation value corresponding to the most recent time synchronization record. The expected deviation is compared with the deviation of the image side relative to the standard time source according to a unified direction rule. The basic offset is determined by the direction and duration of the difference between the two, and the allowable error ranges on both sides are merged to obtain the time reference offset range. If time rollback, abrupt deviation changes, or the extrapolation result exceeds the allowable deviation range, the extrapolation result is not used; instead, the most recent effective deviation and its allowable error range are used. If time synchronization recording is insufficient or has failed, the deviation of the image side relative to the standard time source and its allowable error range shall be used as the time reference offset range.

[0028] The reference table is established item by item based on data source type and status type, recording the effective time range, allowable deviation range, standard-specified bounded recovery time range, recovery start conditions, sampling interval, and applicable version of the time synchronization mechanism. Each record item is pre-determined based on applicable technical specifications and actual operating configuration; if a corresponding record item is missing or the record item version is invalid, time correlation and persistence judgment are not automatically executed, and a configuration verification task is not generated. When the data source type, technical specifications, or operating configuration is updated, a new record item is enabled after verification, and the version identifier is retained.

[0029] The time base offset range includes the basic offset direction, the basic offset duration, and the allowable error range. Taking the first time base as the nominal center, the basic offset duration and the allowable error range are expanded along the basic offset direction, and the allowable error range is expanded in the opposite direction; when the basic offset duration is zero, it is expanded equally forward and backward according to the allowable error range to obtain the associated time window.

[0030] Furthermore, the first modal data and the second modal data are associated as state data pairs, including: extracting the identification presentation features corresponding to the object identifier from the on-site verification image data as the image-side identifier; extracting the identifier information of the record annotation from the logical activity data as the record-side identifier; determining whether the image-side identifier and the record-side identifier point to the same object identifier; if they point to the same object identifier, associating the first modal data with the second modal data generated from the logical activity data retrieved within the association time window as a state data pair; if they point to different object identifiers, no state data pair is generated, and it is recorded as an anomaly to be associated.

[0031] Further, determining whether the state data pair satisfies the causal relationship includes: determining whether the connection state data indicates that the corresponding object identifier is in a connected state, and whether the power supply state data indicates that it is in a powered-on state; if both the connection state data and the power supply state data indicate that the precondition is met, then the state data pair is determined to satisfy the causal correspondence; if either the connection state data or the power supply state data indicates that the precondition is not met, then it is determined whether there is an activity record in the logical activity data after the end time of the associated time window; if there is an activity record, then the state data pair is determined not to satisfy the causal correspondence; if there is no activity record, then the state data pair is determined to satisfy the causal correspondence.

[0032] Specifically, the identifier presentation features corresponding to the object identifier are extracted from the on-site verification image data and converted into image-side identifiers with the same encoding rules as the logical layer operation data. Identification information labeled on the record itself is extracted from the logical activity data and used as the record-side identifier; this identifier information is an existing label carried by the logical activity data during its generation, and this method directly reads and uses it without further limiting the generation process of this label. It is determined whether the image-side identifier and the record-side identifier point to the same object identifier: if they point to the same object identifier, the first modal data and the second modal data generated by the logical activity data are associated as a state data pair; if they point to different object identifiers, no state data pair is generated, and it is recorded as an pending association anomaly. If the object identifier conversion fails, the retrieval interface returns a failure, or the data source is not fully read, it is recorded as a data acquisition anomaly and the consistency judgment is exited; this situation is not considered as a non-existent activity record.

[0033] The second modality data is formed based on the search results. When the search is successful and an activity record exists, the search key, associated time window, time of the activity event, and recording side identifier are recorded. When the search is successful but no activity record exists, the search key, associated time window, time of query completion, and empty result flag are recorded. The search key is used as the associated identifier to form a status data pair with the first modality data that has the same image side identifier, and no recording side identifier is generated. The empty result flag only indicates that no activity record of the corresponding object was found within the successfully read data source and associated time window, and is stored separately from data acquisition anomalies.

[0034] Logical activity data refers to the real-time activity records generated by the object corresponding to the object identifier. The generation of these activity records requires that the object be in a physically connected state and powered on, which distinguishes them from records generated by other objects or not generated in real time. The recording time is the moment the activity event occurs. This method only performs causal correspondence judgment on logical activity data that meets the above conditions.

[0035] The causal correspondence is a necessary condition for data consistency verification. It is a necessary condition for data consistency verification. It is a correspondence that should be satisfied between the connection and power conditions indicated by the connection status data and power supply status data and the activity status indicated by the logical activity data. The establishment of connection and power conditions is a prerequisite for the generation of logical activity records, but not a sufficient condition for the generation of logical activity records.

[0036] The system determines whether the connection status data indicates a connected state and whether the power supply status data indicates a powered-on state. If both indicate that the preconditions are met, the status data pair is determined to satisfy the causal correspondence, regardless of whether there are any activity records in the logical activity data within the associated time window. If either the connection status data or the power supply status data indicates that the preconditions are not met, the system further determines whether there are any activity records in the logical activity data after the end time of the associated time window: if there are activity records, the status data pair is determined not to satisfy the causal correspondence; if there are no activity records, the status data pair is determined to satisfy the causal correspondence.

[0037] Furthermore, the process for outputting transient deviation and service anomaly determination results includes: acquiring logical activity data multiple times at preset intervals within the standard-defined bounded recovery time range to obtain a sequence of changes in activity records over time; determining whether the change sequence shows a process of gradual change in activity records that matches connection status data and power supply status data; if the change sequence shows the change process and reaches a match within the standard-defined bounded recovery time range, it is determined to be a transient deviation; if the change sequence does not show a change process, or fails to reach a match even after exceeding the standard-defined bounded recovery time range, it is determined to be a persistent contradiction, and the service anomaly determination result is output, wherein the service anomaly determination result includes object identifier, connection status data, power supply status data, and change sequence.

[0038] Specifically, the standard specifies a bounded recovery time range, which refers to the time range required for the recovery of a state type during normal state transition, as determined by applicable technical specifications and matching the state type corresponding to the connection status data and power supply status data. This time range is obtained from a pre-established reference table based on the state type. The preset interval is determined jointly based on the standard-specified bounded recovery time range corresponding to the state type recorded in the same reference table, to ensure that multiple sampling points reflecting the changes in the activity record are obtained within this time range. The reference table also records the applicable objects, recovery start conditions, range end point, and version; if there is no corresponding valid record, the process proceeds to the review stage, and automatic routing is not executed.

[0039] For state data pairs that are first confirmed to lack the necessary consistency conditions after the associated time window ends, the confirmation completion time is taken as the starting point for recovery observation, and the observation endpoint is determined according to the reference table. Before the observation begins, it is confirmed that the original connection state data and power supply state data are still within the valid time period; if the re-collection results during the observation period indicate that the connection state or power supply state has actually changed, the original change sequence is terminated, and state data pairs are re-formed based on the new state. Each sampling period only counts logical activity records whose event occurrence time falls within that period; if consistency is confirmed by subsequent sampling periods after a match is achieved, it is determined to be a transient deviation; if a match is first achieved at the observation endpoint and subsequent confirmation is lacking, a review task is generated, and it is not directly determined to be a transient deviation; if there is no match at all or if there is a mismatch after a match, it is determined to be a persistent contradiction.

[0040] Furthermore, the data acquisition process for verification includes: based on the verification task data, re-collecting on-site verification image data and corresponding logical activity data for the object identifier; extracting original features from the re-collected on-site verification image data that are at the same representation level as the reference features used to generate the first modality data; comparing the original features with the reference features to determine whether the reference features have changed; if the reference features have not changed, extracting connection status data and power supply status data based on the original features; determining the difference between the obtained data and the status data pair as a change in the corresponding status of the object identifier; if the reference features have changed, updating the reference features with the original features; and re-extracting connection status data and power supply status data based on the updated reference features to obtain the verification data.

[0041] Specifically, based on the topological connection data, records determined to be persistent contradictions are clustered according to the connection affiliation of the involved object identifiers in the topological connection relationship: if multiple object identifiers belong to the same parent connection node, review task data pointing to that parent connection node is generated; if the object identifiers do not have a common parent connection node, review task data pointing to each object identifier is generated separately.

[0042] Based on the verification task data, on-site verification image data and corresponding logical activity data were re-collected for the identified objects. From the re-collected on-site verification image data, the original features that are at the same representation level as the reference features used to generate the first modality data were extracted. The representation level refers to the transformation stage of the feature; both the reference features and the original features belong to the level of presentation features directly extracted from the image before being converted into connection status data and power supply status data. The original features were compared with the reference features to determine whether the reference features had changed. If they had not changed, connection status data and power supply status data were extracted based on the original features. If they had changed, the connection status and power supply status corresponding to the object identifier were obtained through direct on-site confirmation. Only when the status obtained through direct on-site confirmation was consistent with the status extracted based on the original features were the reference features updated with the original features, and the connection status data and power supply status data were re-extracted based on the updated reference features. In both of the above scenarios, the connection status data and power supply status data that are re-extracted are used as verification data. In the case where no changes have occurred, the difference between the verification data and the original status data pair is determined as the actual change in the status corresponding to the object identifier. The re-collected logical activity data is associated with the verification data as a verification status data pair, and a new judgment is made based on the causal correspondence: if satisfied, the original business anomaly judgment result is revoked; if not satisfied, the verification confirmation result is output.

[0043] Example 2: Based on Example 1, this embodiment focuses on explaining the specific implementation method of forming indirect inconsistency clues based on connection dependencies, and of conducting reliability assessment of on-site verification image data and triggering image re-acquisition.

[0044] Furthermore, the connection dependencies between object identifiers are determined based on the topology connection relationship data. If, based on the connection dependency, the physical connection path of the second object identifier is determined to be provided solely by the first object identifier, and there are no other object identifiers independent of the first object identifier that can provide a connection path for the second object identifier, and the persistent contradiction corresponding to the first object identifier has been verified and confirmed, then it is determined whether there is an activity record in the logical activity data corresponding to the second object identifier within the time range corresponding to the persistent contradiction. If there is an activity record, the record corresponding to the second object identifier is marked as an indirect contradiction and included in the business anomaly judgment result. If there is no activity record, it is not marked.

[0045] Specifically, in addition to being used to generate verification task data for persistent contradiction clustering, the topological connectivity data also records the source relationships of the physical connection paths of each object identifier. That is, whether the physical connection of a certain object identifier needs to be realized through a connection path provided by another object identifier. The result determined after reading the source relationship is used as the connection dependency. When determining the connection dependency, all connection path source records corresponding to the second object identifier in the topological connectivity data are read: if the record only points to the first object identifier, it is determined that the physical connection path of the second object identifier is only provided by the first object identifier; if the record also points to other object identifiers, it is determined that there are other connection paths independent of the first object identifier, which does not meet the applicable premise of indirect causal inference.

[0046] After determining that the physical connection path of the second object identifier is provided solely by the first object identifier, the time range used to determine a persistent contradiction when reading the state data pair corresponding to the first object identifier is the time range corresponding to the observed activity record change sequence within the bounded recovery time range specified in the standard. This time range is directly reused as the time range for determining whether there is an activity record in the logical activity data corresponding to the second object identifier, without setting a new time range. If there is an activity record in the logical activity data corresponding to the second object identifier within this time range, it is determined to be an indirect contradiction: an indirect contradiction refers to a situation where the object identifier itself has not collected on-site verification image data, has not generated first modality data and state data pairs, but based on the connection dependency relationship and the persistent contradiction already confirmed by the upstream object identifier, the result is a determination that its logical activity record contradicts the actual state of the physical layer; the business anomaly determination result in the case of indirect contradiction includes the second object identifier, the first object identifier, the connection dependency relationship, and the determination result on which the persistent contradiction corresponding to the first object identifier is based, to explain that the contradiction is not derived from the state data pair of the second object identifier itself, but is inferred from the connection dependency relationship and the persistent contradiction already confirmed by the first object identifier.

[0047] Furthermore, before adjusting the color presentation features based on the difference between the shooting condition information and the shooting condition information corresponding to the reference features, the method further includes: determining whether the shooting condition information is within the range of shooting condition differences applicable to the adjustment, and whether the area occupied by the connection entity presentation features and color presentation features is complete; if the shooting condition information exceeds the range of shooting condition differences applicable to the adjustment, or the area occupied is incomplete, then the on-site verification image data corresponding to the area is determined to be unreliable, and its color presentation features are not adjusted, nor is it used to determine connection status data or power supply status data; if the shooting condition information is within the range of shooting condition differences applicable to the adjustment, and the area occupied is complete, then it is determined to be reliable, and subsequent adjustments and comparisons are performed accordingly.

[0048] Furthermore, for the object identifier corresponding to the unreliable on-site verification image data, image re-acquisition task data is generated; on-site verification image data for the object identifier is re-acquisitioned based on the image re-acquisition task data, and connection status data and power supply status data are determined for the re-acquisitioned on-site verification image data; if the re-acquisition is still determined to be unreliable, the object identifier is recorded as pending supplementary verification, and no first modality data is generated.

[0049] Specifically, the reliability of on-site verification image data is determined based on two factors: first, whether the shooting conditions are within the effective applicable range of the color presentation feature adjustment method itself; and second, the completeness of the extracted connection entity presentation features and color presentation features in the area. The effective applicable range for the former is determined by referring to a pre-established reference table recording the effective range of color presentation feature adjustment methods under different shooting conditions, based on the area type. This reference table originates from image processing-related technical specifications or existing practical experience. The completeness determination for the latter is determined by referring to the same reference table based on the area type. This reference table is independent of the reference table described in Example 1, which records the bounded recovery time range corresponding to each state type, and serves two different types of judgments: image processing effectiveness determination and state recovery time determination. When the shooting conditions exceed the effective applicable range, even if the adjustment result is calculated according to the adjustment method, the correspondence between the result and the actual connection and power supply status no longer holds, and therefore it is not used for subsequent status determination. If the completeness of the presentation does not reach the minimum level for comparison, it is considered unreliable due to occlusion or incomplete presentation area.

[0050] If any of the above conditions, such as the shooting conditions or the completeness of the presentation, are not met, the on-site verification image data is deemed unreliable. Its color presentation characteristics are not adjusted, nor is it used to determine connection status or power supply status data. Image re-acquisition task data is generated, recording the object identifier and verification area corresponding to the unreliability. This data is submitted to the on-site verification task scheduling stage as part of the task list for the next on-site verification activity, with the specific execution timing following the existing on-site verification task scheduling arrangements. After re-acquiring on-site verification image data based on the image re-acquisition task data, the aforementioned reliability assessment and first-mode data generation process are repeated. If the data is still deemed unreliable after re-acquisition, no new image re-acquisition task data is generated. Instead, the corresponding object identifier is recorded as pending supplementary verification, and the specific area and reason for the unreliability are marked, to be confirmed later through methods other than on-site verification. This is not included in the current causal consistency judgment process. When reliable images or direct on-site confirmation results are subsequently obtained, the confirmation status, confirmation time, and processing result are written back to the image re-acquisition task data, and the pending supplementary verification status is closed. If the input conditions are met, the necessary condition consistency judgment process is re-entered.

[0051] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A business anomaly detection method based on multimodal data fusion, characterized in that, include: Connection status data and power supply status data are extracted from the collected on-site verification image data to generate the first modality data; The first time reference and the deviation relative to the standard time source of the on-site verification image data are obtained. The time reference offset range is obtained by combining the time synchronization records in the logic layer operation data, and the associated time window is determined. Within the associated time window, retrieve logical activity data from the logical layer's operational data to generate second modal data, and associate the first modal data and the second modal data as a state data pair; Based on the causal correspondence between connection status data, power supply status data, and logical activity data, determine whether the status data pairs satisfy the causal correspondence. If the status data that does not meet the requirements is observed within the bounded recovery time range specified by the standard for the connection status data and power supply status data, and the recovery is consistent within the range, it is judged as a transient deviation; if it still does not meet the requirements outside the range, it is judged as a persistent contradiction and the service anomaly judgment result is output. Based on the topological connection relationship data, persistent contradictions are clustered to generate review task data, and review data is obtained.

2. The business anomaly detection method based on multimodal data fusion according to claim 1, characterized in that: The first modal data acquisition process includes: if the state presentation mode of the object identifier corresponding to the on-site verification image data is unchangeable and has a physical connection form, it is determined as the image data to be extracted; in the image data to be extracted, the region with a fixed positional correspondence with the object identifier is determined, and the color presentation features, connection entity presentation features and corresponding shooting condition information of the region are extracted; based on the difference between the shooting condition information and the shooting condition information corresponding to the reference features of the same region in a known state, the color presentation features are adjusted; the adjusted color presentation features and connection entity presentation features are compared with the reference features respectively, and the connection status data and power supply status data are determined based on the comparison results to generate the first modal data.

3. The business anomaly detection method based on multimodal data fusion according to claim 1, characterized in that: The process of determining the associated time window includes: reading the number of time synchronization records in the logical layer running data; if the number of time synchronization records is more than two, and the time interval between the first time reference and the calibration time corresponding to the most recent time synchronization record does not exceed the effective time range of the time synchronization mechanism corresponding to the data source, determining the trend of deviation over time based on the change in the deviation value between two adjacent time synchronization records, and calculating the expected deviation of the corresponding time of the on-site verification image data based on the trend and the time interval between the first time reference and the most recent time synchronization record, and obtaining the time reference offset range based on the expected deviation and the deviation relative to the standard time source; if the number of time synchronization records is less than two, or the aforementioned time interval exceeds the effective time range, then the deviation relative to the standard time source is used as the time reference offset range; and determining the associated time window by expanding forward and backward in both directions according to the time reference offset range, with the first time reference as the center.

4. The business anomaly detection method based on multimodal data fusion according to claim 1, characterized in that: Associating the first modal data and the second modal data as a state data pair includes: extracting the identifier presentation features corresponding to the object identifier from the on-site verification image data as the image-side identifier; extracting the identifier information of the record annotation from the logical activity data as the record-side identifier; determining whether the image-side identifier and the record-side identifier point to the same object identifier; if they point to the same object identifier, associating the first modal data with the second modal data generated from the logical activity data retrieved within the association time window as a state data pair; if they point to different object identifiers, no state data pair is generated, and it is recorded as an anomaly to be associated.

5. The service anomaly detection method based on multimodal data fusion according to claim 1, characterized in that: Determining whether a pair of status data satisfies a causal relationship includes: determining whether the connection status data indicates that the corresponding object identifier is in a connected state, and whether the power supply status data indicates that it is in a powered-on state; if both the connection status data and the power supply status data indicate that the preconditions are met, then the status data pair is determined to satisfy the causal correspondence; if either the connection status data or the power supply status data indicates that the preconditions are not met, then it is determined whether there is an activity record in the logical activity data after the end time of the associated time window; if there is an activity record, then the status data pair is determined not to satisfy the causal correspondence; if there is no activity record, then the status data pair is determined to satisfy the causal correspondence.

6. The service anomaly detection method based on multimodal data fusion according to claim 1, characterized in that: The process for outputting transient deviation and service anomaly determination results includes: acquiring logical activity data multiple times at preset intervals within a standard-defined bounded recovery time range to obtain a sequence of changes in activity records over time; determining whether the change sequence shows a process of activity records gradually matching connection status data and power supply status data; if the change sequence shows the change process and reaches matching within the standard-defined bounded recovery time range, it is determined to be a transient deviation; if the change sequence does not show a change process, or fails to reach matching even after exceeding the standard-defined bounded recovery time range, it is determined to be a persistent contradiction, and the service anomaly determination result is output. The service anomaly determination result includes object identifier, connection status data, power supply status data, and change sequence.

7. The business anomaly detection method based on multimodal data fusion according to claim 2, characterized in that: The data acquisition process for verification includes: based on the verification task data, re-collecting on-site verification image data and corresponding logical activity data for the object identifier; extracting original features from the re-collected on-site verification image data that are at the same representation level as the reference features used to generate the first modality data; comparing the original features with the reference features to determine whether the reference features have changed; if the reference features have not changed, extracting connection status data and power supply status data based on the original features; determining the difference between the obtained data and the status data pair as a change in the corresponding status of the object identifier; if the reference features have changed, updating the reference features with the original features; and re-extracting connection status data and power supply status data based on the updated reference features to obtain the verification data.

Images