A multi-modal spatio-temporal data standardized addressing warehousing monitoring method

Abstract

The application provides a multi-modal spatio-temporal data standardization addressing warehousing monitoring method, comprising: identifying homologous collection chains, derived blood relationship chains and cross-modal corresponding relationships by analyzing the multi-modal evidence source chain, merging evidence from the same collection device or the same alarm chain to obtain a multi-modal homologous observation link; analyzing independent multi-modal fact sets, identifying support paths of different modal fact nodes in a grid unit, evaluating the source independence, modal complementarity and propagation position of each path, and determining the core fact nodes supported across modalities; updating evidence warehousing records according to the core fact nodes, identifying original collection evidence and associated modal evidence in homologous observation, evaluating the multi-modal refresh qualification of the Cube layer unit, and generating a multi-modal layer warehousing task.

Description

Technical Field

[0001] This invention relates to the field of information technology, and in particular to a method for monitoring the standardized addressing and data storage of multimodal spatiotemporal data. Background Technology

[0002] Multimodal spatiotemporal data warehousing and monitoring plays an irreplaceable role in urban governance, emergency response, and social fact verification. It requires integrating various types of evidence, such as images, vector data, and alarms, into a unified grid and time coordinate system to support fact-finding. Ideally, the degree of mutual corroboration among multiple observations can be quickly assessed and credibility derived by statistically analyzing the number of evidence modalities. However, in actual operation, it has been found that seemingly abundant warehousing results do not match the actual sufficiency of observations; the count of factual evidence continuously inflates, and the update frequency of layer units is driven by artificially high levels of duplicate evidence. The root cause of this failure lies in the existing approach's assumption that different modalities are independent of each other, ignoring the potential shared collection sources and derivation relationships between pieces of evidence. When multiple pieces of evidence fall within the same grid code and the same time code, and originate from a single collection activity of the same acquisition device or a single triggering event of the same alarm source, these pieces of evidence are considered spatiotemporally homogeneous evidence. Spatiotemporally homogeneous evidence refers to evidence that, although having different data forms or modal markers, corresponds to the same collection spatiotemporal range and shares the same original information source. Meanwhile, an image frame is automatically parsed into a vector target, and an alarm may trigger subsequent alarms. When original evidence is processed through target recognition, primitive extraction, or alarm triggering to generate subsequent evidence, the traceable generation relationship formed between evidence at each level is called the derived lineage; the vector targets, derived primitives, or subsequent alarms formed along this generation relationship are called derivatives; the original image frame or initial alarm record that directly originates from the acquisition activity or initial triggering event and serves as the basis for the generation of subsequent evidence is called the parent. Although derivatives enter the statistics with a new modality, their substantive information still originates from the corresponding parent. To simplify the evaluation logic, the conventional approach is to use the number of evidence modalities as an important benchmark for credibility. Although this method based on quantitative accumulation can intuitively reflect the breadth of modality coverage and improve the evaluation speed, if evidence from the same source and derivatives are weighted together with the parent, the same fact will be repeatedly accumulated, and the conflict rate and duplication rate in the quality report will deviate from the actual situation. This is because existing assessment methods neglect the shared sources of evidence and the derivative lineage between parent and derivative evidence. The more modalities there are, the more likely it is to mask the scarcity of truly independent information sources, causing the determination of factual credibility to deviate from reality. Therefore, in multimodal data warehousing monitoring, identifying the spatiotemporal homology and derivative lineage between evidence, ensuring that cross-modal evidence within the same grid code and time code participates in weighting only according to truly independent information sources, and avoiding the repeated superposition of evidence collected from the same source and derivative evidence as multiple independent facts, has become a pressing technical challenge. Summary of the Invention

[0003] This invention provides a method for monitoring the standardized addressing and data storage of multimodal spatiotemporal data, the method comprising: Acquire image frames, derived primitives, and alarm records containing spatial coordinates and timestamps, label them as visual, primitive, and recording modalities, complete cross-modal standardized addressing according to the same grid code and time code, and form a multimodal common spatiotemporal evidence group; Extract the acquisition device identifier, derived primitive tracing marker, and alarm record trigger source identifier from the multimodal common spatiotemporal evidence group to form a multimodal evidence source chain; Analyze the multimodal evidence source chain, identify the homologous acquisition chain, the derived lineage chain and the cross-modal correspondence, merge evidence from the same acquisition device or the same alarm chain, and obtain the multimodal homologous observation link; By comparing the content regions of derived primitives and image frames and the triggering sources of alarm records along the multimodal homogeneous observation link, cross-modal homogeneous duplication is identified, evidence of derived duplication is eliminated, and an independent multimodal fact set is formed. Analyze the independent multimodal fact set, identify the support paths of each modal fact node within the grid cell, evaluate the path's source independence, modal complementarity, and propagation location, and determine the core fact nodes supporting cross-modal functions; Update the evidence entry record based on the core fact nodes, evaluate the multimodal refresh eligibility of the Cube layer unit, and generate a multimodal layer entry task; The conflict entry database is used to retrieve the multimodal layer entry tasks, analyze cross-modal conflicts and duplicate entry traces within the same grid code and time code, evaluate the consistency of core fact nodes, and output fact acceptance conclusions.

[0004] Furthermore, the acquisition of image frames containing spatial coordinates and timestamps, derived primitives, and alarm records, labeled as visual, primitive, and recording modalities, includes: Visual modalities are labeled by extracting image pixel information from image frames, geometric contours and attribute fields are read from derived primitives and labeled by primitive modalities, and alarm records are parsed by parsing trigger times and event descriptions and labeled by record modalities, resulting in a set of original evidence entries with modal labels; The spatial coordinates of the original evidence entry set are mapped to a unified hierarchical grid code according to equal latitude and longitude. The timestamps are assigned to fixed time codes according to preset time slices. Modal entries that fall into the same grid code and time code are aggregated into the same container.

[0005] Furthermore, the extraction of the acquisition device identifier, derived primitive tracing marker, and alarm record trigger source identifier from the multimodal shared spatiotemporal evidence set includes: Read the acquisition device identifier from the visual modality partition, read the source tracing mark from the primitive modality partition to obtain the parent image frame number, and read the trigger source identifier and associated fields from the recording modality partition; Alarm records under the same trigger source identifier are sorted according to the trigger time, and the associated fields pointing to the previous alarm are connected in series to form an alarm association sequence. Image frames and derived primitives with the same acquisition device identifier are connected according to the source tracing mark, and the alarm association sequence is attached according to the shared grid code and time code to form a multimodal evidence source chain.

[0006] Furthermore, the analysis of the multimodal evidence source chain, identifying the homologous acquisition chain, the derived kinship chain, and the cross-modal correspondence, includes: Extract the acquisition device identifier, traceability mark and alarm association sequence from the multimodal evidence source chain. Image frames with adjacent shooting times under the same acquisition device identifier are strung together to form a homogeneous acquisition chain. Derived primitives are attached to the parent image frame node of the homogeneous acquisition chain according to the parent image frame number pointed to by the traceability mark, thus obtaining a set of homogeneous acquisition chains with attached derivation branches. For each parent image frame node, recursively traverse the derived primitives with that parent along the source marker, and connect them in the parent-child direction to form an image derivation bloodline chain. The alarm association sequence extends from the head of the chain along the associated field to the last derived alarm to form an alarm derivation lineage chain. The two are combined to obtain a set of derivation lineage chains.

[0007] Furthermore, the two combined form a set of derived bloodline chains, including: For homogeneous acquisition chains and alarm derivation chains that fall into the same grid code and time code, determine whether the acquisition device identifier and alarm chain head number of the two are registered as directly linked in the multimodal evidence source chain; Registered direct connection relationships are merged into cross-modal correspondence relationships and evidence entries are combined. For duplicate derived branch nodes under the same parent image frame number, duplicate nodes are removed and only one is retained to obtain a multimodal co-source observation link.

[0008] Furthermore, the step of comparing the content regions of derived primitives and image frames along the multimodal homogeneous observation link to identify cross-modal homogeneous repetition includes: The geometric contour of the primitive modal entry is converted to the pixel coordinate system of the visual modal entry according to the imaging parameters of the parent image frame to obtain the content area. If the content area falls within the imaging boundary, the primitive modal entry is determined to be a derivative. The trigger source identifier of the recorded modal entry is compared bit by bit with the acquisition device identifier of the visual modal entry. If the strings match, the recorded modal entry is determined to be a derived trigger product. The derived product and the derived trigger product are removed to form an independent multimodal fact set.

[0009] Furthermore, the analysis of the independent multimodal fact set, identifying the support paths of each modal fact node within the grid cell, and evaluating the path's source independence, modal complementarity, and propagation location includes: The data are grouped according to the same grid code to form a grid cell fact set. Each entry is regarded as a fact node. The data is traced back to the source of the data collection along the connection relationship in the multimodal co-source observation link to obtain the support path. The acquisition device identifier is read from the visual side node of the path, and the alarm chain head number is read from the recording side node. If the acquisition device identifier and alarm chain head number are different for different paths, the source independence is determined. The modal complementarity is determined according to the number of modal markers covered by the path. The propagation position is obtained by reading the spatial coordinates from the end node of the path. When the following three conditions are met simultaneously: the data acquisition device identifier and the alarm chain head number of the support path are different from each other, the data covers at least two types of modal markers, and the propagation position falls within the latitude and longitude boundary of the grid cell, the corresponding fact node is determined as the core fact node of cross-modal support.

[0010] Furthermore, the step of updating the evidence entry record based on the core fact node and evaluating the multimodal refresh eligibility of the Cube layer unit includes: The core fact node grid code, time code and modal type mark are used to register the content carried into the evidence warehouse record. The visual modal item reached by locating upward along the attachment path is the original collected evidence. The original collected evidence is registered as the attached graphic element modal item and the record modal item of the shared collection device identifier as the associated modal evidence. If the registered modal type marker set and the newly added modal type marker set under the same time code are different, the modal type marker set is deemed eligible for refresh. The modal type marker set is then assembled into a multimodal layer loading task based on the grid code, time code, and newly added modal type marker set.

[0011] Furthermore, the step of retrieving the multimodal layer loading task using the conflict entry library and analyzing cross-modal conflicts and duplicate loading traces within the same mesh code and time code includes: The conflict entry library is pre-set with conflict pattern items indexed by grid code and time code; Candidate conflict mode items are located by using the grid codes and time codes of the original collection evidence and related modal evidence of the warehousing task. Cross-modal conflicts are determined by comparing the event description fields of different modal entries under the same grid code and time code to see if they give mutually exclusive categories or state values. Duplicate warehousing traces are determined by comparing the original collection evidence numbers of different warehousing tasks to see if they are the same. After backfilling the markers, a set of warehousing tasks with attached conflict and duplicate annotations is obtained.

[0012] Furthermore, obtaining the inbound task set with accompanying conflicts and duplicate annotations includes: If the cross-modal conflict marker of the core fact node in the warehouse task set is empty and the duplicate warehouse entry trace marker is empty, then cross-modal consistency is determined. If the modal coverage of the associated modal evidence is not less than two and the original collected evidence number is unique under this grid code and time code, then the basis for acceptance is established. The core fact node is output as an admissible fact and written into the evidence warehouse entry record table.

[0013] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects: This invention discloses a method for standardized addressing and monitoring of multimodal spatiotemporal data during data warehousing. Addressing the problems of inconsistent spatiotemporal addressing, ambiguous source links, cross-modal duplicate warehousing, and difficulty in identifying conflicts in the warehousing process of multi-source heterogeneous data such as image frames, derived primitives, and alarm records, the method standardizes cross-modal addressing by using the same grid code and time code for visual, primitive, and record-based modalities. This constructs a common spatiotemporal evidence set for multimodal data. Furthermore, it extracts the acquisition device identifier, derived source tracing marker, and alarm trigger source identifier to form a source chain through serial generation, thereby merging data acquired from the same source and... By deriving kinship to obtain the same observation link, comparing the content region with the trigger source to eliminate derived duplicate evidence and form an independent set of facts, and then evaluating the source independence, modal complementarity and propagation position of the supporting path to determine the core fact nodes, the entry record is updated accordingly and a layer entry task is generated. Finally, by using the conflict entry library to retrieve cross-modal conflicts and duplicate entry traces, the cross-modal consistency of the core nodes is evaluated and the acceptance conclusion is output. This achieves unified addressing, deduplication and credible entry of cross-modal evidence, which significantly improves the consistency, traceability and entry quality of spatiotemporal data. Attached Figure Description

[0014] Figure 1 This is a flowchart of a multimodal spatiotemporal data standardization addressing and warehousing monitoring method according to the present invention.

[0015] Figure 2 This is a schematic diagram of a multimodal spatiotemporal data standardization addressing and warehousing monitoring method according to the present invention.

[0016] Figure 3 This is another schematic diagram of a multimodal spatiotemporal data standardization addressing and warehouse entry monitoring method according to the present invention. Detailed Implementation

[0017] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0018] like Figures 1-3 This embodiment of a multimodal spatiotemporal data standardization addressing and warehousing monitoring method may specifically include: Step S101: Obtain image frames, derived primitives, and alarm records containing spatial coordinates and timestamps, and mark them as visual, primitive, and recording modalities respectively. Complete cross-modal standardized addressing according to the same grid code and time code to form a multimodal common spatiotemporal evidence group.

[0019] Image frames, derived primitives, and alarm records with spatial coordinates and timestamps are acquired. For image frames, pixel information of the captured image is extracted and visual modality is labeled. For derived primitives, the geometric contours and attribute fields of vector targets are read and primitive modalities are labeled. For alarm records, the trigger time and event description are parsed and the record modality is labeled, resulting in a set of original evidence entries with modal tags. For the spatial coordinates in the original evidence entry set, a uniform latitude and longitude division method is used to map the latitude and longitude to a unified level of grid coding. For the timestamps, they are assigned to fixed-time codes according to a preset time slice length, forming a cross-modal consistent grid code and time code combination address, completing the standardized addressing process. Based on the combined address output by the standardized addressing process, visual modal, primitive modal, and record modal entries falling into the same grid code and time code are aggregated into the same container according to the combined address. The modal tags, geometric contours, and event description fields of each entry are retained within the container, resulting in a multimodal common spatiotemporal evidence group for cross-modal comparison.

[0020] In one possible implementation, multimodal spatiotemporal data inbound monitoring targets urban governance and emergency response scenarios. The acquisition end receives image frames captured by satellite remote sensing and low-altitude drones, vector-derived primitives output from the target recognition process, and alarm records reported by monitoring stations and sensing devices. The three types of data have inconsistent forms, field structures, and spatial description precision before entering the storage, and must undergo modal annotation and address normalization processing before entering a unified evidence container.

[0021] Specifically, the processing procedure for the image frame is as follows: read the pixel information of the captured image from the original file, and extract the shooting latitude and longitude, shooting time and sensor number from the image metadata. Use the image pixel information as the content of the evidence, add visual modality tags and acquisition device identifiers to form a visual modality entry.

[0022] For example, the latitude and longitude corresponding to a frame of UAV orthophoto imagery are taken from the center point coordinates recorded by the onboard positioning module, the time is taken from the shutter trigger timestamp, and the acquisition device identifier is formed by concatenating the onboard payload's device category code and factory serial number. The processing procedure for derived primitives is as follows: read the geometric contour and attribute fields of the vector target. The geometric contour refers to the target's point, line, and surface coordinate sequence, and the attribute fields include the target category name, confidence level value, and tracing mark. Each derived primitive is assigned a unique number upon generation, and the tracing mark records the direct parent number of the derived primitive: when the direct parent is an image frame, the parent image frame number is filled in; when the direct parent is another derived primitive, i.e., a further derivative of the primitive, the parent primitive's own number is filled in. The above content is used as the carrying content and an primitive modal tag is added to form a primitive modal entry. The processing procedure for alarm records is as follows: parse the trigger time and event description in the alarm message. The event description includes the alarm category, alarm level, and trigger source identifier. The above content is used as the carrier content and a record modality marker is attached to form a record modality entry. The three types of entries are combined to form the original evidence entry set.

[0023] Preferably, the standardized addressing process uses an equal latitude and longitude division method to map spatial coordinates.

[0024] Specifically, the Earth's surface is divided into regular grids according to preset longitude and latitude steps, and each grid is assigned a unique grid code. For each spatial coordinate in the original evidence entry set, the grid interval into which its longitude and latitude values ​​fall is determined, and the corresponding grid code is written into the address field of that entry. If the geometric outline of a derived graphic element spans multiple grids, a unified placement rule is adopted: for area graphic elements, the longitude and latitude of the center point of its circumscribed rectangle are used as the placement coordinates; for linear graphic elements, the longitude and latitude of the midpoint of the line connecting its first and last points are used as the placement coordinates; for point graphic elements, its own coordinates are used directly. The grid where the placement coordinates are located is the placement grid of that derived graphic element. For timestamp processing, the continuous time axis is divided into equal-length time segments according to a preset time slice length, and each time segment is assigned a unique time code; for the timestamp of each piece of evidence, the time segment into which it falls is determined and the corresponding time code is written. The grid code and time code are concatenated to form a combined address, thereby obtaining a unified address expression for the three types of entries spanning the visual modality, primitive modality, and recording modality.

[0025] It should be noted that the preset time slice length is set according to the business observation cycle.

[0026] For example, time slices of 5 to 30 seconds are used for urban street event monitoring, while time slices of 5 to 30 minutes are used for regional remote sensing inspections, ensuring that evidence from different observation times still falls into a consistent time code. Based on the combined address output by the standardized addressing process, visual modal entries, primitive modal entries, and record modal entries that fall into the same grid code and time code are aggregated into the same container according to the combined address. The container is internally divided into three partitions, each indexed by a modal marker. The visual modal partition stores image pixel information and sensor device IDs. The image pixel information is used for pixel-level alignment and connection with the geometric contours in the primitive modal partition during the subsequent source chain construction stage to verify whether the contour boundary landing point matches the pixel grayscale jump position. The sensor device ID is written into the source chain as a source attribute. The primitive modal partition stores geometric contours, attribute fields, and tracing markers. The record modal partition stores the trigger time, event description, trigger source identifier, and associated fields pointing to the previous alarm number. The trigger source identifier is taken from the alarm message header field, and the associated fields are taken from the previous alarm reference bit in the message body. Both support the reading and backtracking of the subsequent source chain. The partitions are interconnected through combined addresses.

[0027] It is understandable that containers under the same combined address constitute a multimodal shared spatiotemporal evidence set. Entries within this evidence set share a unified grid code and time code, and cross-modal fields are fully preserved, providing a consistent data foundation for subsequent identification of evidence sources and derivation relationships.

[0028] Step S102: Extract the acquisition device identifier, derived primitive tracing mark, and alarm record trigger source identifier of the image frame from the multimodal common spatiotemporal evidence group, trace the alarm trigger timing, connect the cross-modal generation relationship, and form a multimodal evidence source chain.

[0029] The acquisition device identifier is read from the visual modality partition of the multimodal shared spatiotemporal evidence group. The source marker is read from the primitive modality partition to obtain the parent image frame number corresponding to the derived primitive. The trigger source identifier and the associated field pointing to the preceding alarm number are read from the recording modality partition to obtain a cross-modal source attribute set. For the recording modality portion of the source attribute set, alarm records under the same trigger source identifier are sorted according to the order of triggering time. It is determined whether the associated field between adjacent alarms points to the preceding alarm number. If they do, they are chained together to form an alarm triggering time sequence link, resulting in an alarm association sequence with a sequential order. Based on the acquisition device identifier, the source marker, and the alarm association sequence, the image frame corresponding to the same acquisition device identifier and its derived primitive are connected according to the source marker. The alarm records in the alarm association sequence are aligned and linked with the image frame and derived primitive according to the shared grid code and time code, forming a multimodal evidence source chain spanning the visual, primitive, and recording modalities.

[0030] In one possible implementation, the evidence source identification stage of multimodal spatiotemporal data inbound monitoring directly uses the aforementioned multimodal common spatiotemporal evidence group as input. This evidence group has been grouped into the same container according to combined addresses, categorizing visual modality entries, primitive modality entries, and record modality entries. The container stores the carrying fields of each entry according to modal marker partitions, providing a consistent data foundation for extracting cross-modal source attributes. The acquisition device identifier in the visual modality partition is formed by concatenating the payload device category code and payload sequence number registered in the image frame metadata. After reading, it is directly written into the acquisition device identifier field of the image frame. The parent image frame number in the primitive modality partition originates from the target recognition process. This process takes a single frame image from the visual partition as input, and the YOLOv8 detection network selects and classifies targets such as vehicles, people, and abandoned objects in the image. The output includes the target category, bounding rectangle coordinates, confidence level, and the unique number of the corresponding image frame. The unique number of the corresponding image frame is written into the parent image frame number field of the derived primitive, achieving reverse tracing from primitive to source frame. The trigger source identifier in the recorded modal partition is taken from the alarm message header field, pointing to the monitoring station or sensing device that issued the alarm. At the same time, the association field in the alarm message is read to establish cross-partition association with the first two types of modal data.

[0031] It should be noted that the aforementioned association field is used in the alarm message to record the hierarchical relationship between the current alarm and its predecessors. The field content is the unique number of the predecessor alarm. When an alarm is triggered by a predecessor alarm, this field is filled with the predecessor alarm number; when the alarm is the initial trigger, this field is empty. The source attributes of the three modalities are combined to form a cross-modal source attribute set.

[0032] Preferably, for the record modal portion in the source attribute set, the records are grouped according to the trigger source identifier, and the alarm records under the same trigger source identifier are sorted according to the order of trigger time to form a candidate sequence.

[0033] For example, a monitoring station issues multiple alarms during a continuous observation process. Regardless of whether the alarms fall into the same address combination or are scattered into adjacent address combinations, they are all grouped into the same candidate sequence according to the trigger source identifier, thus covering the situation where alarms span multiple timecodes on the timeline. Adjacent alarm entries in the candidate sequence are traversed, and the associated field of the next alarm is read. If the number in the associated field matches the unique number of the previous alarm, they are determined to have a derived trigger relationship and are chained together; if the associated field is empty, it is considered an independent trigger starting point, and a new link is established; if the number in the associated field does not belong to the number set of this candidate sequence, the alarm, along with its preceding number, the trigger source identifier, and the timestamp, is temporarily stored in the pending buffer. The buffer is indexed by its preceding sequence number. After the link construction of each trigger source group is completed, a reverse lookup is performed on the buffer using the unique number of all alarms already in the chain within that group as the key: those that match are removed from the buffer and attached to the corresponding chain tail; those that do not match remain in the buffer. If no match is found after 30 minutes, the alarm is discarded and counted as an orphan. The comparison of cross-trigger source identifiers follows the principle of globally unique numbers, meaning that alarms generated by different trigger sources share the same number space. The comparison key is the preceding sequence number + time window, with the time window set to 5 seconds before and after. Alarms can only be reattached if they match and the time difference does not exceed this window, avoiding misattachment of the same number. After full sequence traversal and round-by-round reverse lookup, an alarm association sequence with sequential order and derivation pointers is obtained. The alarm association sequence has the initial alarm as the chain head and the last derived alarm as the chain tail, and each link in the chain can be traced back to the preceding alarm.

[0034] Specifically, the cross-modal linking process uses the acquisition device identifier, the tracing marker, and the alarm association sequence as three parallel threads. Multiple images corresponding to the same acquisition device identifier are sorted into an image frame sequence according to their capture time. For each image frame, the tracing marker is used to find the derived primitive with its unique number as the parent element, and the derived primitive is linked to the parent image frame, forming a parent-child relationship between visual modal entries and primitive modal entries. Further, each alarm record in the alarm association sequence is aligned and linked with the image frame sequence and derived primitive set sharing the same combined address within the container, based on its own grid code and time code. When the head alarm of the alarm association sequence shares a combined address with a certain image frame and a certain derived primitive, the two serve as spatiotemporal co-occurrence evidence of the alarm head and are linked to the head of the chain; subsequent alarms on the chain are also aligned with the visual modal entries and primitive modal entries under their combined addresses in the same way.

[0035] Understandably, after the aforementioned extraction of source attributes, concatenation of alarm association sequences, and cross-modal linking, visual modal entries, primitive modal entries, and record modal entries are linked together into a unified multimodal evidence source chain through acquisition device identifiers, tracing markers, and alarm association sequences. This source chain retains the modal attribution and source orientation of each piece of evidence, providing a traceable linking structure for subsequent identification of spatiotemporal homology and derivational lineage.

[0036] Step S103: By analyzing the multimodal evidence source chain, identify the homologous acquisition chain, the derived lineage chain, and the cross-modal correspondence, merge evidence from the same acquisition device or the same alarm chain, and obtain the multimodal homologous observation link.

[0037] Extract the acquisition device identifier, tracing marker, and alarm association sequence from the multimodal evidence source chain. For each image frame, group them according to the acquisition device identifier. Image frames with the same acquisition device identifier and adjacent shooting times are concatenated into a common-source acquisition chain. For each derived image element, read the parent image frame number pointed to by the tracing marker, and attach the derived image element to the corresponding parent image frame node in the common-source acquisition chain to obtain a set of common-source acquisition chains with attached derivation branches. For each parent image frame node in the set of common-source acquisition chains, recursively traverse the derived image elements with it as the parent and the lower-level derived image elements derived from it along the tracing marker, and concatenate them according to the parent-child direction to form an image derivation lineage chain. For the alarm association sequence, take the first alarm of the chain as the lineage starting point, and extend it level by level along the association field to the last derived alarm to form an alarm derivation lineage chain. The two are combined to obtain a set of derivation lineage chains. Based on the set of homologous acquisition chains and the set of derived bloodline chains, for a homologous acquisition chain and an alarm derived bloodline chain that fall into the same grid code and the same time code, it is determined whether the corresponding acquisition device identifier and alarm chain head number are registered as having a direct connection relationship in the multimodal evidence source chain. If registered, the two links are merged into a cross-modal correspondence and the evidence entries are merged. For the derived branch nodes that are repeatedly attached under the same parent image frame number, duplicates are removed and one is retained to obtain the multimodal homologous observation link.

[0038] In one possible implementation, the source observation identification step of multimodal spatiotemporal data entry monitoring uses the aforementioned multimodal evidence source chain as input. The multimodal evidence source chain has linked visual modal entries, primitive modal entries and recording modal entries into a traceable hook structure through acquisition device identification, traceability markers and alarm association sequences.

[0039] It should be noted that the homologous observation identification link needs to solve two types of sharing problems: one is the problem that multiple frames of images and their derived primitives generated by the same acquisition device during a continuous observation process belong to the same acquisition source; the other is the problem that a series of subsequent alarms triggered by the head of the same alarm chain belong to the same initial trigger source. The former is reflected in the consistency of the acquisition device identifier, and the latter is reflected in the consistency of the chain head number of the alarm association sequence.

[0040] Specifically, the acquisition device identifier mounted on each image frame is sequentially read from the multimodal evidence source chain. This identifier is formed by splicing a device category code and a device serial number, and the values of the device category code include low-altitude UAV inspection payload, satellite remote sensing imaging payload, ground fixed surveillance camera, etc., which is used to make the abstract acquisition device terms mentioned above correspond to the specific inspection scenarios one by one; all image frames are grouped using the complete acquisition device identifier as the grouping key, after the image frames under the same acquisition device identifier are sorted in ascending order by shooting time, the corresponding continuous observation threshold interval is retrieved according to the device category code, the low-altitude UAV inspection scenario takes a lower limit of 2 seconds and an upper limit of 10 seconds, the satellite remote sensing inspection scenario takes a lower limit of 60 seconds and an upper limit of 300 seconds, and the ground surveillance device scenario takes a lower limit of 1 second and an upper limit of 3 seconds; let the shooting time interval between two adjacent frames be d, the lower limit of the interval be Lmin, and the upper limit of the interval be Lmax, the judgment rule distinguishes three situations: when Lmin<=d<=Lmax, it is determined that the two frames are within the same continuous observation rhythm, and they are connected as adjacent nodes of the same homologous acquisition chain; when d<Lmin, it is determined as repeated sampling or jitter redundant frames at the same moment, only the frame with higher definition is retained and merged into the same node, so as to avoid virtual length increase of the homologous chain; when d>Lmax, it is regarded as an observation interruption, and is truncated into two independent homologous acquisition chains. Further, for each derived primitive in the multimodal evidence source chain, the parent image frame number contained in its traceability mark is read, the image frame node corresponding to this number is searched in the node set of the homologous acquisition chain, and the derived primitive is connected as a subordinate branch of this node, so as to obtain a homologous acquisition chain set with derived branches.

[0041] Preferably, the construction of the derived bloodline chain is carried out separately for the two parallel clues of images and alarms. For the image clue, traverse each parent image frame node in the homologous acquisition chain set, and recursively go down along the traceability mark: use the parent image frame number as the key to search for the first-level derived primitive, then use the own number of this first-level derived primitive as the key to search for the second-level derived primitive that takes it as the parent, expand layer by layer according to the same rule, when a certain level of derived primitive is no longer registered as a parent by any lower-level derived primitive, this level is regarded as the end of the bloodline chain and the traversal is terminated, and the derived primitives at all levels are connected into an image derived bloodline chain according to the parent-child direction.

[0042] For example, in an urban street monitoring scenario, a frame of surveillance image outputs vehicle vector primitives through a target recognition process. These vehicle vector primitives then output trajectory segment primitives through a trajectory extraction process. The image frame, vehicle vector primitives, and trajectory segment primitives are sequentially linked to form an image derivation lineage chain, with the original image frame at the head of the chain. For alarm clues, the alarm at the head of the alarm association sequence in the multimodal evidence source chain is used as the lineage starting point. The unique number of the alarm at the head of the chain is read and registered as the alarm chain head number. The chain extends level by level along the association field to the last derived alarm, and each link in the chain is connected sequentially to form an alarm derivation lineage chain. The image derivation lineage chain and the alarm derivation lineage chain merge to form a derivation lineage chain set. The identification of cross-modal correspondences uses the same source acquisition chain set and the derivation lineage chain set as the comparison points, employing a hierarchical matching method of a main anchor at the head of the chain plus auxiliary anchors on the chain, rather than treating the entire chain as falling into the same time code. Specifically, the process first retrieves the grid code and time code of the first image frame in the same-source acquisition chain, and then retrieves the grid code and time code of the first alarm in the alarm derivative chain. If the grid codes are identical and the time code difference does not exceed two time slices, the first main anchor is considered to have been hit. If the first main anchor is not hit, the remaining image frame nodes in the same-source acquisition chain and the remaining derived alarm nodes in the alarm derivative chain are traversed. As long as any pair of nodes satisfies the same grid code and the time code difference does not exceed two time slices, the auxiliary anchor on the chain is considered to have been hit. After a hit, the acquisition device identifier of the same-source acquisition chain and the alarm chain head number of the alarm derivative chain are read, and the cross-modal attachment registration table is checked to see if there is a direct attachment record with the acquisition device identifier as one end and the alarm chain head number as the other end. The cross-modal attachment registration table is not generated independently, but is written synchronously with each parent-child association and alignment attachment during the multimodal evidence source chain construction stage. That is, when a cross-modal attachment is completed on the image side and the alarm side based on the acquisition device identifier, traceability mark and alarm association sequence, a record is added to the table. The record fields include the acquisition device identifier on the image side, the alarm chain head number on the alarm side, the writing time and the shared combination address that triggered this attachment, i.e., grid code plus time code, for this step to query and call according to the identifier.

[0043] Specifically, if a directly linked record is found, the same-source acquisition chain and the alarm-derived lineage chain are determined to have a cross-modal correspondence, and the two chains are merged into a single candidate observation chain; if the query does not find a match, the two chains remain independent. The determination of cross-modal correspondence is performed one by one on all link pairs under the same combined address until the comparison is complete.

[0044] It should be noted that multiple registrations are allowed for derived primitives attached to the same parent image frame node: on the one hand, the acquisition end may report the same derived primitive multiple times within the sliding window period due to retransmission and re-acquisition, and each report will add a record to the cross-modal attachment registration table; on the other hand, the same derived primitive may be reached multiple times along different alarm lineage links, and each reach will write an attachment record, with the write time field recorded as a millisecond-level timestamp. Furthermore, for the merged candidate observation links, deduplication processing is performed on the derived branch nodes under the same parent image frame number: using the derived primitive's own number as the deduplication key, if there are multiple derived primitive branches with the same own number under the same parent image frame node, the registration order is determined according to the write time field in the cross-modal attachment registration table, and only the branch with the earliest write time is retained, while the remaining duplicate branches are removed. During removal, the write time and source link identifier of the removed branch are simultaneously filled back into the remarks field of the retained branch for subsequent traceability.

[0045] Understandably, after the aforementioned grouping, recursive traversal, cross-modal merging, and deduplication of derived branches, visual modal entries, primitive modal entries, and record modal entries falling into the same grid code and the same time code are aggregated into a unified connection structure according to the actual acquisition source and initial trigger source, thus obtaining the multimodal co-source observation link. This multimodal co-source observation link uses the acquisition device identifier and the alarm chain head number as the attribution identifiers at both ends of the link. The link is internally expanded according to the derivational lineage hierarchy, reflecting the real co-source attribution relationship between cross-modal evidence.

[0046] Step S104: Extract modal types and content along the multimodal co-source observation link, compare the content areas of derived primitives and image frames with the triggering sources of alarm records, identify cross-modal co-source duplication, retain the original collected evidence and remove derived duplicate evidence to form an independent multimodal fact set.

[0047] The visual modality entries, primitive modality entries, and record modality entries on the multimodal co-origin observation link are traversed sequentially. The modality type marker and content field attached to each entry are read. The image pixel information of the visual modality entries, the geometric contour and attribute fields of the primitive modality entries, and the trigger time and event description of the record modality entries are used as comparison materials to obtain a set of comparison materials for modal aggregation. For each primitive modal entry in the comparison material set, the parent image frame number indicated by its source marker is read, and the corresponding visual modal entry is retrieved. The geometric contour of the primitive modal entry is converted into the image pixel coordinate system of the visual modal entry according to the imaging parameters registered in the parent image frame metadata to obtain the content region. If the content region falls completely within the imaging boundary of the image pixel information, the primitive modal entry is determined to be a derivative of the visual modal entry. For each recording modal entry, its trigger source identifier is read and compared bit by bit with the acquisition device identifier carried by the visual modal entry. If the two strings are consistent, the recording modal entry is determined to be a derived trigger product of the corresponding visual modal entry, forming a cross-modal same-source duplicate annotation result. Based on the cross-modal homogeneous repeated labeling results, the primitive modal entries labeled as derived products and the record modal entries labeled as derived trigger products on the link are removed from the evidence entry list. Only the visual modal entries as the parent and the primitive modal entries and record modal entries that are not hit by any parent label are retained. The remaining entries are classified into a unified list according to modal type to form an independent multimodal fact set.

[0048] In one possible implementation, the independent fact identification step of multimodal spatiotemporal data inbound monitoring takes the aforementioned multimodal co-source observation link as input. The multimodal co-source observation link has aggregated visual modal entries, primitive modal entries and recording modal entries under a unified connection structure according to the acquisition device identifier and alarm chain head number. The link is expanded according to the derivation lineage hierarchy.

[0049] It should be noted that multiple entries falling into the same grid code and time code, although with different modal labels, may actually originate from the same source. The vector target output by the target recognition process of the image pixel information carried by the visual modal entry becomes the primitive modal entry, and the alarm triggered by the alarm reporting mechanism attached to the same acquisition device during the imaging process becomes the record modal entry. If these two types of derivatives participate in the evidence counting together with the parent frame, it will cause the same fact to be repeatedly accumulated. Therefore, they must be identified according to the correspondence between the parent and the derivative. When reading each entry to be verified, its grid code and time code are first recorded as address tags for archiving and retrieval. The source tracing and trigger source comparison are performed directly on all primitive entries and record entries, regardless of whether the combined address of the entries is the same, so as to cover the cross-slice derivation case where the parent frame falls into adjacent time slices. Through this connection, the grid code and time code are transformed from preliminary descriptions into the first filtering key, forming a series link with the subsequent source tracing and identification comparison.

[0050] Specifically, each entry along the multimodal co-origin observation link is traversed sequentially, its modality type marker is read to determine its grouping, and the content fields are read. The content of visual modality entries includes image pixel information and imaging parameter fields; the content of primitive modality entries includes geometric contours, attribute fields, and tracing markers; and the content of record modality entries includes trigger time, event description, and trigger source identifier. The three types of content are classified into visual subset, primitive subset, and record subset according to modality type, forming a set of comparison materials for modality-based aggregation.

[0051] For example, a common-source observation link in an urban street monitoring scenario may include a traffic monitoring image frame, vehicle vector primitives and pedestrian vector primitives identified from the image frame, and a line crossing alarm record issued by the alarm module on the monitoring pole at the same intersection. The three fall into the above three subsets according to their modalities.

[0052] Preferably, content region comparison is performed on the primitive modal entries in the comparison material set. The parent image frame number pointed to by the tracing marker of the primitive modal entry is read, and the image pixel information and its imaging parameter fields of the visual modal entry corresponding to that number are retrieved. The imaging parameter fields include image resolution, shooting field of view, ground sampling distance, and geographic coordinates corresponding to the four corners of the image. A one-to-one correspondence is established between the geographic coordinates of the four corners of the image and the pixel coordinates of the image pixel matrix defined by the image resolution. A conversion relationship from geographic coordinates to image pixel coordinates is established according to affine projection, i.e., six conversion parameters a, b, c, d, e, and f are calculated based on the four-corner correspondence. Here, a, b, d, and e constitute linear transformation components with 2 scales, 1 rotation, and 1 shearing; c and f are 2 translations, satisfying u = a × x + b × y + c and v = d × x + e × y + f, where x and y are geographic coordinates, and u and v are pixel coordinates. Since the correspondence of the four sets of corner points can be used to list eight linear equations, while there are only six unknown parameters, it belongs to an overdetermined system of equations. The least squares method is used to solve it. A coefficient matrix A and an observation vector L are constructed. The parameter vector X is solved using the normal equation system X = inv(transpose of A * A) * transpose of A * L. The root mean square of the sum of squared residuals is calculated. If the root mean square residual is no greater than 0.5 pixels, the solution is accepted; otherwise, the corner point with the largest deviation is removed, and the solution is recalculated using the remaining corner points. The geographic coordinates of each vertex on the geometric contour of the primitive modality entry are substituted into the transformation relationship to obtain the pixel position in the image pixel coordinate system. Connecting the transformed pixel positions yields the content area of ​​the primitive modality entry in the image pixel information. It is determined whether the minimum bounding rectangle of the content area completely falls within the imaging boundary of the image pixel information. If it completely falls within, the primitive modality entry is determined to be a derivative of the visual modality entry; if it partially falls within or does not fall within, the primitive modality entry is maintained as a non-derived entry.

[0053] In one embodiment, in a satellite remote sensing inspection scenario, the primitive modal entry corresponding to a satellite image is a ground feature outline vector. The conversion is performed using the four corner coordinates registered in the satellite image metadata and the image resolution. The vast majority of ground feature outline vectors fall within the imaging boundary and are thus identified as derivatives of that satellite image. In a low-altitude UAV inspection scenario, the four corner coordinates recorded by the UAV's onboard payload also support the above conversion.

[0054] Specifically, trigger source comparison is performed on the record modality entries in the comparison material set. The trigger source identifier of the record modality entry and the acquisition device identifier of the visual modality entry are read, and the two are compared bit by bit according to their identifier fields. If the two strings are completely identical, the record modality entry is determined to be a derived trigger product of the corresponding visual modality entry; if the two strings are inconsistent, the record modality entry is maintained as a non-derived entry. The trigger source comparison result is combined with the content region comparison result to form a cross-modal homogeneous duplicate annotation result.

[0055] For example, the identifier of the acquisition device on a monitoring pole at an intersection and the trigger source identifier of the alarm module on that pole are registered with the same number in a pre-established equipment ledger. During comparison, the two identifier strings match, and the cross-line alarm record is determined to be a derived trigger product of the monitoring image at that intersection. The equipment ledger has already recorded the identifier correspondence between each acquisition device and its attached alarm module during the acquisition device access phase. Further, based on the cross-modal same-source duplicate labeling results, the list of evidence entries on the link is processed for elimination. All graphic element modal entries labeled as derived products and record modal entries labeled as derived trigger products are removed from the evidence entry list, retaining only the parent visual modal entries, and graphic element modal entries and record modal entries that are not matched by any parent label. The remaining entries are categorized into a unified list according to their modal type; visual modal entries are still grouped by visual, graphic element modal entries by graphic element, and record modal entries by record.

[0056] Understandably, after content extraction, content region comparison, trigger source comparison, and derivation elimination, cross-modal evidence falling into the same grid code and the same time code is retained as having a truly independent information source. Duplicate counts under the same collection event and the same alarm trigger are eliminated, resulting in an independent multimodal fact set. Each entry in this independent multimodal fact set corresponds to an independent information source, providing a deduplicated evidentiary basis for identifying core fact nodes supported by cross-modal data.

[0057] Step S105: Analyze the independent multimodal fact set, identify the support paths of different modal fact nodes within the grid cell, evaluate the source independence, modal complementarity and propagation location of each path, and determine the core fact nodes for cross-modal support.

[0058] The grid code attached to each entry is read from the independent multimodal fact set. Entries with the same grid code are grouped to form a grid cell fact set. For each visual modal entry, primitive modal entry, and recording modal entry within the grid cell fact set, each entry is considered a fact node. The node is traced back to the acquisition source along its connection in the multimodal co-source observation link, and the support path of that fact node within the grid cell is obtained. For each support path, the acquisition device identifier is read from the visual side node, and the alarm chain head number is read from the recording side node. It is determined whether the acquisition device identifier and alarm chain head number of different support paths within the same grid cell are mutually exclusive. If they are mutually exclusive, the paths are determined to have source independence. Modal complementarity is determined based on the coverage quantity of visual, primitive, and recording modal markers on the path. The propagation position is obtained by reading the spatial coordinates from the end node of the path, resulting in a support path description set with source independence, modal complementarity, and propagation position attributes. Based on the set of support path descriptions, for each support path, it is determined whether it simultaneously meets the following conditions: the acquisition device identifier and alarm chain head number are different from those of other paths in the same grid cell, it covers at least two types of modal markers, and the latitude and longitude of the propagation location fall within the latitude and longitude boundary of the grid cell. If all conditions are met, the fact node corresponding to the support path is retained and determined as the core fact node of cross-modal support.

[0059] In one possible implementation, the core fact node identification step of multimodal spatiotemporal data inbound monitoring uses the aforementioned independent multimodal fact set as input. Each entry in the independent multimodal fact set has undergone derivation and elimination, and is accompanied by a grid code, time code, modality type marker, and content field, providing a deduplicated evidentiary basis for path evaluation by grid unit.

[0060] Specifically, the grid code attached to each entry is read sequentially from the independent multimodal fact set, and entries with the same grid code are grouped into the same grid cell fact set. Visual modal entries, primitive modal entries, and recording modal entries within the same grid cell fact set are registered as fact nodes. For each fact node, the connection relationship registered in the multimodal co-source observation link is traced upwards: visual modal entries are traced directly to their corresponding acquisition device identifier; primitive modal entries are traced first to the parent image frame via the tracing marker, and then to the acquisition device identifier; recording modal entries are traced first to the alarm chain head via the association field, and then to the trigger source identifier corresponding to the alarm at that chain head. The nodes traversed are connected in sequence to form the support path of the fact node within the grid cell.

[0061] For example, in a grid cell within an urban street monitoring scenario, there may be three supporting paths: one directly composed of low-altitude drone patrol image frames, with the acquisition time recorded as t1; one composed of video frames from fixed cameras on the exterior walls of street-side shops, which are acquired from different sources than the drone image frames and do not fall within the drone's parent imaging boundary, with the acquisition time recorded as t2, and the time difference between t2 and t1 not exceeding 30 seconds; and one traced from a line-crossing alarm issued by an intersection monitoring pole through the alarm chain head, tracing back to the original alarm entry output by the event detection module embedded in the pole, with the acquisition time recorded as t3. All three supporting paths have passed the derivation elimination verification of S104, meaning that the starting points of each path correspond to mutually unrelated acquisition entities, and the starting point entry of any path does not fall within the imaging boundary or detection field of view of the starting points of other paths, thus falling into the same grid cell fact set as independent evidence entries for subsequent attribute evaluation. As for the vehicle vector primitives identified by the UAV image frames, since they fall within the imaging boundary of the t1 image frame, they have been removed from the evidence entry list according to S104 and are only retained as internal derived annotations of the image frame path, and are no longer counted separately in the number of support paths.

[0062] Preferably, source independence is determined for the support paths. The acquisition device identifier is read from the visual-side node of each support path, and the alarm chain head number is read from the recording-side node. All support paths within the same grid cell are compared pairwise: if the acquisition device identifiers and alarm chain head numbers of two paths are different, they are determined to be source independent; if any field is the same, they are determined not to be source independent. The source independence determination result of each support path with other paths within the same grid cell is recorded as the source independence marker for that path. Further, modal complementarity is evaluated for the support paths. Taking all support paths within the fact set of this grid cell as the overall statistical object, the modal type markers attached to the fact nodes of each support path are read, the number of modal categories appearing in the fact set of this grid cell is counted, and written back to the modal complementarity field of each support path: 1 is recorded if only a single modal marker is covered in the fact set of this grid cell; 2 is recorded if two modal markers are covered; and 3 is recorded if visual, primitive, and recording modal markers are covered. All support paths within the same grid cell share the same modal complementarity value, reflecting the cross-modal coverage breadth of this grid cell.

[0063] In one embodiment, a certain grid cell fact set simultaneously contains a support path carrying traffic monitoring images, a support path carrying vehicle vector primitives, and a support path carrying a line crossing alarm. All three types of modal tags are complete, and the modal complementarity field of each support path in the grid cell is registered as 3.

[0064] Specifically, the landing point location is extracted for the support path. The landing point location refers to the coordinates of the access point of the fact node of the support path within this grid cell. It describes where the path enters the cell and participates in subsequent fusion, and its naming is based on the landing position of the fact node on the grid plane. The spatial coordinates attached to the fact node are read from its end and registered as the landing point location in the landing point location field of the path. Thus, each support path within the same grid cell carries three attributes: source independence marker, modal complementarity value, and landing point location coordinates, which are summarized into a support path description set. Furthermore, for each support path in the support path description set, the following three conditions are sequentially determined to be met simultaneously: First, the source independence marker of the path shows that its acquisition device identifier and alarm chain head number are different from other paths in the same grid cell; if the path is directly composed of visual modal entries without recording side nodes, its alarm chain head number field is registered with an empty placeholder, and only the acquisition device identifier is used for the comparison of different paths; if the path is composed of recording side nodes without visual acquisition entries, its acquisition device identifier field is registered with an empty placeholder, and only the alarm chain head number is used for the comparison of different paths; during the comparison process, the empty placeholder is determined to be different from any non-empty value and from another empty placeholder, that is, the empty placeholder does not constitute a duplicate; Second, the modal complementarity value of the path is not less than 2; Third, the longitude and latitude values ​​of the path's landing point coordinates fall within the rectangular range defined by the longitude and latitude boundaries of this grid cell, respectively.

[0065] In one embodiment, if all three conditions are met, the fact node corresponding to the support path is retained and registered as a core fact node for cross-modal support; if any one of the three conditions is not met, the fact node corresponding to the support path is not registered as a core fact node. After traversing the entire grid cells, a set of core fact nodes covering the current monitoring range is obtained.

[0066] Understandably, after constructing the support path, evaluating three attributes, and determining three conditions, the fact nodes within the grid cell that are truly supported by different acquisition sources, span at least two modalities, and whose propagation positions fall within the geometric range of the grid cell are identified. These are the core fact nodes with cross-modal support, providing a factual basis for the subsequent layer entry qualification assessment that has cross-modal support capabilities.

[0067] Step S106: Update the evidence entry record according to the core fact node, identify the original collected evidence and related modal evidence in the same source observation, evaluate the multimodal refresh qualification of the Cube layer unit, and generate a multimodal layer entry task.

[0068] Based on the core fact nodes, the grid code, time code, and modality type marker attached to each node are read. The content carried by that node is registered as an entry to be entered into the evidence entry record table. The table fields include the grid code, time code, modality type marker, content carried, and core fact node number, resulting in an evidence entry record marked with the core fact node. The core fact node is located upwards along its attachment path in the multimodal co-source observation link, and the visual modality entry reached along this path is taken as the original collected evidence. The primitive modality entries attached to this original collected evidence, as well as the record modality entries that share the acquisition device identifier with this original collected evidence, are registered together as associated modality evidence. If the core fact node itself is a visual modality entry, then it is used as the original collected evidence. The original collected evidence and associated modality evidence are merged and written into the associated entry field of the evidence entry record, resulting in an entry entry group that distinguishes between original and associated identities. For each entry group, the corresponding Cube layer unit is assigned according to the grid code. The modal type tags of all entries in the entry group are merged and deduplicated to form a new set of modal type tags. It is determined whether the set of modal type tags already registered by the Cube layer unit under the same time code is different from the new set of modal type tags. If there is a difference, the Cube layer unit is determined to be qualified for multimodal refresh. For Cube layer units that are qualified for refresh, they are assembled into multimodal layer entry tasks according to their grid code, time code, and new set of modal type tags.

[0069] In one possible implementation, the layer loading and assembly process for multimodal spatiotemporal data loading and monitoring uses the aforementioned core fact nodes as input. These core fact nodes have undergone path evaluation at the grid cell level and are accompanied by grid codes, time codes, and modality type markers; their content corresponds to a truly independent information source.

[0070] It should be noted that a Cube layer unit refers to a layer container divided by grid in a multimodal spatiotemporal data warehouse. Each Cube layer unit uses a fixed grid code as the spatial key and a fixed time code as the time key to register the multimodal entries that have been stored in the warehouse within that grid and time range, as well as their set of modal type tags, for use in quality reports and conflict comparisons.

[0071] Specifically, the core fact nodes are traversed, and for each node, its grid code, time code, and modality type marker are read. The content carried by that node is then registered as a pending entry into the evidence entry record table. The evidence entry record table contains fields such as grid code, time code, modality type marker, content carried, and core fact node number. After registration, an evidence entry record with core fact node annotations is formed.

[0072] Preferably, the location of original collected evidence is performed according to the modal type label attached to the core fact node itself, forming a unified location rule: if the modal type label of the core fact node itself is visual, then it is directly registered as original collected evidence; if the modal type label of the core fact node itself is primitive, then the parent image frame number pointed to by its tracing mark is read, and the visual modal entry registered with that number in the multimodal co-source observation link is registered as original collected evidence; if the modal type label of the core fact node itself is record, then its trigger source identifier is read, and entries with the same trigger source identifier and modal type label of visual are selected in the multimodal co-source observation link. The absolute difference between the shooting time of the visual entry and the trigger time of the record entry is calculated, and the visual entry with the smallest absolute difference is registered as original collected evidence. The three cases are executed mutually exclusively, and each core fact node corresponds to one and only one piece of original collected evidence. Furthermore, the registration rules for associated modal evidence are as follows: All graphic element modal entries attached to the original acquired evidence are retrieved and registered as graphic element-type associated evidence; all record modal entries sharing the acquisition device identifier with the original acquired evidence are retrieved and registered as record-type associated evidence. The graphic element-type associated evidence and the record-type associated evidence are merged and written into the associated entry field of the evidence entry record, forming an entry group together with the original acquired evidence. Each entry in the entry group is labeled with an original or associated identity tag.

[0073] For example, in a city street monitoring scenario, a core fact node is a frame of traffic monitoring image, which is registered as the original collected evidence; the vehicle vector primitives identified from the frame of image under the same grid unit and time code are used as primitive-type associated evidence; the cross-line alarm issued by the monitoring pole at the same intersection is used as record-type associated evidence; the three constitute a group of entries to be stored, sharing the same number of the original collected evidence.

[0074] Specifically, the entry group is assigned to the corresponding Cube layer unit according to the grid code. The modal type tags of all entries in the entry group are merged and deduplicated to form the new modal type tag set for this entry. The modal type tag sets already registered under the same time code of the Cube layer unit are queried, and the two sets are compared item by item according to modal type: if at least one item in the new modal type tag set is not included in the registered modal type tag set, then a difference exists between the two sets, and the Cube layer unit is qualified for multimodal refresh; if the new modal type tag set is completely covered by the registered modal type tag set, then the Cube layer unit remains unchanged and is not refreshed.

[0075] In one embodiment, a Cube layer cell previously only had visual modal markers registered under the same timecode. When a new primitive modal marker and a record modal marker are added to the current entry group, and the two sets show differences, the Cube layer cell is determined to be eligible for multimodal refresh. Further, for Cube layer cells eligible for multimodal refresh, a multimodal layer entry task is assembled based on its grid code, timecode, set of newly added modal type markers, the number of the original collected evidence in the entry group, and the list of related modal evidence entries. Multiple multimodal layer entry tasks falling into the same Cube layer cell are numbered sequentially according to their timecodes for subsequent sequential retrieval.

[0076] It is understandable that after the entry record registration, identification of original and related evidence, comparison of newly added modal tags, and task assembly, the multimodal layer entry task carries the core fact nodes and their original collected evidence and related modal evidence that have been screened and retained through the aforementioned comparison process. The screening is based on the constraints that the new modal tags and the original evidence have a timestamp deviation of no more than 2 seconds, a spatial coordinate deviation of no more than 5 meters, and a semantic tag consistency of no less than 0.85. Isolated evidence nodes with a comparison score of less than 0.6 are eliminated, and only fact nodes with at least two types of modal evidence that corroborate each other are retained. This provides an entry payload with clear identity distinction for subsequent conflict item database retrieval and factual acceptance conclusion output.

[0077] Step S107: Use the pre-established conflict entry library to retrieve the facts contained in the multimodal layer entry task, analyze the cross-modal conflicts and duplicate entry traces within the same grid code and time code, evaluate the cross-modal consistency and acceptance basis of the core fact nodes, output the fact acceptance conclusion, and complete the multimodal spatiotemporal data standardization addressing entry monitoring.

[0078] A pre-established conflict entry library is used to retrieve each fact entry in the multimodal layer entry task. The conflict entry library contains conflict pattern items indexed by grid code and time code, each recording a conflict type label and the field features required to trigger it. For each entry task containing original collected evidence and associated modal evidence, candidate conflict pattern items are located in the library based on their grid code and time code. The modal type label, content field, and field features of the evidence entry are compared one by one to obtain a retrieval hit set with candidate conflict pattern item labels. For the retrieval hit set, the event description key fields indicated by the candidate conflict pattern item labels are compared to see if the event description fields of different modal entries under the same grid code and time code provide mutually exclusive category names or status values ​​to determine cross-modal conflicts. The original collected evidence numbers of different entry tasks are compared to determine duplicate entry traces. The determined cross-modal conflict tags and duplicate entry trace tags are then backfilled into the core fact nodes of the corresponding entry tasks, resulting in an entry task set with conflict and duplicate annotations. Based on the annotations attached to each core fact node in the inbound task set, if the cross-modal conflict marker is empty and the duplicate inbound trace marker is empty, then the core fact node is determined to have cross-modal consistency; if the modal coverage of its associated modal evidence is not less than 2 and the original collected evidence number is unique under this grid code and time code, then the basis for acceptance is established, the core fact node is output as an admissible fact and written into the evidence inbound record table, thus completing the multimodal spatiotemporal data standardized addressing inbound monitoring.

[0079] In one possible implementation, the fact-acceptance stage of multimodal spatiotemporal data inbound monitoring uses the aforementioned multimodal layer inbound task as input. The multimodal layer inbound task carries core fact nodes and their original collected evidence and associated modal evidence. Core fact nodes refer to candidate nodes that, under the same grid code and time code location, have at least two types of modal evidence that can mutually corroborate each other, and whose mutual corroboration score is not lower than 0.6. Candidate nodes that do not meet this corroboration condition are eliminated during the layer inbound stage and do not proceed to the acceptance stage. Each task includes a grid code, a time code, a set of newly added modal type markers, and the original collected evidence number, for the acceptance stage to locate data according to the grid code and time code.

[0080] It should be noted that the conflict entry library is a pre-established collection of pattern items. Each conflict pattern item in the library consists of four types of fields: grid code value range, time code value range, conflict type label, and field features required to trigger the conflict. The conflict type label covers three categories: category mutual exclusion, state mutual exclusion, and source duplication. The field features, constrained by the modal type tag, list the key field names and value sets of the event descriptions required to trigger the conflict. The library also includes a mutual exclusion value pair registration table, which is entered by the business operations team according to business rules during the initialization phase of the conflict entry library. This table records mutually exclusive value pairs under each conflict type label, such as motor vehicles and non-motor vehicles under the vehicle category, and passage and closure under road conditions. The conflict entry library is organized with grid codes as the primary index and time codes as the secondary index, supporting rapid location of candidate pattern items by both spatial and temporal keys.

[0081] Specifically, the conflict entry library is used to retrieve each fact entry in the multimodal layer warehousing task. For each warehousing task containing original collected evidence and associated modal evidence, its grid code and time code are read. Candidate conflict mode items whose value range covers the grid code and time code are located in the conflict entry library. Furthermore, the modal type marker of the evidence entry is compared with the modal constraints of the candidate conflict mode items. The content fields of the evidence entry are aligned with the key fields listed in the candidate conflict mode items, resulting in a retrieval hit set with candidate conflict mode item tags. Each record in the retrieval hit set is labeled with the hit conflict mode item tag and the hit entry number.

[0082] For example, in a city street monitoring scenario, within a specific grid code and time code, a conflict pattern item is registered in the conflict item database. Its conflict type label is "category mutual exclusion," and the field feature specifies that the vehicle category name of the visual modality item and the vehicle category name of the record modality item belong to the same namespace. During retrieval, when the inbound task under this grid code and time code matches this pattern item, it proceeds to the subsequent conflict determination stage.

[0083] Preferably, the determination of cross-modal conflicts relies on the retrieved hit set. For each hit conflict mode item, based on the event description key field indicated by its field features, the event description field values ​​of different modal entries under the same grid code and the same time code are read. If the type label is mutually exclusive, the category names of different modal entries are compared to see if they belong to different values ​​under the same namespace. Here, the namespace is a set of category names pre-registered according to business dimensions, divided according to event theme. All selectable values ​​under the same theme are grouped into the same namespace, and values ​​under different themes are grouped into different namespaces. During registration, a mapping table is formed using the theme identifier as the key and the list of values ​​under that theme as the value. For example, a namespace with weather phenomena as the theme includes four values: sunny, cloudy, rainy, and snowy. A namespace with road traffic status as the theme includes four values: smooth, slow, etc. The system considers four values: line, congestion, interruption, and interruption. The criteria for determining which modality entry belongs to which is the namespace based on the theme identifiers of the entries. If the theme identifiers match, they are considered to belong to the same namespace. Then, the specific values ​​of the two entries are checked to see if they are different items in the value list of that namespace. For example, if one entry is "sunny" and another is "rainy" and both belong to the weather phenomenon namespace, they are considered to have a mutually exclusive category match. If the type label is "state mutually exclusive," the system retrieves the set of value pairs under that type label from the mutually exclusive value pair registration table and determines whether the state values ​​of different modal entries match both ends of a certain value pair in the registration table. For any pair of entries with mutually exclusive category names or state values, the cross-modal conflict marker is set to "conflicted" and backfilled to the core fact node of the corresponding entry task. For pair of entries without mutual exclusion, the cross-modal conflict marker remains empty. Furthermore, the determination of duplicate entry traces is carried out for multiple entry tasks under the same grid code and time code. The original evidence numbers of two entry tasks are read and compared pairwise. If the original evidence numbers of two entry tasks are the same, the duplicate entry trace is marked as duplicated and backfilled into the core fact node of the corresponding task; if the numbers are different, the duplicate entry trace mark remains empty. The cross-modal conflict mark and the duplicate entry trace mark together constitute the conflict and duplicate annotation of the core fact node, which are summarized into an entry task set with conflict and duplicate annotations.

[0084] In one embodiment, within a certain grid code and time code in an urban street monitoring scenario, two entry tasks are triggered by monitoring poles at different intersections, and the original collected evidence numbers are different from each other, with the duplicate entry trace marker remaining empty; at the same time, the vehicle category names of the two tasks are the same, and the cross-modal conflict marker also remains empty, and this core fact node enters the acceptance and judgment stage.

[0085] Specifically, the acceptance determination is divided into two levels: First, read the cross-modal conflict marker and duplicate entry trace marker attached to each core fact node. If both are empty, the core fact node is determined to have cross-modal consistency. The empty duplicate entry trace marker means that the original collected evidence number has passed the uniqueness verification under this grid code and time code, and will not be reviewed in subsequent stages. Second, for core fact nodes with cross-modal consistency, read the deduplication set of modal type markers after merging the original collected evidence and related modal evidence. Let the modal coverage number be M, where M ranges from 1 to 3. Let N be the number of valid evidence items participating in the merging, where N is not less than 2. If M is not less than 2 and N is not less than 2, the acceptance basis is determined to be valid. If M equals 1 or N is less than 2, the acceptance basis is determined to be invalid. Furthermore, for core fact nodes where the basis for acceptance is established, the node is output as an admissible fact, and the acceptance conclusion field is written into the evidence entry record table; for core fact nodes where the basis for acceptance is not established or the cross-modal consistency judgment fails, the node is output as a fact to be reviewed and is simultaneously written into the review mark field of the evidence entry record table, thus completing the standardized addressing and entry monitoring of multimodal spatiotemporal data.

[0086] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the concept of this application. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A method for monitoring the standardized addressing and data storage of multimodal spatiotemporal data, characterized in that, The method includes: Acquire image frames, derived primitives, and alarm records containing spatial coordinates and timestamps, label them as visual, primitive, and recording modalities, complete cross-modal standardized addressing according to the same grid code and time code, and form a multimodal common spatiotemporal evidence group; Extract the acquisition device identifier, derived primitive tracing marker, and alarm record trigger source identifier from the multimodal common spatiotemporal evidence group to form a multimodal evidence source chain; Analyze the multimodal evidence source chain, identify the homologous acquisition chain, the derived lineage chain and the cross-modal correspondence, merge evidence from the same acquisition device or the same alarm chain, and obtain the multimodal homologous observation link; By comparing the content regions of derived primitives and image frames and the triggering sources of alarm records along the multimodal homogeneous observation link, cross-modal homogeneous duplication is identified, evidence of derived duplication is eliminated, and an independent multimodal fact set is formed. Analyze the independent multimodal fact set, identify the support paths of each modal fact node within the grid cell, evaluate the path's source independence, modal complementarity, and propagation location, and determine the core fact nodes supporting cross-modal functions; Update the evidence entry record based on the core fact nodes, evaluate the multimodal refresh eligibility of the Cube layer unit, and generate a multimodal layer entry task; The conflict entry database is used to retrieve the multimodal layer entry tasks, analyze cross-modal conflicts and duplicate entry traces within the same grid code and time code, evaluate the consistency of core fact nodes, and output fact acceptance conclusions.

2. The method for monitoring the standardized addressing and data storage of multimodal spatiotemporal data according to claim 1, characterized in that, The acquisition of image frames, derived primitives, and alarm records containing spatial coordinates and timestamps, labeled as visual, primitive, and recording modalities, includes: Visual modalities are labeled by extracting image pixel information from image frames, geometric contours and attribute fields are read from derived primitives and labeled by primitive modalities, and alarm records are parsed by parsing trigger times and event descriptions and labeled by record modalities, resulting in a set of original evidence entries with modal labels; The spatial coordinates of the original evidence entry set are mapped to a unified hierarchical grid code according to equal latitude and longitude. The timestamps are assigned to fixed time codes according to preset time slices. Modal entries that fall into the same grid code and time code are aggregated into the same container.

3. The method for monitoring the standardized addressing and data storage of multimodal spatiotemporal data according to claim 1, characterized in that, The extraction of the acquisition device identifier, derived primitive tracing marker, and alarm record trigger source identifier from the multimodal spatiotemporal evidence set includes: Read the acquisition device identifier from the visual modality partition, read the source tracing mark from the primitive modality partition to obtain the parent image frame number, and read the trigger source identifier and associated fields from the recording modality partition; Alarm records under the same trigger source identifier are sorted according to the trigger time, and the associated fields pointing to the previous alarm are connected in series to form an alarm association sequence. Image frames and derived primitives with the same acquisition device identifier are connected according to the source tracing mark, and the alarm association sequence is attached according to the shared grid code and time code to form a multimodal evidence source chain.

4. The method for monitoring the standardized addressing and data storage of multimodal spatiotemporal data according to claim 1, characterized in that, The analysis of the multimodal evidence source chain, identifying homologous acquisition chains, derived lineage chains, and cross-modal correspondences, includes: Extract the acquisition device identifier, traceability mark and alarm association sequence from the multimodal evidence source chain. Image frames with adjacent shooting times under the same acquisition device identifier are strung together to form a homogeneous acquisition chain. Derived primitives are attached to the parent image frame node of the homogeneous acquisition chain according to the parent image frame number pointed to by the traceability mark, thus obtaining a set of homogeneous acquisition chains with attached derivation branches. For each parent image frame node, recursively traverse the derived primitives with that parent along the source marker, and connect them in the parent-child direction to form an image derivation bloodline chain. The alarm association sequence extends from the head of the chain along the associated field to the last derived alarm to form an alarm derivation lineage chain. The two are combined to obtain a set of derivation lineage chains.

5. The method for monitoring the standardized addressing and data storage of multimodal spatiotemporal data according to claim 4, characterized in that, The two combined form a set of derived bloodline chains, including: For homogeneous acquisition chains and alarm derivation chains that fall into the same grid code and time code, determine whether the acquisition device identifier and alarm chain head number of the two are registered as directly linked in the multimodal evidence source chain; Registered direct connection relationships are merged into cross-modal correspondence relationships and evidence entries are combined. For duplicate derived branch nodes under the same parent image frame number, duplicate nodes are removed and only one is retained to obtain a multimodal co-source observation link.

6. The method for monitoring the standardized addressing and data storage of multimodal spatiotemporal data according to claim 1, characterized in that, The step of comparing the content regions of derived primitives and image frames along the multimodal homogeneous observation link, and identifying cross-modal homogeneous repetitions, includes: The geometric contour of the primitive modal entry is converted to the pixel coordinate system of the visual modal entry according to the imaging parameters of the parent image frame to obtain the content area. If the content area falls within the imaging boundary, the primitive modal entry is determined to be a derivative. The trigger source identifier of the recorded modal entry is compared bit by bit with the acquisition device identifier of the visual modal entry. If the strings match, the recorded modal entry is determined to be a derived trigger product. The derived product and the derived trigger product are removed to form an independent multimodal fact set.

7. The method for monitoring the standardized addressing and data storage of multimodal spatiotemporal data according to claim 1, characterized in that, The analysis of the independent multimodal fact set identifies the supporting paths of each modal fact node within the grid cell, and evaluates the path's source independence, modal complementarity, and propagation location, including: The data are grouped according to the same grid code to form a grid cell fact set. Each entry is regarded as a fact node. The data is traced back to the source of the data collection along the connection relationship in the multimodal co-source observation link to obtain the support path. The acquisition device identifier is read from the visual side node of the path, and the alarm chain head number is read from the recording side node. If the acquisition device identifier and alarm chain head number are different for different paths, the source independence is determined. The modal complementarity is determined according to the number of modal markers covered by the path. The propagation position is obtained by reading the spatial coordinates from the end node of the path. When the following three conditions are met simultaneously: the data acquisition device identifier and the alarm chain head number of the support path are different from each other, the data covers at least two types of modal markers, and the propagation position falls within the latitude and longitude boundary of the grid cell, the corresponding fact node is determined as the core fact node of cross-modal support.

8. The method for monitoring the standardized addressing and data storage of multimodal spatiotemporal data according to claim 1, characterized in that, The process of updating the evidence entry record based on the core fact nodes and evaluating the multimodal refresh eligibility of the Cube layer unit includes: The core fact node grid code, time code and modal type mark are used to register the content carried into the evidence warehouse record. The visual modal item reached by locating upward along the attachment path is the original collected evidence. The original collected evidence is registered as the attached graphic element modal item and the record modal item of the shared collection device identifier as the associated modal evidence. If the registered modal type marker set and the newly added modal type marker set under the same time code are different, the modal type marker set is deemed eligible for refresh. The modal type marker set is then assembled into a multimodal layer loading task based on the grid code, time code, and newly added modal type marker set.

9. The method for monitoring the standardized addressing and data storage of multimodal spatiotemporal data according to claim 1, characterized in that, The process of retrieving the multimodal layer loading task using a conflict entry database and analyzing cross-modal conflicts and duplicate loading traces within the same mesh code and time code includes: The conflict entry library is pre-set with conflict pattern items indexed by grid code and time code; Candidate conflict mode items are located by using the grid codes and time codes of the original collection evidence and related modal evidence of the warehousing task. Cross-modal conflicts are determined by comparing the event description fields of different modal entries under the same grid code and time code to see if they give mutually exclusive categories or state values. Duplicate warehousing traces are determined by comparing the original collection evidence numbers of different warehousing tasks to see if they are the same. After backfilling the markers, a set of warehousing tasks with attached conflict and duplicate annotations is obtained.

10. A method for monitoring the standardized addressing and data storage of multimodal spatiotemporal data according to claim 9, characterized in that, The obtained inbound task set with attached conflicts and duplicate annotations includes: If the cross-modal conflict marker of the core fact node in the warehouse task set is empty and the duplicate warehouse entry trace marker is empty, then cross-modal consistency is determined. If the modal coverage of the associated modal evidence is not less than two and the original collected evidence number is unique under this grid code and time code, then the basis for acceptance is established. The core fact node is output as an admissible fact and written into the evidence warehouse entry record table.

Images