Space-time synchronization fusion playback method and device for multi-source monitoring data

Abstract

The invention relates to the field of Internet of Things and data visualization, in particular to a time-space synchronization fusion playback method and device for multi-source monitoring data, and the method comprises the steps: receiving a data packet uploaded by a mobile monitoring terminal; analyzing the data packet; associating the timestamp, the position stamp, the event stamp and the storage index of the media data under the same equipment identifier, and constructing a time-space fusion data association relationship; a playback request from the client is responded, a target data set is retrieved based on the time-space fusion data association relation according to the playback request, and the target data set comprises a continuous position track sequence in a target time period, all event marks and media data storage indexes associated with the event marks; and packaging the target data set into a fusion data packet, and issuing the fusion data packet to the client for the client to perform integrated playback. According to the method, the space-time relevance and the backtracking efficiency can be improved.

Description

Technical Field

[0001] This application relates to the fields of Internet of Things and data visualization technology, and in particular to a method and apparatus for spatiotemporal synchronous fusion and playback of multi-source monitoring data. Background Technology

[0002] In applications of mobile intelligent monitoring devices with GPS and communication capabilities (such as vehicle cameras and portable trackers), the devices will continuously or event-triggeredly generate various heterogeneous data streams, including continuous video streams, continuous audio streams, discrete or timed GPS location sequence, and discrete event markers triggered by sensors (such as vibration and voice control).

[0003] In current mainstream solutions, video, audio, and trajectory data are usually stored and managed separately. When users view video recordings, they cannot intuitively know the specific geographical location at the time of recording; when viewing driving trajectories, they cannot directly retrieve the on-site images and sounds at the corresponding locations; correlation retrieval is difficult. When an abnormal event (such as a collision) occurs, users need to manually search for data of roughly the same time period in the video recording list, GPS trajectory log, and event alarm records for manual comparison and correlation, which is cumbersome and prone to errors; the playback experience is fragmented, and users cannot continuously and contextually review the entire event process in a single view by manipulating the timeline or map trajectory points.

[0004] Therefore, there is an urgent need for a technical solution that can automatically integrate multi-source heterogeneous monitoring data, provide a unified spatiotemporal perspective, and realize intelligent retrieval and playback, so as to solve the problems of poor spatiotemporal correlation and low retrieval and tracing efficiency in existing technologies. Summary of the Invention

[0005] Therefore, it is necessary to provide a method and apparatus for spatiotemporal synchronization fusion and playback of multi-source monitoring data that can improve spatiotemporal correlation and backtracking efficiency, in order to address the above-mentioned technical problems.

[0006] In a first aspect, this application provides a method for spatiotemporal synchronization fusion and playback of multi-source monitoring data, the method comprising: Receive data packets uploaded from a mobile monitoring terminal, wherein the data packets include media data spatiotemporally tagged with timestamps and location stamps, and event-tagged data spatiotemporally tagged with event stamps, location stamps, and timestamps, wherein the media data includes video data and audio data; Parse the data packet to extract the device identifier, timestamp, location stamp, event stamp, and storage index of the media data. Associate the timestamps, location stamps, event stamps, and storage indexes of the media data under the same device identifier to construct a spatiotemporal fusion data association relationship; In response to a playback request from a client, and based on the playback request, a target dataset is retrieved based on the spatiotemporal fusion data association relationship. The target dataset includes a continuous location trajectory sequence within a target time period, all event markers, and media data storage indexes associated with each event marker. The target dataset is encapsulated into a fusion data packet and sent to the client for integrated playback.

[0007] In one embodiment, associating timestamps, location stamps, event stamps, and the storage index of the media data under the same device identifier to construct a spatiotemporal fusion data association includes: Establish a main index table with device identifier as the primary key and timestamp as the sorting field. Each record in the main index table is associated with the corresponding location stamp and media data storage index. A time-location mapping table is generated based on the main index table, wherein the mapping table uses timestamps as keys and location stamps as values. Construct an event association sub-table, which records the event stamp, the corresponding event trigger timestamp, the event trigger location stamp, and the media data storage index; By using the device identifier and timestamp fields, the main index table, the time and location mapping table, and the event association sub-table are linked and bound together to form a spatiotemporal fusion data association relationship.

[0008] In one embodiment, the spatiotemporal synchronization fusion and playback method for multi-source monitoring data further includes: If there are missing location stamps corresponding to consecutive timestamps, the coordinates of the missing location stamps are calculated using linear interpolation based on the coordinates and time interval of the adjacent valid location stamps. If the media data storage index corresponding to an event stamp is missing, the corresponding event is marked as having no associated media data and a missing log is recorded.

[0009] In one embodiment, the response originates from a playback request from a client, and retrieving the target dataset based on the spatiotemporal fusion data associations according to the playback request includes: Responding to a playback request from a client, the client's playback request including at least a device identifier and a time range; Using the device identifier as the primary search condition, the main index table, time and location mapping table, and event association sub-table corresponding to the target monitoring device are located based on the spatiotemporal fusion data association relationship; Based on the main index table, time and location mapping table, and event association sub-table corresponding to the target monitoring device, all location stamps within the time range are obtained to form a continuous location trajectory sequence, and all event stamps and event stamp association information within the time range are obtained. Integrate all the acquired information to form the target dataset.

[0010] In one embodiment, the client includes a user interface, which includes a first display area and a second display area; The first display area is used to render a timeline control based on an absolute timeline, and the second display area is used to render an electronic map; On the electronic map, the device movement trajectory is dynamically drawn or replayed based on the sequence of continuous location trajectory points in the fused data packet; Event markers are visually marked on the timeline control and at the corresponding positions on the electronic map's trajectory.

[0011] In one embodiment, the user interface further includes a third display area for playing the media data; When the user drags the time slider of the timeline control, the highlighted position of the electronic map is updated synchronously, and the third display area jumps to the media data at the corresponding time point and plays it. When a user clicks on any event marker on the electronic map, the time slider of the timeline control is synchronously positioned to the corresponding time point, and the third display area plays media data for a preset duration before and after the corresponding time point.

[0012] In one embodiment, when the mobile monitoring terminal collects the media data, it simultaneously collects the GPS location information at the current moment, encapsulates the GPS location information as a location stamp and the collection time as a timestamp, and binds it with the corresponding media data and stores it in the data packet; When the mobile monitoring terminal detects that a preset event has been triggered, it triggers an event marker generation process, synchronously collects GPS location information and time information at the moment the event is triggered, and encapsulates them into a location stamp and a timestamp corresponding to the event, respectively. At the same time, it encapsulates the type information of the triggered event into an event stamp, and associates and binds the event stamp, the location stamp, and the timestamp with the media data within the corresponding time period, and stores them together in the data packet.

[0013] Secondly, this application also provides a spatiotemporal synchronization fusion and playback device for multi-source monitoring data. The device includes: The data packet receiving module is used to receive data packets uploaded from the mobile monitoring terminal. The data packets include media data that is spatiotemporally tagged with timestamps and location stamps, and event marker data that is spatiotemporally tagged with event stamps, location stamps, and timestamps. The media data includes video data and audio data. The data packet parsing module is used to parse the data packet and extract the device identifier, timestamp, location stamp, event stamp, and storage index of the media data from the data packet. The data association module is used to associate timestamps, location stamps, event stamps, and storage indexes of the media data under the same device identifier to build a spatiotemporal fusion data association relationship; The target data retrieval module is used to respond to playback requests from clients and retrieve target datasets based on the spatiotemporal fusion data associations according to the playback requests. The target datasets include continuous location trajectory sequences within the target time period, all event markers, and media data storage indexes associated with each event marker. The data sending module is used to encapsulate the target dataset into a fused data packet and send the fused data packet to the client for integrated playback.

[0014] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps: Receive data packets uploaded from a mobile monitoring terminal, wherein the data packets include media data spatiotemporally tagged with timestamps and location stamps, and event-tagged data spatiotemporally tagged with event stamps, location stamps, and timestamps, wherein the media data includes video data and audio data; Parse the data packet to extract the device identifier, timestamp, location stamp, event stamp, and storage index of the media data. Associate the timestamps, location stamps, event stamps, and storage indexes of the media data under the same device identifier to construct a spatiotemporal fusion data association relationship; In response to a playback request from a client, and based on the playback request, a target dataset is retrieved based on the spatiotemporal fusion data association relationship. The target dataset includes a continuous location trajectory sequence within a target time period, all event markers, and media data storage indexes associated with each event marker. The target dataset is encapsulated into a fusion data packet and sent to the client for integrated playback.

[0015] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps: Receive data packets uploaded from a mobile monitoring terminal, wherein the data packets include media data spatiotemporally tagged with timestamps and location stamps, and event-tagged data spatiotemporally tagged with event stamps, location stamps, and timestamps, wherein the media data includes video data and audio data; Parse the data packet to extract the device identifier, timestamp, location stamp, event stamp, and storage index of the media data. Associate the timestamps, location stamps, event stamps, and storage indexes of the media data under the same device identifier to construct a spatiotemporal fusion data association relationship; In response to a playback request from a client, and based on the playback request, a target dataset is retrieved based on the spatiotemporal fusion data association relationship. The target dataset includes a continuous location trajectory sequence within a target time period, all event markers, and media data storage indexes associated with each event marker. The target dataset is encapsulated into a fusion data packet and sent to the client for integrated playback.

[0016] In summary, this application includes the following beneficial technical effects: By adding spatiotemporal tags to media data and event marker data at the data acquisition source, and associating timestamps, location stamps, event stamps, and storage indexes of the media data under the same device identifier, a spatiotemporal fusion data association relationship is constructed. This fundamentally solves the problem of poor spatiotemporal correlation of multi-source monitoring data in existing technologies, allowing users to associate all relevant data directly through the spatiotemporal dimension without manually comparing multiple data lists. Responding to playback requests from clients, the target dataset is retrieved based on the spatiotemporal fusion data association relationship, solving the problems of cumbersome and inefficient retrieval processes during event backtracking in existing technologies. The target dataset is encapsulated into a fusion data package and sent to the client, supporting integrated playback. This allows for the simultaneous presentation of spatiotemporal trajectories, event markers, and media data within the same interactive interface, eliminating the need to switch between multiple views and greatly improving playback efficiency. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a spatiotemporal synchronization fusion and playback method for multi-source monitoring data in one embodiment. Figure 2 This is a flowchart illustrating the spatiotemporal synchronization fusion and playback method for multi-source monitoring data in another embodiment; Figure 3This is a structural block diagram of a spatiotemporal synchronization fusion and playback device for multi-source monitoring data in one embodiment. Detailed Implementation

[0018] This invention provides a method and apparatus for spatiotemporal synchronization fusion and playback of multi-source monitoring data.

[0019] The embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the accompanying drawings and embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0020] In the description of the embodiments disclosed in this invention, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0021] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 One embodiment of the spatiotemporal synchronization fusion and playback method for multi-source monitoring data in this invention includes: S100 receives data packets uploaded from the mobile monitoring terminal.

[0022] Specifically, during the data reception phase, the server continuously receives data packets uploaded from the mobile monitoring terminal. These data packets contain media data and event tagging data. The media data consists of video and audio data collected by the mobile monitoring terminal, and each frame of media data is spatiotemporally tagged with timestamps and location stamps. The event tagging data consists of discrete data generated by the sensors of the mobile monitoring terminal after detecting a preset event. It is also spatiotemporally tagged with event stamps, location stamps, and timestamps to ensure that the data has spatiotemporal traceability capabilities.

[0023] S200, parse data packets.

[0024] Specifically, after receiving the data packets uploaded by the mobile monitoring terminal, the server parses and standardizes them, extracting core information from the data packets, including device identifier, timestamp, location stamp, event stamp, and media data storage index. Among them, the device identifier is used to distinguish different mobile monitoring terminals to ensure the uniqueness of data ownership; the timestamp corresponds to the precise time of media data collection or event occurrence; the location stamp corresponds to the geographical coordinates (latitude and longitude information) of media data collection or event occurrence; the event stamp records key attributes such as the type and level of the event; and the media data storage index is the unique access address of the media data in the server's storage system, facilitating rapid retrieval later.

[0025] S300 associates timestamps, location stamps, event stamps, and storage indexes of media data under the same device identifier to build spatiotemporal fusion data association relationships.

[0026] Specifically, for all parsed data under the same device identifier, the server uses an absolute timeline as a benchmark to deeply associate and bind timestamps, location stamps, event stamps, and media data storage indexes. Specifically, based on the timestamp as the sorting criterion, the location stamps corresponding to each time point are sequentially linked to form a continuous chain of device movement trajectory data. At the same time, the event stamps are associated with the corresponding timestamps, location stamps, and media data storage indexes, so that each event marker can be accurately mapped to a specific time, geographical location, and on-site media data.

[0027] In this embodiment, by constructing a spatiotemporal fusion data association relationship, the originally scattered media data, trajectory data and event data are integrated into a complete spatiotemporal data chain. Each data node contains three attributes: time, space and content. This fundamentally solves the problem of poor spatiotemporal correlation of multi-source data in the prior art and provides core support for subsequent efficient retrieval and integrated playback.

[0028] S400 responds to playback requests from clients and retrieves the target dataset based on spatiotemporal fusion data relationships according to the playback requests.

[0029] Specifically, when a client initiates a playback request, the server needs to quickly retrieve the target dataset that meets the user's needs based on the established spatiotemporal fusion data relationships. The client's playback request must include at least two core parameters: device identifier and time range. Users can also add filtering conditions such as event type and location region based on their actual needs. After accepting the playback request, the server uses the device identifier as the primary search condition to locate all associated data for that device. Then, based on the time range and other filtering conditions in the playback request, it retrieves the dataset that meets the requirements. The dataset includes a sequence of continuous location trajectory points within the target time period, all event markers, and the media data storage index associated with each event marker.

[0030] In this embodiment, data is retrieved based on the spatiotemporal fusion data correlation relationship, eliminating the need for manual cross-system comparison, which can significantly shorten the retrieval time and solve the problem of low event backtracking efficiency in the prior art.

[0031] The S500 encapsulates the target dataset into a fusion data package and sends the fusion data package to the client for integrated playback.

[0032] Specifically, after retrieving the target dataset, the server needs to standardize and encapsulate it before sending it to the client to ensure efficient parsing and integrated playback. During encapsulation, the server integrates the location trajectory sequence, event marker set, and media data storage index in the target dataset into a fused data packet according to a preset format. After encapsulation, the server dynamically adjusts the transmission strategy based on the client's network status (such as bandwidth and signal strength). If the network bandwidth is sufficient, the TCP protocol is used to ensure data integrity; if the network is unstable, the UDP protocol combined with data fragmentation and retransmission mechanisms is used to ensure that the data packet is sent to the client efficiently and completely. After receiving the data packet, the client does not need to convert the format. It can directly present the device movement trajectory and event marker distribution synchronously on the same interactive interface based on the spatiotemporal correlation information, and supports users to play media data associated with any event marker, realizing integrated playback of "spatiotemporal-event-media" and solving the problem of fragmented playback experience in traditional solutions.

[0033] In one embodiment, such as Figure 2 As shown, S300 includes: S310, Create a primary index table with device identifier as the primary key and timestamp as the sorting field; S320, Generate a time-location mapping table based on the master index table; S330, construct the event-related sub-table; S340 uses the device identifier and timestamp fields to associate and bind the main index table, the time and location mapping table, and the event association sub-table to form a spatiotemporal fusion data association relationship.

[0034] Specifically, firstly, the server establishes a primary index table with the device identifier as the primary key and the timestamp as the sorting field. This table is the core foundation for data association, and each record is associated with a corresponding location stamp and media data storage index, enabling rapid location of any device's location information and corresponding media data resources at a specific point in time. Based on this, a time-location mapping table is generated from the primary index table. This mapping table uses the timestamp as the key and the location stamp as the value, establishing a direct correspondence between time and location through key-value pairs. This allows for rapid querying of the geographical location information corresponding to a device at any given time, thus forming a continuous and complete chain of device movement trajectory data. Simultaneously, an event association sub-table is constructed to record in detail the event stamp, the corresponding event trigger timestamp, the event trigger location stamp, and the associated media data storage index, clarifying the correspondence between each event and spatiotemporal information and media data, enabling precise event traceability. At the data association logic level, the primary index table, the time-location mapping table, and the event association sub-table are linked and bound together through the two key common fields of device identifier and timestamp, forming a logically rigorous and interconnected organic whole from the various types of data stored separately.

[0035] In one embodiment, the spatiotemporal synchronization fusion and playback method for multi-source monitoring data further includes: If there are missing location stamps corresponding to consecutive timestamps, the coordinates of the missing location stamps are calculated using linear interpolation based on the coordinates and time intervals of the adjacent valid location stamps. If there are missing media data storage indexes corresponding to event stamps, the corresponding events are marked as having no associated media data status, and a missing log is recorded.

[0036] Specifically, regarding the handling of missing location stamps, when a missing location stamp corresponding to a consecutive timestamp is detected, the coordinates of the missing location stamp are automatically calculated using linear interpolation based on the coordinates and time interval of adjacent valid location stamps. This interpolation method can restore the device's movement trajectory within the missing time period to the greatest extent possible, ensuring the continuity of the location trajectory sequence and avoiding trajectory breaks due to local data loss, thus ensuring that users can view the complete device movement path. Regarding the handling of missing media data storage indexes, if a missing media data storage index corresponding to an event stamp is detected, the event is automatically marked as having no associated media data, and a detailed missing log is recorded, including key information such as the event occurrence time, device identifier, event type, and reason for the missing data. This design allows users to clearly understand that there is currently no corresponding audio or video evidence for the event, avoiding misjudgment, and provides accurate basis for subsequent system maintenance, data investigation, and repair, ensuring that the overall data availability is not excessively affected by local missing data.

[0037] In one embodiment, responding to a playback request from a client and retrieving the target dataset based on spatiotemporal fusion data correlations according to the playback request includes: In response to a playback request from the client, which includes at least the device identifier and time range, the system uses the device identifier as the primary search condition. Based on the spatiotemporal fusion data correlation, it locates the main index table, time-location mapping table, and event-related sub-table corresponding to the target monitoring device. Based on the main index table, time-location mapping table, and event-related sub-table corresponding to the target monitoring device, it obtains all location stamps within the time range to form a continuous location trajectory sequence, and obtains all event stamps and event stamp association information within the time range. Finally, it integrates all the obtained information to form the target dataset.

[0038] Specifically, the playback request initiated by the client includes at least two core parameters: device identifier and time range. The device identifier is used to accurately locate the target monitoring device, avoiding retrieval errors due to device confusion. The time range limits the data interval for playback; users can flexibly set specific start and end times according to actual needs to achieve precise data filtering. After receiving the playback request, the server uses the device identifier as the primary search condition to quickly locate the corresponding main index table, time-location mapping table, and event association sub-table, significantly reducing the search scope and improving search efficiency. Subsequently, based on the time range in the request, it filters all location stamps within that time period from the located tables, sorts these location stamps chronologically to form a continuous and complete location trajectory sequence, visually presenting the device's movement path within that time period. Simultaneously, it extracts all event stamps within that time period and associates them with information such as trigger time, trigger location, event type, and media data storage address. Finally, it integrates and summarizes all information, including the extracted location trajectory sequence, event marker set, and associated media data storage index, to form a clearly structured and complete target dataset, ensuring that the client can quickly parse and present the data.

[0039] In one embodiment, the client includes a user interface, which includes a first display area and a second display area; the first display area is used to render a timeline control based on an absolute time axis, and the second display area is used to render an electronic map; on the electronic map, the device movement trajectory is dynamically drawn or replayed according to the sequence of continuous location trajectory points in the fused data package; and event markers are visually marked on the timeline control and at the corresponding positions of the trajectory on the electronic map.

[0040] Specifically, the client's user interface includes a first display area and a second display area, which work synchronously and interact with each other. The first display area renders a timeline control based on an absolute time axis. This timeline control clearly presents the distribution of all data within the target time period. Users can quickly locate specific time points of interest by dragging the time slider, achieving precise navigation in the time dimension. The second display area renders an electronic map. Based on the continuous location trajectory point sequence in the parsed fusion data packet, the client dynamically draws or replays the device's movement trajectory on the electronic map. Users can intuitively view the device's travel route, direction of travel, and stopping position in geographic space. Simultaneously, event markers are indicated on the timeline control and the corresponding positions on the electronic map using differentiated visual icons. Different types of events use different styles or colors of icons; for example, a red exclamation mark icon indicates a collision event, and a yellow horn icon indicates a voice-triggered event. This allows users to quickly identify the time and geographical location of events, achieving precise positioning in both time and space dimensions.

[0041] In one embodiment, the user interface further includes a third display area for playing media data. When the user drags the time slider of the timeline control, the highlighted position of the electronic map is updated synchronously, and the third display area jumps to the media data at the corresponding time point and plays it. When the user clicks any event marker on the electronic map, the time slider of the timeline control is positioned synchronously to the corresponding time point, and the third display area plays media data of a preset duration before and after the corresponding time point.

[0042] Specifically, the third display area is used to play media data, forming a collaborative interface with the timeline control in the first display area and the electronic map in the second display area, enabling comprehensive and interconnected data presentation. When the user drags the time slider of the timeline control, the highlighted position on the electronic map updates synchronously, accurately locating the device's position at the corresponding time point. Simultaneously, the third display area automatically jumps to the media data at the corresponding time point and begins playback, achieving real-time synchronization of time, space, and media data, allowing users to directly see the device's status at a specific time and location. When the user clicks on any event marker on the electronic map, the time slider of the timeline control synchronously positions itself to the time point of the event, and the third display area automatically plays media data for a preset duration before and after that time point. The preset duration can be flexibly configured according to the needs of the actual application scenario (e.g., 10 seconds, 30 seconds, or 60 seconds).

[0043] In this embodiment, the linkage of the first display area, the second display area and the third display area allows users to quickly view the complete scene before and after the event without switching between multiple interfaces, greatly improving the efficiency and convenience of playback.

[0044] In one embodiment, when the mobile monitoring terminal collects media data, it simultaneously collects the GPS location information at the current moment, encapsulates the GPS location information as a location stamp and the collection time as a timestamp, and binds them with the corresponding media data and stores them in a data packet. When the mobile monitoring terminal detects that a preset event has been triggered, it triggers an event marker generation process, simultaneously collects the GPS location information and time information at the moment the event is triggered, encapsulates them as the location stamp and timestamp corresponding to the event, and encapsulates the type information of the triggered event as an event stamp. The event stamp, location stamp, and timestamp are then associated and bound with the media data within the corresponding time period and stored together in a data packet.

[0045] Specifically, when collecting media data, the mobile monitoring terminal synchronously collects precise latitude and longitude location information at the current moment through its built-in GPS module, encapsulating this location information as a location stamp. Simultaneously, it records the media data collection time using its built-in high-precision real-time clock (RTC), encapsulating the collection time as a timestamp. The location stamp and timestamp are then bound to the corresponding media data frames and stored in the data packet. When the mobile terminal detects a preset event (such as a collision or emergency braking), it immediately triggers the event marker generation process, synchronously collecting the GPS location and time information at the moment the event is triggered, encapsulating them as the corresponding location stamp and timestamp. At the same time, the type information of the triggering event (such as collision trigger, emergency braking trigger, etc.) is encapsulated as an event stamp. The event stamp, location stamp, and timestamp are then associated and bound to the media data within the corresponding time period and stored in the data packet.

[0046] In one embodiment, such as Figure 3 As shown, a spatiotemporal synchronous fusion and playback device for multi-source monitoring data is provided, comprising: a data packet receiving module 10, a data packet parsing module 20, a data association module 30, a target data retrieval module 40, and a data sending module 50, wherein: The data packet receiving module 10 is used to receive data packets uploaded from the mobile monitoring terminal. The data packets include media data that is spatiotemporally tagged with timestamps and location stamps, and event marker data that is spatiotemporally tagged with event stamps, location stamps, and timestamps. The media data includes video data and audio data. The data packet parsing module 20 is used to parse data packets and extract device identifiers, timestamps, location stamps, event stamps, and storage indexes of media data from the data packets; The data association module 30 is used to associate timestamps, location stamps, event stamps and storage indexes of media data under the same device identifier to build spatiotemporal fusion data association relationships; The target data retrieval module 40 is used to respond to playback requests from clients and retrieve target datasets based on spatiotemporal fusion data associations according to the playback requests. The target datasets include continuous location trajectory sequences within the target time period, all event markers, and media data storage indexes associated with each event marker. The data sending module 50 is used to encapsulate the target dataset into a fused data packet and send the fused data packet to the client for integrated playback.

[0047] In one embodiment, the data association module 30 is further configured to establish a main index table with device identifier as the primary key and timestamp as the sorting field, wherein each record in the main index table is associated with the corresponding location stamp and media data storage index; generate a time-location mapping relationship table based on the main index table, wherein the mapping relationship table uses timestamp as the key and location stamp as the value; construct an event association sub-table, wherein the event association sub-table records the event stamp, the corresponding event trigger timestamp, the event trigger location stamp, and the media data storage index; and associate and bind the main index table, the time-location mapping relationship table, and the event association sub-table through the device identifier and timestamp fields to form a spatiotemporal fusion data association relationship.

[0048] In one embodiment, the spatiotemporal synchronization fusion playback device for multi-source monitoring data further includes a data missing processing module, which is used to calculate the coordinates of the missing location stamps based on the coordinates and time intervals of adjacent valid location stamps if there are missing location stamps corresponding to consecutive timestamps; and to mark the corresponding event as an unrelated media data state and record the missing log if there are missing media data storage indexes corresponding to event stamps.

[0049] In one embodiment, the target data retrieval module 40 is further configured to respond to a playback request from a client, the client's playback request including at least a device identifier and a time range; using the device identifier as the primary retrieval condition, locate the main index table, time and location mapping table, and event association sub-table corresponding to the target monitoring device based on the spatiotemporal fusion data association relationship; based on the main index table, time and location mapping table, and event association sub-table corresponding to the target monitoring device, obtain all location stamps within the time range to form a continuous location trajectory sequence, and obtain all event stamps and event stamp association information within the time range; integrate all the obtained information to form a target dataset.

[0050] In one embodiment, the spatiotemporal synchronization fusion playback device for multi-source monitoring data further includes an interface display module, which is used to dynamically draw or play back the device movement trajectory on an electronic map based on the continuous location trajectory point sequence in the fusion data package; and to visually mark event markers on the timeline control and the trajectory corresponding positions on the electronic map.

[0051] In one embodiment, the spatiotemporal synchronization fusion playback device for multi-source monitoring data further includes an interface display module that is also used to synchronously update the highlighted position of the electronic map when the user drags the time slider of the timeline control, and the third display area jumps to the media data at the corresponding time point and plays it; when the user clicks any event marker on the electronic map, the time slider of the timeline control is synchronously positioned to the corresponding time point, and the third display area plays media data of a preset duration before and after the corresponding time point.

[0052] In one embodiment, the spatiotemporal synchronization fusion playback device for multi-source monitoring data includes a data encapsulation module. This module is used to simultaneously collect GPS location information at the current moment when the mobile monitoring terminal collects media data, encapsulate the GPS location information as a location stamp and the collection time as a timestamp, and bind them to the corresponding media data and store them in a data packet. When the mobile monitoring terminal detects a preset event trigger, it triggers an event marker generation process, simultaneously collecting the GPS location information and time information at the moment the event is triggered, encapsulating them as the location stamp and timestamp corresponding to the event, and encapsulating the type information of the triggered event as an event stamp. The event stamp, location stamp, and timestamp are then associated and bound to the media data within the corresponding time period and stored together in a data packet.

[0053] In one embodiment, this application discloses a computer device including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor loads the computer program, it executes a spatiotemporal synchronization fusion playback method for multi-source monitoring data according to the above embodiment.

[0054] In one embodiment, this application discloses a computer-readable storage medium storing a computer program, wherein when the computer program is loaded by a processor, it executes a spatiotemporal synchronization fusion playback method for multi-source monitoring data as described above.

[0055] The above are all preferred embodiments of this application and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A method for spatiotemporal synchronous fusion and playback of multi-source monitoring data, characterized in that, include: Receive data packets uploaded from a mobile monitoring terminal, wherein the data packets include media data spatiotemporally tagged with timestamps and location stamps, and event-tagged data spatiotemporally tagged with event stamps, location stamps, and timestamps, wherein the media data includes video data and audio data; Parse the data packet to extract the device identifier, timestamp, location stamp, event stamp, and storage index of the media data. Associate the timestamps, location stamps, event stamps, and storage indexes of the media data under the same device identifier to construct a spatiotemporal fusion data association relationship; In response to a playback request from a client, and based on the playback request, a target dataset is retrieved based on the spatiotemporal fusion data association relationship. The target dataset includes a continuous location trajectory sequence within a target time period, all event markers, and media data storage indexes associated with each event marker. The target dataset is encapsulated into a fusion data packet and sent to the client for integrated playback.

2. The spatiotemporal synchronization fusion and playback method for multi-source monitoring data according to claim 1, characterized in that, The step of associating timestamps, location stamps, event stamps, and the storage index of the media data under the same device identifier to construct a spatiotemporal fusion data association includes: Establish a main index table with device identifier as the primary key and timestamp as the sorting field. Each record in the main index table is associated with the corresponding location stamp and media data storage index. A time-location mapping table is generated based on the main index table, wherein the mapping table uses timestamps as keys and location stamps as values. Construct an event association sub-table, which records the event stamp, the corresponding event trigger timestamp, the event trigger location stamp, and the media data storage index; By using the device identifier and timestamp fields, the main index table, the time and location mapping table, and the event association sub-table are linked and bound together to form a spatiotemporal fusion data association relationship.

3. The spatiotemporal synchronization fusion and playback method for multi-source monitoring data according to claim 2, characterized in that, Also includes: If there are missing location stamps corresponding to consecutive timestamps, the coordinates of the missing location stamps are calculated using linear interpolation based on the coordinates and time interval of the adjacent valid location stamps. If the media data storage index corresponding to an event stamp is missing, the corresponding event is marked as having no associated media data and a missing log is recorded.

4. The spatiotemporal synchronization fusion and playback method for multi-source monitoring data according to claim 1, characterized in that, The response originates from a playback request from the client, and the retrieval of the target dataset based on the spatiotemporal fusion data correlation according to the playback request includes: Responding to a playback request from a client, the client's playback request including at least a device identifier and a time range; Using the device identifier as the primary search condition, the main index table, time and location mapping table, and event association sub-table corresponding to the target monitoring device are located based on the spatiotemporal fusion data association relationship; Based on the main index table, time and location mapping table, and event association sub-table corresponding to the target monitoring device, all location stamps within the time range are obtained to form a continuous location trajectory sequence, and all event stamps and event stamp association information within the time range are obtained. Integrate all the acquired information to form the target dataset.

5. The spatiotemporal synchronization fusion and playback method for multi-source monitoring data according to claim 1, characterized in that, The client includes a user interface, which includes a first display area and a second display area. The first display area is used to render a timeline control based on an absolute timeline, and the second display area is used to render an electronic map; On the electronic map, the device movement trajectory is dynamically drawn or replayed based on the sequence of continuous location trajectory points in the fused data packet; Event markers are visually marked on the timeline control and at the corresponding positions on the electronic map's trajectory.

6. The spatiotemporal synchronization fusion and playback method for multi-source monitoring data according to claim 5, characterized in that, The user interface also includes a third display area, which is used to play the media data. When the user drags the time slider of the timeline control, the highlighted position of the electronic map is updated synchronously, and the third display area jumps to the media data at the corresponding time point and plays it. When a user clicks on any event marker on the electronic map, the time slider of the timeline control is synchronously positioned to the corresponding time point, and the third display area plays media data for a preset duration before and after the corresponding time point.

7. The spatiotemporal synchronization fusion and playback method for multi-source monitoring data according to claim 1, characterized in that, When the mobile monitoring terminal collects the media data, it simultaneously collects the GPS location information at the current moment, encapsulates the GPS location information into a location stamp and the collection time into a timestamp, and binds it with the corresponding media data and stores it in the data packet; When the mobile monitoring terminal detects that a preset event has been triggered, it triggers an event marker generation process, synchronously collects GPS location information and time information at the moment the event is triggered, and encapsulates them into a location stamp and a timestamp corresponding to the event, respectively. At the same time, it encapsulates the type information of the triggered event into an event stamp, and associates and binds the event stamp, the location stamp, and the timestamp with the media data within the corresponding time period, and stores them together in the data packet.

8. A spatiotemporal synchronous fusion and playback device for multi-source monitoring data, characterized in that, include: The data packet receiving module is used to receive data packets uploaded from the mobile monitoring terminal. The data packets include media data that is spatiotemporally tagged with timestamps and location stamps, and event marker data that is spatiotemporally tagged with event stamps, location stamps, and timestamps. The media data includes video data and audio data. The data packet parsing module is used to parse the data packet and extract the device identifier, timestamp, location stamp, event stamp, and storage index of the media data from the data packet. The data association module is used to associate timestamps, location stamps, event stamps, and storage indexes of the media data under the same device identifier to build a spatiotemporal fusion data association relationship; The target data retrieval module is used to respond to playback requests from clients and retrieve target datasets based on the spatiotemporal fusion data associations according to the playback requests. The target datasets include continuous location trajectory sequences within the target time period, all event markers, and media data storage indexes associated with each event marker. The data sending module is used to encapsulate the target dataset into a fused data packet and send the fused data packet to the client for integrated playback.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images