A behavior process evidence storage method, system and terminal based on multi-modal data

Abstract

This invention discloses a method, system, and terminal for storing behavioral process evidence based on multimodal data, belonging to the field of data storage and public security technology. It aims to solve the problems of limited types of behavioral process evidence, difficulty in cross-verification, and difficulty in post-event verification in existing technologies. This invention acquires the user's multimodal data, constructs a dynamic physiological data baseline using at least heart rate data, and monitors multiple triggering events: real-time data deviating from the baseline by more than a threshold, user-initiated commands, and external system alerts. When any triggering event occurs, the recorded data is annotated and / or enhanced, and a user post-event annotation function is also provided. This invention can automatically and multidimensionally record behavioral processes without the user's awareness or with simple operation, forming a mutually corroborating chain of evidence. It is applicable to various scenarios such as evidence collection for acts of helping others to prevent false accusations and proving innocence in court, and has advantages such as high evidence credibility, good user experience, and strong privacy protection.

Description

Technical Field

[0001] This invention relates to the fields of data storage and public safety technology, and more specifically, to a method, system, and terminal for storing behavioral process evidence based on multimodal data. Background Technology

[0002] In public settings, individuals often face situations where they need to prove their innocence or uncover the truth of an event. For example, helping someone who has fallen might lead to misunderstandings and being mistaken for the perpetrator; victims of verbal threats, physical altercations, or traffic accidents often struggle to provide objective evidence. Existing solutions primarily rely on video surveillance, mobile phone recordings, or eyewitness accounts. However, these methods have significant limitations:

[0003] Incomplete surveillance coverage: Public cameras cannot cover all areas, and recordings may be overwritten due to limited storage space, making it difficult to guarantee that the required scenes are fully recorded.

[0004] Difficulties in filming: In emergency situations, the parties involved or bystanders may not have time to start recording, or may be unable to operate due to fear or coercion; even if there is a video, its integrity and authenticity (whether it has been edited or staged) are often questioned, requiring complex judicial appraisal, which increases the cost of protecting rights.

[0005] Limited type of evidence: A single video or audio recording cannot fully reflect the process of an event and the state of the parties involved. For example, it cannot prove the trajectory of movement before the behavior occurred, the relative distance between the two parties, or the physiological reactions of the parties involved (such as tension or fear). This results in insufficient persuasiveness of the evidence and easily leads to a dilemma of "each side sticking to their own story".

[0006] Difficulty in obtaining evidence after the fact: Even if data exists, it is scattered across different devices or platforms, making it difficult to link them to form a complete chain of evidence; moreover, ordinary users lack professional means of preserving evidence, making it difficult to corroborate each other, which affects its effective use in judicial proceedings.

[0007] Therefore, the market urgently needs a technological solution that can automatically and multidimensionally record behavioral processes without the user's awareness or with simple operation, and ensure the authenticity, integrity, mutual verifiability, and traceability of evidence. Summary of the Invention

[0008] To address the problems existing in the prior art, the present invention provides a method, system and terminal for storing evidence of behavioral processes based on multimodal data, which can automatically or according to triggers annotate recorded data to form a chain of evidence that can corroborate each other, and realize the objective reconstruction of behavioral processes.

[0009] To achieve the above objectives, the present invention provides the following technical solution:

[0010] A method for storing behavioral process evidence based on multimodal data, executed by a user-worn or held terminal, includes the following steps:

[0011] (1) Data Acquisition: Acquire the user's multimodal data. The multimodal data is a data set that can describe the user's state and surrounding environment from different dimensions, including at least heart rate data. Optionally, it also includes location data, motion data, skin conductance response data, environmental audio data, environmental video data, and relative spatial relationship data with the target object.

[0012] (2) Dynamic baseline construction: at least the heart rate data is used to construct a dynamic physiological data baseline, specifically including: using a sliding window of a set duration, performing aggregation operations on the heart rate data in the window, calculating its statistical characteristic values ​​(such as arithmetic mean, median, etc.), and using it as the dynamic physiological data baseline at the current moment; the baseline is updated over time to adapt to the slow changes in individual physiological state.

[0013] The dynamic physiological data baseline is obtained by statistically calculating historical heart rate data using a sliding window of a set duration, and is adaptively updated over time.

[0014] (3) Trigger event monitoring: Monitor at least one of the following trigger events in real time:

[0015] Automatic anomaly detection: When the deviation between the real-time heart rate data and the dynamic heart rate data baseline exceeds the preset suspected event threshold, the system can call other multimodal data for cross-validation to improve the confidence of the triggering conditions.

[0016] The automatic anomaly detection can be performed on one or more physiological data separately. Each physiological data or physiologically related data (such as heart rate, body temperature, skin conductance response, etc.) can have its corresponding dynamic physiological data baseline. The system monitors the deviation of various physiological data from their respective baselines in real time. When the deviation of any physiological data exceeds a preset trigger event threshold, it constitutes trigger event a.

[0017] The cross-validation includes, but is not limited to: when heart rate data deviates and triggers a suspected event, calling motion data to determine whether it is caused by physical activity, and calling skin conductance data to determine whether it is caused by emotional fluctuations.

[0018] User-initiated instructions: Receive annotation instructions input by the user through the human-computer interaction interface.

[0019] External input alert signal: An alert signal is received from an external device or system. The system parses the metadata contained in the signal (such as effective time, valid area, authorized confirmation, etc.) and, in conjunction with the user's current context information, determines whether the signal meets the preset activation conditions. Only when the signal is determined to be valid for the current user will it be recognized as a valid trigger event.

[0020] (4) Event Labeling: When the system detects any triggering event, it labels the multimodal data being recorded, that is, inserts an event label tag into each data entry. Users can also add event label tags to the data in the cache by time or geographical location before the data is overwritten. Post-event labeling is only valid before the data is overwritten. The corresponding multimodal data is extracted from the evidence cache and labeled for evidence storage.

[0021] Multimodal data that has been annotated with events is sent to a storage area outside the cache for evidence storage, based on the user's authorization.

[0022] (5) Two-level recording strategy: When no triggering event is detected, multimodal data is acquired and stored cyclically at the basic sampling frequency; when a triggering event is detected and the event is labeled, the sampling frequency can be increased and more types of data can be acquired.

[0023] (6) Adaptive Stop: During the enhanced recording process, the enhanced recording will automatically stop and revert to the basic recording when any of the following conditions are met: receiving a stop command input by the user through the human-computer interaction interface; the triggering event no longer meets the preset activation conditions; the enhanced recording duration reaches the preset maximum duration.

[0024] It should be noted that the relative spatial relationship data with the target object is obtained based on two-way communication between the user terminal and the terminal held by the target object. There are two methods: 1. Both terminals need to be compatible with the same short-range wireless communication protocol (such as UWB, satellite flash, etc.) and calculate the relative distance and azimuth between them in real time by measuring the signal flight time or angle of arrival; 2. Both parties obtain each other's satellite position data through the communication module and calculate the relative distance and azimuth through the data.

[0025] Furthermore, it should be noted that the core of this invention lies in constructing a dynamic heart rate data baseline using at least heart rate data and monitoring triggering events; heart rate data is essential data. Other multimodal data (including but not limited to location data, motion data, environmental data, and relative spatial relationship data with the target object) are optional acquisitions, and corresponding dynamic data baselines can be established accordingly. When certain data types cannot be acquired due to limitations such as terminal hardware configuration, user authorization, network conditions, or on-site environment, such data types can be omitted without affecting the implementation of the core function of this invention. For example, when the user terminal is not equipped with a microphone or the user has not authorized the collection of environmental audio, environmental audio data can be omitted; when the target object does not possess a compatible terminal, relative spatial relationship data can be omitted; when satellite signals are weak and positioning is impossible, location data can be omitted.

[0026] The present invention also provides a terminal for performing the above-described method, and a system comprising the terminal and a server.

[0027] The terminal includes a data acquisition module for acquiring the multimodal data; a memory for storing computer programs and data; a processor for executing the computer program to implement the steps of the method; a communication module for data interaction with external devices or systems, including a wide area network module and a short-range wireless network module; and a human-computer interaction interface for receiving user annotation instructions.

[0028] The server is configured to communicate with the terminal, receive and store the data stored by the terminal, or send warning signals and instructions to the terminal.

[0029] Compared with the prior art, the present invention has the following significant advantages:

[0030] 1. Constructing a multimodal, mutually corroborating evidence chain to enhance the judicial validity of evidence: This invention simultaneously acquires various types of data, including location, motion, physiological, distance, and audio / video data, enabling mutual corroboration from multiple dimensions such as space, behavior, physiological state, and environment, forming a complete storyline. Compared to single audio or video evidence, this multi-dimensional, mutually corroborating evidence chain greatly improves the objectivity and credibility of evidence, reduces the difficulty of forensic identification, and increases the acceptance rate.

[0031] 2. Introducing adaptive dynamic baseline technology for accurate personalized anomaly detection: This invention overcomes the shortcomings of fixed thresholds in traditional methods by constructing a user-specific dynamic data baseline in real time through a sliding window. This baseline can adaptively learn the user's daily behavioral patterns and physiological characteristics, such as the range of heart rate data at rest and the intensity of exercise while walking, thereby effectively avoiding false triggers caused by normal physical activity and accurately capturing recording requests triggered by emotional fluctuations.

[0032] 3. Design multiple redundant triggering mechanisms to ensure reliable initiation of critical events: This invention combines three triggering methods—automatic anomaly detection, user-initiated commands, and external alerts—to form a complementary system. When users are unable to operate in time (e.g., due to sudden conflict), dare not operate (e.g., under duress), or are unable to perceive the risk (e.g., due to toxic gas leaks), the automatic or external triggering mechanism can reliably initiate recording, covering all possible critical scenarios.

[0033] 4. User-controlled cancellation and post-event review mechanism, balancing privacy protection and evidence collection needs: This invention returns ultimate control over data acquisition to the user through a mechanism that allows users to cancel recordings at any time, fully demonstrating respect for personal privacy. Simultaneously, the unique background loop caching and post-event annotation functions enable users to securely and conveniently retrieve critical data that they "forgot to record at the time," addressing a fundamental pain point that existing technologies have failed to resolve.

[0034] 5. Possesses a wide range of application scenarios and significant social benefits: This invention can be integrated into various terminals such as smartwatches, wristbands, mobile phones, and dashcams, and is applicable to various scenarios such as helping people prevent extortion and evidence collection, and preserving evidence at the scene of disputes. It helps to restore the truth, resolve social conflicts, maintain fairness and justice, and build a harmonious and stable society under the rule of law, and has good market prospects and social benefits.

[0035] The present invention also provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, it implements the steps of any of the methods described above. The storage medium includes, but is not limited to, any carrier capable of storing and running programs, such as a terminal's built-in memory, a mobile storage device (e.g., a USB flash drive, a portable hard drive), or network storage space. Attached Figure Description

[0036] Figure 1 A flowchart illustrating the behavioral process evidence preservation method in this embodiment of the invention.

[0037] Figure 2 System architecture diagram of terminal-server interaction in this embodiment of the invention Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the architecture of the terminal will be described first. For example... Figure 2 As shown, the terminal includes:

[0039] Data acquisition module: This module integrates multiple sensors to acquire multimodal data from the user. Specifically, it includes, but is not limited to, heart rate sensors (such as PPG sensors), accelerometers, gyroscopes, skin conductance sensors, BeiDou / GPS positioning modules, microphones, and short-range wireless communication modules (such as UWB / starburst) for measuring relative distance. The sampling frequency of each sensor can be dynamically adjusted according to the strategy of basic and augmented recording (see Example 1 for details).

[0040] It should be noted that, as described in the invention summary, the core functionality of this invention relies solely on heart rate data; other multimodal data are optional. In the following embodiments, for ease of explanation, it is assumed that the relevant data is available; however, in actual deployment, when certain data types are unavailable due to hardware or environmental limitations, these data can be omitted without affecting the core implementation of this invention.

[0041] After the raw signals output by each sensor integrated in the data acquisition module are processed and converted by the preprocessing submodule, the data acquisition module may include:

[0042] (a) Analog signal conditioning circuit: For analog output sensors (such as microphones), the raw signal is amplified and filtered, and then converted into a digital signal by an analog-to-digital converter (ADC);

[0043] (b) Digital signal preprocessing: For digital output sensors (such as IMU, PPG), the microcontroller (MCU) reads data through buses such as I²C and SPI, and can perform calibration, normalization and other operations;

[0044] (c) Preprocessing dedicated to heart rate data: This may also include adaptive filtering based on IMU motion data (such as the least mean square algorithm) for motion artifact compensation, and data normalization using the formula (x - x_min) / (x_max - x_min) to transform the data into a uniform interval for subsequent fusion calculation.

[0045] Through the above acquisition and preprocessing process, the multimodal data acquired by the system provides reliable input for subsequent fusion computing and decision-making.

[0046] Memory: Used to store computer programs and data. The memory is divided into multiple circular buffers, including a real-time analysis cache (stores data from the most recent few seconds for baseline calculation and cross-validation) and an evidence data cache (stores multimodal data for a preset duration, such as 48 hours). For the specific storage structure, see Example 1.

[0047] Processor: Coupled with the data acquisition module and the memory, it executes the computer program stored in the memory to implement the various steps of the method described in this invention, including dynamic baseline construction, trigger event monitoring, event labeling, two-level recording strategy control, adaptive stopping, etc. The processor has local computing capabilities and can independently complete the core functions without a network connection.

[0048] Communication module: Supports wide area network communication (such as 4G / 5G, NB-IoT) and short-range wireless communication (such as StarFlash, UWB, Wi-Fi, Bluetooth) for data interaction with servers, other terminals, external devices (such as smoke detectors), and rescue platforms. The communication module can adaptively select the communication path according to network conditions. The communication module includes a wide area network module and a short-range wireless module.

[0049] Human-computer interaction interface: including physical buttons, touch screen, microphone (for voice commands) and input / output devices such as vibration motor, speaker, and indicator lights, used to receive user annotation commands and output reminder signals.

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The embodiments described are only for explaining the invention, and the data in the embodiments are exemplified only to illustrate the principles, and are not intended to limit the scope of the invention.

[0051] Example 1: Sensor data acquisition, two-level caching, and usage (see flowchart) Figure 1 )

[0052] I. Data Acquisition

[0053] Using these built-in sensors, the terminal acquires the following multimodal data (sampling frequency is for illustrative purposes only):

[0054]

[0055] II. Two-level circular cache

[0056] The terminal allocates two independent circular buffers in memory for each data type:

[0057] a. Real-time analysis cache: Stores multimodal data from the last 5-30 seconds (adjustable by technicians based on the actual scenario) for:

[0058] (1) Construction of dynamic physiological data baseline

[0059] The process of constructing a dynamic physiological data baseline (taking the construction of a dynamic heart rate data baseline as an example): Using a 10-second sliding window, the arithmetic mean of the heart rate data within the window is calculated and used as the dynamic heart rate data baseline for the current moment. The system adopts a first-in, first-out (FIFO) update method: Taking heart rate data as an example, if the sampling frequency is 2Hz, that is, a new heart rate data sample is received every 0.5 seconds, the old data from 10 seconds ago is discarded, completing one window update. Then, the calculated average of 20 heart rate data in the 10-second window is calculated, thus forming the construction of the dynamic physiological data baseline (here referred to as the dynamic heart rate data baseline). This baseline can adaptively change with changes in the user's heart rate data.

[0060] In addition to the arithmetic mean, other methods such as the median and exponentially weighted moving average can also be used to construct the average.

[0061] Optionally, physiological data such as body temperature can also be used to construct a corresponding dynamic physiological data baseline. For example, body temperature data reflects a user's physiological state, but its changes are relatively slow. A sliding window of 5 minutes is used to calculate the arithmetic mean of body temperature as the dynamic body temperature data baseline.

[0062] (2) Use of dynamic physiological data baseline

[0063] Use of dynamic heart rate data baseline: When real-time heart rate data is compared with dynamic heart rate data baseline, if the deviation exceeds the preset first threshold (or suspected event threshold, such as 20%), a "suspected event" signal is triggered; if the deviation exceeds the preset second threshold (or triggered event threshold, such as 30%), event labeling and enhanced recording are directly triggered.

[0064] Use of other dynamic physiological data baselines: When the real-time body temperature deviates from its corresponding dynamic body temperature data baseline by more than a preset first threshold (e.g., 1.5℃), an "abnormal body temperature" signal or a low-grade fever signal is generated; if the deviation exceeds a preset second threshold (e.g., 3℃), an "excessively abnormal body temperature" signal or a high-grade fever signal is generated.

[0065] Optionally, abnormal signals generated by comparing real-time physiological data with the corresponding dynamic physiological data baseline can directly trigger event labeling and enhanced recording. For example, when body temperature data generates a "super-abnormal body temperature" signal, event labeling and enhanced recording can be directly triggered, just like heart rate data.

[0066] Technicians can adjust the window size or threshold according to the needs of the actual scenario.

[0067] Explanation regarding window size (i.e., length):

[0068] The duration of the window needs to be chosen in a trade-off between sensitivity and stability.

[0069] Too short a duration (e.g., 2 seconds): If the sampling frequency is 1Hz, the baseline is generated by aggregating two data points. Adding a sudden data point will cause the baseline to change too quickly, failing to keep up with real-time data fluctuations, resulting in a lack of differentiation from the baseline, thus leading to sluggish perception and a high risk of missed detections.

[0070] Conversely, if the duration is too long (e.g., 60 seconds): the baseline changes slowly and is easily triggered by instantaneous noise, resulting in false alarms.

[0071] Data simulations and field tests show that 5-30 seconds is a suitable choice, as it can smooth out noise during heartbeat intervals and respond promptly to continuous changes in condition.

[0072] Abnormal signals generated by real-time physiological data can also be used to cross-validate trigger events generated by real-time heart rate data: when heart rate data deviates from a suspected trigger event signal, the system calls motion data and skin conductance data for verification.

[0073] If the exercise intensity increases synchronously and the skin conductance remains stable → it is determined to be caused by normal physical activity (such as running), and the event labeling is not triggered;

[0074] If the exercise intensity is stable and the skin conductance increases → it is determined to be caused by emotional abnormality, triggering event labeling and enhanced recording.

[0075] (3) Identify the contextual information that needs to be included in the evidence: for example:

[0076] Location data shows that the user is on a highway (a high-risk area) → Strengthening the confidence of heart rate data deviation signals can lower the trigger threshold.

[0077] Location data shows that users are in school classrooms or at home (safe areas) → weakening the confidence of heart rate data deviation signals can increase the trigger threshold.

[0078] The ambient audio detected keywords such as "help" and "fire" → directly serving as evidence of the trigger signal.

[0079] The data in this cache is updated in real time and is only used for judgment at the current moment; it is not stored for a long time.

[0080] It should be noted that when cross-validating multimodal data or incorporating contextual information for identification, the multimodal data can be prioritized or ranked by weight. Multimodal data that may endanger the user's life (such as body temperature of 39°C or above, audio of "help" etc.) can be given higher priority or weight to ensure the timeliness and accuracy of evidence preservation.

[0081] b. Evidence data caching: Stores multimodal data for a preset duration (e.g., 48 hours). When storage space is insufficient, the oldest data is automatically overwritten by the newest data.

[0082] Data storage format: The evidence storage data cache adopts a fixed-period data frame structure. For example, each data frame contains the following in sequence:

[0083] Frame header (4 bytes): contains synchronization code and version number

[0084] Timestamp (4 bytes): Number of seconds since the epoch.

[0085] For example, the lower 15 bits of the event label code are used to identify the trigger source, and the highest bit is used to indicate whether it is a post-event label (1 indicates post-event addition, 0 indicates real-time labeling). The specific encoding method can be flexibly set by technical personnel according to the system design. For example:

[0086] 0x0001: Heart rate data deviation triggered

[0087] 0x0002: Triggered by user-initiated command

[0088] 0x0003: External warning signal triggered

[0089] 0x8001: Post-processing annotation (a 1 in the highest bit indicates post-processing; the specific encoding method can be set by technical personnel).

[0090] Checksum (2 bytes): CRC16 checksum

[0091] Heart rate data (2 bytes)

[0092] Satellite positioning data (12 bytes): 4 bytes each for longitude, latitude, and altitude.

[0093] Acceleration data (6 bytes): 2 bytes each for X, Y, and Z axes

[0094] Skin conductance data (2 bytes)

[0095] Audio / Video Data Index (Optional): If the terminal acquires audio or video data during enhanced recording, this data is stored in a separate file, and its storage location, start and end times, and other information are associated with the data frame through an index field.

[0096] Alternate fields (several)

[0097] It should be noted that high-volume data such as audio and video, after being acquired during the augmentation recording process, can be stored as independent files. Their index information is associated with periodic data frames to ensure timeline alignment and data integrity. This storage method optimizes storage space utilization while ensuring evidence traceability.

[0098] For post-event annotation, the system reads multimodal data for the user-specified time period from the evidence storage data cache, inserts a "post-event annotation" flag (e.g., setting the highest bit to 1) into the event annotation code of each data frame, recalculates the checksum, and then moves these data frames to a non-volatile evidence storage area (such as flash memory or the cloud) to ensure that they are not overwritten by loops. This preserves both the timestamp and content of the original data and clearly distinguishes the source of the annotation.

[0099] The data and data frame structures listed in this embodiment are exemplary, demonstrating the principle, and are not essential features of the technical solution.

[0100] Example 2: Triggering events generated by external alerts

[0101] (1) User-initiated commands

[0102] The terminal human-machine interface receives user annotation commands, including but not limited to: voice commands, key commands, and gesture commands.

[0103] (2) External warning signals

[0104] The terminal receives warning signals from external devices or systems through its communication modules (including short-range wireless modules and wide area network modules), including:

[0105] Warnings for IoT devices connected via short-range wireless modules (such as smoke detectors).

[0106] Warnings from public safety platforms connected to the WAN module (such as area control notices issued by the police).

[0107] The terminal parses metadata such as the effective time and valid area in the signal, and combines it with the current time and location to determine whether the activation conditions are met.

[0108] Example 3: Examples of the different roles of different types of data

[0109] (1) Heart rate data baseline - the main trigger source

[0110] Heart rate data reflects physiological stress levels, and the changes are gradual. Using a 10-second sliding window, the arithmetic mean of heart rate data is calculated as a dynamic baseline. When the relative change rate of real-time heart rate data from the baseline is >30%, a "suspected stress" signal is generated.

[0111] (2) Baseline of skin conductance – the main source of supporting evidence

[0112] Skin conductance response is sensitive to emotions but easily affected by noise. A median skin conductance was calculated using a 60-second sliding window as the dynamic baseline. When the absolute difference between real-time skin conductance and baseline > 5 μs, an "emotional fluctuation" signal was generated to corroborate whether abnormal heart rate data was caused by emotions.

[0113] (3) Exercise intensity baseline – auxiliary trigger / supporting source

[0114] The root mean square value within a 30-second window is calculated using the composite amplitude of triaxial acceleration as the baseline for exercise intensity. When the relative rate of change of real-time exercise intensity is >50%, an "abnormal exercise" signal is generated, which, in conjunction with the heart rate data signal, can distinguish between "exercise-induced increase in heart rate data" and "stress-induced increase in heart rate data".

[0115] (4) Other data – without establishing a baseline, directly used for event identification or context.

[0116] Motion posture: Records the user's action sequence (such as running, walking, standing still, posture changes) using gyroscope and acceleration time-series data as an objective record of the behavior process.

[0117] Satellite positioning: Establish "resident areas" by clustering historical locations to determine whether the location is unconventional.

[0118] Ambient audio: Identify abnormal sounds such as "calls for help," "impacts," and "arguments" through volume thresholds or spectrum matching.

[0119] Relative distance: Directly records the distance change curve between the target object and the target object.

[0120] Example 4: Multi-modal data collaborative decision-making logic

[0121] Scenario 1: Increased heart rate during exercise (not triggered)

[0122] User running: Heart rate increased from 70 bpm to 120 bpm (deviation +71%), exercise intensity increased simultaneously (deviation +120%), skin conductance remained stable (no emotional fluctuations). This was determined to be exercise-induced and not triggered.

[0123] Scenario 2: Sudden fright (triggered)

[0124] The user experienced a sudden fright: heart rate rose from 70 bpm to 130 bpm (deviation +86%), exercise intensity remained stable (at rest), and skin conductance increased (deviation +8 μS). This was determined to be a dual physiological-emotional abnormality, triggering the event.

[0125] Scenario 3: Post-event data verification – Did the user fall?

[0126] Suppose a user is accused of knocking down another person during a conflict, and the other person claims they also fell. The evidence data from this invention is retrieved afterward:

[0127] Motion data: The accelerometer shows that the user remained standing throughout, and no impact waveform was detected upon falling.

[0128] Gyroscope data: The user's attitude angle remained stable during the event, with no change in horizontal attitude after the fall;

[0129] Heart rate data: Although the heart rate data showed an increase (consistent with a state of tension), there was no dramatic fluctuation after the fall.

[0130] These three types of data corroborate each other, proving that the user did not fall, thus effectively refuting the other party's false accusations. This scenario demonstrates how the raw data recorded by this invention can be used to passively prove facts, rather than actively judge events.

[0131] Example 5: Application Scenario – Helping People and Preventing Accusations

[0132] When a user wearing a smartwatch integrated with the method of this invention encounters someone who has fallen and needs assistance, the user may fall into one of three situations:

[0133] For meticulous users: They proactively issue the voice command "Start recording". Upon receiving the command, the system immediately annotates the event and initiates enhanced recording.

[0134] A kind-hearted user hesitated due to excitement before offering assistance. During his agitation, a sudden change in his heart rate data triggered automatic detection, and the system automatically labeled the event and initiated enhanced recording.

[0135] Careless user: Calmly helps the person who has fallen without triggering any events. Afterwards, the user remembers that evidence is needed and marks the incident within 48 hours (i.e., the memory cycle overwrite period) by selecting the time "10:00-10:10 AM" and the location "a certain intersection" through the APP. The system then retrieves the basic data for that time period from the evidence storage cache and stores it as evidence.

[0136] Examples of using multimodal data in blackmail situations:

[0137] Assuming a user is accused of knocking down a person who has fallen, the data recorded in this invention can be logically self-verified from the following dimensions:

[0138] (1) Logical self-evidence in the spatial dimension: The location trajectory shows that the user approached the fallen person from 10 meters away 10 seconds before the incident, rather than staying next to the fallen person the whole time. The relative distance data records that the distance between the user and the fallen person gradually decreased from 8 meters to 0.5 meters, which is consistent with the physical law of "approaching" rather than "collision".

[0139] (2) Logical self-evidence of the behavioral dimension: Through acceleration and gyroscope time sequence analysis, the motion posture data identifies a complete action sequence of "deceleration-bending-squatting-standing up", which is completely consistent with the "helping" behavior, rather than the acceleration impact mode required for "collision".

[0140] (3) Logical self-justification from the physiological dimension: The heart rate data steadily increased from 75 bpm to 110 bpm when approaching the person who fell, which reflects exertion or tension, rather than the "panic heart rate data surge" (usually >140 bpm) that should occur after the impact. The skin conductance data increased synchronously, which corroborates that the emotional fluctuation was "tension" rather than "fear".

[0141] (4) Inevitable correlation on the time axis: The above four types of data are strictly aligned on the time axis—as the positioning distance shortens, the movement posture changes, and the heart rate data increases synchronously. This inherent consistency of multimodal data cannot be forged.

[0142] Compared to single video evidence, the multimodal data of this invention have an inherent physical consistency relationship (such as the bidirectional symmetry of distance data and the correlation between heart rate data and movement), making it difficult for forgers to simultaneously forge all modal data and maintain their inherent consistency, thereby significantly improving the anti-counterfeiting capability of the evidence.

[0143] (5) The essential difference from fake videos: A single video can be edited, taken out of context, or questioned as staged. However, the multimodal data recorded in this invention forms a logical closed loop—any accusation that attempts to refute one type of data will be refuted by the other types of data. For example, if the other party claims that the video is staged, then the heart rate data must show "calm" rather than "tense"—and the person staging the video cannot control their own heart rate data.

[0144] Example 6: Data Interaction with the Server and Privacy Protection

[0145] When the terminal detects a trigger event or when the user annotates the event afterward, it performs a SHA-256 hash calculation on the relevant data packet to generate a data fingerprint. The terminal uploads the fingerprint to the blockchain evidence storage platform to obtain an evidence storage certificate containing a timestamp. The original data can be optionally encrypted and stored locally on the terminal, or uploaded to the cloud (or mobile phone) for backup with user authorization.

[0146] It should be noted that the core functions of this invention (triggering, labeling, and local evidence storage) can be completed independently on the terminal, and the interaction with the server (or mobile phone) is only for enhancing functionality.

[0147] Explanation of server architecture:

[0148] like Figure 2 As shown, the server in this invention is an optional enhancement component used to provide cloud-based evidence storage, data analysis, and instruction distribution services. The server mainly includes the following modules:

[0149] Communication interface module: Establishes secure communication connections with terminals, external devices (such as smoke detectors, police platforms) and rescue platforms (such as 110 / 120), and is responsible for receiving and sending data.

[0150] The evidence storage module receives multimodal data with event annotations uploaded by the terminal, performs hash calculations (such as SHA-256) on the data packets, generates a data fingerprint, uploads it to the blockchain evidence storage platform, and returns an evidence storage certificate containing a timestamp. Simultaneously, the evidence storage module can encrypt and store the original data in the server database for retrieval after user authorization.

[0151] Command module: Based on preset rules or manual instructions, it sends warning signals (such as area control notices issued by the police), configuration update instructions, or data retrieval requests to terminals within a specific area. When issuing commands, geofencing technology can be used to ensure that the commands only apply to the relevant terminals.

[0152] Analysis module (optional): Performs spatiotemporal correlation analysis on aggregated multi-terminal data to generate regional risk situation maps or group behavior profiles, assisting decision-makers in emergency dispatch (see Examples 8 and 9). This module can integrate machine learning models to achieve risk prediction and early warning.

[0153] It should be noted that the various modules of the server can be flexibly configured according to actual deployment needs, such as using cloud servers, edge servers, local servers, or user mobile phones (which act as lightweight servers by installing corresponding software). The core functionality of this invention does not depend on the server; the server is only designed to enhance user experience and enable large-scale collaboration.

[0154] Example 7: Evidence Preservation of Bullying

[0155] When bullied, victims may not actively instruct the event to be marked due to fear or lack of time, but the terminal detects the following multimodal anomalies:

[0156] Because users are afraid, heart rate data spikes (e.g., from 75 bpm to 140 bpm, a deviation of +87%).

[0157] A surge in skin conductance (e.g., a deviation from +10 μS, reflecting fear-induced cold sweats)

[0158] The accelerometer detected violent limb movements (struggle).

[0159] The gyroscope detected body tremors (high-frequency micro-vibrations).

[0160] The system automatically triggers event annotation and starts enhanced logging.

[0161] The record includes:

[0162] Location data (proving the location of the incident)

[0163] Physiological data (heart rate data, skin conductance, demonstrating the victim's state of fear)

[0164] Motion data (accelerometer, gyroscope readings, demonstrating struggle and trembling)

[0165] Ambient audio (recording ambient sound)

[0166] If there is a camera, it can also record video (although it may be incomplete).

[0167] Afterwards, students or parents can use the app to annotate the event and extract data from the time period as evidence. This multimodal data corroborates each other: heart rate data and skin conductance data prove the victim's state of fear at the time, and motion data proves the struggle process, forming a complete and irrefutable chain of evidence.

[0168] It should be noted that the duration of the enhancement record is related to the state of the triggering event. As long as the triggering event continues to meet the activation conditions, the enhancement record will continue until the user actively stops it, the triggering event no longer meets the activation conditions, or the enhancement record duration reaches the preset maximum duration.

[0169] Example 8: Two-way evidence preservation at the scene of a conflict

[0170] In physical altercations that occur in public places, if both parties are wearing terminals integrated with the method of this invention, a complete two-way chain of evidence can be formed.

[0171] Data Acquisition: Both terminals synchronously record the following data at 0.5-second intervals:

[0172] Absolute location data (to prove the location of the incident)

[0173] The relative distance data with the other party is used to adaptively select the measurement method based on the real-time distance: when both parties are within the effective range of short-range wireless communication, UWB / star-flash two-way ranging is preferred to obtain centimeter-level accuracy; when the distance exceeds the short-range range, satellite positioning coordinates are exchanged through a wide area network to calculate the relative distance and azimuth.

[0174] Motion data (accelerometer, gyroscope, recording of motion sequences)

[0175] Physiological data (heart rate data, skin conductance, reflecting emotional state)

[0176] Ambient audio (recording ambient sound)

[0177] Construction of the chain of evidence:

[0178] Spatial relationship evidence: Relative distance data shows that A rapidly approached B from 3 meters away 3 seconds before the conflict, while B's position remained essentially unchanged. This proves that A was the one who actively approached, and B was the one who passively retreated.

[0179] Behavioral evidence: A's motion data shows continuous forward lunging and arm swinging movements, while B's motion data shows backward movement, arm bending, and at a certain moment, a falling impact waveform. This proves that A committed an aggressive act, B adopted a defensive posture, and B did indeed fall to the ground.

[0180] Physiological evidence: A's heart rate was elevated but skin conductance remained stable, consistent with the agitated state during an attack; B's heart rate spiked and skin conductance increased, consistent with a state of fear. This corroborates the different psychological states of the two individuals.

[0181] Audio evidence: Environmental audio recordings of B's ​​cries for help and A's shouts further confirm the nature of the conflict.

[0182] Mutual verification of data:

[0183] The distance between terminal A and terminal B recorded by terminal A and the distance between terminal B and terminal A should be strictly equal within the measurement error range. Any significant inconsistency may indicate that the data has been tampered with. This two-way verification mechanism makes the forgery of evidence virtually impossible.

[0184] Retrospective analysis:

[0185] Police can access data from both parties' terminals and replay the entire conflict on a unified timeline: when A began to approach, when A threw the first punch, when B retreated, when B fell to the ground, and when B called for help. The entire process is like a "black box" with two perspectives, leaving no room for argument.

[0186] This embodiment demonstrates the immense value of the present invention in resolving social conflicts and disputes—when everyone has an objective recording tool, the facts will no longer depend on "who speaks the loudest," but on "whose data is clearest."

[0187] Example 9: Intelligent Response to Public Safety Alerts

[0188] The user's terminal received a warning signal from the police: "Temporary traffic control will be implemented in a certain residential area starting at 23:00." The terminal then analyzed the effective time and affected area.

[0189] Scenario A (Not Activated): At 20:20, the user is not in this cell, and the signal is only used for notification.

[0190] Scenario B (Activated): At 23:05, the user walks to the entrance of the community. The signal meets the activation conditions, triggering event labeling and starting enhanced recording to record whether the user has entered the restricted area.

[0191] Scenario C: User is unconscious in a dangerous area

[0192] A user fell unconscious at home due to a gas leak. The terminal received a "gas leak" warning signal from the IoT platform. The signal activation conditions were met (the user was at home), and the system initiated enhanced recording. Because the user was unconscious and unable to operate the system, and the signal remained valid, enhanced recording continued. Two hours later, the gas leak was resolved, the signal was deactivated, and the system automatically stopped enhanced recording. Afterwards, the user's family can retrieve the data from this period as evidence for accident investigation and claims.

[0193] Example 10: Time-Axis-Based Data Profiling—Intoxicated Individuals Proving Their Innocence

[0194] A fire broke out in a shop in a certain city. Some people reported seeing a man staggering past the shop before the fire started, and others confirmed that the intoxicated man left the vicinity of the shop after the fire. The shop owner also told the police that the intoxicated man had argued with him and threatened to "burn down your shop." When the police found the man, now sober, they discovered a lighter in his pocket and that his account of his actions before and after the fire was incoherent and unclear.

[0195] Thinking of the smartwatch he was wearing (the terminal), the man immediately annotated the multimodal data stored on it, extending the time range to the time before he left the hotel and arrived at the burning shop, so that the data before and after he passed by the shop was saved to his mobile phone or cloud server.

[0196] Then, in front of the police, he imported the multimodal data labeled with the incident into the computer, displaying his walking trajectory and emotional change curves on a timeline, thus clearing his name. During the data import, the system simultaneously displayed the hash value of the data, which could be compared with blockchain-based evidence to verify that the data had not been tampered with.

[0197] He also discovered that at 11:25 p.m. last night, after passing the store entrance and moving 10 meters away, he stumbled and hit his head on the wall, fell to the ground, and fell asleep next to the wall. The smartwatch automatically marked and enhanced the recording of the event because of his head hitting the wall and falling, and the data was also saved to his mobile phone or cloud server.

[0198] These data reveal everything:

[0199] First paragraph: Passing by and an accident

[0200] At 23:20, Zhang passed by the shop entrance; location data (obtained by the satellite positioning module) showed he remained outside and did not enter. At 23:25, he accidentally fell at a corner 10 meters away from the shop, hitting his head on the wall—the severe pain caused his heart rate to spike from 95 bpm to 130 bpm instantly, with a simultaneous increase in skin conductance. This anomaly triggered automatic event labeling and enhanced logging, initiating data acquisition at a higher frequency.

[0201] Part Two: Sleep and Fire

[0202] After falling, Zhang fell into a deep sleep, his heart rate gradually dropped to 65 bpm, his skin conductance remained stable, and his posture remained horizontal for 75 minutes. The fire department determined that the fire started between 11:30 PM and 11:40 PM, which falls within this timeframe—it is impossible for a person with a stable heart rate and a still body to commit arson.

[0203] Third paragraph: Waking up and supplementary notes

[0204] At 00:40, Zhang woke up and saw a fire in the distance. He staggered away.

[0205] Data closed loop:

[0206] Passing by a store (23:00-23:25 tracking) → Not entered (location)

[0207] Severe pain from a fall (heart rate spikes + increased skin conductance, triggering annotation) → Sleep (stable heart rate, no activity) → Fire period (no data fluctuation)

[0208] Wake up and leave (00:40 motion data) → Pass by again (trajectory)

[0209] All the data corroborated each other, fully revealing the facts: Zhang was merely passing by while intoxicated, accidentally fell, and fell asleep. When he awoke, the fire was already raging. He himself couldn't explain it, but the data remembered everything. The case was dropped, and his innocence was proven.

[0210] The value of this embodiment:

[0211] The pain from the fall triggered automatic recording, preserving critical data from the dormant period;

[0212] Subsequent annotations saved the "passing by" trajectory from being overwritten, forming a complete chain of evidence;

[0213] Multimodal data (location, heart rate data, skin conductance, and motion) corroborate each other, making the truth irrefutable.

[0214] When a person is unable to testify due to intoxication, illness, or memory loss, data becomes the best witness.

[0215] It should be noted that the data supplemented after the fact comes from the basic records, and its sampling frequency and data type may be lower than those of the augmented records annotated in real time (for example, it may lack high-volume information such as audio and video). However, the basic records still contain core data such as location, heart rate, and movement, which is sufficient to plot the user's movement trajectory and emotional change curves, providing crucial support for reconstructing the facts. The two annotation methods complement each other, jointly ensuring the integrity and usability of the evidence.

[0216] About computer-readable storage media

[0217] The present invention also provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, it is used to implement any of the methods described above. The physical form of the storage medium includes, but is not limited to, any carrier capable of storing and running programs, such as memory built into a terminal or server (e.g., ROM, RAM, Flash), mobile storage devices (e.g., USB flash drives, portable hard drives), and network storage space.

[0218] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the design concept of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for storing behavioral process evidence based on multimodal data, characterized in that, Includes the following steps: Acquire multimodal data, wherein the multimodal data includes at least user heart rate data; A dynamic physiological data baseline is constructed using at least the heart rate data from the multimodal data, and the dynamic physiological data baseline is used to characterize the data characteristics of the user in a normal state; Monitor at least one of the following triggering events: a. The deviation of one or more real-time physiological data points from their respective corresponding dynamic physiological data baselines exceeds a preset threshold. b. Receive annotation instructions from the user through the human-computer interaction interface; c. A warning signal is received from an external device or system, and the warning signal meets the preset activation conditions; When any of the aforementioned triggering events is detected, the multimodal data being recorded is labeled with the event.

2. The method according to claim 1, characterized in that, The multimodal data also includes at least one of the following data types: location data, motion data, environmental data, and relative spatial relationship data with the target object.

3. The method according to claim 1, characterized in that, When a physiological data point is detected to trigger event a, other multimodal data are further invoked for cross-validation to improve the confidence of the triggering condition.

4. The method according to claim 1, characterized in that, The warning signal meets preset activation conditions, including at least one of the following: The effective time range specified in the warning signal covers the current time. The geographical area specified in the warning signal matches the user's current location; The warning signal has been authorized and confirmed by the user.

5. The method according to claim 1, characterized in that, The event annotations include: Real-time annotation: When the triggering event is detected, an event annotation label is inserted into each recorded data entry; Post-event annotation: Before the multimodal data stored in the circular storage is overwritten, users can add event annotation tags to the data in the cache by time or geographical location; Multimodal data that has been annotated with events is sent to a storage area outside the cache for evidence preservation.

6. The method according to claim 1, characterized in that, A two-level recording strategy is adopted: Basic recording: When no triggering event is detected, the multimodal data is acquired and stored cyclically at a basic sampling frequency; Enhanced recording: After a triggering event is detected and the event is labeled, the sampling frequency of the multimodal data can be increased and / or more types of data can be acquired.

7. The method according to claim 1, characterized in that, It also includes an adaptive stopping step: during the enhancement recording process, the enhancement recording is automatically stopped and the system reverts to the base recording when any of the following conditions are met: Received a stop command from the user via the human-computer interaction interface; The triggering event no longer meets the preset activation conditions; The enhanced recording duration reaches the preset maximum duration.

8. A terminal for implementing the method according to any one of claims 1-7, characterized in that, include: The data acquisition module is used to acquire the multimodal data; Memory, used to store computer programs and data; A processor for executing the computer program to implement the steps of the method; The communication module is used for data interaction with external devices or systems; The human-computer interaction interface is used to receive annotation instructions from users.

9. A behavioral process evidence storage system based on multimodal data, characterized in that, It includes multiple terminals as described in claim 8, and a server; the server is configured to communicate with the terminals, receive and store the data stored by the terminals, or issue warning signals and instructions to the terminals.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

Images