A storage method, device, equipment, storage medium and computer program product of multi-modal data

Abstract

The application discloses a kind of multi-modal data storage method, device, equipment, storage medium and computer program product, belong to data storage field, including: synchronous acquisition to the first time sequence of being collected object in first modality and the second time sequence in second modality;According to the importance degree of event occurring at each time of first time sequence, determine the data transfer accuracy of each time of second time sequence;In the case where the original storage accuracy of data item in second time sequence is higher than the data transfer accuracy of corresponding time, data item is compressed processing.The application associates the time series data storage strategy with the event importance distribution, so that the data storage allocation is more consistent with the importance value structure of the event occurring, achieving dynamic accuracy allocation for long time series data, maintaining the overall use value of the data without being weakened while improving the effective utilization of limited data storage space, balancing data capacity and data quality.

Description

Technical Field

[0001] This application belongs to the field of data storage, and specifically relates to a method, apparatus, device, storage medium, and computer program product for storing multimodal data. Background Technology

[0002] In various systems that require continuous recording or long-term storage of time-series data, such as security monitoring records, environmental monitoring, equipment status tracking, or behavior analysis, continuous monitoring data is usually generated over a long period of time, and this data needs to be stored in storage media for an extended period of time to meet subsequent usage requirements.

[0003] To meet the aforementioned long-term storage requirements, existing technologies typically employ two types of data storage methods: one method preserves data in a lossless manner; the other method uses lossy compression to reduce the data's footprint on the storage medium.

[0004] However, due to the need for continuous storage in certain scenarios, lossless storage would consume a large amount of storage resources; at the same time, lossy storage strategies would lead to a decrease in data precision, thereby affecting the subsequent use value of the data. Summary of the Invention

[0005] This application aims to provide a method, apparatus, device, storage medium, and computer program product for storing multimodal data, at least to solve the problem that existing data storage methods cannot simultaneously address data capacity and data quality.

[0006] In a first aspect, embodiments of this application disclose a method for storing multimodal data, including: The first time-series sequence of the object under the first mode and the second time-series sequence under the second mode are acquired simultaneously. The data transfer accuracy of the second time series is determined based on the importance of events occurring at each time point in the first time series; the data transfer accuracy is positively correlated with the importance level. If the original storage precision of a data item in the second time sequence is higher than the data transfer precision at the corresponding time, the data item is compressed.

[0007] Secondly, embodiments of this application also disclose a storage device for multimodal data, comprising: The acquisition module is used to simultaneously acquire the first time-series sequence of the acquired object in the first mode and the second time-series sequence in the second mode; The tagging module is used to determine the data transfer accuracy of each moment in the second time series based on the importance of events occurring at each moment in the first time series; the data transfer accuracy is positively correlated with the importance level. The conversion module is used to compress the data item when the original storage precision of the data item in the second time sequence is higher than the data transfer precision at the corresponding time.

[0008] Thirdly, embodiments of this application also disclose an electronic device, including a processor and a memory, wherein the memory stores a program or instructions that can run on the processor, and the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0009] Fourthly, embodiments of this application also disclose a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the method described in the first aspect.

[0010] Fifthly, embodiments of this application also disclose a computer program product, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps described in the first aspect.

[0011] In summary, in this embodiment, by synchronously collecting the first and second time-series sequences of the collected objects in their respective modalities, it is ensured that the data sources for subsequent data processing have a common time and event benchmark. Then, based on the importance of events occurring at each moment in the first time-series sequence, differentiated value representations are established for different times, and different data transfer accuracies are established under different importance levels to distinguish the different storage values ​​of data at each moment, thereby achieving differentiated configuration of storage accuracy. On this basis, based on the different importance levels at each moment, data items in the second time-series sequence with storage accuracy higher than transfer accuracy are compressed, so that different data contents are saved according to their matching content value, thereby reducing the data occupation of low-value data contents. While reducing the high resource occupation pressure brought by completely lossless storage, it also avoids the overall data quality degradation caused by completely lossy storage. Therefore, the method based on the embodiments of this application, by associating the storage strategy of time series data with the importance distribution of events, makes the data storage allocation more in line with the importance value structure of the events, realizes dynamic precision allocation of long-term time series data, and improves the effective utilization of limited data storage space while maintaining the overall subsequent use value of the data, thus improving the contradiction between data capacity and data quality. Attached Figure Description

[0012] In the attached diagram: Figure 1 This is a flowchart illustrating the steps of a method for storing multimodal data according to an embodiment of this application; Figure 2This is a flowchart illustrating the steps of another method for storing multimodal data provided in an embodiment of this application; Figure 3 This is a schematic diagram of a time window-based label value adjustment method according to an embodiment of this application; Figure 4 This is a program logic for audio and video processing provided in an embodiment of this application; Figure 5 This is a block diagram of a multimodal data storage device provided in an embodiment of this application; Figure 6 This is a block diagram of an electronic device provided in one embodiment of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0014] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, the "and / or" signifies at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects have an "or" relationship.

[0015] In existing technologies, it is common to simultaneously acquire data streams of different modalities, such as audio and video, environmental sensors and image acquisition, and behavior recording and spatial positioning. Although these modalities are synchronous in time, the amount of information they carry varies greatly in different time segments: some moments may contain sudden events, significant changes, or key content, while other moments may only contain background information or stable states. Existing technologies typically use a uniform storage precision for all modalities, which leads to high-information-content segments and low-information-content segments occupying the same storage resources. Consequently, in long-term acquisition scenarios, a large amount of redundant data is written, resulting in low overall storage efficiency.

[0016] From the perspective of data processing, the key to the problem lies in that the system lacks a mechanism capable of expressing "content value differences" in the time dimension, and also lacks a linkage mode that enables different modalities to automatically adjust storage precision according to the value differences. In other words, although multimodal data are synchronized in time, the storage strategy cannot be adjusted with content changes.

[0017] Therefore, in the present application, an "importance label" is introduced as the value expression structure of time-series data. By identifying content changes of different time segments in a first modality, quantifiable label values are assigned to each moment, so that the labels form a continuous value distribution on the time axis; then the label values are used to drive the storage precision selection of a second modality, enabling the second modality to automatically adjust the storage strategy according to the content changes of the first modality, thereby realizing cross-modal dynamic precision allocation.

[0018] Based on the above idea, as Figure 1 shows, it is a storage method for multimodal data provided by an embodiment of the present application.

[0019] The method may include the following steps: Step 101: synchronously collect a first time-series sequence of a collected object in a first modality and a second time-series sequence of the collected object in a second modality.

[0020] In some embodiments of the present application, in order to enable subsequent importance determination and data to be performed on a unified time basis, it is necessary to synchronously collect the first time-series sequence used for importance judgment and the second time-series sequence used as a compression object. Specifically, it is necessary to ensure that the time-series sequences in respective modalities maintain a one-to-one correspondence on the time axis. By enabling the same time segment to have matchable records in both types of data, the two parties are ensured to share the same event fact background for the same collected object. Through completing the data collection process in this synchronous manner, the subsequent process can generate importance evaluation based on the first time-series sequence, thereby ensuring that no fact dislocation occurs in the compression of the second time-series sequence, and finally ensuring a reasonable and effective correlation between importance judgment and storage amount adjustment.

[0021] In a specific example, during monitoring and collection, a security monitoring system records continuous audio data as the first time-series sequence, and synchronously generates continuous video frames as the second time-series sequence. Audio and video are recorded according to a unified time axis during collection, so that each audio segment can form a synchronous relationship with the video frame at the corresponding moment. After such processing, the importance degree generated subsequently based on audio data can be mapped to the corresponding video frame at the same time point, and used to adjust the storage precision of the video frame, thereby realizing the linkage between label judgment and data compression on the premise of ensuring time synchronization.

[0022] Step 102: Determine the data transfer accuracy of each moment in the second time series based on the importance of the events occurring at each moment in the first time series.

[0023] Among them, the accuracy of data transfer is positively correlated with the degree of importance.

[0024] In some embodiments of this application, in order to differentiate the storage of the second time series based on its content value, the data transfer precision at each moment needs to be determined according to the importance of the first time series at each moment. Specifically, when processing the first time series, the system reads the importance tag for each moment and directly maps the tag to the data transfer precision of the second time series at the corresponding moment, so that the second time series can obtain a precision setting that matches its importance level during storage. Since the data transfer precision is positively correlated with the importance level, moments with higher importance level correspond to higher storage precision, while moments with lower importance level correspond to lower storage precision, thus forming a precision distribution of the second time series that changes with content over time.

[0025] In a specific example, the system acquires a continuous audio stream as the first temporal sequence and a continuous video frame as the second temporal sequence during the acquisition process. The audio stream has already been labeled with different importance levels in previous steps; for example, silent segments correspond to lower importance, while segments containing significant acoustic changes correspond to higher importance. The system then sets the data transfer precision of the corresponding video frames to different precision levels based on these importance levels, ensuring that high-importance segments are stored with higher precision, while low-importance segments are stored with lower precision. Through this processing, the final output second temporal sequence maintains high quality in critical segments while reducing storage usage for ordinary segments, effectively controlling the overall storage size.

[0026] Step 103: If the original storage precision of the data item in the second time sequence is higher than the data transfer precision at the corresponding time, the data item is compressed.

[0027] In some embodiments of this application, after determining the data transfer precision at each moment of the second time series, the system compares the original storage precision of each data item moment by moment. When a data item at a certain moment originally had a storage specification higher than the data transfer precision required at that moment, the system performs compression processing on the data item to reduce its storage precision to a level that matches the data transfer precision. By processing high-precision data items in this way, the second time series can present a storage precision distribution consistent with its content value across different time segments, thereby reducing unnecessary high-precision usage in low-value segments.

[0028] In a specific example, the system acquires consecutive video frames as a second time-series sequence during the acquisition process. Video frames typically have a uniform initial storage precision during acquisition, while in previous steps, different data transfer precisions were assigned to each moment based on the importance of the first time-series sequence. The system then compares the initial storage precision of each video frame with its corresponding data transfer precision: when the initial storage precision of a segment is higher than the required transfer precision, the system compresses the video frames of that segment to reduce its storage precision to the same level as the transfer precision; if the initial storage precision is no higher than the transfer precision, no processing is required for that segment. Through this processing, the final output second time-series sequence maintains the necessary storage precision in critical segments while reducing storage usage in ordinary segments, effectively controlling the overall storage size.

[0029] In summary, in this embodiment, by synchronously collecting the first and second time-series sequences of the collected objects in their respective modalities, it is ensured that the data sources for subsequent data processing have a common time and event benchmark. Then, based on the importance of events occurring at each moment in the first time-series sequence, differentiated value representations are established for different times, and different data transfer accuracies are established under different importance levels to distinguish the different storage values ​​of data at each moment, thereby achieving differentiated configuration of storage accuracy. On this basis, based on the different importance levels at each moment, data items in the second time-series sequence with storage accuracy higher than transfer accuracy are compressed, so that different data contents are saved according to their matching content value, thereby reducing the data occupation of low-value data contents. While reducing the high resource occupation pressure brought by completely lossless storage, it also avoids the overall data quality degradation caused by completely lossy storage. Therefore, the method based on the embodiments of this application, by associating the storage strategy of time series data with the importance distribution of events, makes the data storage allocation more in line with the importance value structure of the events, realizes dynamic precision allocation of long-term time series data, and improves the effective utilization of limited data storage space while maintaining the overall subsequent use value of the data, thus improving the contradiction between data capacity and data quality.

[0030] Figure 2 This is another method for storing multimodal data provided in the embodiments of this application.

[0031] The method may include the following steps: Step 201: Synchronously acquire the first time sequence of the acquired object in the first mode and the second time sequence in the second mode.

[0032] The method shown in this step has been explained in step 101 and will not be repeated here.

[0033] Step 202: Determine the data transfer accuracy of each moment in the second time series based on the importance of the events occurring at each moment in the first time series.

[0034] Among them, the accuracy of data transfer is positively correlated with the degree of importance.

[0035] The method shown in this step has been explained in step 102 and will not be repeated here.

[0036] Optionally, step 202 includes the following sub-steps: Sub-step 2021: Initialize the importance of each time step in the first time series.

[0037] In some embodiments of this application, in order to enable subsequent modifications and diffusion of the importance level of the first time series on a unified data basis, it is necessary to first set initial label values ​​for the importance labels of each time point. Specifically, when processing the first time series, a corresponding importance label is established for each time point in the series, so that these label items have basic values ​​that can be used for subsequent adjustments in the initial state. By completing the initial necessity and sufficiency setting, subsequent modifications to the importance level of each target time point and the diffusion of the importance level to other times can be carried out on a unified data type basis.

[0038] In a specific example, when a monitoring system acquires an audio stream, it divides the continuous audio data into multiple time points in chronological order. Since these time points initially contain only raw acoustic data and lack importance labels for subsequent processing, the system first performs filtering, noise reduction, and signal enhancement on the audio stream during the initialization phase to improve the accuracy of subsequent event detection. Then, it establishes a corresponding importance label for each time point, giving it a unified data structure. For example, for segments that haven't recorded any event information, the system assigns them a preset initial label value, allowing them to participate in the overall importance judgment during subsequent label modification and propagation. For segments that already have complete label information, their original label values ​​are directly retained as the basis for the initialized labels. Through this process, the entire first time-series sequence forms a complete and unified set of importance labels in the initial stage, providing a consistent data foundation for subsequent label enhancement, label propagation, and accuracy correlation.

[0039] Optionally, sub-step 2021 includes the following sub-steps: Sub-step 20211: If the importance level at any time in the first time series is not set, set the unset importance level to the preset baseline importance level.

[0040] In some embodiments of this application, to ensure that the first time series has a complete initial state in the labeling system, it is necessary to supplement the importance information of these moments when there are pure data fragments or incomplete importance information in the sequence. When processing the first time series, the importance information of each moment is checked to see if it has a label value. For moments that only record raw data or contain only partial event information without forming importance information, a preset baseline importance level is assigned to that moment, giving it a basic value in the labeling system that can participate in subsequent modifications and dissemination. By completing this supplementation process, the entire time series forms a complete initial information set in the labeling system, thereby ensuring that subsequent information modifications and dissemination can be carried out on a unified data basis.

[0041] In a specific example, when a monitoring system acquires audio, some segments may only contain raw acoustic data without any event labels; others may only record partial event labels, such as detecting movement changes or environmental quiet, but without numerical labels indicating their importance. For these segments, a preset baseline importance level will be assigned to their importance labels, giving them an initial data format consistent with other segments within the labeling system. After this processing, the entire audio sequence can be processed in subsequent steps based on its complete data format.

[0042] Sub-step 20212: If the importance of each time point in the first time series has been set, retain the importance of each time point in the first time series.

[0043] In some embodiments of this application, when each moment in the first time series already has complete importance label values, it is unnecessary to supplement or assign baseline label values ​​to these moments. In this case, the raw data can be directly used as the initialization result. This allows subsequent label modification and dissemination to be based on the actual labels, thereby maintaining the correspondence between the label system and the actual collected content.

[0044] In a specific example, the monitoring system directly acquires a set of collected audio clips, which have already generated complete importance labels based on sensor detection results. For example, a clip is assigned higher importance because it detects people speaking, and lower importance because it is quiet in the environment. Since these clips are already complete and accurate, there is no need to supplement or reset them. Therefore, these original importance labels are directly used as the initialization results, serving as the basis for further adjustments and diffusion in subsequent steps, and providing a true reference for subsequent data compression.

[0045] Sub-step 2022: determining, among each time instant of the first time sequence, a first target time instant corresponding to each target occurrence event, and modifying the importance degree of the corresponding first target time instant to be greater than or equal to the importance degree of the remaining time instants according to the event complexity respectively represented by each target occurrence event.

[0046] In some embodiments of the present application, in order that the first time sequence can reflect the difference in content value in the importance system, it is necessary to identify the time corresponding to the target occurrence event from the occurrence events expressed at each time in the first time sequence, and improve the importance degree of the corresponding time according to the event complexity of the target occurrence event. In this process, the corresponding first target time can be determined according to the occurrence time of each target occurrence event, and this time is provided with an importance label matching the event complexity in the importance system. A target occurrence event is generally used to reflect a specific content change occurring at a certain time or time segment, for example, a movement or sound detected by a sensor, or other abnormal signals appearing in a scene. By adjusting the importance degree of the first target time, the time can reflect the corresponding content value in the subsequent precision allocation, thereby providing a clear reference for the subsequent label adjustment of other times.

[0047] In a specific example, when processing an audio stream serving as the first time sequence, the system determines whether a certain time segment belongs to different event types such as silence, ordinary noise, human voice or intense sound according to the audio energy, the voice activity detection (VAD) result and the distinguishing ability of a lightweight model between human voice and noise for each time segment, so as to determine the occurrence event situation at each time of the segment, and further determine whether the target occurrence event occurs at each time. The system predefines a plurality of information complexity levels, for example: L0 Silence level: the audio energy is lower than a threshold T1; L1 Noise level: T1 ≤ energy < T2, and no human voice is detected; L2 Human voice level: stable human voice features are detected, and at this time, it can be set that even if the energy does not reach T2, the level is directly promoted to this level; L3 Intense Sound Level: Energy ≥ T3 or significant energy spike (used to mark sudden, abnormal, or high-intensity events). If the audio energy of a segment is significantly higher than the silence threshold T1, but still lower than the human voice detection threshold T2, and the model does not detect human voice, the segment will be classified as L1 noise level. When the model detects stable human voice features, even if the energy does not reach T2, the segment will be directly upgraded to L2 human voice level. When a segment experiences an energy spike or its energy exceeds a higher threshold T3, that time point will be considered an intense sound event, corresponding to level L3. L1, L2, and L3 in the above levels all belong to target occurrence events. When the system identifies these events, it will determine the corresponding time point as the first target moment and increase the importance of that moment according to the event level, so that it reflects a value expression in the labeling system that matches the complexity of the event. For example, L2 human voice level corresponds to a higher label value, while L3 intense sound level corresponds to an even higher label value. By enhancing the importance of the first target moment, this moment becomes a clear reference in subsequent tag diffusion and video precision allocation, thereby enabling the overall importance distribution to accurately reflect the importance of events in the audio.

[0048] Optionally, the first time sequence is audio stream data, and sub-step 2022 includes the following sub-steps: Sub-step 20221: Identify noise and / or human voices in the audio stream data, and determine each time corresponding to the identified noise and / or human voices as a first target time.

[0049] In some embodiments of this application, when processing audio stream data as a first time-series sequence, the system performs time-by-time analysis on consecutive audio segments to identify whether there are acoustic changes of business significance, such as noise or human voices. Specifically, the system performs energy detection, frequency domain feature extraction, or speech activity detection on each moment of the audio stream, enabling segments that reflect acoustic changes to be accurately identified on the time axis. When the system detects noise or human voices at a certain moment, it designates that moment as a first target moment, giving it a clear event marker in the subsequent importance label adjustment process. By identifying acoustic events and determining the corresponding first target moments in this way, the first time-series sequence can form an event distribution consistent with actual acoustic changes in the time dimension, providing a reliable basis for subsequent importance setting.

[0050] In a specific example, when acquiring the audio stream, the system performs speech activity detection on each time segment to determine whether the segment contains stable human voice features; it also identifies ordinary environmental noise through energy change detection. When significant noise or human voice is detected in a segment, the system marks the corresponding time point as the first target time, allowing it to be used as a basis for setting importance labels in subsequent steps. Through this processing, the audio stream forms a set of target times on the timeline that reflects the occurrence of noise or human voice, providing a clear event basis for subsequent importance adjustments.

[0051] Sub-step 20222: The importance of each first target moment is determined by the sound complexity indicated by the noise and / or human voice corresponding to each first target moment.

[0052] Wherein, the sound complexity indicated by noise and / or human voice is greater than or equal to the sound complexity indicated by quiet conditions.

[0053] In some embodiments of this application, after identifying each first target moment in the audio stream, the system assigns an importance level to each moment based on its corresponding acoustic content, matching the actual acoustic changes. Specifically, the system assigns a higher label value than quiet segments to each moment based on the acoustic characteristics of noise or human voice at that moment, enabling it to reflect corresponding content value within the labeling system. For example, when stable human voice characteristics are detected at a moment, the system sets the importance level of that moment to be higher than that of ordinary noise; when significant environmental noise is detected at a moment, the system sets the importance level of that moment to be higher than that of silent segments. By assigning importance levels based on acoustic content in this way, the first target moment possesses a value expression consistent with its acoustic changes within the labeling system, providing a clear basis for subsequent labeling processing.

[0054] In a specific example, when acquiring the audio stream, the system performs speech activity detection on each time segment to determine whether the segment contains stable human voice features; it also identifies ordinary environmental noise through energy change detection. When a human voice is detected in a segment, the system assigns a higher importance level to the corresponding time point than to noise segments; conversely, when only ordinary noise is detected in a segment, its importance level is assigned a higher level than to silent segments. Through this processing, the audio stream forms a set of importance labels on the timeline that reflect the intensity of acoustic changes, enabling the labeling system to accurately express the content value of different segments in subsequent steps.

[0055] Sub-step 2023: Increase the importance of the second target time in the first time series.

[0056] Among them, the second target time and the first target time satisfy a preset event correlation relationship.

[0057] In some embodiments of this application, considering that events are usually not sudden but have a certain event context, in order to make the time corresponding to the event context also have a certain expressive power of importance, the importance of the second target time that meets the preset event correlation relationship with the first target time is also increased, so that the labeling of the target event has more effective coherence. By adjusting the importance of relevant times, the importance system can form a continuous value distribution in the time dimension, thereby making the subsequent precision allocation more coherent and perfect.

[0058] In a specific example, the system has already identified a time segment of audio corresponding to an event of high importance in previous steps. For instance, the system uses energy detection, voice activity detection (VAD), and a lightweight human voice model to determine that stable human voice features appear in the segment. Therefore, the time point corresponding to this segment is marked as the first target moment, and its importance label value is raised to L2 human voice level. After determining the first target moment, the system uses the label level of the first target moment as a reference to select a second target moment in its vicinity, such as the process of a relevant person entering the scene before a speech occurs. At this time, although the actual situation may correspond to L0 silence level, that is, there is no actual sound, because it involves the beginning of a speech event, its original L0 level can be raised to L1 or higher. In this way, the segments near the target moment will form an importance distribution that is correlated with the importance level of the first target moment, so that the importance system presents a continuous value change on the time axis, rather than an abrupt jump at the target moment. This continuous label distribution will provide clearer event information for subsequent video precision allocation, enabling the system to maintain higher video quality before and after the event.

[0059] Optionally, sub-step 2023 includes the following sub-steps: Sub-step 20231: Establish the first time window and the second time window for time-by-time advancement.

[0060] The first time window covers the corresponding window time, and the second time window covers the first time window.

[0061] In some embodiments of this application, considering that the context of an event is usually temporally adjacent to the event itself, a first time window and a second time window that advance over time are needed to achieve time-by-time importance adjustment in the first time series. When processing the first time series, a first time window covering the current inspection time is constructed based on the current inspection time, and a second time window with a larger coverage area is constructed outside of it. The first time window is used to focus on evaluating the label values ​​of the current time and its neighboring times to determine whether it is necessary to adjust the importance of nearby times, while the second time window is used to provide the specific range of adjustment. As the inspection time advances by a preset step size, the two time windows move synchronously, thereby forming a nested window structure that advances and checks adjustments time by time, providing a continuous time interval for subsequent label value adjustments.

[0062] In a specific example, such as Figure 3 As shown, multiple time points are sequentially distributed along the time axis…Ti 2N, Ti N, Ti, Ti+N, Ti+2N… etc. At the current inspection time Ti, a first time window is established, with a range of [Ti…]. The window [N, Ti+N] covers the current inspection time Ti. Simultaneously, a second time window is created outside this window, with a range of [Ti]. [2N, Ti+2N], this window covers the first time window. As the inspection time progresses from Ti to Ti+N, the two windows slide forward synchronously, ensuring that the first time window always covers the current inspection time and the second time window always covers the first time window, thus forming a window structure that progresses step by step, laying a continuous temporal foundation for subsequent label value adjustments.

[0063] Sub-step 20232: If the first time window contains at least one first target time, then in the second time window, each time corresponding to an importance level less than the first importance threshold is determined as a second target time.

[0064] In some embodiments of this application, in order to ensure that the label values ​​form a clear local upward trend around the first target time, it is necessary to determine whether each time in the second time window needs to undergo subsequent boosting operations based on the importance distribution of the first target time included in the first time window. Therefore, the existence of the first target time within the first time window is first determined; then, based on this existence, a second target time that meets the modification conditions is selected in the second time window. The selection of the second target time can usually be done by setting an importance threshold, for example, using a value below a certain specific importance level as a condition for filtering. This will enable the label values ​​in the second time window to form an enhancement interval centered on the first target time in the time dimension, thereby ensuring that the label system exhibits a stable upward trend in a local range, laying the foundation for subsequent label diffusion and overall smoothing.

[0065] In a specific example, such as Figure 3 As shown, multiple time points are sequentially distributed along the time axis…Ti 2N, Ti N, Ti, Ti+N, Ti+2N… etc. At the current inspection time Ti, the range of the first time window is [Ti…]. The range of the second time window is [Ti], where N,Ti+N]. [2N, Ti+2N]. If the first time window contains a first target time, such as a target time located at Ti+N, its importance level is L3, indicating that the corresponding time is a first target time. Based on this condition, all second target times will be selected in the second time window, for example, all times with an importance level lower than L2 will be selected. Subsequently, the importance of these selected times will be increased to make the label values ​​present a more consistent expression of importance on the timeline.

[0066] Sub-step 20233: Increase the importance of each second objective moment to the first importance level range.

[0067] Among them, the minimum value of the first importance interval is greater than or equal to the minimum importance in the first time window, and the maximum value of the first importance interval is less than or equal to the maximum importance in the first time window.

[0068] In some embodiments of this application, to ensure that the second target time points exhibit an increasing trend consistent with the first time window within the labeling system, the importance of these second target time points needs to be increased to a first importance level range. The minimum value of this range is greater than or equal to the minimum importance level within the first time window, and the maximum value is less than or equal to the maximum importance level within the first time window. This design aims to ensure that the increased label values ​​remain within a reasonable range of change in the current window, avoiding label distribution distortion caused by excessive increases. By increasing the importance in this way, the label values ​​in the second time window can form a continuous increasing range centered on the first target time point in the time dimension, thereby maintaining a smooth upward trend in the labeling system within a local range, laying the foundation for subsequent label diffusion and overall smoothing.

[0069] In a specific example, such as Figure 3 As shown, multiple time points are sequentially distributed along the time axis…Ti 2N, Ti N, Ti, Ti+N, Ti+2N… etc. At the current inspection time Ti, the range of the first time window is [Ti…]. The range of the second time window is [Ti], where N,Ti+N]. [2N, Ti+2N]. If the first time window contains a first target time, such as a time located at Ti+N, and its importance is high, the system will select all second target times (e.g., times with importance below a preset threshold) in the second time window and increase the importance of these times to a reasonable range within the importance range of the first time window, thus forming a local enhancement region around Ti+N on the time axis. After this processing, the label values ​​in the second time window show an increasing trend consistent with those in the first time window, providing a continuous foundation for subsequent label diffusion.

[0070] Sub-step 20234: Within the time interval that is located on the time backtracking side within the second time window and outside the first time window, establish a third time window.

[0071] In some embodiments of this application, to enable the tag system to simultaneously increase importance near the current moment and decrease importance at earlier moments on the timeline, thereby maintaining the continuity of tag changes within a local range, a third time window needs to be established on the time backtracking side within the second time window and outside the first time window. Specifically, based on the already constructed first and second time windows, the system selects the time interval within the second time window that is on the time backtracking side and not covered by the first time window as the third time window, using the current inspection moment as a reference. This window specifically covers a time interval earlier than the current inspection moment. The principle behind this window division is that the impact of acoustic events or time-series events usually exhibits an asymmetrical diffusion over time. By separately delineating the backtracking side, bounded reduction processing of earlier tags can be performed without interfering with the current window's increase adjustment. By establishing the third time window, a bidirectional adjustment structure combining current importance increase and importance delay decrease is formed on the tag system's timeline, thereby enabling the overall tag distribution to exhibit a continuous, smooth, and controllable change effect within a local range.

[0072] In a specific example, such as Figure 3 As shown, multiple time points are sequentially distributed along the time axis…Ti 2N, Ti N, Ti, Ti+N, Ti+2N… etc. At the current inspection time Ti, the range of the first time window is [Ti…]. The range of the second time window is [Ti], where N,Ti+N]. [2N, Ti+2N]. When constructing the third time window, the system selects an interval outside the first time window and on the time backtracking side within the range of the second time window, for example, [Ti]. 2N,Ti [N] serves as the third time window, specifically covering time points earlier than Ti. With this division, the first time window is used to assess importance around the current inspection time, while the third time window is used to assess importance delay over earlier times. Together, they create a continuous and controllable adjustment effect on the label distribution along the timeline within a local area.

[0073] Sub-step 20235: If the first target time is not included in the first time window, each time in the third time window whose importance level modification is less than the preset second importance threshold is determined as a third target time.

[0074] In some embodiments of this application, in order to enable the tagging system to perform necessary delay reduction on earlier moments when the first time window does not contain the first target moment, thereby maintaining the overall continuity of tag changes on the timeline, it is necessary to select a moment that meets the conditions in the third time window as the third target moment. Specifically, the system first determines whether the first time window contains the first target moment; if the first target moment does not exist in the window, it means that no important event requiring improvement has appeared near the current inspection moment, and the tag trend should be decreasing rather than increasing in this segment. Therefore, it is necessary to retrospectively reduce the tag value on the time backtracking side. Based on this determination, the system checks the importance modification magnitude of each moment in the third time window and determines the moment with a modification magnitude less than a preset second importance threshold as the third target moment; the reason for using the modification magnitude as the screening criterion is that moments with small changes usually do not have significant event characteristics and are suitable as objects for delay reduction. Through this screening method, the third time window can accurately locate the important tags that need to be reduced, thereby maintaining the overall continuity of tag changes on the timeline. Figure 3 The timeline shown exhibits a retrospective adjustment effect consistent with the overall trend.

[0075] In a specific example, such as Figure 3 As shown, multiple time points are sequentially distributed along the time axis…Ti 2N, Ti N, Ti, Ti+N, Ti+2N… etc. At the current inspection time Ti, the range of the first time window is [Ti…]. [N, Ti+N], the third time window is located on the time backtracking side of the second time window, for example [Ti 2N,Ti [N]. If the first time window does not contain any first target time, the system will check the importance modification magnitude of each time in the third time window, for example, by comparing it with the initial label value or the label value of the previous round. When the modification magnitude of a certain time is less than the preset second importance threshold, the system will mark that time as the third target time, so that it can be reduced to the second importance range in subsequent steps. After this processing, the label values ​​in the third time window can form a continuous delay reduction range on the time axis, which together with the improvement range of the first time window constitutes a complete two-way adjustment structure.

[0076] Sub-step 20236 reduces the importance of each third objective moment to the second importance level range.

[0077] Among them, the maximum value of the second importance interval is less than or equal to the minimum importance level in the first or second time window.

[0078] In some embodiments of this application, in order to ensure that the label values ​​of the third target moments identified in the third time window are consistent with the overall trend near the current moment, thereby creating a continuous delay reduction effect on the time axis, it is necessary to reduce the importance of these third target moments to the second importance level range. Specifically, given that the third target moments have been identified, the system uniformly reduces the importance of these moments, causing them to fall into the second importance level range. The maximum value of this range is less than or equal to the minimum importance level in the first or second time window. The design principle is that the label value on the retrospective side should be lower than the lowest label level of the current window, so as to reflect the decay trend of the event's impact over time rather than directly forming a sudden change. By reducing the importance of the third target moments to this range, the label system forms a clear delay reduction segment in the time dimension, thereby forming a complete two-way adjustment structure together with the enhancement segment of the first time window.

[0079] In a specific example, such as Figure 3 As shown, multiple time points are sequentially distributed along the time axis…Ti 2N, Ti N, Ti, Ti+N, Ti+2N… etc. At the current inspection time Ti, the third time window is located on the time backtracking side of the second time window, for example, [Ti…]. 2N,Ti [N]. If several third target moments have been identified in the window, the system will reduce the importance of these moments to the second importance level range. For example, it will uniformly set their label values ​​to a level lower than the minimum label value in the first time window, so that they form a continuous decreasing range on the time axis.

[0080] Step 203: In response to the importance modification operation of the fourth objective time, adjust the importance level corresponding to the fourth objective time.

[0081] In some embodiments of this application, to enable the importance system to be effectively monitored and corrected, and to realize the value judgment brought about by human intervention, an importance adjustment can be performed on the fourth target time point manually selected in the first time series when a modification operation is received. When processing the first time series, users or system administrators can submit a manual modification request for the importance label of a certain time point, which is then determined as the fourth target time point, and its importance level can be increased or decreased. The importance modification operation is used to introduce manually enhanced label changes into the label system, so that the importance level can not only change based on automatically identified events, but also reflect the results of human intervention when necessary, thereby enhancing the flexibility and controllability of the label system.

[0082] In a specific example, during the data collection process, an administrator might notice a fleeting animal appearing briefly in the real-time monitoring interface, but with relatively quiet sounds. The administrator might therefore determine this moment to be of high importance, but due to the quietness, the system doesn't elevate its importance level. In this case, the administrator can manually label this moment, designating it as the fourth target moment and increasing its importance, thus giving it a higher value representation within the tagging system. Through this process, the tagging system can simultaneously reflect both automatic identification results and manual judgment, making the overall tag distribution more aligned with actual business needs.

[0083] Step 204: If the original storage precision of the data item in the second time sequence is less than or equal to the data transfer precision at the corresponding time, the processing of the data item is ignored.

[0084] In some embodiments of this application, after determining the data transfer precision at each moment of the second time series, the system compares the original storage precision of each data item moment by moment. When the original storage precision of a data item at a certain moment is less than or equal to the data transfer precision required at that moment, it indicates that the current storage specification of the data item already meets the precision requirements at that moment, and no further processing is required. In this case, the system directly retains the original storage form of the data item, allowing it to be written to or retained in the storage space with the existing storage precision, thereby avoiding unnecessary operations on data items that already meet the precision requirements.

[0085] In a specific example, the system acquires consecutive video frames as a second time sequence during the acquisition process. The video frames have a uniform initial storage precision during acquisition, while in previous steps, different data transfer precisions were set for each moment based on the importance of the first time sequence. The system then compares the initial storage precision of each video frame with its corresponding data transfer precision: when the initial storage precision of a segment is no higher than the required transfer precision for that segment, the system directly retains the video frames of that segment without compression or adjustment. Through this processing, segments in the second time sequence that meet the precision requirements maintain their original quality, and the system's processing resources are not used for unnecessary operations.

[0086] Step 205: If the original storage precision of the data item in the second time sequence is higher than the data transfer precision at the corresponding time, the data item is compressed.

[0087] The method shown in this step has been explained in step 103 and will not be repeated here.

[0088] Step 206: Output the second time sequence after compressing the data items synchronously with the first time sequence.

[0089] In some embodiments of this application, to enable the processed data to be used on a unified time basis, a first time series and a second time series that has undergone precision conversion can be output as a common time series. During the output process, based on the correspondence between the two types of data on the time axis, the content of the same time segment is arranged synchronously, ensuring that data from different sources maintains a consistent time order in the output results. Maintaining time synchronization during output allows subsequent analysis, playback, or storage management to perform correlated processing of the two types of data on the same time axis. By achieving joint output in this way, the processed data maintains consistency in the time dimension, enabling subsequent use cases to perform content comparison or joint analysis based on a unified time series.

[0090] In a specific example, during the monitoring and acquisition process, a security monitoring system records continuous audio data as the first time sequence and generates continuous video frames as the second time sequence. The audio data is assigned different levels of importance in the preceding steps, and the video data is compressed accordingly. In the output stage, the audio segments and compressed video frames are arranged along a unified timeline to form synchronized audio and video. This processing ensures that subsequent monitoring playback maintains high-definition output only for events of significant importance, while outputting lower-definition footage during periods of low-relevant information. This ensures that the data remains usable while minimizing data storage overhead.

[0091] In some embodiments of this application, to illustrate the performance of allocating storage precision for a second time-series sequence based on the importance tags of the first time-series sequence in a real-world device, verification can be performed using an audio stream and video stream acquisition scenario. In this scenario, the audio stream serves as the first time-series sequence, reflecting the intensity of acoustic events in different time segments; the video stream serves as the second time-series sequence, recording the corresponding video content. After completing initial tag setting, target time identification, and tag diffusion along the timeline, the audio side forms a tag sequence that reflects the intensity and temporal continuity of events. This tag sequence guides the storage precision allocation on the video side in subsequent steps, enabling the system to maintain high storage precision during events and reduce storage precision during ordinary segments, thereby achieving overall compression.

[0092] To further illustrate the effect of this precision allocation in actual devices, we can statistically analyze video data at different storage precision levels using common video encoding parameters. For example, after analyzing one minute of video data encoded in H.265, we can obtain the typical storage sizes corresponding to different levels: At higher precision levels (such as L3), a 2K resolution and a bitrate of approximately 3 Mbps can be used, corresponding to a storage space of approximately 22.5 MB. At medium precision levels (such as L2), a 1080P resolution and a bitrate of approximately 2 Mbps can be used, occupying approximately 15 MB of space. At lower precision levels (such as L1), a 720P resolution and a bitrate of approximately 0.9 Mbps can be used, occupying approximately 6.75 MB of space. At the lowest precision level (such as L0), a 720P resolution and a bitrate of approximately 0.6 Mbps can be used, occupying approximately 4.5 MB of space. It is evident that there are significant differences in data volume between different storage precision levels, and the lower the precision level, the smaller the storage space required.

[0093] In security monitoring scenarios with synchronized audio and video, the system generates importance tags based on acoustic events in the audio stream. For example, sudden noises usually represent abnormal situations, human voices may represent personnel activity, while background noise corresponds to lower importance. In this way, the audio stream forms a tag sequence that fluctuates with events along the timeline, guiding the selection of storage precision on the video side: when high-importance moments such as abnormal sounds or personnel activity are detected, the system maintains high storage precision; in ordinary noise or silent segments, a higher compression rate is used, thereby reducing overall storage costs while ensuring the quality of critical images.

[0094] Furthermore, it can be seen that this solution also employs similar, but not entirely identical, level selection logic in other audio-visual synchronization scenarios. For example, in dashcam scenarios, the types of acoustic events on the audio side differ from those in security scenarios. Sudden events such as collision sounds and braking sounds typically have higher importance, while ordinary environmental noise has lower importance. Accordingly, the system maintains high video storage precision at the moment of collision or sudden braking to highlight key segments before and after the accident. In intelligent inspection equipment, the audio side may focus more on mechanical abnormalities or equipment malfunctions. The system will increase the storage precision of the corresponding video segments when such acoustic events are detected to facilitate subsequent problem localization. Through the above methods, this solution establishes a level selection logic centered on acoustic events in audio-visual scenarios, and can adjust the event type and importance according to different business needs, making the tag-driven storage precision allocation adaptable to various practical applications.

[0095] Furthermore, this solution is also applicable to other multimodal data combinations. In these scenarios, the system typically receives lightweight sensor time-series data as a first time-series sequence and space-intensive video streams or image sequences as a second time-series sequence. The first time-series sequence reflects the state changes of the device or environment at different time segments, and the system generates corresponding importance labels based on the magnitude of changes, anomalies, or business implications of the sensor data. The second time-series sequence records the video content, and its storage precision is driven by the importance level of the first time-series sequence. Different sensor types and event characteristics exist in different scenarios, therefore the allocation of importance levels also varies, enabling the system to increase storage precision at critical moments and decrease it at ordinary moments according to specific business needs, thereby achieving a balance between retaining critical content and compressing overall data. For example, in a vehicle monitoring scenario, the first time series can be a sequence of acceleration changes over time, reflecting the vehicle's motion state at different time segments. The system assigns importance based on the magnitude of acceleration changes: when acceleration exceeds a preset threshold at a certain moment, it usually represents a high-value event such as rapid acceleration, rapid deceleration, or a collision, and the system marks this moment as having higher importance; while in ordinary driving segments with stable acceleration changes, it is marked as having lower importance. The second time series is video stream data. The system maintains high storage precision at high-importance moments to preserve clearer images in subsequent analysis, while using a higher compression rate for ordinary driving segments to reduce the overall data volume.

[0096] In airborne monitoring scenarios, the first time series can be temperature sensor data. The system assigns importance based on the degree of anomaly in temperature changes: when a device experiences a temperature anomaly at a certain moment, it usually indicates an abnormal device status or potential fault, and the system marks this moment as of higher importance; while during normal periods of stable temperature, it is marked as of lower importance. The second time series is the airborne video stream. The system improves the storage precision of the corresponding video segments at the time of temperature anomaly, ensuring that key images are preserved more clearly.

[0097] In environmental inspection scenarios, the first time series can be current or voltage sensor data. The system assigns importance based on the magnitude of current abrupt changes or voltage fluctuations: abrupt changes or fluctuations usually represent potential fault points and are therefore marked as higher importance; while stable current or voltage corresponds to lower importance. The second time series is the inspection video stream. The system improves storage accuracy at high-importance moments to facilitate subsequent problem localization.

[0098] In security monitoring scenarios, the first time-series data can be from infrared or radar sensors. The system assigns importance based on target approach, changes in movement speed, or abnormal trajectories: when infrared or radar detects target approach or abnormal movement, the system marks that moment as having higher importance; while during periods without targets or with stable target movement, it marks them as having lower importance. The second time-series data is the monitoring video stream. The system maintains high storage accuracy during high-importance moments to preserve clearer images at critical times.

[0099] In satellite remote sensing scenarios, the system typically receives high-resolution image sequences and various auxiliary sensor data simultaneously. The first time-series sequence can be remote sensing auxiliary information that changes over time, such as illumination, cloud cover, orbital position, and ground feature change detection results. The system assigns importance levels based on the imaging conditions or surface changes indicated by the auxiliary information: for example, when cloud cover is low, illumination conditions are good, or ground feature changes are significant, the system marks that moment as having higher importance; while during periods of poor imaging conditions or insignificant changes, it marks them as having lower importance. The second time-series sequence is a sequence of satellite images acquired over time. The system maintains higher image storage accuracy during high-importance moments and uses a higher compression rate during ordinary moments, thereby reducing the overall data volume while ensuring the clarity of key areas.

[0100] Finally, it's important to emphasize that the value judgment of event types can be flexibly set according to actual business needs, regardless of whether it's an audio / video scenario or other multimodal scenarios. It's not fixed. For example, in a motorcycle riding scenario, human voices in the outdoor environment typically don't have high event value, while collision sounds, braking sounds, or other sudden noises are more representative of critical events. Therefore, in different scenarios, the system can set different event types and importance levels according to business needs, increasing the storage precision of the second time series during high-importance moments and decreasing the storage precision during ordinary periods. This achieves a balance between preserving critical content and compressing overall data across various application scenarios.

[0101] Because these sensor data have clear event indication significance in scenarios such as vehicle-mounted, airborne, security, inspection, and satellite applications, they can serve as markers for video or image upgrades: when the importance of the first time series is high, the system maintains high storage precision in the corresponding second time series to preserve clearer images at critical moments; while in periods of lower importance, a higher compression rate is used to reduce the overall data volume while preserving key content. Therefore, this solution can drive the selection of storage precision for space-intensive video or image data in various multimodal scenarios through lightweight sensor data, enabling the system to achieve a balance between preserving key content and compressing overall data in different application scenarios.

[0102] Figure 4 This paper illustrates the overall audio and video processing flow in the embodiments of this application. The entire process revolves around "audio-driven video level adjustment": the system first preprocesses and divides the audio into windows, then determines the video level for the corresponding time period based on the events within the window, and finally downgrades the video according to the level before writing it. In this way, the audio analysis results can affect the video storage specifications in real time, making the data flow form a continuous and controllable processing chain. S1. Device captures audio and video: The system starts the acquisition module to acquire audio and video stream data in the environment in real time and form the raw input signal; S2. Audio and video separation: Separating the acquired composite signal into independent audio and video streams, providing a foundation for subsequent parallel processing; S3. Audio preprocessing: Filtering, noise reduction, and signal enhancement are performed on the audio stream to improve the accuracy of subsequent event detection; S4. Maintain high-level video recording specifications: Before audio analysis is completed, the video is recorded at the highest level by default to ensure that critical events are not lost due to delayed analysis. S5. The audio stream is divided into first to third time windows based on the time frame: the audio stream is divided into multiple window segments according to the time axis for hierarchical detection and label diffusion. S6. The leading edge of the first and second time windows moves to the next unit of time: The large (second time window) and small (first and second time windows) time windows slide along the time axis to achieve continuous detection, enabling the system to update the event status in real time; R1. Determine whether a corresponding level of trigger event has occurred within the current time window based on audio information. The system analyzes the audio characteristics of the current window (energy threshold, human voice detection, burst sound detection, etc.) to determine whether a preset event level has been triggered. S7. Set the video of the first time window to the corresponding level: If a triggered event is detected, the video level of that time window will be upgraded to the corresponding level and kept at the same level to ensure the quality of the event footage. R2. Does the video in the third time window (observation area) already have a rating? If no event is detected, the system determines whether the video in the observation area already has a rating. S8. Video in the third time window is downgraded to the lowest level: If there is no level marker in the observation area, the video in that time window is downgraded to the lowest level to save storage space; S9. During the third time window, the video should maintain the corresponding level: If the observation area already has a level mark, the original level should be maintained to avoid frequent fluctuations. S10. According to the level, the video of the corresponding time period is downgraded and overwritten into the storage space using the timestamp: The system performs hierarchical encoding or compression of the video according to the level label of each time period, and writes it into the storage using the timestamp as the index. S11. Audio and video are aligned using timestamps: In the final output stage, the system resynchronizes the audio and video based on the timestamps to ensure that the event visuals and audio correspond consistently.

[0103] This creates a complete data stream within the system, from audio analysis to video rating adjustment and final storage. The audio side is responsible for identifying events and generating rating labels, while the video side executes the corresponding encoding specifications based on the labels, with both maintaining consistency through timestamps. This allows the system to retain high-quality footage during critical events while automatically reducing the bitrate during ordinary or silent segments, significantly reducing overall storage requirements. The entire process is clearly structured, with smooth data flow, enabling stable operation on real-world devices and producing a clear compression effect.

[0104] In summary, in this embodiment, by synchronously collecting the first and second time-series sequences of the collected objects in their respective modalities, it is ensured that the data sources for subsequent data processing have a common time and event benchmark. Then, based on the importance of events occurring at each moment in the first time-series sequence, differentiated value representations are established for different times, and different data transfer accuracies are established under different importance levels to distinguish the different storage values ​​of data at each moment, thereby achieving differentiated configuration of storage accuracy. On this basis, based on the different importance levels at each moment, data items in the second time-series sequence with storage accuracy higher than transfer accuracy are compressed, so that different data contents are saved according to their matching content value, thereby reducing the data occupation of low-value data contents. While reducing the high resource occupation pressure brought by completely lossless storage, it also avoids the overall data quality degradation caused by completely lossy storage. Therefore, the method based on the embodiments of this application, by associating the storage strategy of time series data with the importance distribution of events, makes the data storage allocation more in line with the importance value structure of the events, realizes dynamic precision allocation of long-term time series data, and improves the effective utilization of limited data storage space while maintaining the overall subsequent use value of the data, thus improving the contradiction between data capacity and data quality.

[0105] refer to Figure 5 It illustrates a multimodal data storage device 30 provided in an embodiment of this application, comprising: The acquisition module 301 is used to simultaneously acquire the first time sequence of the acquired object in the first mode and the second time sequence in the second mode; The tag module 302 is used to determine the data transfer accuracy of each moment in the second time series based on the importance of events occurring at each moment in the first time series; the data transfer accuracy is positively correlated with the importance. The conversion module 303 is used to compress data items when the original storage precision of the data items in the second time sequence is higher than the data transfer precision at the corresponding time.

[0106] Optionally, the label module 302 includes: The initialization submodule is used to initialize the importance of each time step in the first time series. The first tagging submodule is used to determine the first target time corresponding to each target occurrence event in each time step of the first time sequence, and modify the importance of the corresponding first target time to be greater than or equal to the importance of the other time steps according to the event complexity represented by each target occurrence event.

[0107] Optionally, the initialization submodule includes: The marking unit is used to set the unset importance level to a preset baseline importance level when the importance level is not set at any time in the first time series. The confirmation unit is used to retain the importance of each moment in the first time series, provided that the importance of each moment in the first time series has been set.

[0108] Optionally, the label module 302 also includes: The second tag submodule is used to enhance the importance of the second target moment in the first time sequence; the second target moment and the first target moment satisfy a preset event association relationship.

[0109] Optionally, the second tag submodule includes: The first window unit is used to establish a first time window and a second time window that advances step by step; the first time window covers the corresponding window time, and the second time window covers the first time window. The first time unit is used to determine each time in the second time window that corresponds to an importance level less than a first importance threshold as a second target time when the first time window contains at least one first target time. The first upgrade unit is used to upgrade the importance of each second target moment to the first importance level range; the minimum value of the first importance level range is greater than or equal to the minimum importance level in the first time window, and the maximum value of the first importance level range is less than or equal to the maximum importance level in the first time window.

[0110] Optionally, the second tag submodule also includes: The second window unit is used to establish a third time window within the time interval that is located on the time backtracking side and outside the first time window within the second time window; The second time unit is used to determine each time in the third time window whose importance level modification is less than a preset second importance threshold as a third target time when the first target time is not included in the first time window. The second upgrade unit is used to reduce the importance of each third target time to the second importance level range; the maximum value of the second importance level range is less than or equal to the minimum importance level in the first time window or the second time window.

[0111] Optionally, the first time sequence is audio stream data, and the first tag submodule includes: An audio recognition unit is used to identify noise and / or human voices in audio stream data, and to determine each moment corresponding to the identified noise and / or human voices as a first target moment; An audio tagging unit is used to determine the importance of each first target moment based on the sound complexity indicated by noise and / or human voice corresponding to each first target moment; the sound complexity indicated by noise and / or human voice is greater than or equal to the sound complexity indicated under quiet conditions.

[0112] Optionally, the storage device 30 for multimodal data also includes: The retention module is used to ignore the processing of data items if the original storage precision of the data items in the second time sequence is less than or equal to the data transfer precision at the corresponding time.

[0113] Optionally, the storage device 30 for multimodal data also includes: The output module is used to synchronously output the second time-series sequence after the data items are compressed with the first time-series sequence.

[0114] Optionally, the storage device 30 for multimodal data also includes: The modification module is used to adjust the importance level corresponding to the fourth objective time step in response to the operation of modifying the importance of the fourth objective time step.

[0115] In summary, in this embodiment, by synchronously collecting the first and second time-series sequences of the collected objects in their respective modalities, it is ensured that the data sources for subsequent data processing have a common time and event benchmark. Then, based on the importance of events occurring at each moment in the first time-series sequence, differentiated value representations are established for different times, and different data transfer accuracies are established under different importance levels to distinguish the different storage values ​​of data at each moment, thereby achieving differentiated configuration of storage accuracy. On this basis, based on the different importance levels at each moment, data items in the second time-series sequence with storage accuracy higher than transfer accuracy are compressed, so that different data contents are saved according to their matching content value, thereby reducing the data occupation of low-value data contents. While reducing the high resource occupation pressure brought by completely lossless storage, it also avoids the overall data quality degradation caused by completely lossy storage. Therefore, the method based on the embodiments of this application, by associating the storage strategy of time series data with the importance distribution of events, makes the data storage allocation more in line with the importance value structure of the events, realizes dynamic precision allocation of long-term time series data, and improves the effective utilization of limited data storage space while maintaining the overall subsequent use value of the data, thus improving the contradiction between data capacity and data quality.

[0116] Reference Figure 6 The electronic device 500 may include one or more of the following components: processing component 502, memory 504, power supply component 506, multimedia component 508, audio component 510, input / output (I / O) interface 512, sensor component 514, and communication component 516.

[0117] Processing component 502 typically controls the overall operation of electronic device 500, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 502 may include one or more processors 520 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 502 may include one or more modules to facilitate interaction between processing component 502 and other components. For example, processing component 502 may include a multimedia module to facilitate interaction between multimedia component 508 and processing component 502.

[0118] Memory 504 is used to store various types of data to support the operation of electronic device 500. Examples of this data include instructions for any application or method operating on electronic device 500, contact data, phonebook data, messages, pictures, multimedia, etc. Memory 504 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0119] Power supply component 506 provides power to various components of electronic device 500. Power supply component 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 500.

[0120] Multimedia component 508 includes an interface that provides an output interface between electronic device 500 and user. In some embodiments, the interface may include a liquid crystal display (LCD) and a touch panel (TP). If the interface includes a touch panel, the interface may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may not only sense the boundaries of touch or swipe actions but also detect the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 508 includes a front-facing camera and / or a rear-facing camera. When electronic device 500 is in an operating mode, such as shooting mode or multimedia mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0121] Audio component 510 is used to output and / or input audio signals. For example, audio component 510 includes a microphone (MIC) used to receive external audio signals when electronic device 500 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 504 or transmitted via communication component 516. In some embodiments, audio component 510 also includes a speaker for outputting audio signals.

[0122] Input / output (I / O) interface 512 provides an interface between processing component 502 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0123] Sensor assembly 514 includes one or more sensors for providing state assessments of various aspects of electronic device 500. For example, sensor assembly 514 may detect the on / off state of electronic device 500, the relative positioning of components such as the display and keypad of electronic device 500, changes in position of electronic device 500 or a component of electronic device 500, the presence or absence of user contact with electronic device 500, orientation or acceleration / deceleration of electronic device 500, and temperature changes of electronic device 500. Sensor assembly 514 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 514 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 514 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0124] Communication component 516 facilitates wired or wireless communication between electronic device 500 and other devices. Electronic device 500 can access wireless networks based on communication standards, such as WiFi, carrier networks (such as 2G, 3G, 4G, or 5G), or combinations thereof. In one exemplary embodiment, communication component 516 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 516 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0125] In an exemplary embodiment, the electronic device 500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to implement the methods provided in the embodiments of this application.

[0126] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 504 including instructions, which can be executed by a processor 520 of an electronic device 500 to perform the above-described method. For example, the non-transitory storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0127] In an exemplary embodiment, the electronic device 500 may also be provided as a server, including a processing component 502, which further includes one or more processors, and memory resources represented by memory 504 for storing instructions, such as applications, that can be executed by the processing component 502. The applications stored in memory 504 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 502 is configured to execute instructions to perform the methods provided in the embodiments of this application.

[0128] Electronic device 500 may also include a power supply component 506 configured to perform power management of electronic device 500, a wired or wireless communication component 516 configured to connect electronic device 500 to a network, and an input / output (I / O) interface 512. Electronic device 500 may operate on an operating system stored in memory 504, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0129] This application also provides a computer program product, including a computer program, which, when executed by a processor, implements the method of this application embodiment.

[0130] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the application disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims below.

[0131] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the claimed technical solutions.

[0132] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0133] It will be readily apparent to those skilled in the art that any combination of the above embodiments is feasible. Therefore, any combination of the above embodiments is an implementation scheme of this application. However, due to space limitations, this specification will not describe them in detail here.

[0134] It should be noted that, unless otherwise expressly stated, the methods provided in the embodiments of this application are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. Based on the above description, the required structure for constructing a system having the solutions of this application is obvious. Furthermore, this application is not directed to any particular programming language. It should be understood that the content of this application described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of this application.

[0135] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0136] Similarly, it should be understood that, for the purpose of simplification and aiding understanding of one or more aspects of the application, various features of the application are sometimes grouped together in a single embodiment, figure, or description thereof in the above description of exemplary embodiments of the application. However, this disclosure should not be construed as reflecting an intention that the claimed application requires more features than are expressly recited in each of the claimed technical solutions. Rather, as reflected in the claimed technical solutions, the application aspects comprise fewer features than all those in the single embodiment disclosed above. Therefore, the claimed technical solutions following the specific implementation are thus expressly incorporated into that specific implementation, wherein each claimed technical solution is itself a separate embodiment of the application.

[0137] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination of all features disclosed in this application and all processes or units of any method or device so disclosed can be employed. Unless expressly stated otherwise, each feature disclosed in this application may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0138] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the claimed technical solutions, any one of the claimed embodiments can be used in any combination.

[0139] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components in the methods according to the embodiments of this application. This application can also be implemented as a device or apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such an implementation of this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0140] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute the method of the embodiments of the present application.

[0141] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0142] It should be noted that, for the sake of simplicity, the method embodiments of this application are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily essential to the embodiments of this application.

[0143] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the application disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims below.

[0144] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the claimed technical solutions.

[0145] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method for storing multimodal data, characterized in that, include: The first time-series sequence of the object under the first mode and the second time-series sequence under the second mode are acquired simultaneously. Initialize the importance level of each moment in the first time series; In each moment of the first time series, a first target moment corresponding to each target occurrence event is determined, and the importance of the corresponding first target moment is modified to be greater than or equal to the importance of the other moments according to the event complexity represented by each target occurrence event. Establish a first time window and a second time window that advances step by step; the first time window covers the corresponding window time, and the second time window covers the first time window. If the first time window contains at least one of the first target moments, the importance of each moment in the second time window that is less than the first importance threshold is raised to the first importance range. The data transfer accuracy of the second time series is determined based on the importance of events occurring at each time point in the first time series; the data transfer accuracy is positively correlated with the importance level. If the original storage precision of a data item in the second time sequence is higher than the data transfer precision at the corresponding time, the data item is compressed.

2. The method for storing multimodal data as described in claim 1, characterized in that, The initialization of the importance of each moment in the first time series includes: If the importance level at any time in the first time series is not set, the unset importance level is set to a preset baseline importance level; If the importance of each moment in the first time series has been set, the importance of each moment in the first time series is retained.

3. The method for storing multimodal data as described in claim 1, characterized in that, The minimum value of the first importance interval is greater than or equal to the minimum importance value in the first time window, and the maximum value of the first importance interval is less than or equal to the maximum importance value in the first time window.

4. The method for storing multimodal data as described in claim 1, characterized in that, The method for storing multimodal data also includes: A third time window is established within the time interval that is located on the time backtracking side within the second time window and outside the first time window; If the first target time is not included in the first time window, each time in the third time window whose change in importance is less than a preset second importance threshold is determined as a third target time. The importance of each of the third target moments is reduced to a second importance range; the maximum value of the second importance range is less than or equal to the minimum importance in the first time window or the second time window.

5. The method for storing multimodal data as described in claim 1, characterized in that, The first time series is audio stream data. The step of determining the first target time corresponding to each target occurrence event within each moment of the first time series, and modifying the importance of the corresponding first target time to be greater than or equal to the importance of the other moments based on the event complexity represented by each target occurrence event, includes: Identify noise and / or human voices in the audio stream data, and determine each moment corresponding to the identified noise and / or human voices as a first target moment; The importance of each first target moment is determined by the sound complexity indicated by noise and / or human voice corresponding to each first target moment; the sound complexity indicated by noise and / or human voice is greater than or equal to the sound complexity indicated under quiet conditions.

6. The method for storing multimodal data as described in claim 1, characterized in that, The method for storing multimodal data also includes: If the original storage precision of a data item in the second time series is less than or equal to the data transfer precision at the corresponding time, the processing of the data item is ignored.

7. The method for storing multimodal data as described in claim 1, characterized in that, The method for storing multimodal data also includes: The second time series sequence after compressing the data items is output synchronously with the first time series sequence.

8. The method for storing multimodal data as described in claim 1, characterized in that, The method for storing multimodal data also includes: In response to the operation modifying the importance of the fourth target time, the importance level corresponding to the fourth target time is adjusted.

9. A storage device for multimodal data, characterized in that, include: The acquisition module is used to simultaneously acquire the first time-series sequence of the acquired object in the first mode and the second time-series sequence in the second mode; The tagging module is used to determine the data transfer accuracy of each moment in the second time series based on the importance of events occurring at each moment in the first time series; the data transfer accuracy is positively correlated with the importance level. The conversion module is used to compress the data item when the original storage precision of the data item in the second time series is higher than the data transfer precision at the corresponding time. The tag module includes: An initialization submodule is used to initialize the importance of each moment in the first time series. The first tag submodule is used to determine the first target time corresponding to each target occurrence event in each time of the first time sequence, and modify the importance of the corresponding first target time to be greater than or equal to the importance of the other times according to the event complexity represented by each target occurrence event. The first window unit is used to establish a first time window and a second time window that advances step by step; the first time window covers the corresponding window time, and the second time window covers the first time window. The first time unit is used to determine each time in the second time window that corresponds to an importance level less than a first importance threshold as a second target time when the first time window contains at least one first target time. The first upgrade unit is used to elevate the importance of each second objective moment to the first importance level range.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the method for storing multimodal data as described in any one of claims 1 to 8.

11. An electronic device, characterized in that, The system includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method for storing multimodal data as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, The computer program product stores a computer program, which, when executed by a processor, implements the steps of the method for storing multimodal data as described in any one of claims 1 to 8.

Images