Acoustic data storage and retrieval method combined with computer cloud computing

Abstract

The invention provides an acoustic data storage and retrieval method combined with computer cloud computing, which comprises the following steps of: performing time sequence division processing on original acoustic data to generate an audio frame sequence with a timestamp mark; based on a context window of the audio frame sequence, acoustic features are extracted, and a context-aware acoustic feature vector is constructed; constructing a time sequence sensing distributed index structure according to the acoustic feature vector, and dynamically storing the streaming acoustic data to obtain the time sequence sensing distributed index structure; and executing cross-modal similarity retrieval on the query audio and the distributed index structure to obtain a target fragment sequence in space-time association with the query audio. By adopting the method, the retrieval precision and efficiency of the acoustic data in a complex time sequence association scene can be improved through a time sequence perception storage architecture and a cross-modal retrieval mechanism.

Description

Technical Field

[0001] The present invention belongs to the field of data storage and retrieval, and in particular to an acoustic data storage and retrieval method combined with computer cloud computing. Background Art

[0002] With the popularization of cloud computing technology, the storage and retrieval of acoustic data are increasingly widely used in the fields of intelligent monitoring, industrial diagnosis, etc. The existing technology usually adopts a similarity retrieval scheme based on acoustic feature vectors (such as MFCC, Mel spectrum), combined with distributed databases (such as Elasticsearch, Milvus) to build indexes to achieve efficient management of massive data. However, such methods have significant defects: on the one hand, traditional indexes only rely on single-frame acoustic features, ignoring the contextual relevance of sound events in the time dimension, resulting in audio clips with similar semantics but misaligned timing (such as the same abnormal sound before and after equipment failure) being mistakenly judged as irrelevant; on the other hand, the incremental index construction for streaming acoustic data often adopts a random insertion strategy, which destroys the temporal continuity of the index structure, causing the retrieval efficiency to drop sharply with the growth of data scale.

[0003] Therefore, how to achieve time-aware acoustic data storage and retrieval in a cloud computing environment has become a key challenge to improve system accuracy and performance. Summary of the invention

[0004] Based on this, it is necessary to provide an acoustic data storage and retrieval method combined with computer cloud computing to address the above technical problems. It can improve the retrieval accuracy and efficiency of acoustic data in complex time-related scenarios through a time-aware storage architecture and cross-modal retrieval mechanism.

[0005] In a first aspect, the present application provides an acoustic data storage and retrieval method combined with computer cloud computing, comprising:

[0006] Performing time-series division processing on the original acoustic data to generate an audio frame sequence with a timestamp mark;

[0007] Extract acoustic features and construct context-aware acoustic feature vectors based on the context window of the audio frame sequence;

[0008] A time-series-aware distributed index structure is constructed based on the acoustic feature vectors, and streaming acoustic data is dynamically stored to obtain a time-series-aware distributed index structure.

[0009] Cross-modal similarity retrieval is performed on the query audio and the distributed index structure to obtain the target segment sequence that is temporally and spatially associated with the query audio.

[0010] In a possible embodiment, the original acoustic data is processed by time series partitioning to generate an audio frame sequence with timestamp markings, including:

[0011] The original audio stream is framed according to a preset time interval to generate a set of discrete audio frames;

[0012] Each frame in the set of discrete audio frames is processed by applying a window function to generate preprocessed audio frames;

[0013] A unique timestamp is assigned to the preprocessed audio frames, and the temporal context window of its preceding and succeeding related frames is marked to generate an audio frame sequence.

[0014] In a possible embodiment, based on the context window of the audio frame sequence, extracting acoustic features and constructing a context-aware acoustic feature vector includes:

[0015] Based on the temporal context window, the range of the preceding and succeeding related frames of the target frame is defined according to a preset window radius parameter to generate a set of related frame ranges;

[0016] Acoustic feature extraction processing is performed on each frame within the set of related frame ranges to generate a set of high-dimensional feature vectors;

[0017] Using the following formula, attention fusion processing is performed on the set of high-dimensional feature vectors to obtain attention weights:

[0018]

[0019] where, α t,i is the attention weight, q t is the query vector of the target frame, k i is the key vector of the related frame, p i-t is the relative position encoding vector;

[0020] An acoustic feature vector is generated according to the attention weight result.

[0021] In a possible embodiment, a time series-aware distributed index structure is constructed according to the acoustic feature vector to dynamically store the streaming acoustic data, obtaining a time series-aware distributed index structure, including:

[0022] The acoustic feature vectors are projected into a low-dimensional space to generate a set of index feature vectors;

[0023] According to the timestamp order of the set of index feature vectors, a hierarchical graph index is dynamically inserted;

[0024] In each layer of the hierarchical graph index, the nodes are connected based on temporal proximity to generate a distributed index structure.

[0025] In a possible embodiment, connection processing is performed on nodes based on temporal proximity to generate a temporally aware distributed index structure, including:

[0026] Define the temporal proximity weight using the following formula:

[0027]

[0028] where w(u, v) is the temporal proximity weight, t u is the node timestamp, and λ is the time decay coefficient;

[0029] Filter neighboring nodes according to the temporal proximity weight to generate a set of candidate connection edges;

[0030] When it is detected that there are cross-period edges in the set of candidate connection edges, delete the cross-period edges and supplement the intra-period edges to generate a distributed index structure.

[0031] In a possible embodiment, cross-modal similarity retrieval is performed on the query audio and the distributed index structure to obtain a target segment sequence that is spatio-temporally associated with the query audio, including:

[0032] Perform context-aware feature extraction processing on the query audio to generate a query feature vector and its timestamp;

[0033] Perform similarity search processing in the distributed index structure to generate a set of candidate feature vectors;

[0034] Use the following formula to calculate the timestamp alignment penalty value:

[0035] penalty(t q ,t i )=μ·|t q -t i |

[0036] where penalty(t q ,t i ) is the time alignment penalty value, μ is the dynamic penalty coefficient, t q is the query timestamp, and t i is the candidate timestamp;

[0037] Perform weighted processing on the timestamp alignment penalty value and the feature similarity to generate a comprehensive score, and sort the set of candidate feature vectors according to the comprehensive score to obtain the target segment sequence.

[0038] In a possible embodiment, sorting the set of candidate feature vectors according to the comprehensive score to obtain the target segment sequence includes:

[0039] Filter the comprehensive score according to a preset threshold to generate a set of valid candidate segments;

[0040] Perform timestamp continuity verification processing on the set of valid candidate segments to generate verified target segments that pass the verification;

[0041] Output the verified target segments and determine the verified target segments as the target segment sequence.

[0042] In a second aspect, the present application also provides an acoustic data storage and retrieval device combined with computer cloud computing, including:

[0043] A timing division module for performing timing division processing on the original acoustic data to generate an audio frame sequence with timestamp markings;

[0044] A feature extraction module for extracting acoustic features and constructing context-aware acoustic feature vectors based on the context window of the audio frame sequence;

[0045] A distributed indexing module for constructing a timing-aware distributed indexing structure according to the acoustic feature vectors, dynamically storing streaming acoustic data, and obtaining a timing-aware distributed indexing structure;

[0046] A query and retrieval module for performing cross-modal similarity retrieval on the query audio and the distributed indexing structure to obtain a target segment sequence that is spatio-temporally associated with the query audio.

[0047] In a third aspect, the present application also provides a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the above-mentioned acoustic data storage and retrieval method combined with computer cloud computing.

[0048] In a fourth aspect, the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the above-mentioned acoustic data storage and retrieval method combined with computer cloud computing.

[0049] The above-mentioned acoustic data storage and retrieval method combined with computer cloud computing generates an audio frame sequence with timestamp markings by performing timing division processing on the original acoustic data, ensuring the integrity of the time attribute of the data; extracts acoustic features and constructs context-aware acoustic feature vectors based on the timing context window of the audio frame sequence to capture the timing dependence of sound events; constructs a timing-aware distributed indexing structure according to the acoustic feature vectors to realize the dynamic storage of streaming acoustic data and optimize the index update efficiency; finally, performs cross-modal similarity retrieval on the query audio and the distributed indexing structure, jointly analyzes the acoustic feature similarity and the timestamp alignment degree, and generates a spatio-temporally associated target segment sequence. This method effectively improves the retrieval accuracy and response speed of acoustic data in complex timing scenarios through a timing-aware storage architecture and a cross-modal retrieval mechanism. Brief Description of the Drawings

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the drawings required for use in the description of the embodiments or related technologies. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0051] Figure 1 It is a schematic flowchart of a method for acoustic data storage and retrieval combined with computer cloud computing provided by an embodiment of the present invention;

[0052] Figure 2 It is a schematic structural diagram of a device for acoustic data storage and retrieval combined with computer cloud computing provided by an embodiment of the present invention. Detailed Embodiments

[0053] In order to make the objectives, technical solutions and advantages of the present application more clear and understandable, the following further details the present application in conjunction with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0054] First, a brief introduction is made to the nouns involved in the embodiments of the present application.

[0055] Computer cloud computing is a generalized distributed computing model based on the Internet. It centrally manages and schedules a large number of computing resources (including servers, storage devices, network bandwidth, etc.) and provides users with on-demand elastic computing services in a virtualized manner. Users do not need to own and maintain expensive hardware facilities. They can obtain powerful computing capabilities, storage spaces and software applications anytime and anywhere just by connecting to the network. The core advantages of cloud computing lie in its flexibility, scalability and cost-effectiveness, which can meet the diverse needs from individual users to large enterprises and are widely used in multiple fields such as data storage, high-performance computing, artificial intelligence, and the Internet of Things. It is one of the important trends in the development of modern information technology.

[0056] A distributed index structure is an architecture that disperses data indexes across multiple nodes for efficient management and retrieval of large-scale data. By splitting and distributing index information to different servers or storage units, it enables parallel processing and fast access to data. Each node is responsible for storing and managing a portion of the index data and collaborates through a network to jointly handle query requests. This structure can effectively address the storage and retrieval requirements of massive data, featuring high scalability, fault tolerance, and load balancing. It is widely applied in fields such as search engines, big data processing, and distributed file systems, significantly enhancing the efficiency of data retrieval and the overall performance of the system.

[0057] A hierarchical graph index is an efficient indexing method designed for complex graph data structures. It organizes the nodes and edges in a graph according to a certain hierarchical relationship to form a multi-level index structure. Each level of the index is responsible for storing and managing graph element information within a specific range, thereby enabling rapid positioning and querying of graph data. This indexing method can effectively reduce the search space and improve query efficiency, especially suitable for storage and retrieval scenarios of large-scale graph data.

[0058] Based on the above noun explanations, the implementation environment of the acoustic data storage and retrieval method combined with computer cloud computing provided in the embodiments of this application is described. Schematically, this implementation environment includes: a cloud computing server cluster, data storage devices, and terminals. Among them, the cloud computing server cluster, storage devices, and terminals are communicatively connected through network devices, where the network devices can be high-speed switches, routers, load balancers, fiber optic network devices, etc.; the cloud computing server cluster includes but is not limited to high-performance servers, GPU-accelerated servers, distributed storage nodes, etc.; the storage devices can be distributed storage systems, cloud storage services, or local high-performance storage arrays, which are not limited here.

[0059] Combined with the above noun explanations and implementation environment, the application scenarios of the embodiments of this application are described. The acoustic data storage and retrieval method combined with computer cloud computing provided in the embodiments of this application can be applied to, but is not limited to, the following scenarios:

[0060] In the audio monitoring scenarios of urban security or public places, a large number of microphone devices continuously collect environmental acoustic data. The acoustic data is transmitted to the cloud computing server cluster through the network. After time series partitioning and feature extraction, a time series-aware distributed index structure is constructed. When the monitoring personnel need to retrieve abnormal sounds (such as glass breaking sounds, arguing sounds, etc.) within a specific time period, a query request is submitted through the terminal device, and the system can quickly retrieve the target segment sequence that is spatio-temporally associated with the query audio from the massive audio data, helping the monitoring personnel to discover and handle abnormal events in a timely manner.

[0061] In an industrial production environment, the sound generated during the operation of equipment can reflect its operating status. By collecting the sound data of equipment operation through acoustic sensors deployed in the factory workshop and uploading it to the cloud computing platform for processing. When it is necessary to monitor equipment failures, technicians can submit a query request for known fault sound samples through terminal devices. The system can quickly retrieve the fragment sequences similar to the fault sound, helping technicians quickly locate the faulty equipment and the fault time point, so as to achieve efficient equipment maintenance and fault diagnosis.

[0062] In a smart home or smart office environment, smart voice assistant devices need to quickly respond to users' voice commands. The voice data collected by the devices is uploaded to the cloud server through the network and stored in a distributed index structure after processing. When the user issues a new voice command, the system uses cross-modal similarity retrieval technology to quickly match the voice data stored, find the most similar command fragment and execute the corresponding operation, so as to achieve efficient and accurate voice interaction.

[0063] Exemplarily, the acoustic data storage and retrieval method combining computer cloud computing provided in the embodiments of the present application can also be applied to other application scenarios. Only examples are given here, and the specific application scenarios are not limited.

[0064] In an exemplary embodiment, as Figure 1 shown, there is provided an acoustic data storage and retrieval method combining computer cloud computing. This embodiment is illustrated by taking the application of this method to the terminal in the foregoing implementation environment. It can be understood that this system can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server.

[0065] An acoustic data storage and retrieval method combining computer cloud computing provided in the embodiments of the present application includes the following steps 101 to step 104:

[0066] Step 101, perform temporal partitioning processing on the original acoustic data to generate an audio frame sequence with timestamp markings.

[0067] Specifically, by performing temporal partitioning processing on the original acoustic data and generating an audio frame sequence with timestamp markings, the continuous acoustic data can be discretized, facilitating subsequent feature extraction and index construction. The timestamp markings provide an accurate time reference for each audio frame, enabling the system to analyze and retrieve acoustic data in the temporal dimension, providing a basis for processing streaming acoustic data, and improving the efficiency and accuracy of data processing.

[0068] Step 102, based on the context window of the audio frame sequence, extract acoustic features and construct a context-aware acoustic feature vector.

[0069] Specifically, extracting acoustic features based on the context window of the audio frame sequence and constructing a context-aware acoustic feature vector can fully consider the context information of the audio frames in the time series. Compared with the method of only extracting single-frame features, the feature vector generated in this step can more comprehensively represent the acoustic characteristics of the audio frames and their temporal correlations, thereby providing richer semantic information for subsequent index construction and retrieval, and improving the accuracy and semantic relevance of the retrieval results.

[0070] Step 103: Construct a time-series-aware distributed index structure based on the acoustic feature vector to dynamically store the streaming acoustic data, and obtain a time-series-aware distributed index structure.

[0071] Specifically, constructing a time-series-aware distributed index structure based on the acoustic feature vector can efficiently store and manage large-scale streaming acoustic data. The distributed storage method enables the system to dynamically receive and store continuously input acoustic data, while supporting fast data access and retrieval. The time-series-aware index structure utilizes timestamp information to quickly locate audio data within a specific time period, improving the retrieval efficiency and reducing the retrieval time complexity, which is suitable for processing large-scale and real-time acoustic data.

[0072] Step 104: Perform cross-modal similarity retrieval on the query audio and the distributed index structure to obtain a target segment sequence that is spatio-temporally associated with the query audio.

[0073] Specifically, performing cross-modal similarity retrieval on the query audio and the distributed index structure can quickly and accurately find a target segment sequence that is relevant to the query audio both in terms of acoustic features and time. The cross-modal retrieval mechanism enables the system to perform effective similarity matching between acoustic data in different modalities (such as different recording devices, different environmental noises), improving the robustness of the retrieval. At the same time, combined with the time-series-aware index structure, the retrieval results are not only similar to the query audio in terms of acoustic features, but also can reflect their context relationships in the time series, thereby providing more semantically relevant and practical retrieval results for users.

[0074] The above method for acoustic data storage and retrieval combined with computer cloud computing generates an audio frame sequence with timestamp marks through temporal partitioning of the original acoustic data, ensuring the integrity of the time attributes of the data; extracts acoustic features based on the temporal context window of the audio frame sequence and constructs a context-aware acoustic feature vector to capture the temporal dependence relationship of sound events; constructs a time-aware distributed index structure according to the acoustic feature vector to realize the dynamic storage of streaming acoustic data and optimize the index update efficiency; finally, performs cross-modal similarity retrieval on the query audio and the distributed index structure, jointly analyzes the acoustic feature similarity and the timestamp alignment degree, and generates a target segment sequence with spatio-temporal association. This method effectively improves the retrieval accuracy and response speed of acoustic data in complex temporal scenarios through a time-aware storage architecture and a cross-modal retrieval mechanism.

[0075] In a possible embodiment, the temporal partitioning process of the original acoustic data to generate an audio frame sequence with timestamp marks may include:

[0076] Step 201, perform frame division on the original audio stream according to a preset time interval to generate a set of discrete audio frames.

[0077] Exemplarily, the system performs frame division on the original audio stream according to a preset time interval (e.g., 20 ms) to generate a set of discrete audio frames. Specifically, a frame division algorithm can be used to cut the continuous audio stream at a fixed duration to ensure that the time length of each frame of audio data is consistent, thereby generating a set of discrete and time-aligned audio frames. By dividing frames at a fixed time interval, the temporal alignment of the audio data is ensured, avoiding the loss or overlap of audio information caused by dynamic frame division in traditional methods, and providing a structured input for subsequent temporal correlation analysis.

[0078] Step 202, apply a window function to each frame in the set of discrete audio frames to generate preprocessed audio frames.

[0079] Exemplarily, apply a window function (e.g., Hamming window) to each frame in the set of discrete audio frames to generate preprocessed audio frames. Specifically, by multiplying the Hamming window function with each frame of audio data point by point, the spectral leakage phenomenon caused by frame division is eliminated, improving the accuracy of subsequent acoustic feature extraction.

[0080] Step 203, assign a unique timestamp to the preprocessed audio frames and mark the temporal context window of their preceding and succeeding associated frames to generate an audio frame sequence.

[0081] Exemplarily, timestamps accurate to the millisecond level can be generated for each frame according to the start time of the audio stream and the frame segmentation interval. At the same time, a context window range including several frames before and after each frame (for example, the first 5 frames and the last 5 frames) is defined for each frame to form a time-series related audio frame sequence. Through the unique timestamp marking and the context window definition, the time-series correlation relationship between audio frames can be established, directly supporting subsequent context-aware feature extraction and cross-modal retrieval, and reducing the risk of misjudgment caused by time-series misalignment.

[0082] In a possible embodiment, extracting acoustic features and constructing context-aware acoustic feature vectors based on the context window of the audio frame sequence may include:

[0083] Step 301, based on the time-series context window, perform a definition process on the range of the associated frames before and after the target frame according to the preset window radius parameter to generate a set of associated frame ranges.

[0084] Specifically, the context range of the target frame is clarified through the preset window radius parameter to ensure the time-series correlation of the acoustic features, avoiding feature deviation caused by fuzzy window range or the neglect of long-time-series correlation by traditional fixed window methods.

[0085] Step 302, perform acoustic feature extraction processing on each frame in the set of associated frame ranges to generate a set of high-dimensional feature vectors.

[0086] Exemplarily, a pre-trained acoustic feature extraction model (such as Wav2Vec 2.0) can be used to process each frame of audio data, convert each frame of audio into a high-dimensional feature vector, and form a set of high-dimensional feature vectors representing acoustic information. Among them, the acoustic features may include Mel Frequency Cepstral Coefficients (MFCC), energy, zero-crossing rate, etc., to reflect the acoustic characteristics of the audio frame.

[0087] Step 303, use the following formula to perform attention fusion processing on the set of high-dimensional feature vectors to obtain attention weights:

[0088]

[0089] where α t,i is the attention weight, q t is the query vector of the target frame, k i is the key vector of the associated frame, p i-t is the relative position encoding vector.

[0090] Exemplarily, by calculating the attention weights between the target frame and the associated frames, the contribution degree of each associated frame to the features of the target frame is dynamically adjusted; among them, the target frame generates a query vector, the associated frames generate key vectors, and a relative position encoding vector is introduced to capture the temporal position relationship. The features of the associated frames are weighted and summed according to the attention weights to generate an acoustic feature vector that fuses temporal context information. The weights of the associated frames are dynamically adjusted through the attention mechanism, effectively fusing temporal context information and enhancing the sensitivity of the features to the dynamic changes of acoustic events.

[0091] Step 304, generate an acoustic feature vector according to the attention weight result.

[0092] Specifically, the fused acoustic feature vector is normalized to eliminate the feature dimension deviation, generating a standardized context-aware acoustic feature vector to eliminate the feature scale difference, ensure the numerical comparability of different frame features, and improve the stability and accuracy of the subsequent retrieval algorithm.

[0093] In a possible embodiment, a time-series aware distributed index structure is constructed based on the acoustic feature vector to dynamically store the streaming acoustic data. The time-series aware distributed index structure may include:

[0094] Step 401, project the acoustic feature vector into a low-dimensional space to generate a set of index feature vectors.

[0095] Exemplarily, a pre-trained linear projection model can be used to reduce the dimension of the high-dimensional acoustic features, significantly reducing the storage and computational overhead while retaining the core acoustic information, and generating a set of low-dimensional index feature vectors suitable for efficient retrieval.

[0096] Step 402, dynamically insert a hierarchical graph index according to the timestamp order of the set of index feature vectors.

[0097] Specifically, a Hierarchical Navigable Small World (HNSW) structure can be used as the index basic framework, and the streaming data is inserted sequentially according to the timestamp order to ensure that new data is included in the index range in real time, while maintaining the dynamic update ability of the index structure, avoiding the interruption of the retrieval service caused by traditional full-scale reconstruction, and improving the system availability.

[0098] Step 403, in each layer of the hierarchical graph index, connect the nodes based on temporal proximity to generate a distributed index structure.

[0099] Specifically, when constructing the index connection, nodes with similar timestamps are preferentially selected to establish edge connections, and a time decay coefficient is introduced to adjust the connection weight, so that the index structure strengthens the temporal continuity while retaining the feature similarity, generating a distributed index structure that supports efficient spatio-temporal retrieval. Through the time decay coefficient and the adjacent connection strategy, the spatio-temporal correlation of the index structure is optimized, so that the retrieval process takes into account both the acoustic feature similarity and the event temporal continuity, and improves the retrieval accuracy in complex scenarios.

[0100] In a possible embodiment, the nodes are connected based on temporal proximity to generate a temporally aware distributed index structure, which may include:

[0101] Step 501, define the temporal proximity weight using the following formula:

[0102]

[0103] where w(u, v) is the temporal proximity weight, t u is the node timestamp, and λ is the time decay coefficient.

[0104] Specifically, the time decay coefficient is used to adjust the influence degree of time difference on proximity. The time difference is the absolute value difference between the timestamps of two nodes; by combining the time decay coefficient with the time difference calculation, a weight value representing the temporal correlation strength between nodes is generated, and the weight value decays as the time difference increases. By quantifying the influence of time difference on proximity, the temporal correlation strength between nodes is accurately represented, improving the accuracy and reliability of index structure construction and avoiding the temporal misalignment problem caused by only relying on feature similarity in traditional methods.

[0105] Step 502, screen adjacent nodes according to the temporal proximity weight to generate a set of candidate connection edges.

[0106] Specifically, a weight threshold can be set or sorted by weight value, and nodes with weight values meeting the conditions are selected from the temporal proximity range of the current node to form a set of candidate connection edges, ensuring that the connected nodes have both acoustic feature similarity and time continuity.

[0107] Step 503, when it is detected that there are cross-period edges in the set of candidate connection edges, delete the cross-period edges and supplement the in-period edges to generate a distributed index structure.

[0108] Specifically, cross-time edges are judged by a preset maximum time-span threshold (for example, the time difference exceeds 10 seconds), and such edges are deleted to eliminate the interference of time-series breaks. At the same time, within the time period, adjacent nodes are reselected according to the time-series proximity weight to establish connection edges, optimizing the time-series coherence of the index structure, generating a distributed index structure that supports efficient retrieval, ensuring the time-series continuity of the index structure, and enhancing the time coherence of retrieval results in complex scenarios (such as equipment failure event monitoring).

[0109] In a possible embodiment, performing cross-modal similarity retrieval on the query audio and the distributed index structure to obtain a target segment sequence that is spatio-temporally associated with the query audio may include:

[0110] Step 601, performing context-aware feature extraction processing on the query audio to generate a query feature vector and its timestamp.

[0111] Specifically, the same feature extraction model as in the index construction stage is used to process the query audio, generating a query feature vector representing acoustic information, and generating a timestamp accurate to the millisecond level based on the start time of the query audio and the frame interval. Through a unified feature extraction process, the consistency of the feature space between the query and the index data is ensured, avoiding the problem of feature deviation in cross-modal retrieval.

[0112] Step 602, performing similarity search processing in the distributed index structure to generate a candidate feature vector set.

[0113] Specifically, according to the Euclidean distance or cosine similarity between the query feature vector and the index feature vector, candidate feature vectors with a similarity higher than a preset threshold are screened from the index to form a candidate set containing spatio-temporally associated segments. Through the efficient retrieval ability of the distributed index structure, feature vectors with a relatively high similarity to the query audio can be quickly screened out from large-scale acoustic data, significantly improving the retrieval efficiency.

[0114] Step 603, using the following formula to calculate the timestamp alignment penalty value:

[0115] penalty(t q ,t i )=μ·|t q -t i |

[0116] Where penalty(t q ,t i ) is the time alignment penalty value, μ is the dynamic penalty coefficient, t q is the query timestamp, and t i is the candidate timestamp.

[0117] Specifically, according to the dynamically adjusted penalty coefficient and the absolute difference between the query timestamp and the candidate timestamp, the influence degree of the timestamp deviation on the retrieval result is quantified. The penalty value increases linearly with the increase of the timestamp difference, which can adaptively balance the feature similarity and the time continuity, and improve the retrieval accuracy in complex scenarios.

[0118] Step 604: Perform weighted processing on the timestamp alignment penalty value and the feature similarity to generate a comprehensive score, and sort the candidate feature vector set according to the comprehensive score to obtain the target segment sequence.

[0119] Specifically, performing weighted processing on the timestamp alignment penalty value and the feature similarity to generate a comprehensive score, and sorting the candidate feature vector set according to the comprehensive score to obtain the target segment sequence can comprehensively consider the acoustic similarity and the time alignment accuracy. The comprehensive score generated by the above method can more comprehensively reflect the spatio-temporal correlation between the candidate feature vector and the query audio, so that the finally obtained target segment sequence is not only similar to the query audio in terms of acoustic features, but also aligned with the query audio in time, significantly improving the accuracy and practicality of the retrieval result.

[0120] In a possible embodiment, sorting the candidate feature vector set according to the comprehensive score to obtain the target segment sequence may include:

[0121] Step 701: Filter the comprehensive score according to a preset threshold to generate a set of valid candidate segments.

[0122] Specifically, the preset threshold is dynamically set based on historical retrieval data or actual application scenario requirements, and the candidate segments with comprehensive scores higher than the threshold are screened out, and the low-confidence segments are excluded to form a set of valid candidate segments.

[0123] Step 702: Perform timestamp continuity verification processing on the set of valid candidate segments to generate verified target segments that pass the verification.

[0124] Specifically, check whether the timestamps of the candidate segments are continuous or within a preset time window range (for example, the time interval between adjacent segments does not exceed 1 second), eliminate the abnormal segments with time jumps or overlaps, and retain the candidate segments with continuous time series as the target segments that pass the verification. Through the continuity verification mechanism based on the time window, the problems of time series breakage or overlap caused by retrieval noise can be effectively eliminated, and the time coherence of the output segments can be improved.

[0125] Step 703: Output the verified target segments and determine the verified target segments as the target segment sequence.

[0126] Specifically, the target segments that have passed the timestamp continuity verification and successfully passed the verification are output. These segments are determined as the target segment sequence, which is the final retrieval result. It is not only similar to the query audio in acoustic features but also continuous in time, and can accurately reflect the spatio-temporal association of the query audio in the original acoustic data. This method completes the task of screening out the target segment sequence that best meets the query requirements from a large number of candidate segments, providing accurate and complete retrieval results for users.

[0127] In summary, the acoustic data storage and retrieval method provided by the embodiments of the present application combines computer cloud computing. By performing time-series frame processing on the original acoustic data, an audio frame sequence with timestamp tags is generated; based on the context window, the acoustic features of the target frame and its associated frames are extracted, and the time-series context information is fused through the attention mechanism to generate context-aware feature vectors, solving the problem that traditional single-frame features ignore time-series associations; after projecting the feature vectors into a low-dimensional space, they are dynamically inserted into the hierarchical graph index in chronological order, and the node connection strategy is optimized based on time-series proximity, preferentially connecting nodes with close times and suppressing cross-period edge interference, constructing a distributed index structure that takes into account both feature similarity and time-series continuity; after extracting the same context-aware features from the query audio, a similar candidate set is retrieved in the index, and the feature similarity and the timestamp alignment penalty value are jointly calculated to generate a comprehensive score; through dynamic threshold filtering and time continuity verification, a target segment sequence with spatio-temporal association is output. The above technical solutions can achieve the structured organization and dynamic update of acoustic data through a time-series-aware storage architecture, and use a cross-modal retrieval mechanism to fuse acoustic features and time continuity constraints, improving the retrieval accuracy and efficiency in complex time-series association scenarios (such as equipment fault monitoring, environmental event recognition).

[0128] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least a part of other steps or steps in other steps.

[0129] Based on the same inventive concept, an embodiment of the present application further provides an acoustic data storage and retrieval device combined with computer cloud computing for implementing the above-mentioned acoustic data storage and retrieval method combined with computer cloud computing. The solution provided by this device to solve the problem is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the acoustic data storage and retrieval device combined with computer cloud computing provided below can refer to the limitations on the acoustic data storage and retrieval method combined with computer cloud computing in the above text, and will not be repeated here.

[0130] In an exemplary embodiment, as Figure 2 shown, an acoustic data storage and retrieval device 20 combined with computer cloud computing is provided, including:

[0131] A timing division module 21, configured to perform timing division processing on the original acoustic data to generate an audio frame sequence with timestamp markings.

[0132] A feature extraction module 22, configured to extract acoustic features and construct context-aware acoustic feature vectors based on the context window of the audio frame sequence.

[0133] A distributed indexing module 23, configured to construct a timing-aware distributed indexing structure according to the acoustic feature vectors, dynamically store the streaming acoustic data, and obtain a timing-aware distributed indexing structure.

[0134] A query and retrieval module 24, configured to perform cross-modal similarity retrieval on the query audio and the distributed indexing structure to obtain a target segment sequence that is spatio-temporally associated with the query audio.

[0135] In a possible embodiment, the timing division module 21 may include:

[0136] An audio frame division unit 211, configured to divide the original audio stream into frames according to a preset time interval to generate a set of discrete audio frames.

[0137] An audio preprocessing unit 212, configured to apply a window function to each frame in the set of discrete audio frames to generate preprocessed audio frames.

[0138] An audio frame sequence generation unit 213, configured to assign a unique timestamp to the preprocessed audio frames and mark the timing context window of their front and back associated frames to generate an audio frame sequence.

[0139] In a possible embodiment, the feature extraction module 22 may include:

[0140] An associated frame definition unit 221, configured to perform definition processing on the front and back associated frame ranges of the target frame based on the timing context window according to a preset window radius parameter to generate a set of associated frame ranges.

[0141] A high-dimensional feature extraction unit 222 is configured to perform acoustic feature extraction processing on each frame within the associated frame range set, and generate a high-dimensional feature vector set.

[0142] An attention fusion unit 223 is configured to perform attention fusion processing on the high-dimensional feature vector set using the following formula to obtain attention weights:

[0143]

[0144] where α t,i is the attention weight, q t is the query vector of the target frame, k i is the key vector of the associated frame, and p i-t is the relative position encoding vector.

[0145] An acoustic feature vector generation unit 224 is configured to generate an acoustic feature vector according to the attention weight result.

[0146] In a possible embodiment, the distributed index module 23 may include:

[0147] A low-dimensional projection unit 231 is configured to project the acoustic feature vector into a low-dimensional space to generate an index feature vector set.

[0148] A hierarchical graph indexing unit 232 is configured to dynamically insert a hierarchical graph index according to the timestamp order of the index feature vector set.

[0149] A distributed index structure module 234 is configured to perform connection processing on nodes based on temporal proximity in each layer of the hierarchical graph index to generate a distributed index structure.

[0150] In a possible embodiment, the distributed index structure module 234 may include:

[0151] A temporal proximity weight definition subunit 2341 is configured to define a temporal proximity weight using the following formula:

[0152]

[0153] where w(u, v) is the temporal proximity weight, t u is the node timestamp, and λ is the time decay coefficient.

[0154] A candidate connection edge generation subunit 2342 is configured to screen neighboring nodes according to the temporal proximity weight to generate a candidate connection edge set.

[0155] The redundant edge removal subunit 2343 is used to delete the cross-time period edges and supplement the in-time period edges when it is detected that there are cross-time period edges in the candidate connection edge set, so as to generate a distributed index structure.

[0156] In a possible embodiment, the query retrieval module 24 may include:

[0157] The context-aware feature extraction unit 241 is used to perform context-aware feature extraction processing on the query audio to generate a query feature vector and its timestamp.

[0158] The similarity search unit 242 is used to perform similarity search processing in the distributed index structure to generate a set of candidate feature vectors.

[0159] The timestamp alignment penalty calculation unit 243 is used to calculate the timestamp alignment penalty value using the following formula:

[0160] penalty(t q ,t i )=μ·|t q -t i |

[0161] Where penalty(t q ,t i ) is the time alignment penalty value, μ is the dynamic penalty coefficient, t q is the query timestamp, and t i is the candidate timestamp.

[0162] The comprehensive score sorting unit 244 is used to perform weighted processing on the timestamp alignment penalty value and the feature similarity to generate a comprehensive score, and sort the set of candidate feature vectors according to the comprehensive score to obtain the target segment sequence.

[0163] In a possible embodiment, the comprehensive score sorting unit 244 may include:

[0164] The data filtering subunit 2441 is used to filter the comprehensive score according to a preset threshold to generate a set of valid candidate segments.

[0165] The continuity verification subunit 2442 is used to perform timestamp continuity verification processing on the set of valid candidate segments to generate the verified target segments that pass the verification.

[0166] The target segment sequence determination subunit 2443 is used to output the verified target segments and determine the verified target segments as the target segment sequence.

[0167] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps of an acoustic data storage and retrieval method combined with computer cloud computing as described above are implemented.

[0168] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0169] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial descriptions of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present disclosure solution. A person of ordinary skill in the art can understand and implement it without creative efforts.

[0170] The above embodiments only represent several implementation manners of the embodiments of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the embodiments of the application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the embodiments of the present application, several deformations and improvements can still be made, and these all belong to the protection scope of the embodiments of the present application.

Claims

1. A method for storing and retrieving acoustic data in combination with computer cloud computing, characterized in that: The method comprises: Performing time-series division processing on the original acoustic data to generate an audio frame sequence with a timestamp mark; Extracting acoustic features and constructing a context-aware acoustic feature vector based on a context window of the audio frame sequence; Constructing a time-series-aware distributed index structure according to the acoustic feature vector, dynamically storing streaming acoustic data, and obtaining a time-series-aware distributed index structure; A cross-modal similarity retrieval is performed on the query audio and the distributed index structure to obtain a target segment sequence that is temporally and spatially associated with the query audio.

2. The method according to claim 1, characterized in that The time-series division processing of the original acoustic data to generate an audio frame sequence with a timestamp mark includes: Performing frame processing on the original audio stream according to a preset time interval to generate a discrete audio frame set; Applying a windowing function to each frame in the discrete audio frame set to generate a preprocessed audio frame; A unique timestamp is assigned to the preprocessed audio frame, and a timing context window of the preceding and following frames is marked to generate the audio frame sequence.

3. The method according to claim 2, characterized in that The extracting acoustic features and constructing a context-aware acoustic feature vector based on the context window of the audio frame sequence includes: Based on the temporal context window, defining and processing the range of related frames before and after the target frame according to a preset window radius parameter to generate a set of related frame ranges; Performing acoustic feature extraction processing on each frame within the associated frame range set to generate a high-dimensional feature vector set; Use the following formula to perform attention fusion processing on the high-dimensional feature vector set to obtain the attention weight: Among them, α t,i is the attention weight, q t is the query vector of the target frame, k i is the key vector of the associated frame, p i-t is the relative position encoding vector; The acoustic feature vector is generated according to the attention weight result.

4. The method according to claim 1, characterized in that: The step of constructing a time-series-aware distributed index structure according to the acoustic feature vector, dynamically storing streaming acoustic data, and obtaining a time-series-aware distributed index structure includes: Projecting the acoustic feature vector into a low-dimensional space to generate a set of index feature vectors; Dynamically inserting a hierarchical graph index according to the timestamp order of the set of index feature vectors; In each layer of the hierarchical graph index, nodes are connected based on temporal proximity to generate the distributed index structure.

5. The method according to claim 4, characterized in that The connecting process of the nodes based on the temporal proximity to generate the temporal-aware distributed index structure includes: The temporal neighbor weights are defined using the following formula: Wherein, w(u,v) is the temporal neighbor weight, t u is the node timestamp, λ is the time decay coefficient; Filter neighboring nodes according to the temporal proximity weights to generate a set of candidate connection edges; When it is detected that there are cross-time period edges in the candidate connection edge set, the cross-time period edges are deleted and the intra-time period edges are supplemented to generate the distributed index structure.

6. The method according to claim 1, characterized in that The performing cross-modal similarity retrieval on the query audio and the distributed index structure to obtain a target segment sequence that is spatially and temporally associated with the query audio includes: Performing context-aware feature extraction processing on the query audio to generate a query feature vector and its timestamp; Performing similarity search processing in the distributed index structure to generate a set of candidate feature vectors; Use the following formula to calculate the timestamp alignment penalty: penalty(t q ,t i )=μ·|t q -t i | Among them, penalty(t q ,t i ) is the time alignment penalty value, μ is the dynamic penalty coefficient, t q is the query timestamp, t i is the candidate timestamp; The timestamp alignment penalty value and the feature similarity are weighted to generate a comprehensive score, and the candidate feature vector set is sorted according to the comprehensive score to obtain the target fragment sequence.

7. The method according to claim 6, characterized in that The step of sorting the candidate feature vector set according to the comprehensive score to obtain the target segment sequence includes: Filtering the comprehensive score according to a preset threshold to generate a set of valid candidate segments; Performing a timestamp continuity check on the valid candidate segment set to generate a check target segment that passes the check; The verification target segment is output, and the verification target segment is determined as the target segment sequence.

8. An acoustic data storage and retrieval device combined with computer cloud computing, characterized in that: include: A time division module is used to perform time division processing on the original acoustic data to generate an audio frame sequence with a timestamp mark; A feature extraction module, for extracting acoustic features and constructing a context-aware acoustic feature vector based on a context window of the audio frame sequence; A distributed index module, used for constructing a time-series-aware distributed index structure according to the acoustic feature vector, dynamically storing streaming acoustic data, and obtaining a time-series-aware distributed index structure; The query retrieval module is used to perform cross-modal similarity retrieval on the query audio and the distributed index structure to obtain a target segment sequence that is temporally and spatially associated with the query audio.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images