Multimodal sound picture storage platform

Abstract

The application provides a multimodal sound picture storage platform and relates to the technical field of information storage, and comprises a data receiving module, a feature extraction module and the like.The data receiving module is used for receiving audio, video and sensor log data in monitoring, and storing the audio, video and sensor log data into a distributed file device, an object storage service and a relational database respectively according to a time dimension to build a distributed storage system.The feature extraction module is used for extracting acoustic features of the audio and visual features of the video based on the distributed storage system, and associating the acoustic features, the visual features and a sensor log label of the same space-time point through a time stamp and a unique identification code of a monitoring device to generate a multimodal feature group with space-time labels.The application realizes efficient integrated storage and convenient calling of various modal data of sound and picture, and improves the comprehensiveness of information management, the accuracy of retrieval and the flexibility of content use.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of information storage and processing, and particularly relates to a multimodal sound picture storage platform. BACKGROUND

[0002] In the intelligent campus monitoring, the traditional sound picture storage technology has some defects in processing multimodal data. In the traditional technology, the audio, video and sensor data are often stored independently, and there is a lack of unified space-time indexing mechanism. For example, when a student gathering event occurs in the campus playground, the video recorded by the monitoring camera, the noise collected by the surrounding sound pickup and the device use log recorded by the sports equipment sensor may not be quickly associated through the same time + the same area due to the independent storage system. When the video is queried, the corresponding time period of the audio file needs to be manually matched. If the time stamp accuracy difference exceeds 1 second, the misplacement of "the crowd has gathered in the picture but the audio is still quiet" may occur, which affects the complete restoration of the event.

[0003] In addition, the retrieval of the traditional technology is mainly based on the simple label of a single mode, and lacks the cross-modal semantic association ability. For example, when the abnormal event of "running sound + fast moving shadow in the corridor of the teaching building during the lunch break" needs to be retrieved, the traditional system needs to retrieve the video with the label of "corridor" and the audio with the label of "running sound" respectively, and then manually compare the time coincidence degree. However, since the video label may only mark "corridor" without distinguishing "moving shadow", the audio label may only mark "noise" without detailing "running sound", and the label systems of the two are independent, it is easy to miss the retrieval. SUMMARY

[0004] The technical problem to be solved by the application is to provide a multimodal sound picture storage platform, realize efficient integrated storage and convenient calling of multimodal data of sound and picture, and improve the information management efficiency.

[0005] To solve the above technical problems, the technical scheme of the application is as follows:

[0006] In a first aspect, the multimodal sound picture storage platform comprises:

[0007] A data receiving module is configured to receive audio, video and sensor log data in monitoring, and store the data to a distributed file device, an object storage service and a relational database respectively according to the time dimension, so as to construct a distributed storage system;

[0008] A feature extraction module is configured to extract acoustic features of the audio and visual features of the video based on the distributed storage system, and associate the acoustic features, the visual features and the sensor log labels of the same space-time point through the time stamp and the unique identification code of the monitoring device, so as to generate a multimodal feature group with space-time labels;

[0009] An index construction module is configured to extract audio acoustic keywords and video visual keywords based on the multi-modal feature group, bind the keywords and the feature group by a hash mapping algorithm, establish a cross-modal semantic index table, and realize preliminary semantic association of multi-modal data.

[0010] A fusion storage module is configured to input the cross-modal semantic index table into a cross-modal deep learning mechanism, jointly encode audio keywords and video keywords, generate a feature vector of fused acoustic and visual semantics, and store the feature vector and original multi-modal data into a vector database to realize semantic calculation of the storage platform.

[0011] A calibration retrieval module is configured to retrieve the vector database, set multiple discrete reference points in a retrieval space, divide the retrieval space into multiple dynamic partitions based on spatial distribution characteristics of the discrete reference points, generate semantic calibration parameters according to data distribution characteristics of each dynamic partition, and dynamically adjust semantic representation weights of the feature vector.

[0012] Further, based on a distributed storage system, acoustic features of audio and visual features of video are extracted, and audio features, video features and sensor log labels of the same space-time point are associated by a timestamp and a unique identification code of a monitoring device to generate a multi-modal feature group with space-time labels, including:

[0013] For audio data streams stored by a distributed file device, mel frequency cepstral coefficients are calculated by pre-emphasis filtering to generate acoustic spectrum feature vectors.

[0014] For video data streams in an object storage service, depth visual feature vectors are extracted from key frames.

[0015] Based on sensor logs in a relational database, a device unique identification code, a millisecond-level timestamp and an event type code are extracted.

[0016] Taking the device unique identification code and the timestamp as index keys, the acoustic spectrum feature vectors are compensated for time offset according to an audio sampling rate, the depth visual feature vectors are calibrated for spatial position according to a video frame timestamp, and the acoustic and visual feature sequences are aligned by a dynamic time warping algorithm and bound with the event type code.

[0017] The acoustic spectrum feature vectors, the depth visual feature vectors and the event type code after space-time alignment are packaged into a structured data packet including a device unique identification code, an aligned timestamp and a feature version number to generate a multi-modal feature group.

[0018] Further, based on the multi-modal feature group, audio acoustic keywords and video visual keywords are extracted, the keywords and the feature group are bound by a hash mapping algorithm, a cross-modal semantic index table is established, and preliminary semantic association of multi-modal data is realized, including:

[0019] performing power spectral density analysis on the acoustic feature vectors in the multi-modal feature group, and calculating an acoustic segment with energy exceeding a threshold as an audio acoustic keyword;

[0020] performing region convolution response value detection on the visual feature vectors in the multi-modal feature group, and calculating a visual region with a response value exceeding a preset threshold as a video visual keyword;

[0021] forming a keyword pair in a spatio-temporal correlation by combining the audio acoustic keyword and the video visual keyword, and inputting the keyword pair into a local sensitive hashing algorithm for dimension reduction mapping to generate a binary hash code;

[0022] binding the hash code with a device unique identifier code and a timestamp of the multi-modal feature group, constructing an inverted index structure based on the bound hash code, including an acoustic keyword index item, a visual keyword index item, and a spatio-temporal identifier of the associated feature group, forming a cross-modal semantic index table, and realizing preliminary semantic correlation of multi-modal data.

[0023] Further, performing power spectral density analysis on the acoustic feature vectors in the multi-modal feature group, and calculating an acoustic segment with energy exceeding a threshold as an audio acoustic keyword, including:

[0024] based on the acoustic spectral feature vector, performing frame processing with a preset time window length to obtain an equal-interval audio frame sequence, and based on the audio frame sequence, performing discrete Fourier transform on the windowed frame to obtain a frequency domain power spectral density value sequence;

[0025] based on the power spectral density value sequence, calculating a sliding energy mean value within a continuous time window, dynamically generating an energy determination threshold combined with the energy distribution of the sequence, and comparing the power spectral density value sequence with the energy determination threshold to mark the frame exceeding the threshold as a candidate active frame;

[0026] merging the candidate active frames, and when the time interval of adjacent frames is less than a preset time tolerance and the energy difference is less than a preset energy tolerance, obtaining a continuous acoustic segment;

[0027] extracting a start timestamp, a duration, and an average energy intensity from the continuous acoustic segment to obtain a structured audio acoustic keyword.

[0028] Further, inputting the cross-modal semantic index table into a cross-modal deep learning mechanism to jointly encode the audio keyword and the video keyword, generating a feature vector fused with acoustic and visual semantics, and storing the feature vector associated with the original multi-modal data into a vector database to realize semantic calculation of the storage platform, including:

[0029] Based on the cross-modal semantic index table, an audio acoustic keyword sequence corresponding to an acoustic keyword index item and a video visual keyword sequence corresponding to a visual keyword index item are extracted;

[0030] The audio acoustic keyword sequence is input into a bidirectional long short-term memory network to generate a context-aware acoustic semantic vector, and the video visual keyword sequence is input into a graph convolution network to generate a spatially related visual semantic vector;

[0031] The acoustic semantic vector and the visual semantic vector are input into a cross-modal attention alignment layer to calculate an acoustic-visual semantic correlation weight matrix and generate an aligned fusion semantic vector;

[0032] The fusion semantic vector is input into a fully connected layer for dimension compression to generate a unified semantic feature vector, and a mapping relationship is established with a device unique identification code and a timestamp included in the multi-modal feature group;

[0033] The mapping relationship is stored in a vector database to generate a structured entry including a vector data, a device identification and a timestamp triple, and realize semantic calculation of a storage platform.

[0034] Further, the acoustic semantic vector and the visual semantic vector are input into a cross-modal attention alignment layer to calculate an acoustic-visual semantic correlation weight matrix and generate an aligned fusion semantic vector, including:

[0035] The acoustic semantic vector is taken as a query vector, the visual semantic vector is taken as a key vector and a value vector, the correlation weight of the acoustic semantic to the visual semantic is calculated through a multi-head attention mechanism, and an acoustic-visual attention weight matrix is generated;

[0036] Based on the acoustic-visual attention weight matrix, the visual semantic vector is weighted aggregated to generate an acoustic-guided visual feature vector;

[0037] The visual semantic vector is taken as a query vector, the acoustic semantic vector is taken as a key vector and a value vector, the correlation weight of the visual semantic to the acoustic semantic is calculated through a multi-head attention mechanism, and a visual-acoustic attention weight matrix is generated;

[0038] Based on the visual-acoustic attention weight matrix, the acoustic semantic vector is weighted aggregated to generate a visual-guided acoustic feature vector;

[0039] The visual feature vector and the acoustic feature vector are integrated to form a cross-modal interaction feature, and the cross-modal interaction feature is input into a fully connected projection layer for feature fusion and dimension reduction to generate a preliminary fusion vector;

[0040] The preliminary fusion vector, the original acoustic semantic vector and the original visual semantic vector are connected in residual, and the features after residual connection are processed by layer normalization to generate an aligned fusion semantic vector.

[0041] Further, the vector database is searched, a plurality of discrete reference points are set in the search space, and the search space is divided into a plurality of dynamic partitions based on the spatial distribution characteristics of the discrete reference points, semantic calibration parameters are generated according to the data distribution characteristics of each dynamic partition, and the semantic representation weight of the feature vector is dynamically adjusted, including:

[0042] Based on the semantic distribution density of the feature vector in the vector space, the feature vector with a distribution density greater than a preset threshold is determined as a discrete reference point;

[0043] According to the discrete reference points, the vector space is divided into a plurality of dynamic partitions by the Voronoi diagram algorithm, and each dynamic partition includes all feature vectors closest to the corresponding discrete reference points of the partition;

[0044] Based on each divided dynamic partition, the local spatial density value of the feature vector in the partition is calculated, and the cross-modal similarity distribution of the acoustic feature vector and the visual feature vector in the partition is extracted;

[0045] Based on the local spatial density value and the cross-modal similarity distribution, a partition-level weight adjustment coefficient for balancing the contribution degrees of acoustic and visual modalities is generated;

[0046] In the search process, according to the target dynamic partition to which the query vector belongs, the partition-level weight adjustment coefficient is called to dynamically weight the semantic representation weight of the acoustic feature and the visual feature in the similarity calculation.

[0047] Further, based on each divided dynamic partition, the local spatial density value of the feature vector in the partition is calculated, and the cross-modal similarity distribution of the acoustic feature vector and the visual feature vector in the partition is extracted, including:

[0048] Obtain the spatial coordinates of all feature vectors in the dynamic partition to form a partition spatial coordinate set;

[0049] Input the partition spatial coordinate set into the kernel density estimation algorithm to calculate the feature vector distribution density in the unit hyper-volume, and obtain the local spatial density value;

[0050] Based on the local spatial density value, each feature vector in the partition is traversed, the acoustic feature vector sub-component and the visual feature vector sub-component stored therein are extracted, and an acoustic-visual feature pair set is generated;

[0051] For each feature pair in the acoustic-visual feature pair set, a cosine similarity operation is performed to generate a set of cosine similarity values of all feature vectors in the partition;

[0052] Based on the cosine similarity value set, the similarity value range is divided according to a preset number interval, the proportion of the number of feature vectors in each interval is counted, and a probability distribution of acoustic-visual cross-modal similarity, that is, a cross-modal similarity distribution, is generated.

[0053] In a second aspect, a computing device includes:

[0054] one or more processors;

[0055] a storage device storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the platform.

[0056] In a third aspect, a computer-readable storage medium stores a program, which, when executed by a processor, implements the platform.

[0057] The above scheme of the present application at least includes the following beneficial effects:

[0058] The audio, video and sensor log data are classified and stored by the distributed storage system, and the device unique identification code and the time stamp are used as the core link to realize the accurate space-time alignment of the cross-modal data. Compared with the mode in the prior art that the multi-modal data are dispersedly stored and associated by relying on manual matching, the data integration cost is reduced, and it is ensured that the acoustic, visual and sensor information of the same event form a complete closed loop in the space-time dimension. According to the cross-modal semantic index table and the deep learning joint coding mechanism, the platform constructs the deep semantic association of the audio and video features, breaks through the limitation of the traditional single-modal label retrieval, and when a complex event such as "argument under the dormitory building + conflict of body movement" is retrieved, the associated data can be directly located by fusing the semantic vectors without manually comparing the audio and video, so that the missed detection caused by the independent label system and the time misalignment is avoided, and the retrieval accuracy and efficiency are improved compared with the traditional mode.

[0059] The dynamic partitioning and semantic calibration mechanism can adjust the weight of the acoustic and visual features in the semantic calculation in real time according to the data distribution characteristics (such as local spatial density and cross-modal similarity). The platform realizes the associated storage of the fused semantic vectors and the original data through the vector database, not only supports efficient retrieval, but also provides a structured data basis for secondary analysis. In the field of intelligent security and event tracing, the associated mode of "abnormal sound + abnormal behavior" can be quickly mined based on the multi-modal data stored by the platform, and the data basis for security warning and responsibility identification is provided, so as to promote the transformation of data from "passive storage" to "active empowerment". BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 is a schematic diagram of the multi-modal acoustic picture storage platform provided by the embodiment of the present application.

[0061] Figure 2 is a flowchart of a process for power spectral density analysis of acoustic feature vectors in a multi-modal feature group, according to an embodiment of the present application. DETAILED DESCRIPTION

[0062] Exemplary embodiments of the present disclosure will be described more fully hereinafter with reference to the accompanying drawings; however, they are not limited to the embodiments set forth herein but can be implemented in various forms. The present disclosure will be described herein with reference to the drawings, in which like reference numerals indicate like elements throughout the various figures. The embodiments of the present disclosure are provided to more completely explain the present disclosure to one of ordinary skill in the art and thereby fully convey the scope of the present disclosure to those skilled in the art.

[0063] As shown in Figure 1 , an embodiment of the present application proposes a multi-modal acoustic picture storage platform, comprising:

[0064] a data receiving module for receiving audio, video and sensor log data in monitoring, and storing the data to a distributed file device, an object storage service and a relational database respectively according to time dimension, to build a distributed storage system;

[0065] a feature extraction module for extracting acoustic features of audio and visual features of video based on the distributed storage system, and associating audio features, video features and sensor log labels of the same space-time point through a timestamp and a unique identification code of a monitoring device, to generate a multi-modal feature group with space-time markers;

[0066] an index construction module for extracting audio acoustic keywords and video visual keywords based on the multi-modal feature group, and binding the keywords and the feature group through a hash mapping algorithm, to establish a cross-modal semantic index table and realize preliminary semantic association of multi-modal data;

[0067] a fusion storage module for inputting the cross-modal semantic index table into a cross-modal deep learning mechanism, to jointly encode audio keywords and video keywords, generate feature vectors with fused acoustic and visual semantics, and store the feature vectors and original multi-modal data to a vector database after association, to realize semantic calculation of the storage platform;

[0068] a calibration retrieval module for retrieving the vector database, setting multiple discrete reference points in a retrieval space, dividing the retrieval space into multiple dynamic partitions based on spatial distribution characteristics of the discrete reference points, generating semantic calibration parameters according to data distribution characteristics of each dynamic partition, and dynamically adjusting semantic representation weights of the feature vectors.

[0069] In the embodiments of the present application, by storing audio, video and sensor log data respectively into a distributed file device, an object storage service and a relational database in the time dimension, a unified distributed storage system is constructed, solving the problem of mixed storage of multi-modal data, format incompatibility in traditional storage, not only meeting the continuous read-write demand of audio stream, the large-capacity storage demand of video file, but also adapting to the structured query demand of sensor log, improving the pertinence and efficiency of data storage. For example, in a smart campus, the classroom teaching audio can be stored with low delay through the distributed file device, the playground monitoring video can be expanded massively relying on the object storage service, and the classroom light, temperature and humidity sensor logs can be quickly retrieved through the relational database. Each type of data is independently managed and associated based on time.

[0070] By extracting audio acoustic features and video visual features, and performing spatio-temporal alignment based on device unique identification code and timestamp, the problem of "spatio-temporal dislocation" of multi-modal data in traditional technology is solved. For example, through timestamp calibration and dynamic time warping algorithm, the "vehicle honking audio" and "vehicle fast passing video frame" in the campus road monitoring can be accurately bound, avoiding "sound and picture out of sync" caused by the difference between audio sampling rate and video frame rate. At the same time, the aligned features are packaged into structured data packets containing device identification, timestamp and feature version number, forming a standardized multi-modal feature group, ensuring the consistency of different modal features in format and dimension, and reducing the redundant cost of cross-module data conversion.

[0071] By extracting audio acoustic keywords and video visual keywords, and using hash mapping algorithm to establish a cross-modal semantic index table, the limitations of traditional single-modal index are broken through. The acoustic keyword index item, visual keyword index item and spatio-temporal identifier contained in the index table realize one-key association of multi-modal data, shorten the positioning time from keywords to original data, even in the face of massive monitoring data, the relevant feature group can be quickly matched through hash coding, the response speed of retrieval is improved compared with traditional way. Through cross-modal deep learning mechanism, the keywords are jointly encoded to generate a feature vector that integrates acoustic and visual semantics, solving the problem of only storing data without semantic association in traditional storage. At the same time, the associated storage of feature vector and original data enables the vector database to support fast retrieval based on semantics, and also enables the original audio and video to be located reversely through semantic vector, realizing bidirectional mapping of semantics-data. This integrated design of storage+semantic computing upgrades the platform from a simple data warehouse to a semantic hub with intelligent analysis capability.

[0072] By dynamic partitioning and semantic calibration parameter adjustment, the problem of poor scene adaptability caused by fixed weight in traditional retrieval is solved. For example, in a campus gym, when retrieving a falling athlete, the weight of video visual features (such as body posture) will be automatically increased due to the high background noise; while in a quiet study room, the weight of audio features will be increased when retrieving abnormal sound, ensuring that the retrieval result meets the actual needs of the scene. In addition, based on the design of discrete reference point partitioning dynamic partitioning, the retrieval space can be dynamically adjusted according to the data distribution, avoiding the local data matching error caused by the global unified retrieval strategy. For example, according to the data distribution difference of different areas (teaching area, sports area, living area) in the campus, region-specific calibration parameters can be generated to keep the retrieval accuracy stable in each scene.

[0073] In a preferred embodiment of the present application, based on the distributed storage system, the acoustic features of the audio and the visual features of the video are extracted, and the audio features, video features and sensor log labels of the same space-time point are associated through the timestamp and the unique identification code of the monitoring device to generate a multi-modal feature group with space-time markers, which can include:

[0074] For the audio data stream stored by the distributed file device, the mel frequency cepstral coefficient is calculated by pre-emphasis filtering to generate an acoustic spectrum feature vector;

[0075] For the video data stream in the object storage service, the key frame is extracted to obtain a deep visual feature vector;

[0076] Based on the sensor log in the relational database, the device unique identification code, millisecond timestamp and event type code are extracted;

[0077] Taking the device unique identification code and the timestamp as the index key, the acoustic spectrum feature vector is compensated for the time offset according to the audio sampling rate; the deep visual feature vector is calibrated for the spatial position according to the video frame timestamp, and the acoustic and visual feature sequences are aligned through the dynamic time warping algorithm, and are bound with the event type code;

[0078] The acoustic spectrum feature vector, the deep visual feature vector and the event type code after space-time alignment are packaged as a structured data packet including the device unique identification code, the alignment timestamp, the feature version number, to generate a multi-modal feature group.

[0079] In the embodiment of the present application, the audio data stream is pre-emphasis filtered, which is to enhance the energy of high frequency signals in the audio. Specifically, each sample in the audio data is processed, and the current sample is subtracted by the product of a fixed coefficient and the previous sample. The fixed coefficient is 0.97. For example, when the nth sample is processed, the pre-emphasis sample value is the original nth sample value minus 0.97 times the original (n-1)th sample value. After pre-emphasis, a framing operation is performed to divide the continuous audio signal into a plurality of short frames. The length of each frame is generally 20-40 milliseconds, and 25 milliseconds is selected as a frame here. There is 50% overlap between adjacent two frames, that is, each frame is moved by 12.5 milliseconds. When framing, each frame is multiplied by a Hanning window function to reduce the spectral leakage of the frame edge. The calculation of the Hanning window is that, for each position in a frame, 0.5 is subtracted by 0.5 times the cosine value corresponding to the position. The angle of the cosine value is 2π times the index of the position in the frame divided by the length of the frame minus 1.

[0080] Then, Fourier transform is performed to convert each frame from the time domain to the frequency domain to obtain the spectrum of each frame. Then, the power spectrum is calculated, that is, the real part and the imaginary part of the Fourier transform result are squared and added, and then divided by the length of the frame. Then, the grey wolf optimization algorithm is introduced to optimize the design of the mel filter bank. First, the number of mel filters required is determined, which is assumed to be 40. These filters are regarded as "prey", and each individual in the grey wolf group represents a combination of filter parameters, including the center frequency and bandwidth of each filter. According to the selection mechanism of alpha, beta and delta wolves in the grey wolf optimization algorithm, the fitness of each individual in the group is calculated. The fitness function is set to maximize the discrimination of the filter bank to the audio features. For example, the difference in features of different types of audio signals after passing through the filter bank is calculated to determine the alpha wolf, which represents the final filter parameter combination. The beta wolf is the second highest fitness, and the delta wolf is the third highest fitness.

[0081] Then, other gray wolves (omega wolves) learn from alpha, beta and delta wolves by updating their own positions, constantly adjusting filter parameters. In each iteration, omega wolves update parameters according to the distance from alpha, beta and delta wolves. The distance is calculated based on the Euclidean distance in the parameter space. Through multiple iterations, the final Mel filter bank parameters are found. The power spectrum is filtered using the adjusted Mel filter bank. Each filter corresponds to a frequency band. The energy of the power spectrum in each filter band is added to obtain 40 Mel frequency coefficients. The coefficients are then logarithmically transformed to obtain the log Mel frequency spectrum. Finally, the discrete cosine transform is performed on the log Mel frequency spectrum to remove the correlation between the coefficients. The first 12-13 coefficients are taken as the Mel frequency cepstral coefficients to generate the acoustic spectral feature vector.

[0082] After reading the video data stream from the object storage service, a decoding operation is performed to convert the video stream into a frame-by-frame image. Then, the extraction interval of key frames is determined, which is set to 10 key frames per second. In the process of extracting key frames, the gray wolf optimization algorithm is introduced to select the final key frames. The video frame sequence is regarded as a search space, and each key frame candidate is regarded as a "prey". Each individual in the gray wolf population represents a set of key frame indexes. The fitness function is defined as the selected key frames that can retain the important information of the video to the greatest extent. For example, by calculating the total sum of pixel differences between adjacent frames, the greater the difference, the more likely the frame is a key frame, and the higher the fitness.

[0083] According to the steps of the gray wolf optimization algorithm, the key frame combinations corresponding to alpha, beta and delta wolves are first selected. Other individuals update their key frame indexes by learning from the three wolves. For example, omega wolves will adjust the position of the selected key frame according to the distance from alpha wolves. The closer the distance, the smaller the adjustment range, so as to approach the final key frame combination. After multiple iterations, the final key frame set is determined. For each extracted key frame, preprocessing is performed, including adjusting the image size to a fixed size, such as 224x224 pixels, and then normalizing the RGB values of each pixel from the range of 0-255 to the range of 0-1, i.e. dividing each pixel value by 255.

[0084] Then, the pre-trained convolutional neural network is used to extract deep visual features. The pre-processed key frame is input into the first convolutional layer of the convolutional neural network. Each convolution kernel is convolved with the corresponding region of the key frame. Through the sliding window method, the convolution kernel is multiplied by the pixel value of the corresponding region and summed, and then a bias value is added to obtain a feature value of the layer. Then, a pooling operation is performed. A 2x2 pooling window is selected, and the maximum value of the feature values in the window is taken as the result of the pooling. Such convolution and pooling operations will go through multiple levels to gradually extract the deep visual features of the key frame. Finally, the feature values output by the fully connected layer are combined to generate a deep visual feature vector.

[0085] The sensor log data is read from the relational database. Each log data contains multiple fields. First, the device unique identification code is extracted, which is a pre-set string or number combination. It can be directly read from the corresponding field of the log, such as obtaining the identification code "SN123456789" from the "device_id" field. For the extraction of millisecond timestamps, there is usually a field in the log that records the event occurrence time, such as "timestamp". Its format may be "YYYY-MM-DD HH:MM:SS.XXX", and the specific millisecond value is extracted, such as "2025-07-11 10:30:25.123". The event type code is extracted according to the description of the event in the log, and different types of events are mapped to pre-defined codes, such as "device startup" corresponding to code "001", "data transmission" corresponding to code "002", etc. The corresponding code can be matched directly according to the event description field in the log.

[0086] In the extraction process, the grey wolf optimization algorithm is introduced to adjust the efficiency of data extraction. The records in the database are regarded as "prey", and each individual of the grey wolf population represents a data extraction path or strategy, such as which fields to extract first and what query method to use. The fitness function is set as the time and accuracy of extracting data. The shorter the time and the higher the accuracy, the better the fitness. Through the update and iteration of individuals in the grey wolf optimization algorithm, the final extraction strategy is found to improve the extraction efficiency.

[0087] The time offset compensation of the acoustic spectrum feature vector is processed, the audio sampling rate is known, such as 44100Hz, that is, 44100 samples per second, for each acoustic spectrum feature vector, the offset from the standard timestamp is calculated according to the corresponding sampling time, for example, if the sampling time of a certain feature vector is 0.002 seconds earlier than the standard timestamp, the feature vector needs to be moved back 0.002 seconds on the time axis, the specific calculation of the movement is to adjust the position of the feature vector in the sequence so that the corresponding time is consistent with the standard timestamp, in this process, the grey wolf optimization algorithm is used to adjust the calculation of the offset, the possible offset range is used as the search space, each grey wolf individual represents an offset value, the fitness function is defined as the matching degree of the feature vector after the offset compensation and other related features in time, the higher the matching degree, the better the fitness, through the search and iteration of the grey wolf group, the most accurate offset is found, and the accuracy of the time compensation is improved.

[0088] For spatial position calibration of deep visual feature vectors, the position of the frame in the entire video sequence is determined according to the timestamp of the video frame, if the video has jitter or displacement, the spatial position of the current frame needs to be calibrated according to the positional relationship of adjacent frames, for example, if the current frame is offset by 5 pixels in the horizontal direction compared with the previous frame, the feature vector of the current frame is moved back by 5 pixels in the horizontal direction to calibrate the spatial position; the grey wolf optimization algorithm is introduced, the offset of the spatial position is used as the search space, each individual represents a set of offset parameters (the number of offset pixels in the horizontal and vertical directions), the fitness function is the spatial continuity of the calibrated feature vector and the surrounding features, the better the continuity, the higher the fitness, and the final offset parameters are found through algorithm iteration, improving the accuracy of spatial calibration.

[0089] Then the dynamic time warping algorithm is used to align the acoustic and visual feature sequences, the distance between each point in the two feature sequences is calculated, such as the Euclidean distance between a point in the acoustic feature sequence and a point in the visual feature sequence, that is, the square of the difference in each dimension is added and then the square root is taken, then a final path is found, so that the total distance of the two sequences along the path is minimized; in the dynamic time warping process, the grey wolf optimization algorithm is used to adjust the search of the path, the possible path is used as the "prey", the grey wolf individual represents different path choices, the fitness function is the total distance corresponding to the path, the smaller the total distance, the higher the fitness, through the search of the grey wolf group, the final alignment path is found, the accuracy of the feature sequence alignment is improved, and finally the aligned acoustic and visual feature sequences are bound with the event type code, that is, the event type code corresponding to each feature sequence is attached to the sequence to form a one-to-one correspondence.

[0090] The integrated acoustic spectrum feature vector, depth visual feature vector and event type code after time and space alignment is first added with a device unique identification code to ensure the clear attribution of the feature, and then the aligned timestamp is determined, which is a unified time marker after calibration and alignment, accurate to the millisecond level. The determination of the feature version number is based on the feature extraction and processing process. If a new algorithm or parameter is used in the processing process, the version number is updated, such as from "V1.0" to "V1.1". In the process of packaging the structured data packet, the grey wolf optimization algorithm is introduced to adjust the structure of the data packet. The arrangement mode and storage format of each component of the data packet (such as the identification code, timestamp, feature vector, etc.) are taken as the search space, and each grey wolf individual represents a structure scheme. The fitness function is set as the storage efficiency and reading speed of the data packet. The higher the efficiency and the faster the speed, the better the fitness. Through the algorithm, the final data packet structure is determined, and then each item of content is packaged according to the structure to generate a multi-modal feature group.

[0091] In the process of generating acoustic spectrum feature vectors and depth visual feature vectors, the grey wolf optimization algorithm is introduced to adjust the mel filter bank and key frame selection, so that the extracted features can better reflect the essential features of audio and video, reduce the interference of irrelevant information, and improve the accuracy of the features. Through the grey wolf optimization algorithm, the time offset calculation of the acoustic feature vector and the spatial position calibration parameter of the visual feature vector are adjusted, and the alignment path is adjusted in dynamic time warping, so that the acoustic and visual feature sequences can be more accurately aligned, avoiding the problem of feature mismatch caused by time and space offset. In the process of sensor log information extraction and data packet packaging, the grey wolf optimization algorithm is used to find the final extraction strategy and data packet structure, reducing the data extraction time and data packet storage occupancy, while speeding up the data reading speed, and improving the overall data processing efficiency. Through the grey wolf optimization algorithm in the whole processing process, the consistency of the generated multi-modal feature group in time and space is better, the binding of the features and the event types is more accurate, and the introduction of the version number also facilitates the management and tracing of the features, thereby enhancing the reliability and usability of the multi-modal feature group. Accurate alignment and reliable multi-modal feature group provide convenient conditions for the fusion analysis of different modal data, making it possible to more effectively mine the associated information between audio, video and sensor data, and expanding the application scenarios of multi-modal data.

[0092] As shown in Figure 2 In a preferred embodiment of the present application, based on the multi-modal feature group, audio acoustic keywords and video visual keywords are extracted, and the keywords are bound to the feature group through a hash mapping algorithm to establish a cross-modal semantic index table, realizing the preliminary semantic association of multi-modal data, which can include:

[0093] The power spectrum density analysis is performed on the acoustic feature vectors in the multi-modal feature group, and the acoustic segments with energy exceeding a threshold are calculated as audio acoustic keywords, specifically comprising: based on the acoustic spectrum feature vectors, frame processing is performed with a preset time window length to obtain an equal-interval audio frame sequence, and based on the audio frame sequence, discrete Fourier transform is performed on the windowed frames to obtain a frequency domain power spectrum density value sequence; based on the power spectrum density value sequence, sliding energy mean is calculated within a continuous time window, energy determination threshold is dynamically generated combined with the energy distribution of the sequence, and the power spectrum density value sequence is compared with the energy determination threshold, and the frames exceeding the threshold are marked as candidate active frames; the candidate active frames are merged, and when the time interval of adjacent frames is less than a preset time tolerance and the energy difference is less than a preset energy tolerance, continuous acoustic segments are obtained; the start time stamp, duration and average energy intensity of the continuous acoustic segments are extracted to obtain structured audio acoustic keywords.

[0094] The region convolution response value detection is performed on the visual feature vectors in the multi-modal feature group, and the visual regions with response values exceeding a preset threshold are calculated as video visual keywords.

[0095] The audio acoustic keywords and the video visual keywords are combined to form a spatio-temporal associated keyword pair, and the keyword pair is input into a local sensitive hash algorithm for dimension reduction mapping to generate a binary hash code.

[0096] The hash code is bound with a device unique identification code and a time stamp of the multi-modal feature group, and based on the bound hash code, an inverted index structure is constructed, including an acoustic keyword index item, a visual keyword index item and a spatio-temporal identifier of the associated feature group, to form a cross-modal semantic index table, realizing preliminary semantic association of multi-modal data.

[0097] In the embodiment of the application, acoustic feature vectors in the multi-modal feature group are collected, which are obtained by processing the original audio signal. For example, the audio signal is first sampled at a fixed sampling frequency (such as 44100 times per second), and each sampling obtains a value representing the sound intensity at that instant. These values are arranged in chronological order to form an acoustic feature vector, and the unit of each value is amplitude (such as millivolt). When performing power spectrum density analysis, the time interval of segment division is first determined, such as 20 milliseconds. Assuming that the audio duration corresponding to the acoustic feature vector is 1 minute, a total of 3000 segments will be divided (60 seconds ÷ 0.02 seconds = 3000).

[0098] The total energy of each 10-millisecond segment is calculated. First, the number of sampling points in the 10-millisecond segment is determined. For example, if the sampling frequency is 44100 times per second, then there are 441 sampling points in 10 milliseconds (44100 times per second x 0.01 second = 441). The amplitude values of the 441 sampling points are squared one by one (for example, if the amplitude of a sampling point is 0.5 millivolts, the square is 0.25 square millivolts). Then, the sum of all squared values is obtained, which is the total energy of the segment (unit: square millivolts). Next, the energy threshold is determined. For example, 1000 pieces of audio of different scenes are collected, and the total energy of all segments of each piece of audio is calculated according to the above method. Then, the distribution of the total energy is counted, and the energy range with a higher frequency is found. For example, if the energy of 90% of the segments is lower than 5000 square millivolts, then 5000 square millivolts is set as the initial threshold. The threshold can be adjusted according to the amount of key audio information to be retained in actual application. For example, if more possible key words are to be retained, the threshold is appropriately lowered. If the key words are to be simplified, the threshold is increased. Finally, the total energy of each 10-millisecond segment is compared with the threshold. If the total energy of the segment is greater than the threshold, the 10-millisecond segment is marked as an acoustic segment corresponding to the audio acoustic key word, and the start time and end time of the segment are recorded.

[0099] The visual feature vector in the multi-modal feature group comes from each frame of image of the video. Each frame of image is composed of a large number of pixel points, each of which has a corresponding brightness value (such as an integer between 0 and 255) and color value (such as the value of RGB three channels, each channel is also an integer between 0 and 255). The values of these pixel points are arranged in row and column order to form a visual feature vector. When performing regional convolution response value detection, the division method of the image region is first determined. For example, a 1920x1080 resolution image is divided into 192x108 small regions (each region is 10x10 pixels). For each 10x10 pixel region, a 3x3 convolution kernel (the values in the convolution kernel are preset according to common image features, such as the convolution kernel for edge detection may be a combination of 0 in the center and positive and negative values around the center) is used for convolution operation. The center of the convolution kernel is aligned with a certain pixel in the region, and then the values at each position of the convolution kernel are multiplied by the brightness value of the corresponding pixel (for example, if the convolution kernel value is 1 and the corresponding pixel brightness value is 100, the product is 100). The sum of the 9 product results is the convolution value of the pixel position. In the same way, the convolution values of each pixel position in the region are calculated. Then, the maximum value among these convolution values is found as the preliminary response value of the region.

[0100] Collect 500 frames of images of different scenes, calculate the initial response value of all areas of each frame of image, and count the distribution of these response values, such as finding that 80% of the area response values are lower than 2000, and setting 2000 as the initial threshold, and adjusting according to the importance of the visual features to be recognized in practice, when the initial response value of a certain area is greater than the preset threshold, the area is marked as the visual area corresponding to the video visual keyword, and the coordinate position of the area in the image (such as the row and column numbers of the upper left corner pixel and the row and column numbers of the lower right corner pixel) is recorded.

[0101] When the keywords of spatiotemporal association are composed, first, the time information of all audio acoustic keywords is sorted, each acoustic keyword has a starting time (such as 1.2 seconds) and an ending time (such as 1.4 seconds), and then the time information of all video visual keywords is sorted, each visual keyword comes from a frame of video, and the frame of video has a corresponding timestamp (such as 1.3 seconds); then time matching is performed, for each acoustic keyword, find the visual keyword whose timestamp is within its starting time to ending time range, such as a certain acoustic keyword whose time range is 1.2-1.4 seconds, find the visual keyword whose timestamp is 1.3 seconds; if the time range of the acoustic keyword is 2.0-2.5 seconds, find the visual keywords whose timestamps are 2.1 seconds, 2.3 seconds and 2.4 seconds, respectively, to form keyword pairs, if a certain acoustic keyword has no corresponding visual keyword within its time range, or a certain visual keyword has no corresponding acoustic keyword, then temporarily do not form a keyword pair.

[0102] When the keyword pair is input into the local sensitive hashing algorithm, first, the feature information of the keyword pair is converted into a high-dimensional vector, the features of the audio acoustic keyword include the total energy and duration of the segment, and the features of the video visual keyword include the average brightness and area (number of pixels) of the region, arrange these feature values in order to form a high-dimensional vector (such as containing 5 features, which is a 5-dimensional vector), when dimensionality reduction mapping is performed, the algorithm will generate a plurality of random straight lines (each straight line represents a division of a dimension), such as generating 32 random straight lines, for a high-dimensional vector, calculate its projection position on each straight line, if the projection position is on one side of the midpoint of the straight line, record it as 0; on the other side, record it as 1, arrange the 0 or 1 corresponding to the 32 straight lines in order, and a 32-bit binary hash code (such as 010110…101) is formed.

[0103] When binding information, the device unique identification code is a unique string (such as "device_001_2025") for identifying the device collecting the multi-modal data; the timestamp is the time when the multi-modal data starts to be collected (such as "2025-07-11 10:00:00.000"); the information of the three of the hash code, the device unique identification code and the timestamp is put together to form an association record (such as "hash code: 0101...101; device code: device_001_2025; timestamp: 2025-07-11 10:00:00.000"); when constructing the inverted index structure, the acoustic keyword index item is established first, all acoustic keywords are collected, each acoustic keyword is taken as an index key, and the corresponding value is a list containing the hash code, the corresponding device code and the timestamp of all acoustic keywords, such as the acoustic keyword "fragment A" corresponding list is [(hash code 1, device code 1, timestamp 1), (hash code 3, device code 2, timestamp 2)]; similarly, each visual keyword is taken as an index key, and the corresponding value is a list containing the hash code, the device code and the timestamp of the visual keyword.

[0104] The spatiotemporal identifier of the associated feature group is composed of the device unique identification code and the timestamp (such as "device_001_2025_2025-07-11 10:00:00.000"); when constructing the inverted index structure, the acoustic keyword index item, the visual keyword index item and the spatiotemporal identifier are organized in the following manner, taking the hash code as the intermediate association point, each hash code corresponds to a record, the record contains the acoustic keyword, the visual keyword, the device code and the timestamp corresponding to the hash code, then the index is established according to the acoustic keyword and the visual keyword, through the index, all hash codes containing the keyword and the corresponding device and time information can be quickly found, thus forming the cross-modal semantic index table, when querying a certain acoustic keyword, the visual keyword and the corresponding device and time information associated with the acoustic keyword can be found through the index table, and the preliminary semantic association of the multi-modal data is realized.

[0105] The acoustic keyword is determined by calculating the energy of each audio segment in detail, and based on specific sampling, segmentation, energy calculation and threshold comparison, the key segment and irrelevant segment in the audio can be accurately distinguished, which completely relies on actual numerical calculation and avoids ambiguous judgment standard, so that the extracted acoustic keyword can accurately reflect the important content in the audio, such as the key sentence in the speech and the climax part in the music, which provides accurate audio basis for subsequent cross-modal correlation; The regional convolution response value detection is adopted, the specific response value is obtained through the convolution operation of each pixel in the region, and then compared with the preset threshold, which can accurately lock the region with characteristics in the video frame. This method based on actual pixel numerical calculation can effectively exclude the unimportant background area in the image, highlight the key visual information such as face and object edge, and the extracted visual keyword has strong pertinence, so that the extraction of video features is more accurate and reliable, and provides high-quality visual basis for cross-modal correlation; The keyword pair of time-space correlation is matched by strict time range matching, which ensures the correlation of audio and visual keywords in time, such as the picture of the speaker in the video and the corresponding voice segment, and the local sensitive hash algorithm converts high-dimensional features into binary hash code, which not only greatly reduces the storage amount of data (binary code occupies small space), but also makes the comparison and matching between different keyword pairs more rapid, such as by comparing the difference between two hash codes, whether they are related can be quickly judged, the efficiency of data processing is improved, and the key features of keyword pairs are reserved; The hash code is bound with the device unique identification code and the timestamp, so that each group of multi-modal data has a unique identity and time record, which is convenient for tracing the source and time of data, and the inverted index structure contains various index items, which can quickly find the corresponding visual keyword and related data through acoustic keyword, and vice versa, realizing the effective correlation of audio and video data. This index table makes the query and management of multi-modal data efficient, such as finding the video segment containing the sound of "car horn", the corresponding visual picture can be quickly found through acoustic keyword index, which improves the application value and processing efficiency of multi-modal data.

[0106] In a preferred embodiment of the present application, the cross-modal semantic index table is input into the cross-modal deep learning mechanism, the audio keyword and the video keyword are jointly semantically encoded, the feature vector fused with acoustic and visual semantics is generated, and the feature vector is associated with the original multi-modal data and stored in the vector database, realizing semantic calculation of the storage platform, which can include:

[0107] Based on the cross-modal semantic index table, the audio acoustic keyword sequence corresponding to the acoustic keyword index item and the video visual keyword sequence corresponding to the visual keyword index item are extracted;

[0108] The audio acoustic keyword sequence is input into a bidirectional long short-term memory network to generate a context-aware acoustic semantic vector; and the video visual keyword sequence is input into a graph convolution network to generate a spatially-related visual semantic vector;

[0109] The acoustic semantic vector and the visual semantic vector are input into a cross-modal attention alignment layer to calculate an acoustic-visual semantic correlation weight matrix and generate an aligned fusion semantic vector, specifically including: taking the acoustic semantic vector as a query vector, the visual semantic vector as a key vector and a value vector, calculating the correlation weight of the acoustic semantic vector to the visual semantic vector through a multi-head attention mechanism to generate an acoustic-visual attention weight matrix; based on the acoustic-visual attention weight matrix, the visual semantic vector is weighted and aggregated to generate an acoustic-guided visual feature vector; taking the visual semantic vector as a query vector and the acoustic semantic vector as a key vector and a value vector, the correlation weight of the visual semantic vector to the acoustic semantic vector is calculated through a multi-head attention mechanism to generate a visual-acoustic attention weight matrix; based on the visual-acoustic attention weight matrix, the acoustic semantic vector is weighted and aggregated to generate a visual-guided acoustic feature vector; the visual feature vector and the acoustic feature vector are integrated to form a cross-modal interaction feature, and the cross-modal interaction feature is input into a fully connected projection layer for feature fusion and dimension reduction to generate a preliminary fusion vector; the preliminary fusion vector, the original acoustic semantic vector and the original visual semantic vector are connected in residual, and the features after the residual connection are normalized by layers to generate an aligned fusion semantic vector;

[0110] The fusion semantic vector is input into a fully connected layer for dimension compression to generate a unified semantic feature vector, and a mapping relationship is established with a device unique identification code and a timestamp included in the multi-modal feature group;

[0111] The mapping relationship is stored into a vector database to generate a structured entry including a vector data, a device identification and a timestamp triple, realizing semantic calculation of a storage platform.

[0112] In the embodiment of the present application, when the keyword sequence is extracted from the cross-modal semantic index table, the specific positions of the acoustic keyword index item and the visual keyword index item are first determined. For the extraction of the audio acoustic keyword sequence, the staff will check each audio keyword recorded in the index item one by one, and check the time mark of the keyword in the original audio file. For example, the index item records that the sound of rain appears at the 10th second of the audio, the sound of thunder appears at the 15th second, and the sound of wind appears at the 20th second. According to the time sequence of 10 seconds, 15 seconds and 20 seconds, the sound of rain, the sound of thunder and the sound of wind are arranged into an audio acoustic keyword sequence. When the video visual keyword sequence is extracted, the spatial positions of each visual keyword in the video image are first observed. For example, the tree is on the left side of the image, the house is in the middle of the image, and the road is at the bottom of the image. Then, combined with the time sequence of their appearance, if the tree appears at the 5th frame, the house appears at the 10th frame, and the road appears at the 15th frame, the three keywords are arranged into a video visual keyword sequence according to the spatial position association and the time sequence.

[0113] After the audio acoustic keyword sequence is input into the bidirectional long short-term memory network, the network will first process each keyword in the sequence separately. Taking the sequence of the sound of rain, the sound of thunder and the sound of wind as an example, the network will first convert the sound of rain into an initial vector, which contains semantic information related to rain, such as the sound characteristics of water droplets falling, etc. Then, the sound of thunder is processed. During processing, the vector of the sound of thunder is combined with the vector information of the sound of rain, so that the vector of the sound of thunder not only contains its own semantics, but also contains the association with the sound of rain. Then, the sound of wind is processed, which is combined with the vector information of the sound of rain and the sound of thunder to form a vector containing the entire context. After that, the network will process the sequence again from back to front. The vector of the sound of wind will affect the vector of the sound of thunder, and the vector of the sound of thunder will affect the vector of the sound of rain. Finally, the vectors obtained by processing twice are integrated to generate a context-aware acoustic semantic vector, which comprehensively contains the semantic association of the entire audio keyword sequence.

[0114] When the video visual keyword sequence is input into the graph convolution network, taking the sequence of tree, house and road as an example, the network will treat each keyword as a node, first analyze the spatial association between the nodes, and then establish connections between the nodes. For example, the tree and the house are adjacent in the image, and the house and the road are connected, so connections are established between these nodes. Then, the vector of the tree absorbs part of the information in the vector of the house because they are adjacent; the vector of the house absorbs part of the information in the vectors of the tree and the road; and the vector of the road absorbs part of the information in the vector of the house. After such information transmission, the vector of each node contains the spatial association information of the surrounding nodes, and finally a spatially associated visual semantic vector is generated.

[0115] After the acoustic semantic vector and the visual semantic vector are input into the cross-modal attention alignment layer, the layer first analyzes each part in the two vectors, such as the part about the sound of rain in the acoustic semantic vector and the part about the wetness of trees in the visual semantic vector, calculates the semantic similarity between them, judges the correlation degree of the sound of rain and the wetness of trees, and forms an acoustic-visual semantic correlation weight matrix by performing such similarity calculation on all parts of the acoustic semantic vector and the visual semantic vector. Each value in the matrix represents the correlation degree of a certain acoustic part and a certain visual part. The greater the value, the closer the correlation. Then, according to the weight matrix, the acoustic semantic vector and the visual semantic vector are fused. The parts with high correlation degree will have a greater proportion in the fusion. For example, the sound of rain and the wetness of trees have a high correlation degree, so the semantic correlation between the two will be highlighted in the fusion, and finally an aligned fused semantic vector is generated. This vector contains both acoustic rain-related semantics and visual tree wetness-related semantics, and the two are closely related.

[0116] After the fused semantic vector is input into the full connection layer, the layer processes each element in the vector. The fused semantic vector may contain many elements, and the full connection layer filters and integrates these elements, retains key semantic information, and removes redundant information, thereby reducing the dimension of the vector. For example, the original fused semantic vector has 1000 elements, and after processing by the full connection layer, it may be reduced to 100 elements to form a unified semantic feature vector. Although the dimension of this vector is reduced, it still contains the core semantics of the original fused semantic vector. Then, the device unique identifier code, such as "Device_001", and the timestamp, such as "2025-07-11 10:00:00", are found in the multi-modal feature group. The unified semantic feature vector is corresponded with "Device_001" and "2025-07-11 10:00:00" to determine that this unified semantic feature vector is generated by the multi-modal data generated by "Device_001" at "2025-07-11 10:00:00", and a one-to-one mapping relationship is established.

[0117] When storing the mapping relationship to the vector database, the uniform semantic feature vector is first taken as vector data, then the corresponding device unique identification code "Device_001" and the time stamp "2025-07-11 10:00:00" are combined with the vector data, then according to the storage format requirements of the vector database, the three parts are arranged into a structured entry, which clearly shows what the vector data is, which device identification it corresponds to, and what the time stamp is, for example, the entry will be in the form of "vector data: [...], device identification: Device_001, time stamp: 2025-07-11 10:00:00", forming a structured entry containing the vector data, device identification and time stamp triple, to complete the semantic calculation of the storage platform.

[0118] In the extraction of the keyword sequence, the arrangement is strictly in time and space order, so that the extracted sequence can accurately reflect the actual occurrence of the keywords in the original multi-modal data, avoiding semantic understanding deviation caused by incorrect keyword order and improving the initial data quality of the entire processing process; the bidirectional long short-term memory network, when processing the audio acoustic keyword sequence, transmits information in both directions, so that the generated acoustic semantic vector contains the context semantics of each keyword in the entire sequence, which can more comprehensively understand the audio content; the graph convolution network processes the video visual keyword sequence by transmitting spatial correlation information between nodes, so that the visual semantic vector can accurately reflect the spatial relationship of elements in the video picture, thereby improving the semantic expression accuracy of acoustic and visual semantic vectors; the cross-modal attention alignment layer can accurately capture the internal relationship between acoustic and visual semantics by calculating the acoustic-visual semantic correlation weight matrix, so that the fused semantic vector realizes accurate alignment of acoustic and visual semantics, avoiding disconnection of the two modal semantics, and enabling the fused semantic vector to more completely and accurately express the overall semantics of the multi-modal data; the fully connected layer performs dimension compression on the fused semantic vector to generate a uniform semantic feature vector, realizing unified representation of different modal semantics, and establishing a mapping relationship with the device unique identification code and the time stamp to quickly trace the original data source and generation time corresponding to each semantic feature vector, improving the management efficiency and traceability of multi-modal data; storing the structured entry containing the vector data, device identification and time stamp triple to the vector database makes the storage of multi-modal data more standardized and orderly, forming structured data with clear semantic correlation, facilitating efficient semantic calculation by the storage platform, and improving the processing capacity and utilization value of the storage platform for multi-modal data.

[0119] In a preferred embodiment of the present invention, a vector database is searched, multiple discrete reference points are set in the search space, and the search space is divided into multiple dynamic partitions based on the spatial distribution characteristics of the discrete reference points. Semantic calibration parameters are generated according to the data distribution characteristics of each dynamic partition, and the semantic representation weight of the feature vector is dynamically adjusted. This may include:

[0120] Based on the semantic distribution density of the feature vectors in the vector space, determining the feature vectors whose distribution density is greater than a preset threshold as discrete reference points;

[0121] According to the discrete reference points, the vector space is divided into multiple dynamic partitions by the Voronoi diagram algorithm. Each dynamic partition includes all the eigenvectors closest to the discrete reference point corresponding to the partition.

[0122] Based on each dynamic partition of the division, the local spatial density value of the feature vector in the partition is calculated, and the cross-modal similarity distribution of the acoustic feature vector and the visual feature vector in the partition is extracted, specifically including: obtaining the spatial coordinates of all feature vectors in the dynamic partition to form a partition spatial coordinate set; inputting the partition spatial coordinate set into the kernel density estimation algorithm, calculating the feature vector distribution density in the unit hypervolume, and obtaining the local spatial density value; based on the local spatial density value, traversing each feature vector in the partition, extracting the internally stored acoustic feature vector sub-components and visual feature vector sub-components, and generating an acoustic-visual feature pair set; performing a cosine similarity operation on each feature pair in the acoustic-visual feature pair set to generate a cosine similarity value set of all feature vectors in the partition; based on the cosine similarity value set, dividing the similarity value range into a preset number of intervals, counting the proportion of the number of feature vectors in each interval, and generating a probability distribution of acoustic-visual cross-modal similarity, that is, a cross-modal similarity distribution;

[0123] Based on the local spatial density value and cross-modal similarity distribution, a partition-level weight adjustment coefficient is generated to balance the contribution of acoustic and visual modalities;

[0124] During the retrieval process, the partition-level weight adjustment coefficient is called according to the target dynamic partition to which the query vector belongs, and the semantic representation weights of acoustic features and visual features in the similarity calculation are dynamically weighted.

[0125] In the embodiment of the present application, all the feature vectors in the vector space are labeled one by one, such as being given a unique serial number, for the convenience of tracking and statistics, and then an inspection range is defined for each feature vector. The size of the range can be determined as follows: first, the average distance between all the feature vectors in the vector space is calculated, and then half of the average distance is taken as the initial range value, and then the range is fine-tuned according to the density of the feature vector distribution. If the overall distribution is relatively dense, the range is adjusted to be smaller; if the distribution is relatively sparse, the range is adjusted to be larger. Then, for each feature vector, the number of other feature vectors within the inspection range is counted. For example, for feature vector A, all the feature vectors with serial numbers are checked to see which feature vectors are within the range of A, and the number of these feature vectors is counted, which is the distribution density of feature vector A. Then, a preset threshold is determined. First, the sum of the distribution densities of all the feature vectors is calculated, and then the average distribution density is obtained by dividing the total number of feature vectors. Then, the threshold is set according to the actual search requirements. If more discrete reference points are desired, the threshold is set to be lower than the average value; if the reference points are desired to be more representative and fewer in number, the threshold is set to be higher than the average value. For example, if the average distribution density is 20 and the actual requirement is that the reference points can reflect a relatively dense area, the threshold is set to 30. Finally, the distribution density of each feature vector is compared with the preset threshold. If the distribution density of a feature vector is greater than the threshold, it is determined as a discrete reference point, and the serial number and position information of the discrete reference point in the vector space are recorded.

[0126] After all the discrete reference points are obtained, each feature vector in the vector space is processed. For each feature vector, the distance to each discrete reference point is calculated. When calculating the distance, first, the values in each dimension of the feature vector and the discrete reference point are compared. For example, the value of the feature vector in the first dimension is 5, and the value of the discrete reference point in the first dimension is 3, so the difference between the two values is 2. Then, the values in the second dimension are compared. The value of the feature vector is 8, and the value of the discrete reference point is 6, so the difference is also 2. The values in all dimensions are calculated in this way, and then the differences are added up to obtain the distance from the feature vector to the discrete reference point (here, no square or square root operation is involved, only the differences in each dimension are simply added up).

[0127] After calculating the distance of a feature vector to all discrete reference points, the distance values are listed, and the smallest distance value is found. Then, the feature vector is divided into the dynamic partition corresponding to the discrete reference point. For example, the distance of feature vector B to discrete reference point 1 is 10, to discrete reference point 2 is 8, and to discrete reference point 3 is 12. The smallest distance is 8, which corresponds to discrete reference point 2. Therefore, feature vector B is divided into the dynamic partition corresponding to discrete reference point 2. In this way, all feature vectors in the vector space are processed until each feature vector is divided into the corresponding dynamic partition, and each dynamic partition contains all feature vectors closest to the corresponding discrete reference point.

[0128] For each dynamic partition, the calculation of the local space density value is processed first. In the partition, a local check range is set for each feature vector, which is smaller than the range determined in step one, such as half of the range. The purpose is to more accurately reflect the local density. Then, for each feature vector in the partition, the number of other feature vectors in the local check range is counted, which is the local density of the feature vector. For example, feature vector C in the partition has 15 other feature vectors in its local check range, so its local density is 15. Then, the local densities of all feature vectors in the partition are added up to get the total, and then divided by the total number of feature vectors in the partition to get the local space density value of the partition. For example, there are 20 feature vectors in the partition, and the total of their local densities is 300. Therefore, the local space density value of the partition is 300 ÷ 20 = 15.

[0129] Next, the extraction of the cross-modal similarity distribution is considered. In each dynamic partition, acoustic feature vectors and visual feature vectors are distinguished, such as marking which are acoustic feature vectors and which are visual feature vectors. Then, for each acoustic feature vector, it is compared with each visual feature vector in the partition to determine their similarity. Specifically, the matching of the corresponding feature dimensions is compared, such as the association between the frequency dimension of a certain acoustic feature vector and the brightness dimension of a certain visual feature vector. According to the closeness of this association, a similarity score is given, with a range of 0 to 10, where 0 represents complete dissimilarity and 10 represents very similar.

[0130] The similarity scores between all acoustic feature vectors and visual feature vectors are collected, and the number of occurrences of different scores is counted, for example, the score of 6 occurs 20 times, the score of 7 occurs 15 times, the score of 5 occurs 10 times, etc. Then, the proportion of each score in the total number of times is calculated according to the number of occurrences, for example, the total number of times is 100, the score of 6 accounts for 20%, the score of 7 accounts for 15%, etc. The number of occurrences and the proportion of different scores constitute the cross-modal similarity distribution of acoustic feature vectors and visual feature vectors in the partition.

[0131] First, analyze the local spatial density value. If the local spatial density value of a certain partition is large, it means that the feature vectors in the partition are densely distributed, and the differences between different feature vectors may be small. At this time, the weights of acoustic and visual modalities need to be balanced to avoid the excessive influence of one of the modalities. If the local spatial density value is small, it means that the feature vectors in the partition are sparsely distributed, and the weight of the more representative modality may need to be highlighted. Then, analyze the cross-modal similarity distribution. If the proportion of high scores (such as 8-10) in the distribution is large, it means that the acoustic and visual feature vectors in the partition are strongly correlated, and their contribution is relatively balanced. If the proportion of low scores (such as 0-3) is large, it means that the correlation between the two modalities is weak, and one of the modalities may need to be emphasized, for example, if the acoustic features in the partition are more stable, the weight of the acoustic features needs to be increased. The local spatial density value and the cross-modal similarity distribution are combined to determine the partition-level weight adjustment coefficient, for example, when the local spatial density value is high and the proportion of high scores in the cross-modal similarity distribution is large, the weight ratio of acoustic features and visual features is set to 5:5; when the local spatial density value is low and the proportion of low scores in the cross-modal similarity distribution is large, and the acoustic features are more reliable, the ratio is set to 7:3, to balance the contribution of the two modalities.

[0132] In the retrieval process, when the query vector is input, first calculate the distance from the query vector to each discrete reference point, the calculation method is the same as the calculation of the distance from the feature vector to the discrete reference point, that is, the numerical difference of the query vector and the discrete reference point in each dimension is accumulated, then the distances are compared, and the discrete reference point corresponding to the minimum distance is found. The dynamic partition where the discrete reference point is located is the target dynamic partition of the query vector; then, the partition level weight adjustment coefficient corresponding to the target dynamic partition is called. When calculating the similarity between the query vector and the feature vector in the partition, the weight adjustment coefficient is used to allocate the weights of acoustic features and visual features, for example, if the weight adjustment coefficient is 6:4, then in the similarity calculation, the influence degree of acoustic features is calculated according to 60%, and the influence degree of visual features is calculated according to 40%; Specifically, assuming that the similarity of acoustic features is calculated separately to obtain a basic score, and the similarity of visual features is calculated separately to obtain another basic score, then the final similarity score is the product of the acoustic feature basic score multiplied by 60% plus the product of the visual feature basic score multiplied by 40%, so as to dynamically weight the semantic representation weights of the two features in the similarity calculation.

[0133] When determining the discrete reference points, the distribution density of each feature vector is calculated in detail and compared with the preset threshold, so that the feature vectors in the densely distributed area can be accurately screened as reference points. These reference points can accurately reflect the aggregation of features in the vector space, so that the partition is more consistent with the actual distribution of features. When using the Voronoi diagram algorithm to divide the dynamic partition, the distance from each feature vector to each discrete reference point is calculated and the attribution is determined, so that the distance between the feature vectors in each partition and the corresponding reference point is the shortest, making the features in the partition have stronger correlation and similarity, reducing the mutual interference of features in different areas. The calculation of the local space density value and the extraction of the cross-modal similarity distribution can deeply mine the local features of each partition. The local space density value reflects the density of the features in the partition, and the cross-modal similarity distribution reflects the correlation of the two modal features. These detailed local information makes the weight adjustment more targeted and avoids the deviation caused by using a unified standard for the entire vector space. The partition level weight adjustment coefficient is generated based on the local space density value and the cross-modal similarity distribution, which can balance the contribution of acoustic and visual modal according to the unique characteristics of each partition, making the weight adjustment more accurate. Different weight coefficients are used in different partitions to avoid the problem that a single weight cannot adapt to all partitions, improving the adaptability to different feature distribution areas. In the retrieval process, the corresponding weight adjustment coefficient is called according to the partition to which the query vector belongs, and the semantic representation weights of the two features are dynamically weighted, which can make the similarity calculation more consistent with the feature characteristics of the partition. The calculated similarity is more accurate, which can improve the relevance of the retrieval result and reduce the occurrence of irrelevant results, thereby improving the efficiency and accuracy of the retrieval and making the retrieval result more consistent with the actual needs of the user.

[0134] Embodiments of the present application also provide a computing device, comprising a processor, a memory storing a computer program, the computer program being executed by the processor to perform the method as described above. All implementation manners in the above method embodiments are applicable to this embodiment and can achieve the same technical effects.

[0135] Embodiments of the present application also provide a computer readable storage medium storing instructions, which, when executed on a computer, cause the computer to perform the method as described above. All implementation manners in the above method embodiments are applicable to this embodiment and can achieve the same technical effects.

[0136] The above is the preferred embodiment of the present application. It should be pointed out that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should also be considered as the protection scope of the present application.

Claims

1. Multimodal audio and video storage platform, characterized by: include: The data receiving module is used to receive audio, video and sensor log data from monitoring, and store them in distributed file devices, object storage services and relational databases according to the time dimension to build a distributed storage system; The feature extraction module is used to extract the acoustic features of audio and the visual features of video based on the distributed storage system. It also associates the audio features, video features, and sensor log tags at the same time and space point through the timestamp and the unique identification code of the monitoring device to generate a multimodal feature group with time and space labels. The index construction module is used to extract audio acoustic keywords and video visual keywords based on multimodal feature groups, bind the keywords to the feature groups through a hash mapping algorithm, and establish a cross-modal semantic index table to achieve preliminary semantic association of multimodal data; The fusion storage module is used to input the cross-modal semantic index table into the cross-modal deep learning mechanism, perform joint semantic encoding on audio keywords and video keywords, generate feature vectors that integrate acoustic and visual semantics, associate the feature vectors with the original multimodal data, and store them in the vector database. This generates structured entries consisting of triplets of vector data, device identifiers, and timestamps, thus realizing semantic computing on the storage platform. A calibration retrieval module is used to search a vector database, set multiple discrete reference points in the retrieval space, and divide the retrieval space into multiple dynamic partitions based on the spatial distribution characteristics of the discrete reference points. Semantic calibration parameters are generated according to the data distribution characteristics of each dynamic partition, and the semantic representation weight of the feature vector is dynamically adjusted. The semantic calibration parameters are based on the local spatial density value and the cross-modal similarity distribution to generate partition-level weight adjustment coefficients for balancing the contribution of acoustic and visual modalities.

2. The multimodal audio and video storage platform according to claim 1, characterized in that: Based on a distributed storage system, the acoustic features of audio and the visual features of video are extracted. The audio features, video features, and sensor log tags at the same spatiotemporal point are associated through timestamps and the unique identification code of the monitoring device to generate a multimodal feature set with spatiotemporal labels, including: For the audio data stream stored in the distributed file device, the Mel-frequency cepstral coefficients are calculated through pre-emphasis filtering to generate the acoustic spectrum feature vector; For the video data stream in the object storage service, extract key frames to obtain deep visual feature vectors; Extract device unique identification codes, millisecond timestamps, and event type codes from sensor logs in a relational database. Using the device's unique identifier and timestamp as index keys, the acoustic spectrum feature vector is time-offset compensated according to the audio sampling rate. The deep visual feature vector is spatially calibrated according to the video frame timestamp. The acoustic and visual feature sequences are aligned using a dynamic time warping algorithm and bound to the event type code. The spatiotemporally aligned acoustic spectrum feature vectors, deep visual feature vectors, and event type codes are encapsulated into a structured data packet including a unique device identification code, alignment timestamp, and feature version number to generate a multimodal feature group.

3. The multimodal audio and video storage platform according to claim 2, characterized in that: Based on multimodal feature groups, we extract audio acoustic keywords and video visual keywords, bind the keywords to the feature groups through a hash mapping algorithm, and establish a cross-modal semantic index table to achieve preliminary semantic association of multimodal data, including: Perform power spectral density analysis on the acoustic feature vectors in the multimodal feature group, and calculate the acoustic segments whose energy exceeds the threshold as audio acoustic keywords; Perform regional convolution response value detection on the visual feature vectors in the multimodal feature group, and calculate the visual regions whose response values ​​exceed the preset threshold as video visual keywords; The audio acoustic keywords and the video visual keywords are combined into a spatiotemporally associated keyword pair, and the keyword pair is input into the local sensitive hashing algorithm for dimensionality reduction mapping to generate a binary hash code; The hash code is bound to the device unique identification code and timestamp of the multimodal feature group, and an inverted index structure is constructed based on the bound hash code, including acoustic keyword index items, visual keyword index items and spatiotemporal identifiers of the associated feature group, forming a cross-modal semantic index table to achieve preliminary semantic association of multimodal data.

4. The multimodal audio and video storage platform according to claim 3, characterized in that: Perform power spectral density analysis on the acoustic feature vectors in the multimodal feature group and calculate the acoustic segments with energy exceeding the threshold as audio acoustic keywords, including: Based on the acoustic spectrum feature vector, frame processing is performed with a preset time window length to obtain a sequence of equally spaced audio frames. Based on the audio frame sequence, a discrete Fourier transform is performed on the windowed frames to obtain a sequence of frequency domain power spectral density values; Based on the power spectrum density value sequence, the sliding energy mean is calculated in a continuous time window. The energy judgment threshold is dynamically generated based on the energy distribution of the sequence. The power spectrum density value sequence is compared with the energy judgment threshold, and the frames exceeding the threshold are marked as candidate activation frames. Merging candidate activation frames to obtain continuous acoustic segments when the time interval between adjacent frames is less than a preset time tolerance and the energy difference is less than a preset energy tolerance; The start timestamp, duration and average energy intensity of continuous acoustic segments are extracted to obtain structured audio acoustic keywords.

5. The multimodal audio and video storage platform according to claim 4, characterized in that: The cross-modal semantic index table is input into the cross-modal deep learning mechanism to perform joint semantic encoding on audio and video keywords, generating a feature vector that integrates acoustic and visual semantics. The feature vector is then associated with the original multimodal data and stored in a vector database, implementing semantic computing on the storage platform, including: Based on the cross-modal semantic index table, extract the audio acoustic keyword sequence corresponding to the acoustic keyword index item and the video visual keyword sequence corresponding to the visual keyword index item; The audio acoustic keyword sequence is input into the bidirectional long short-term memory network to generate a context-aware acoustic semantic vector; the video visual keyword sequence is input into the graph convolutional network to generate a spatially associated visual semantic vector; The acoustic semantic vector and the visual semantic vector are input into the cross-modal attention alignment layer, the acoustic-visual semantic association weight matrix is ​​calculated, and the aligned fusion semantic vector is generated; The fused semantic vector is input into the fully connected layer for dimensionality compression to generate a unified semantic feature vector, and a mapping relationship is established with the device unique identification code and timestamp included in the multimodal feature group; The mapping relationship is stored in the vector database, and structured entries including vector data, device identification and timestamp triples are generated to realize semantic computing of the storage platform.

6. The multimodal audio and video storage platform according to claim 5, characterized in that: The acoustic semantic vector and the visual semantic vector are input into the cross-modal attention alignment layer, the acoustic-visual semantic association weight matrix is ​​calculated, and the aligned fusion semantic vector is generated, including: Using the acoustic semantic vector as the query vector and the visual semantic vector as the key vector and value vector, the multi-head attention mechanism is used to calculate the association weight of acoustic semantics to visual semantics and generate the acoustic-visual attention weight matrix. Based on the acoustic-visual attention weight matrix, the visual semantic vectors are weighted and aggregated to generate acoustically guided visual feature vectors; Using the visual semantic vector as the query vector and the acoustic semantic vector as the key vector and value vector, the multi-head attention mechanism is used to calculate the association weight of the visual semantics to the acoustic semantics, and generate the visual-acoustic attention weight matrix; Based on the visual-acoustic attention weight matrix, the acoustic semantic vectors are weighted and aggregated to generate the visually guided acoustic feature vectors; Integrate the visual feature vector and the acoustic feature vector to form a cross-modal interaction feature, and input the cross-modal interaction feature into the fully connected projection layer for feature fusion and dimensionality reduction to generate a preliminary fusion vector; The preliminary fusion vector is residually connected with the original acoustic semantic vector and the original visual semantic vector, and the features after residual connection are layer-normalized to generate the aligned fusion semantic vector.

7. The multimodal audio and video storage platform according to claim 6, characterized in that: Search the vector database, set multiple discrete reference points in the search space, and divide the search space into multiple dynamic partitions based on the spatial distribution characteristics of the discrete reference points. Generate semantic calibration parameters based on the data distribution characteristics of each dynamic partition, and dynamically adjust the semantic representation weight of the feature vector, including: Based on the semantic distribution density of the feature vectors in the vector space, determining the feature vectors whose distribution density is greater than a preset threshold as discrete reference points; According to the discrete reference points, the vector space is divided into multiple dynamic partitions by the Voronoi diagram algorithm. Each dynamic partition includes all the eigenvectors closest to the discrete reference point corresponding to the partition. Based on each dynamic partition, the local spatial density value of the feature vector in the partition is calculated, and the cross-modal similarity distribution of the acoustic feature vector and the visual feature vector in the partition is extracted; Based on the local spatial density value and cross-modal similarity distribution, a partition-level weight adjustment coefficient is generated to balance the contribution of acoustic and visual modalities; During the retrieval process, the partition-level weight adjustment coefficient is called according to the target dynamic partition to which the query vector belongs, and the semantic representation weights of acoustic features and visual features in the similarity calculation are dynamically weighted.

8. The multimodal audio and video storage platform according to claim 7, characterized in that: Based on each dynamic partition, the local spatial density value of the feature vector in the partition is calculated, and the cross-modal similarity distribution of the acoustic feature vector and the visual feature vector in the partition is extracted, including: Obtain the spatial coordinates of all eigenvectors in the dynamic partition to form a partition spatial coordinate set; Input the partition space coordinate set into the kernel density estimation algorithm, calculate the eigenvector distribution density within the unit hypervolume, and obtain the local space density value; Based on the local spatial density value, each feature vector in the partition is traversed, and the internally stored acoustic feature vector quantum components and visual feature vector quantum components are extracted to generate a set of acoustic-visual feature pairs; For each feature pair in the acoustic-visual feature pair set, perform a cosine similarity operation to generate a set of cosine similarity values ​​of all feature vectors in the partition; Based on the cosine similarity value set, the similarity value range is divided into a preset number of intervals, and the proportion of the number of feature vectors in each interval is counted to generate a probability distribution of acoustic-visual cross-modal similarity, that is, the cross-modal similarity distribution.

9. A computing device, characterized in that include: one or more processors; A storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the platform according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a program, which, when executed by a processor, implements the platform according to any one of claims 1 to 8.

Images