Audio scene data analysis method and system, electronic device, storage medium

Abstract

This disclosure provides an audio scene data analysis method, system, electronic device, and storage medium, belonging to the field of data processing technology. The method includes: extracting audio features from audio data of a target scene; extracting video features from video data of the target scene; fusing the audio features and video features based on an attention mechanism to obtain fused features; and inputting the fused features into a data analysis model to obtain the analysis results of the target scene. The audio scene data analysis method, system, electronic device, and storage medium provided in this disclosure can improve the accuracy of audio scene analysis.

Description

Technical Field

[0001] This disclosure belongs to the field of data processing technology, and more specifically, relates to an audio scene data analysis method and system, electronic device, and storage medium. Background Technology

[0002] In numerous fields such as smart homes, public safety, and traffic management, accurate audio scene analysis is gradually becoming a core requirement for applications such as human-computer interaction and environmental monitoring. For example, in smart homes, distinguishing between door opening sounds, water flow sounds, and appliance operating sounds allows the system to make corresponding automatic controls; in traffic management, differentiating between vehicle driving sounds, braking sounds, and horn sounds provides a basis for traffic condition monitoring.

[0003] To better meet the application needs of various fields, it is necessary to further improve the accuracy of audio scene analysis. Summary of the Invention

[0004] The purpose of this disclosure is to provide an audio scene data analysis method, system, electronic device, and storage medium to improve the accuracy of audio scene analysis.

[0005] A first aspect of this disclosure provides an audio scene data analysis method, including: Audio features are extracted from audio data of the target scene, and video features are extracted from video data of the target scene. The audio features and video features are fused based on an attention mechanism to obtain fused features; The fused features are input into the data analysis model to obtain the analysis results of the target scene.

[0006] A second aspect of this disclosure provides an audio scene data analysis system, comprising: The feature extraction module is used to extract audio features from audio data of the target scene and video features from video data of the target scene. The feature fusion module is used to fuse the audio features and the video features based on an attention mechanism to obtain fused features; The audio scene analysis module is used to input the fused features into the data analysis model to obtain the analysis results of the target scene.

[0007] A third aspect of this disclosure provides an electronic device including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the audio scene data analysis method described above.

[0008] A fourth aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the audio scene data analysis method described above.

[0009] The beneficial effects of the audio scene data analysis method and system, electronic device, and storage medium provided in this disclosure are as follows: In this embodiment, audio features can characterize changes in audio signals in the target scene, and video features can characterize changes in video signals in the target scene. Based on an attention mechanism, the correlation and importance weights between audio and video features are automatically learned, and then the audio and video features are fused based on the importance weights, so that the final fused features contain the parts of different modal features most relevant to the analysis task. Target scene analysis based on fused features can more accurately identify and understand the target scene, thereby improving the accuracy of the target scene analysis results. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A flowchart illustrating an audio scene data analysis method provided in an embodiment of this disclosure; Figure 2 This is a structural block diagram of an audio scene data analysis system provided in an embodiment of the present disclosure; Figure 3 This is a schematic block diagram of an electronic device provided according to an embodiment of the present disclosure. Detailed Implementation

[0012] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, so as to provide a thorough understanding of the embodiments of this disclosure. However, those skilled in the art will understand that this disclosure may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this disclosure with unnecessary detail.

[0013] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following description will be provided in conjunction with the accompanying drawings and specific embodiments.

[0014] Please refer to Figure 1 , Figure 1This is a flowchart illustrating an audio scene data analysis method provided in an embodiment of the present disclosure. The method includes: S101: Extract audio features from audio data of the target scene, and extract video features from video data of the target scene.

[0015] In this embodiment, audio acquisition devices such as microphones can be set up in the target scene to collect audio data. Raw audio data is usually a complex time-series signal, which is difficult and inefficient to process directly. By extracting features, key information in the audio data can be obtained, which is convenient for subsequent model learning and processing, while reducing the amount of data and improving processing efficiency.

[0016] Audio features include time-domain features and frequency-domain features. Time-domain features include zero-crossing rate, energy, and other features, which can be obtained through statistical methods. Frequency-domain features include the spectrum, Mel-frequency cepstral coefficients (MFCCs), etc.

[0017] Correspondingly, image acquisition devices such as cameras can be set up in the target scene to collect video data. Video data contains a large number of image frames, which are rich in information but highly redundant. By extracting video features, valuable information can be filtered out, reducing data dimensionality and highlighting key information in the video.

[0018] Video features can include static features and dynamic features. Static features include the image's color histogram, texture features, etc., while dynamic features include optical flow features, which are used to describe the motion information of objects.

[0019] S102: Based on the attention mechanism, audio features and video features are fused to obtain fused features.

[0020] In this embodiment, considering the potential limitations of audio scene analysis based on single-modal data—for example, in intelligent security systems, video surveillance can display the movement of people or objects within the monitored area, but it is difficult to determine some subtle anomalies based solely on video—combining audio monitoring allows the system to more accurately determine whether unauthorized intrusion has occurred and issue an alarm in a timely manner if abnormal climbing sounds or friction sounds are detected in the audio while the video shows people approaching a wall.

[0021] Specifically, attention mechanisms can be used to automatically learn the correlation between audio and video features. By calculating and normalizing attention scores, the weight of each feature in the fusion process is obtained. Then, audio and video features are weighted and fused according to their respective weights to generate fused features. The fused features contain the most relevant parts of the features from different modalities to the analysis task, fully leveraging the complementary advantages of multimodal data.

[0022] S103: Input the fused features into the data analysis model to obtain the analysis results of the target scene.

[0023] In this embodiment, the fused features are input into a pre-trained data analysis model. The data analysis model performs calculations and inferences based on the input fused features and finally outputs the analysis results of the target scene.

[0024] The analysis results of audio scenes can include audio scene classification, such as distinguishing between indoor, outdoor, busy street, quiet area, office, restaurant and other scenes; the analysis results of audio scenes can also include audio event detection, identifying whether the audio contains specific events or sounds, such as vehicle driving sounds, human voices, animal calls, alarm sounds, door closing sounds, etc., and by analyzing the characteristics of the audio signal, determining whether an event of interest has occurred, and determining information such as the time and duration of the event.

[0025] The data analysis model can be trained based on existing neural network models, such as convolutional neural networks (CNN), recurrent neural networks (RNN), and Transformers.

[0026] As can be seen from the above, in this embodiment, audio features can characterize changes in audio signals in the target scene, and video features can characterize changes in video signals in the target scene. Based on the attention mechanism, the correlation and importance weight between audio features and video features are automatically learned, and then the audio features and video features are fused based on the importance weight, so that the final fused features contain the parts of different modal features that are most relevant to the analysis task. Target scene analysis based on fused features can more accurately identify and understand the target scene, thereby improving the accuracy of the target scene analysis results.

[0027] In one embodiment of this disclosure, audio features and video features are fused based on an attention mechanism to obtain fused features, including: The first weight of audio features and the second weight of video features are calculated based on an attention mechanism; The audio and video features are weighted and summed based on the first and second weights to obtain the fused features.

[0028] In this embodiment, a dot product attention mechanism can be used to perform dot product operations on audio and video features with a learnable query vector, respectively. Then, the vectors are normalized using the Softmax function to obtain a first weight and a second weight. This allows for dynamic adjustment of the attention given to audio and video features, capturing real-time changes in the importance of different modal features. For example, in a warehouse intrusion monitoring scenario, if the sound of picking the lock is heard, the first weight of the audio feature will be higher.

[0029] The audio and video features are weighted and summed based on the first and second weights. The resulting fused features contain richer semantics, which helps to understand the target scene more accurately. When inputting into the data analysis model, it can provide more comprehensive and accurate information to the data analysis model and improve the model's ability to judge abnormal situations.

[0030] As can be seen from the above, this embodiment utilizes the attention mechanism to automatically identify key parts in audio and video features, assigns higher weights to features containing more important information, and performs feature fusion based on dynamically assigned weights, so that the fused features focus on key content, thereby improving the accuracy of data analysis.

[0031] In one embodiment of this disclosure, the audio scene data analysis method further includes: If the visibility of the target scene is less than the first threshold, the first weight and the second weight are adjusted based on the preset ratio parameter to increase the first weight.

[0032] In this embodiment, considering that in low-visibility weather such as rain, snow, or fog, objects in the video footage may become blurry and difficult to distinguish details, thus reducing the accuracy of target object recognition. In such cases, the first weight of audio features can be appropriately increased to highlight the role of audio features in audio scene analysis and avoid the reduction in accuracy of audio scene analysis caused by abnormal weather.

[0033] Specifically, a visibility detector can be set up in the target scene to detect the visibility of the target scene, and a first threshold and a ratio parameter greater than 1 can be preset. When the visibility of the target scene is less than the first threshold, it indicates that the visibility of the target scene is low. At this time, the ratio parameter can be multiplied by the first weight to increase the value of the first weight. Then, based on the sum of the first weight and the second weight being 1, the value of the second weight can be determined.

[0034] As can be seen from the above, this embodiment dynamically adjusts the first and second weights based on the environmental conditions of the target scene, thus avoiding the reduction in the accuracy of audio scene analysis caused by abnormal weather.

[0035] In one embodiment of this disclosure, the data analysis model is trained based on sample data, which includes a first fusion feature corresponding to the first output data, and a second fusion feature and a third fusion feature corresponding to the second output data; the first output data is used to indicate that the analysis result of the audio target scene is normal, and the second output data is used to indicate that the analysis result of the audio target scene is abnormal. The first fusion feature is obtained based on the first audio sample data and the first video sample data, and the second fusion feature is obtained based on the second audio sample data and the second video sample data. The third fusion feature is obtained based on the third audio sample data and the first video sample data, or The third fusion feature is obtained based on the first audio sample data and the third video sample data; the third audio sample data and the third video sample data are obtained based on SHAP value analysis of the second output data.

[0036] In this embodiment, considering that in security monitoring scenarios, events of interest are often low-probability events, and a large amount of monitoring data consists of audio and video data under normal conditions, the number of negative samples used to train the data analysis model is small, thus affecting the accuracy of the data analysis model.

[0037] To address the aforementioned issues, this embodiment performs SHAP value analysis on the abnormal output results of the model (i.e., the second output result). SHAP value analysis can explain the importance of each feature in the model analysis results and uncover more critical and discriminative feature combinations for the abnormal output results. Therefore, by performing SHAP value analysis on the second output result, it can be determined whether the features causing the second output result are primarily audio or video features. If the features causing the second output result are primarily audio features, then the second audio sample data corresponding to the second output result is determined to be abnormal sample data and is denoted as the third audio sample data. If the features causing the second output result are primarily video features, then the second video sample data corresponding to the second output result is determined to be abnormal sample data and is denoted as the third video sample data.

[0038] Based on the obtained third audio sample data and third video sample data, the first audio sample data and the third video sample data can be fused together, or the first audio sample data and the third video sample data can be fused together, to obtain the third fusion feature corresponding to the abnormal output result, thereby realizing the expansion of negative sample data.

[0039] As can be seen from the above, this embodiment can determine the third audio sample data and the third video sample data corresponding to the abnormal output results by performing SHAP value analysis on the abnormal output results. Then, based on the third audio sample data and the third video sample data, the negative samples of the data analysis model are expanded, thereby improving the training accuracy of the data analysis model.

[0040] In one embodiment of this disclosure, the audio scene data analysis method further includes: SHAP value analysis is performed on the second output to obtain the SHAP value corresponding to each feature; each feature includes each audio feature and each video feature. Calculate the average of the absolute values ​​of the SHAP values ​​of multiple audio features to obtain the first average value; Calculate the average of the absolute values ​​of the SHAP values ​​of multiple video features to obtain the second average; If the first average value is greater than or equal to the second average value, then the second audio sample data corresponding to the second output result is determined as the third audio sample data; If the first average value is less than the second average value, then the second video sample data corresponding to the second output result will be determined as the third video sample data.

[0041] In this embodiment, both audio and video features include multiple features. The average SHAP value of all audio features (i.e., the first average value) and the average SHAP value of all video features (i.e., the second average value) can be calculated. If the average SHAP value of the audio features is higher than the average SHAP value of the video features, it is determined that the audio features play a major role in the second output result, and the abnormality of the second audio sample data leads to the second output result (is abnormal). Therefore, the second audio sample data can be regarded as abnormal sample data and recorded as the third audio sample data. Conversely, if the average SHAP value of the video features is higher, it is determined that the video features play a major role in the second output result, and the abnormality of the second video sample data leads to the second output result (is abnormal). Therefore, the second video sample data can be regarded as abnormal sample data and recorded as the third video sample data.

[0042] As can be seen from the above, this embodiment quantifies the importance of audio and video features by calculating the average SHAP values ​​of multiple audio features and the average SHAP values ​​of multiple video features respectively, and selects the third audio sample data and the third video sample data accordingly, which helps to improve the accuracy of the selection results.

[0043] In one embodiment of this disclosure, audio features are extracted based on audio data of a target scene, and video features are extracted based on video data of the target scene, including: The audio and video data are segmented according to the same time period; Calculate the average value of the audio data and the average value of the video data within each segment; Audio features are extracted based on the average value of the audio data within each segment, and video features are extracted based on the average value of the video data within each segment.

[0044] In this embodiment, segmenting audio and video data according to the same time interval ensures that the audio and video data are aligned in the time dimension. For example, in a security monitoring scenario, segmenting audio and video data into 10-second intervals ensures that the audio and video data correspond within each 10-second time period, facilitating subsequent analysis and fusion.

[0045] Meanwhile, by averaging the audio and video data within each segment, the impact of noise and fluctuations is reduced. Audio features are extracted based on the average value of the audio data, and video features are extracted based on the average value of the video data within each segment. This allows for the extraction of more valuable information from relatively stable data representations.

[0046] As can be seen from the above, this embodiment segments audio and video data according to the same time frame, ensuring that the audio and video data are aligned in the time dimension. This avoids errors in the feature relationships acquired by the attention network, which could lead to unreasonable weight allocation and affect the accuracy of the fusion results. Simultaneously, the process of calculating the average value smooths the original data, reducing the impact of noise and short-term fluctuations, thus making the extracted features more stable.

[0047] In one embodiment of this disclosure, audio data and video data are segmented according to the same time period, including: The first relative standard deviation of the audio data and the second relative standard deviation of the video data are calculated based on the set sliding window. If the first relative standard deviation is greater than the second threshold, or the second relative standard deviation is greater than the second threshold, the audio data is segmented based on the first time interval; If both the first relative standard deviation and the second relative standard deviation are less than or equal to the second threshold, the audio data is segmented based on the second time interval; the second time interval is greater than the first time interval.

[0048] In this embodiment, the first relative standard deviation of the audio data and the second relative standard deviation of the video data are calculated using a set sliding window, which can reflect the fluctuations of the audio and video data. For audio data, a larger first relative standard deviation indicates that the audio signal fluctuates significantly within that time period, such as the mixing of multiple sounds or a sharp change in sound intensity; for video data, a larger second relative standard deviation may indicate that the video image has changed significantly within that time period, such as scene switching or rapid movement of objects.

[0049] When the first relative standard deviation is greater than the second threshold, or vice versa, it indicates that the audio or video data fluctuates significantly within that time period. In this case, the audio and video data can be segmented based on a smaller first time interval to capture more detailed information about the changing parts. When both the first and second relative standard deviations are less than or equal to the second threshold, it indicates that the audio and video data are relatively stable with minimal changes within that time period. In this case, the audio and video data can be segmented based on a larger second time interval to reduce the amount of data processing.

[0050] The first time interval and the second time interval are both preset constants. Those skilled in the art can flexibly design the specific values ​​of the first time interval and the second time interval, for example, setting the first time interval to 1s, 2s or 3s, and setting the second time interval to 1min, 5min or 10min.

[0051] As can be seen from the above, this embodiment flexibly adjusts the segmentation interval according to the changes in audio and video data. While ensuring the data analysis effect, it minimizes the number of segments, reduces the complexity of data processing, improves the overall processing efficiency, and thus saves computing resources and time costs.

[0052] In one embodiment of this disclosure, the audio scene data analysis method further includes: Determine the location of the audio source based on audio data; The angle of the video acquisition device is adjusted based on the location of the audio source; the video acquisition device is a device for acquiring video data.

[0053] In this embodiment, the location of the audio source can be determined by processing and analyzing the audio data. For example, multiple audio acquisition devices (such as a microphone array) can be used to collect audio signals, and the spatial coordinates of the audio source can be calculated based on information such as the time difference of arrival (TDOA) or signal strength difference (RSSI) of the audio signals received by different microphones, combined with the corresponding positioning algorithm. Alternatively, a method based on a sound propagation model can be used to infer the location of the audio source by considering the propagation characteristics of sound in different environments.

[0054] Building upon this, by adjusting the angle of the video capture device to align with the audio source, better matching and fusion of audio and video information can be achieved. Once the audio source location is determined, pointing the camera at that location allows for the capture of visual information related to the audio event, contributing to a more comprehensive understanding of the scene.

[0055] For example, if an abnormal sound is detected in a certain location within the monitored area, adjusting the camera angle in time can help check whether an abnormal event has occurred at that location (such as someone breaking in, equipment malfunction, etc.).

[0056] As can be seen from the above, this embodiment determines the location of the audio source based on the audio data, and then adjusts the angle of the video acquisition device based on the location of the audio source. This enables the video acquisition device to acquire video information related to the audio event more effectively, thus improving the accuracy and effectiveness of information acquisition.

[0057] Corresponding to the audio scene data analysis method in the above embodiments, Figure 2 This is a structural block diagram of an audio scene data analysis system provided according to an embodiment of the present disclosure. For ease of explanation, only the parts relevant to the embodiment of the present disclosure are shown. References Figure 2 The audio scene data analysis system 20 includes: a feature extraction module 21, a feature fusion module 22, and an audio scene analysis module 23. Among them, the feature extraction module 21 is used to extract audio features based on audio data of the target scene and to extract video features based on video data of the target scene; Feature fusion module 22 is used to fuse audio features and video features based on an attention mechanism to obtain fused features; The audio scene analysis module 23 is used to input the fused features into the data analysis model to obtain the analysis results of the target scene.

[0058] In one embodiment of this disclosure, the feature fusion module 22 is specifically used for: The first weight of audio features and the second weight of video features are calculated based on an attention mechanism; The audio and video features are weighted and summed based on the first and second weights to obtain the fused features.

[0059] In one embodiment of this disclosure, the feature fusion module 22 is further configured to: If the visibility of the target scene is less than the first threshold, the first weight and the second weight are adjusted based on the preset ratio parameter to increase the first weight.

[0060] In one embodiment of this disclosure, the data analysis model is trained based on sample data, which includes a first fusion feature corresponding to the first output data, and a second fusion feature and a third fusion feature corresponding to the second output data; the first output data is used to indicate that the analysis result of the audio target scene is normal, and the second output data is used to indicate that the analysis result of the audio target scene is abnormal. The first fusion feature is obtained based on the first audio sample data and the first video sample data, and the second fusion feature is obtained based on the second audio sample data and the second video sample data. The third fusion feature is obtained based on the third audio sample data and the first video sample data, or The third fusion feature is obtained based on the first audio sample data and the third video sample data; the third audio sample data and the third video sample data are obtained based on SHAP value analysis of the second output data.

[0061] In one embodiment of this disclosure, the audio scene analysis module 23 is specifically used for: SHAP value analysis is performed on the second output to obtain the SHAP value corresponding to each feature; each feature includes each audio feature and each video feature. Calculate the average of the absolute values ​​of the SHAP values ​​of multiple audio features to obtain the first average value; Calculate the average of the absolute values ​​of the SHAP values ​​of multiple video features to obtain the second average; If the first average value is greater than or equal to the second average value, then the second audio sample data corresponding to the second output result is determined as the third audio sample data; If the first average value is less than the second average value, then the second video sample data corresponding to the second output result will be determined as the third video sample data.

[0062] In one embodiment of this disclosure, the feature extraction module 21 is specifically used for: The audio and video data are segmented according to the same time period; Calculate the average value of the audio data and the average value of the video data within each segment; Audio features are extracted based on the average value of the audio data within each segment, and video features are extracted based on the average value of the video data within each segment.

[0063] In one embodiment of this disclosure, the feature extraction module 21 is further configured to: The first relative standard deviation of the audio data and the second relative standard deviation of the video data are calculated based on the set sliding window. If the first relative standard deviation is greater than the second threshold, or the second relative standard deviation is greater than the second threshold, the audio data and video data are segmented based on the first time interval; If both the first relative standard deviation and the second relative standard deviation are less than or equal to the second threshold, the audio and video data are segmented based on the second time interval; the second time interval is greater than the first time interval.

[0064] In one embodiment of this disclosure, the feature extraction module 21 is specifically used for: Determine the location of the audio source based on audio data; The angle of the video acquisition device is adjusted based on the location of the audio source; the video acquisition device is a device for acquiring video data.

[0065] See Figure 3 , Figure 3 This is a schematic block diagram of an electronic device provided according to an embodiment of the present disclosure. Figure 3The electronic device 300 in this embodiment may include one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The processors 301, input devices 302, output devices 303, and memories 304 communicate with each other via a communication bus 305. The memories 304 store computer programs, including program instructions. The processors 301 execute the program instructions stored in the memories 304. Specifically, the processors 301 are configured to invoke the program instructions to perform the functions of each module / unit in the above-described device embodiments, for example... Figure 2 The functions of modules 21 to 23 are shown.

[0066] It should be understood that, in the embodiments of this disclosure, the processor 301 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0067] Input device 302 may include a touchpad, a fingerprint sensor (for collecting the user's fingerprint information and fingerprint orientation information), a microphone, etc., and output device 303 may include a display (LCD, etc.), a speaker, etc.

[0068] The memory 304 may include read-only memory and random access memory, and provides instructions and data to the processor 301. A portion of the memory 304 may also include non-volatile random access memory. For example, the memory 304 may also store device type information.

[0069] In specific implementations, the processor 301, input device 302, and output device 303 described in the embodiments of this disclosure can execute the implementation methods described in the first and second embodiments of the audio scene data analysis method provided in the embodiments of this disclosure, or they can execute the implementation methods of the electronic devices described in the embodiments of this disclosure, which will not be repeated here.

[0070] In another embodiment of this disclosure, a computer-readable storage medium is provided. This computer-readable storage medium stores a computer program, which includes program instructions. When executed by a processor, the program instructions implement all or part of the processes in the methods described above. Alternatively, the computer program can instruct related hardware to implement these processes. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include any entity or device capable of carrying computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0071] The computer-readable storage medium can be an internal storage unit of the electronic device in any of the foregoing embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device. Furthermore, the computer-readable storage medium can include both internal and external storage units of the electronic device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0072] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0073] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the electronic devices and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0074] In the several embodiments provided in this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces or units, or it may be an electrical, mechanical, or other form of connection.

[0075] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this disclosure, depending on actual needs.

[0076] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0077] The above are merely specific embodiments of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this disclosure, and these modifications or substitutions should all be covered within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. An audio scene data analysis method, characterized in that, include: Audio features are extracted from audio data of the target scene, and video features are extracted from video data of the target scene. The audio features and video features are fused based on an attention mechanism to obtain fused features; The fused features are input into the data analysis model to obtain the analysis results of the target scene; The data analysis model is trained based on sample data, which includes a first fusion feature corresponding to the first output data, and a second fusion feature and a third fusion feature corresponding to the second output data. The first output data is used to indicate that the analysis result of the audio target scene is normal, and the second output data is used to indicate that the analysis result of the audio target scene is abnormal. The first fusion feature is obtained based on the first audio sample data and the first video sample data, and the second fusion feature is obtained based on the second audio sample data and the second video sample data; The third fusion feature is obtained by fusing the third audio sample data and the first video sample data, or the third fusion feature is obtained by fusing the first audio sample data and the third video sample data. The third audio sample data and the third video sample data are obtained based on SHAP value analysis of the second output data.

2. The audio scene data analysis method as described in claim 1, characterized in that, The process of fusing the audio features and the video features based on an attention mechanism to obtain fused features includes: The first weight of the audio features and the second weight of the video features are calculated based on an attention mechanism; The audio features and the video features are weighted and summed based on the first weight and the second weight to obtain the fused features.

3. The audio scene data analysis method as described in claim 2, characterized in that, Also includes: If the visibility of the target scene is less than a first threshold, the first weight and the second weight are adjusted based on a preset ratio parameter to increase the first weight.

4. The audio scene data analysis method as described in claim 1, characterized in that, Also includes: SHAP value analysis is performed on the second output to obtain the SHAP value corresponding to each feature; Each feature includes both audio and video features; Calculate the average of the absolute values ​​of the SHAP values ​​of multiple audio features to obtain the first average value; Calculate the average of the absolute values ​​of the SHAP values ​​of multiple video features to obtain the second average; If the first average value is greater than or equal to the second average value, then the second audio sample data corresponding to the second output result is determined as the third audio sample data; If the first average value is less than the second average value, then the second video sample data corresponding to the second output result will be determined as the third video sample data.

5. The audio scene data analysis method as described in claim 1, characterized in that, Audio features are extracted from audio data based on the target scene, and video features are extracted from video data based on the target scene, including: The audio data and the video data are segmented according to the same time period; Calculate the average value of the audio data and the average value of the video data within each segment; The audio features are extracted based on the average value of the audio data within each segment, and the video features are extracted based on the average value of the video data within each segment.

6. The audio scene data analysis method as described in claim 5, characterized in that, The step of segmenting the audio data and the video data according to the same time period includes: The first relative standard deviation of the audio data and the second relative standard deviation of the video data are calculated based on the set sliding window. If the first relative standard deviation is greater than the second threshold, or the second relative standard deviation is greater than the second threshold, the audio data and the video data are segmented based on the first time interval; If both the first relative standard deviation and the second relative standard deviation are less than or equal to the second threshold, the audio data and the video data are segmented based on the second time interval; the second time interval is greater than the first time interval.

7. An audio scene data analysis system, characterized in that, include: The feature extraction module is used to extract audio features from audio data of the target scene and video features from video data of the target scene. The feature fusion module is used to fuse the audio features and the video features based on an attention mechanism to obtain fused features; The audio scene analysis module is used to input the fused features into the data analysis model to obtain the analysis results of the target scene; The data analysis model is trained based on sample data, which includes a first fusion feature corresponding to the first output data, and a second fusion feature and a third fusion feature corresponding to the second output data. The first output data is used to indicate that the analysis result of the audio target scene is normal, and the second output data is used to indicate that the analysis result of the audio target scene is abnormal. The first fusion feature is obtained based on the first audio sample data and the first video sample data, and the second fusion feature is obtained based on the second audio sample data and the second video sample data; The third fusion feature is obtained by fusing the third audio sample data and the first video sample data, or the third fusion feature is obtained by fusing the first audio sample data and the third video sample data. The third audio sample data and the third video sample data are obtained based on SHAP value analysis of the second output data.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

Images