Earthquake survivor identification method and device based on audio and video fusion

Abstract

The present application provides a method and device for earthquake survivor identification based on audio and video fusion, the method comprising: determining a target video, wherein the target video comprises N video clips, where N is a positive integer; extracting audio features and visual features corresponding to each of the N video clips; based on a self-attention mechanism and a cross-modal attention mechanism, fusing the same-modal features and cross-modal features of the audio features and visual features corresponding to each of the video clips, and determining the temporal audio features and temporal visual features corresponding to each of the video clips; based on the temporal audio features and temporal visual features, determining an earthquake survivor identification result of the target video, thereby improving the accuracy of survivor identification.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method and device for identifying earthquake survivors based on audio and video fusion. Background Art

[0002] During the post-earthquake rescue process, how to confirm whether there are survivors in the honeycomb-shaped holes of collapsed buildings is a basic rescue task. For example, based on the video or audio data of the earthquake scene, the detection of human targets through artificial intelligence algorithms can speed up disaster rescue.

[0003] With the application of deep learning, existing object detection tasks have made great progress. For example, Redmon et al. regard the object detection problem as a regression problem and use a separate network to solve it, while Ren et al. generate candidate regions based on the region candidate network and then obtain the location information of the target through the regressor.

[0004] However, existing detection methods are difficult to cope with complex and changeable earthquake scenes with harsh environments. Specifically, the complex and changeable scenes after the disaster will cause the single-modal human target features to lack effective information, thus affecting the accuracy of the survivor identification task. Summary of the invention

[0005] The present application provides an earthquake survivor identification method and device based on audio and video fusion, so as to solve the defect of low accuracy of survivor identification tasks in the prior art and improve the accuracy of survivor identification.

[0006] The present application provides an earthquake survivor identification method based on audio and video fusion, comprising:

[0007] Determine a target video, where the target video includes N video clips, where N is a positive integer;

[0008] Extracting audio features and visual features corresponding to each of the N video clips;

[0009] Based on the self-attention mechanism and the cross-modal attention mechanism, the same-modal features and cross-modal features of the audio features and visual features corresponding to the respective video clips are integrated to determine the temporal audio features and temporal visual features corresponding to the respective video clips;

[0010] Based on the temporal audio features and the temporal visual features, an earthquake survivor recognition result of the target video is determined.

[0011] According to an earthquake survivor identification method based on audio and video fusion provided by the present application, the step of extracting audio features and visual features corresponding to each of the N video clips includes:

[0012] Determining audio information and visual information corresponding to each of the N video clips;

[0013] Extracting backbone features of the audio information and visual information, wherein the backbone features include audio backbone features, visual two-dimensional backbone features, and visual three-dimensional backbone features;

[0014] The backbone features are mapped to audio features and visual features corresponding to each of the video clips.

[0015] According to an earthquake survivor identification method based on audio and video fusion provided by the present application, the self-attention mechanism and the cross-modal attention mechanism are based on fusing the same-modal features and cross-modal features of the audio features and visual features corresponding to each video clip, and determining the temporal audio features and temporal visual features corresponding to each video clip, including:

[0016] Based on the audio features and visual features corresponding to the respective video clips, determining a homomodal temporal audio feature corresponding to each of the audio features, a cross-modal temporal visual feature corresponding to each of the audio features, a homomodal temporal visual feature corresponding to each of the visual features, and a cross-modal temporal visual feature corresponding to each of the visual features;

[0017] Based on the audio features of each video clip, the isomodal temporal audio features and the cross-modal temporal visual features, the temporal audio features of each video clip are determined, and based on the visual features of each video clip, the isomodal temporal visual features and the cross-modal temporal audio features, the temporal audio features of each video clip are determined.

[0018] According to an earthquake survivor identification method based on audio and video fusion provided by the present application, based on the audio features and visual features corresponding to the respective video clips, determining the same-modal temporal audio features corresponding to each audio feature, the cross-modal temporal visual features corresponding to each audio feature, the same-modal temporal visual features corresponding to each visual feature, and the cross-modal temporal visual features corresponding to each visual feature, including:

[0019] Determine an audio-to-audio similarity between the audio feature of each video clip and the audio features of the N video clips, and determine a same-modal temporal audio feature corresponding to each of the audio features based on the audio-to-audio similarity;

[0020] Determine the audio-visual similarity between the audio feature of each video clip and the visual features of the N video clips, and determine the cross-modal temporal visual feature corresponding to each of the audio features based on the audio-visual similarity;

[0021] Determine a visual-to-visual similarity between a visual feature of each video clip and the visual features of the N video clips, and determine a same-modal temporal visual feature corresponding to each of the visual features based on the visual-to-visual similarity;

[0022] Determine the visual and audio similarity between the visual feature of each video clip and the audio features of the N video clips, and determine the cross-modal temporal audio feature corresponding to each of the visual features based on the visual and audio similarity.

[0023] According to an earthquake survivor identification method based on audio and video fusion provided by the present application, the step of determining the audio-to-audio similarity between the audio features of each video clip and the audio features of the N video clips includes:

[0024] Based on the audio features of the N video clips and an audio-to-audio similarity calculation formula, determine the audio-to-audio similarity between each of the audio features and the audio features of the N video clips;

[0025] The audio-to-audio similarity calculation formula is as follows:

[0026]

[0027] in, represents the audio features of the kth segment of the target video, where k is a positive integer less than N. represents the audio features of the jth segment of the target video, where j is a positive integer less than N, express and The audio-to-audio similarity, T represents the transposition operation, d represents the dimension of the feature, represents the audio features of the qth segment of the target video, q is a positive integer less than N, and exp represents an exponential function with the natural constant e as the base;

[0028] The determining, based on the audio-to-audio similarity, a same-modal temporal audio feature corresponding to each of the audio features comprises:

[0029] Based on the audio-to-audio similarity between the audio feature of each video clip and the audio features of the N video clips, using a homomodal temporal audio feature calculation formula, determine a homomodal temporal audio feature corresponding to each of the audio features;

[0030] The same-modal temporal audio feature calculation formula is as follows:

[0031]

[0032] in, Representation and The corresponding same-modal temporal audio features, Represents the audio features of the sth segment of the target video, where s is a positive integer less than N. express and The audio-to-audio similarity.

[0033] According to an earthquake survivor identification method based on audio and video fusion provided by the present application, the step of determining the audio-visual similarity between the audio features of each video clip and the visual features of the N video clips includes:

[0034] Determining the audio-visual similarity between each of the audio features and the visual features of the N video clips based on the audio features of the N video clips and an audio-visual similarity calculation formula;

[0035] The audio-visual similarity calculation formula is as follows:

[0036]

[0037] in, represents the audio features of the kth segment of the target video, where k is a positive integer less than N. represents the visual features of the jth segment of the target video, where j is a positive integer less than N. express and The audio visual similarity between them, T represents the transposition operation, d represents the dimension of the feature, represents the visual features of the qth segment of the target video, q is a positive integer less than N, and exp represents an exponential function with the natural constant e as the base;

[0038] The determining, based on the audio-visual similarity, a cross-modal temporal visual feature corresponding to each of the audio features comprises:

[0039] Based on the audio-to-audio similarity between the audio feature of each video clip and the visual features of the N video clips, a cross-modal temporal visual feature calculation formula is used to determine the cross-modal temporal visual feature corresponding to each of the audio features;

[0040] The cross-modal temporal visual feature calculation formula is as follows:

[0041]

[0042] in, Representation and The corresponding cross-modal temporal visual features, Represents the visual features of the sth segment of the target video, where s is a positive integer less than N. express and Audio-visual similarity.

[0043] The present application also provides an earthquake survivor identification device based on audio and video fusion, comprising:

[0044] An input module, used to determine a target video, wherein the target video includes N video clips, where N is a positive integer;

[0045] A feature extraction module, used to extract audio features and visual features corresponding to each of the N video clips;

[0046] A cross-modal hybrid attention module, configured to determine temporal audio features and temporal visual features corresponding to each video clip by fusing homomodal features and cross-modal features of the audio features and visual features corresponding to each video clip;

[0047] A classification module is used to determine the earthquake survivor recognition result of the target video based on the temporal audio features and the temporal visual features.

[0048] The present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for identifying earthquake survivors based on audio and video fusion as described above is implemented.

[0049] The present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for identifying earthquake survivors based on audio and video fusion as described in any one of the above is implemented.

[0050] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for identifying earthquake survivors based on audio and video fusion.

[0051] The earthquake survivor identification method and device based on audio and video fusion provided in the present application determine the temporal audio features and temporal visual features corresponding to each video clip by fusing the homomodal features and cross-modal features of the audio features and visual features corresponding to each video clip, and then identify the earthquake survivors based on the temporal audio features and temporal visual features. The self-attention mechanism and the cross-modal attention mechanism are used to fuse information of different modalities, thereby overcoming the problem of limited information of single modal data and improving the accuracy of earthquake survivor identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0053] Figure 1 It is a flowchart of an earthquake survivor identification method based on audio and video fusion provided by this application;

[0054] Figure 2 It is a schematic diagram of the network structure provided by this application;

[0055] Figure 3 It is a flowchart of determining the temporal audio features and temporal visual features corresponding to each video clip based on the self-attention mechanism and the cross-modal attention mechanism provided by the present application, fusing the same-modal features and cross-modal features of the audio features and visual features corresponding to each video clip;

[0056] Figure 4 It is a schematic diagram of positive and negative samples of videos in the constructed dataset provided by this application;

[0057] Figure 5 It is a structural schematic diagram of an earthquake survivor identification device based on audio and video fusion provided by the present application;

[0058] Figure 6 It is a structural schematic diagram of the electronic device provided by this application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0060] In order to overcome the problem of inaccurate perception caused by incomplete single-modal information, audio and video fusion is a reasonable solution. On the one hand, visual information can overcome the identification of survivors without audio information. On the other hand, audio information can overcome the identification of survivors in the case of severe human target occlusion. Audio and video fusion information is generally used in speech recognition, event location and emotion classification tasks. Compared with the above tasks, the scene of post-disaster survivor identification is more complex and changeable. Specifically, the post-disaster scene has poor visibility, many interferences, and diverse target state changes. At the same time, the audio information is weak, contains noise, and is easily disturbed by the environment.

[0061] Figure 1 is a flow chart of an earthquake survivor identification method based on audio and video fusion provided in an embodiment of the present application, such as Figure 1 As shown, the earthquake survivor identification method based on audio and video fusion includes: step 100, step 101, step 102 and step 103.

[0062] Step 100: Determine a target video, where the target video includes N video clips, where N is a positive integer.

[0063] Optionally, for a given target video P, the target video may be divided into N video segments. The target video may be a video shot at an earthquake site. The video segments refer to dividing the target video according to the time dimension to obtain N segments.

[0064] Step 101: extracting audio features and visual features corresponding to each of the N video clips.

[0065] It can be understood that each segment in the target video includes not only audio information, but also image information, i.e., visual information. Where P i is the i-th video clip in P, A i is the audio information corresponding to the i-th video clip, V i is the visual information corresponding to the i-th video clip, and i is a positive integer less than or equal to N.

[0066] The audio features and visual features of each video clip can be obtained through feature extraction.

[0067] In some embodiments, step 101 includes:

[0068] Determining audio information and visual information corresponding to each of the N video clips;

[0069] Extracting backbone features of the audio information and visual information, wherein the backbone features include audio backbone features, visual two-dimensional backbone features, and visual three-dimensional backbone features;

[0070] The backbone features are mapped to audio features and visual features corresponding to each of the video clips.

[0071] Optionally, Figure 2 is a schematic diagram of the network structure provided in the embodiment of the present application, such as Figure 2 As shown, for audio A i With visual information V i First, we use the pre-trained model of audio-visual perception, that is, the backbone network to extract A i and V iThe backbone features obtained after extraction include: audio backbone features Visual 2D backbone features and visual 3D backbone features

[0072] In order to obtain unimodal high-order features, the feature mapping module maps the backbone audio features into And fuse 2D and 3D visual features into Then use the feature mapping module to map the backbone features into audio features and visual features In step 102 and step 103, the cross-modal feature fusion module is used to obtain and The effective timing features are obtained and the effective timing features are classified to obtain the final result.

[0073] Step 102: Based on the self-attention mechanism and the cross-modal attention mechanism, the homomodal features and cross-modal features of the audio features and visual features corresponding to the respective video clips are integrated to determine the temporal audio features and temporal visual features corresponding to the respective video clips.

[0074] It should be noted that since a video is intended to describe an event, the video clips in the target video are continuous and repetitive. Specifically, the audio events or visual events in the target video usually occur repeatedly in the video unimodal data or cross-modal data, i.e., audio-visual synchronization data.

[0075] The cross-modal mixed attention module can be used to search for homomodal information and cross-modal information similar to each audio feature and visual feature from all video clips, fuse the homomodal features and cross-modal features of the audio features and visual features corresponding to each video clip, and determine the temporal audio features and temporal visual features corresponding to each video clip.

[0076] like Figure 3 As shown, in some embodiments, step 102 includes: step 300 and step 301 .

[0077] Step 300: Based on the audio features and visual features corresponding to the various video clips, determine the homomodal temporal audio features corresponding to each of the audio features, the cross-modal temporal visual features corresponding to each of the audio features, the homomodal temporal visual features corresponding to each of the visual features, and the cross-modal temporal visual features corresponding to each of the visual features.

[0078] Optionally, homomodal features include two types: homomodal features between audio features and homomodal features between visual features; correspondingly, cross-modal features also include two types: cross-modal temporal visual features corresponding to audio features and cross-modal temporal audio features corresponding to visual features.

[0079] Optionally, step 300 is implemented by a cross-modal mixed attention module, which includes a self-attention mechanism and a cross-modal attention mechanism.

[0080] Among them, the self-attention mechanism is used to capture the internal correlation of unimodal features, and the self-attention mechanism includes audio self-attention mechanism and visual self-attention mechanism; the cross-modal self-attention mechanism is used to capture cross-modal temporal context information, and the cross-modal attention mechanism includes audio-visual attention mechanism and visual-audio attention mechanism.

[0081] That is to say, there are four types of attention mechanisms: audio self-attention mechanism, visual self-attention mechanism, audio-visual attention mechanism, and visual-audio attention mechanism. These four types of attention mechanisms are introduced in detail below.

[0082] 1. Audio self-attention mechanism.

[0083] Optionally, an audio-to-audio similarity between the audio feature of each video clip and the audio features of the N video clips is determined, and based on the audio-to-audio similarity, a same-modal temporal audio feature corresponding to each of the audio features is determined.

[0084] Optionally, for a given target video, the audio features of the kth segment First measure and the audio features of the target video The audio-to-audio similarity of each audio clip in :

[0085]

[0086] in, represents the audio features of the kth segment of the target video, where k is a positive integer less than N. represents the audio features of the jth segment of the target video, where j is a positive integer less than N, express and The audio-to-audio similarity, T represents the transposition operation, d represents the dimension of the feature, represents the audio features of the qth segment of the target video, q is a positive integer less than N, and exp represents an exponential function with the natural constant e as the base.

[0087] It is understandable that the target video has N segments, so N audio-to-audio similarities can be obtained:

[0088] Based on the audio-to-audio similarity between the audio feature of each video clip and the audio features of the N video clips, a homomodal temporal audio feature calculation formula is used to determine the homomodal temporal audio feature corresponding to each of the audio features.

[0089] The same-modal temporal audio feature calculation formula is as follows:

[0090]

[0091] in, Representation and The corresponding same-modal temporal audio features, Represents the audio features of the sth segment of the target video, where s is a positive integer less than N. express and The audio-to-audio similarity.

[0092] in, Representation and Related homomodal temporal audio features, Represents the audio features of the sth segment of the target video, where s is a positive integer less than N. express and The audio-to-audio similarity.

[0093] Based on the audio-to-audio similarity, the integration is Correlated homomodal temporal audio features.

[0094] 2. Audio-visual attention mechanism.

[0095] Optionally, the audio-visual similarity between the audio feature of each video clip and the visual features of the N video clips is determined, and based on the audio-visual similarity, the cross-modal temporal visual feature corresponding to each of the audio features is determined.

[0096] The audio features of the same modality of the video are correlated, and the cross-modal visual features of each segment of the video It will also be consistent with the current audio characteristics Therefore, this application uses audio visual similarity to measure the audio features of the kth segment And each fragment visual feature of relevance.

[0097] Optionally, based on the audio features of the N video clips and an audio-visual similarity calculation formula, determining the audio-visual similarity between each of the audio features and the visual features of the N video clips;

[0098] The audio-visual similarity calculation formula is as follows:

[0099]

[0100] in, represents the audio features of the kth segment of the target video, where k is a positive integer less than N. represents the visual features of the jth segment of the target video, where j is a positive integer less than N. express and The audio visual similarity between them, T represents the transposition operation, d represents the dimension of the feature, represents the visual features of the qth segment of the target video, q is a positive integer less than N, and exp represents an exponential function with the natural constant e as the base.

[0101] It is understandable that the target video has N segments, so N audio-visual similarities can be obtained:

[0102] With the help of audio-visual similarity, we can get Related cross-modal temporal visual features:

[0103]

[0104] in, Representation and Related cross-modal temporal visual features, Represents the visual features of the sth segment of the target video, where s is a positive integer less than N. express and Audio-visual similarity.

[0105] 3. Visual self-attention mechanism.

[0106] Optionally, a visual-to-visual similarity between the visual feature of each video clip and the visual features of the N video clips is determined, and based on the visual-to-visual similarity, a homomodal temporal visual feature corresponding to each of the visual features is determined.

[0107] Similar to audio self-attention, for a given target video, the visual features of the kth segment are First measure and the visual features of the target video The visual similarity between each visual segment in:

[0108]

[0109] in, represents the visual features of the jth segment of the target video, express and The visual and visual similarity of is the visual feature of the qth segment of the target video, q is a positive integer less than N, N is the number of segments of the target video, T represents the transposition operation, d represents the dimension of the feature, and exp represents an exponential function with the natural constant e as the base.

[0110] It is understandable that the target video has N segments, so N visual-to-visual similarities can be obtained:

[0111] Based on visual and visual similarity, we can integrate Related homomodal temporal visual features:

[0112]

[0113] in, Representation and Related homomodal temporal visual features, Represents the visual features of the sth segment of the target video, where s is a positive integer less than N. express and The visual and visual similarity.

[0114] 4. Visual audio attention mechanism.

[0115] Optionally, the visual and audio similarities between the visual features of each video clip and the audio features of the N video clips are determined, and based on the visual and audio similarities, the cross-modal temporal audio features corresponding to each of the visual features are determined.

[0116] The same modality visual features of the video are correlated, and the cross-modal audio features of each segment of the video are It will also be consistent with the current visual features. Therefore, this application uses visual audio similarity to measure the visual features of the kth segment. And each segment audio feature Relevance:

[0117]

[0118] in, represents the visual features of the kth segment of the target video, is the audio feature of the jth segment of the target video, for and The visual audio similarity of is the audio feature of the qth segment of the target video, q is a positive integer less than N, N is the number of segments of the target video, T represents the transposition operation, d represents the dimension of the feature, and exp represents the exponential function with the natural constant e as the base.

[0119] It is understandable that the target video has N segments, so N visual-audio similarities can be obtained:

[0120] With the help of visual audio similarity, we can get Related cross-modal temporal audio features:

[0121]

[0122] in, Representation and Related cross-modal temporal audio features, Represents the audio features of the sth segment of the target video, where s is a positive integer less than N. express and The visual-audio similarity.

[0123] In order to integrate cross-modal features, this application uses a cross-modal mixed attention module to search for the same modality information and cross-modal information similar to each audio feature and visual feature from all video clips, overcoming the problem of limited single modality data information and improving the accuracy of survivor identification.

[0124] Step 301: determine the temporal audio features of each video clip based on the audio features of each video clip, the homomodal temporal audio features and the cross-modal temporal visual features, and determine the temporal audio features of each video clip based on the visual features of each video clip, the homomodal temporal visual features and the cross-modal temporal audio features.

[0125] Optionally, based on the audio features of each video clip, the same modal features between the audio features, and the cross-modal temporal visual features corresponding to the audio features, the temporal audio features can be obtained.

[0126] Audio features for the kth segment of the target video This application integrates the video with Related Homomodal Audio Features and cross-modal visual features And use the skip connection to save its own audio features, and finally get the updated temporal audio features:

[0127]

[0128] for The corresponding temporal audio features, For Related homomodal temporal audio features, For Related cross-modal temporal visual features.

[0129] It should be noted that skip connections are usually used in residual networks to alleviate the gradient explosion and gradient vanishing problems during the training of deeper networks. Additional audio features output by the feature mapping module

[0130] Based on the visual features of each video clip, the same modal features between the visual features, and the cross-modal temporal audio features corresponding to the visual features, the temporal visual features can be obtained.

[0131] Visual features of the kth segment of the target video This application integrates the video with Related Homomodal Audio Features and cross-modal visual features And use the jump connection to save its own visual features, and finally get the updated temporal visual features:

[0132]

[0133] for The corresponding temporal visual features, For Related homomodal temporal visual features, For Related cross-modal temporal visual features.

[0134] Step 103: Determine the earthquake survivor recognition result of the target video based on the temporal audio features and the temporal visual features.

[0135] Optionally, in order to achieve video classification, the present application is based on the updated temporal audio features and temporal visual features Categorize each fragment.

[0136] Using a shared fully connected layer The time series segment features and Project it into the category space and use the sigmoid function (σ(·)) to get the classification probability:

[0137]

[0138]

[0139] Representing temporal audio features The classification results, Representing temporal visual features The classification results.

[0140] Finally, the single-mode prediction results of each segment of the video are integrated to classify the video, that is, the prediction results of all modal categories of all segments in the video are summed up and comprehensively considered to obtain the classification probability of the final video. The calculation formula is as follows:

[0141]

[0142] The earthquake survivor identification method based on audio and video fusion provided in the present application determines the temporal audio features and temporal visual features corresponding to each video clip by fusing the homomodal features and cross-modal features of the audio features and visual features corresponding to each video clip, and then identifies the earthquake survivors based on the temporal audio features and temporal visual features. It uses the self-attention mechanism and the cross-modal attention mechanism to fuse information of different modalities, thereby overcoming the problem of limited information of single modal data and improving the accuracy of earthquake survivor identification.

[0143] In some embodiments, the audio features and visual features corresponding to each of the N video clips are extracted; based on the self-attention mechanism and the cross-modal attention mechanism, the homomodal features and cross-modal features of the audio features and visual features corresponding to each of the video clips are integrated to determine the temporal audio features and temporal visual features corresponding to each of the video clips; based on the temporal audio features and temporal visual features, the earthquake survivor recognition results of the target video are determined, including: inputting the N video clips into an earthquake survivor recognition model to obtain the earthquake survivor recognition results output by the earthquake survivor recognition model.

[0144] The earthquake survivor identification model is obtained by constructing a training set for training.

[0145] With the rapid development of the new generation of artificial intelligence technology, the establishment of intelligent survivor identification tasks based on audio and video data has received increasing attention. However, the realization of intelligent survivor identification tasks mainly faces the problem of lack of audio and video data of real scenes. Due to the instantaneous and destructive nature of earthquake disasters, it is difficult for existing monitoring equipment to capture real earthquake disaster scenes, and therefore it is impossible to obtain real data.

[0146] In order to construct post-earthquake disaster simulation data for real scenarios, this application selects video clips from earthquake scenes shot in earthquake movies to construct a post-disaster survivor identification dataset.

[0147] In order to simulate a real post-disaster scene, this embodiment of the application selected 341 10-second video data samples from three movies, including data in both visual and audio modalities, to construct the Audio-Visual Earthquake Person Recognition Dataset (AVEPR).

[0148] like Figure 4 As shown in Figure 2, the data can be divided into positive samples and negative samples according to whether the video sample contains human target information. Specifically, the positive sample video includes three situations:

[0149] like Figure 4 In video sample 1, the visual data contains human target information but the audio does not contain human target speech; in video sample 2, the visual data contains no human target information but the audio data can be judged to contain human target speech; in video sample 3, both the visual and audio data contain human target information.

[0150] Video negative samples are those that do not contain human target information in both visual and audio data, such as Figure 4 Medium video sample 4.

[0151] In addition, each video data can be divided into visual data and audio data for unimodal tasks, such as object detection, speech recognition, etc. The positive and negative sample distribution of unimodal visual data can be 236:105, and the positive and negative sample distribution of audio data can be 188:153.

[0152] Each video sample and sample label is input into the earthquake survivor recognition model before training for training. The earthquake survivor recognition model is based on the self-attention mechanism and the cross-modal attention mechanism. It integrates the homomodal features and cross-modal features of the audio features and visual features corresponding to each video clip of the video sample, determines the temporal audio features and temporal visual features corresponding to each video clip of the video sample, and determines the video classification probability of the video sample based on the temporal audio features and temporal visual features.

[0153] Combined with the real label Y of the videoV and video classification probability The video classification loss can be obtained as:

[0154]

[0155] Where CE represents the cross entropy loss.

[0156] When the cross entropy loss is greater than or equal to the target value, the model parameters are updated and the model is trained. When the cross entropy loss is less than or equal to the target value, the model parameters are saved to obtain an earthquake survivor identification model.

[0157] Table 1 is a table of experimental results of the present application on multiple methods and data sets. It can be seen that the method of the present invention has a significant effect on the accuracy of earthquake survivor identification, where w / o A and w / o V are the methods of the present application that only use visual data and audio data, respectively, and AVF is the method of the present invention based on cross-modal feature fusion.

[0158] Table 1 Experimental results of this application on multiple methods and datasets

[0159] method Task1 Task2 Task3 Task4 Task5 Avg LSTM 70.37 86.80 87.82 85.39 81.07 82.29 GRU 70.13 84.64 88.12 85.99 83.97 82.57 w / o V 77.79 76.90 77.06 78.44 77.24 77.49 w / o A 76.61 74.20 77.67 76.30 79.16 76.79 AVF 73.71 88.05 87.48 88.21 81.84 82.86

[0160] The earthquake survivor identification device based on audio and video fusion provided in the present application is described below. The earthquake survivor identification device based on audio and video fusion described below and the earthquake survivor identification method based on audio and video fusion described above can refer to each other.

[0161] like Figure 5 As shown, the earthquake survivor identification device 500 based on audio and video fusion includes: an input module 510, a feature extraction module 520, a cross-modal mixed attention module 530 and a classification module 540.

[0162] An input module 510 is used to determine a target video, where the target video includes N video clips, where N is a positive integer;

[0163] A feature extraction module 520, configured to extract audio features and visual features corresponding to each of the N video clips;

[0164] A cross-modal hybrid attention module 530, configured to determine temporal audio features and temporal visual features corresponding to each video segment by fusing homomodal features and cross-modal features of the audio features and visual features corresponding to each video segment;

[0165] The classification module 540 is used to determine the earthquake survivor recognition result of the target video based on the temporal audio features and the temporal visual features.

[0166] Optionally, the feature extraction module 520 is further used to:

[0167] Determining audio information and visual information corresponding to each of the N video clips;

[0168] Extracting backbone features of the audio information and visual information, wherein the backbone features include audio backbone features, visual two-dimensional backbone features, and visual three-dimensional backbone features;

[0169] The backbone features are mapped to audio features and visual features corresponding to each of the video clips.

[0170] Optionally, the cross-modal mixed attention module 530 is further used to:

[0171] Based on the audio features and visual features corresponding to the respective video clips, determining a homomodal temporal audio feature corresponding to each of the audio features, a cross-modal temporal visual feature corresponding to each of the audio features, a homomodal temporal visual feature corresponding to each of the visual features, and a cross-modal temporal visual feature corresponding to each of the visual features;

[0172] Based on the audio features of each video clip, the isomodal temporal audio features and the cross-modal temporal visual features, the temporal audio features of each video clip are determined, and based on the visual features of each video clip, the isomodal temporal visual features and the cross-modal temporal audio features, the temporal audio features of each video clip are determined.

[0173] Optionally, the cross-modal mixed attention module 530 is further used to:

[0174] Determine an audio-to-audio similarity between the audio feature of each video clip and the audio features of the N video clips, and determine a same-modal temporal audio feature corresponding to each of the audio features based on the audio-to-audio similarity;

[0175] Determine the audio-visual similarity between the audio feature of each video clip and the visual features of the N video clips, and determine the cross-modal temporal visual feature corresponding to each of the audio features based on the audio-visual similarity;

[0176] Determine a visual-to-visual similarity between a visual feature of each video clip and the visual features of the N video clips, and determine a same-modal temporal visual feature corresponding to each of the visual features based on the visual-to-visual similarity;

[0177] Determine the visual and audio similarity between the visual feature of each video clip and the audio features of the N video clips, and determine the cross-modal temporal audio feature corresponding to each of the visual features based on the visual and audio similarity.

[0178] Optionally, determining the audio-to-audio similarity between the audio feature of each video clip and the audio features of the N video clips includes:

[0179] Based on the audio features of the N video clips and an audio-to-audio similarity calculation formula, determine the audio-to-audio similarity between each of the audio features and the audio features of the N video clips;

[0180] The audio-to-audio similarity calculation formula is as follows:

[0181]

[0182] in, represents the audio features of the kth segment of the target video, where k is a positive integer less than N. represents the audio features of the jth segment of the target video, where j is a positive integer less than N, express and The audio-to-audio similarity, T represents the transposition operation, d represents the dimension of the feature, represents the audio features of the qth segment of the target video, q is a positive integer less than N, and exp represents an exponential function with the natural constant e as the base;

[0183] The determining, based on the audio-to-audio similarity, a same-modal temporal audio feature corresponding to each of the audio features comprises:

[0184] Based on the audio-to-audio similarity between the audio feature of each video clip and the audio features of the N video clips, using a homomodal temporal audio feature calculation formula, determine a homomodal temporal audio feature corresponding to each of the audio features;

[0185] The same-modal temporal audio feature calculation formula is as follows:

[0186]

[0187] in, Representation and The corresponding same-modal temporal audio features, Represents the audio features of the sth segment of the target video, where s is a positive integer less than N. express and The audio-to-audio similarity.

[0188] Optionally, determining the audio-visual similarity between the audio feature of each video clip and the visual features of the N video clips includes:

[0189] Determining the audio-visual similarity between each of the audio features and the visual features of the N video clips based on the audio features of the N video clips and an audio-visual similarity calculation formula;

[0190] The audio-visual similarity calculation formula is as follows:

[0191]

[0192] in, represents the audio features of the kth segment of the target video, where k is a positive integer less than N. represents the visual features of the jth segment of the target video, where j is a positive integer less than N. express and The audio visual similarity between them, T represents the transposition operation, d represents the dimension of the feature, represents the visual features of the qth segment of the target video, q is a positive integer less than N, and exp represents an exponential function with the natural constant e as the base;

[0193] The determining, based on the audio-visual similarity, a cross-modal temporal visual feature corresponding to each of the audio features comprises:

[0194] Based on the audio-to-audio similarity between the audio feature of each video clip and the visual features of the N video clips, a cross-modal temporal visual feature calculation formula is used to determine the cross-modal temporal visual feature corresponding to each of the audio features;

[0195] The cross-modal temporal visual feature calculation formula is as follows:

[0196]

[0197] in, Representation and The corresponding cross-modal temporal visual features, Represents the visual features of the sth segment of the target video, where s is a positive integer less than N. express and Audio-visual similarity.

[0198] It should be noted here that the above-mentioned device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those in the method embodiment will not be described in detail here.

[0199] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the earthquake survivor identification method based on audio and video fusion, and the method includes:

[0200] Determine a target video, where the target video includes N video clips, where N is a positive integer;

[0201] Extracting audio features and visual features corresponding to each of the N video clips;

[0202] Based on the self-attention mechanism and the cross-modal attention mechanism, the same-modal features and cross-modal features of the audio features and visual features corresponding to the respective video clips are integrated to determine the temporal audio features and temporal visual features corresponding to the respective video clips;

[0203] Based on the temporal audio features and the temporal visual features, an earthquake survivor recognition result of the target video is determined.

[0204] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0205] On the other hand, the present application further provides a computer program product, the computer program product comprising a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer can execute the earthquake survivor identification method based on audio and video fusion provided by the above methods, the method comprising:

[0206] Determine a target video, where the target video includes N video clips, where N is a positive integer;

[0207] Extracting audio features and visual features corresponding to each of the N video clips;

[0208] Based on the self-attention mechanism and the cross-modal attention mechanism, the same-modal features and cross-modal features of the audio features and visual features corresponding to the respective video clips are integrated to determine the temporal audio features and temporal visual features corresponding to the respective video clips;

[0209] Based on the temporal audio features and the temporal visual features, an earthquake survivor recognition result of the target video is determined.

[0210] On the other hand, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the earthquake survivor identification method based on audio and video fusion provided by the above methods, the method comprising:

[0211] Determine a target video, where the target video includes N video clips, where N is a positive integer;

[0212] Extracting audio features and visual features corresponding to each of the N video clips;

[0213] Based on the self-attention mechanism and the cross-modal attention mechanism, the same-modal features and cross-modal features of the audio features and visual features corresponding to the respective video clips are integrated to determine the temporal audio features and temporal visual features corresponding to the respective video clips;

[0214] Based on the temporal audio features and the temporal visual features, an earthquake survivor recognition result of the target video is determined.

[0215] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0216] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0217] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for identifying earthquake survivors based on audio and video fusion, characterized in that: include: Determine a target video, where the target video includes N video clips, where N is a positive integer; Extracting audio features and visual features corresponding to each of the N video clips; Based on the self-attention mechanism and the cross-modal attention mechanism, the same-modal features and cross-modal features of the audio features and visual features corresponding to the respective video clips are integrated to determine the temporal audio features and temporal visual features corresponding to the respective video clips; Determining an earthquake survivor recognition result of the target video based on the temporal audio features and the temporal visual features; The extracting audio features and visual features corresponding to each of the N video clips includes: Determining audio information and visual information corresponding to each of the N video clips; Extracting backbone features of the audio information and visual information, wherein the backbone features include audio backbone features, visual two-dimensional backbone features, and visual three-dimensional backbone features; The backbone features are mapped to audio features and visual features corresponding to each of the video clips.

2. The earthquake survivor identification method based on audio and video fusion according to claim 1 is characterized in that: The method of fusing the same modal features and cross-modal features of the audio features and visual features corresponding to each video clip based on the self-attention mechanism and the cross-modal attention mechanism to determine the temporal audio features and temporal visual features corresponding to each video clip includes: Based on the audio features and visual features corresponding to the respective video clips, determining a homomodal temporal audio feature corresponding to each of the audio features, a cross-modal temporal visual feature corresponding to each of the audio features, a homomodal temporal visual feature corresponding to each of the visual features, and a cross-modal temporal audio feature corresponding to each of the visual features; Based on the audio features of each video clip, the isomodal temporal audio features and the cross-modal temporal visual features, the temporal audio features of each video clip are determined, and based on the visual features of each video clip, the isomodal temporal visual features and the cross-modal temporal audio features, the temporal visual features of each video clip are determined.

3. The earthquake survivor identification method based on audio and video fusion according to claim 2 is characterized in that: The determining, based on the audio features and visual features corresponding to the respective video clips, a homomodal temporal audio feature corresponding to each audio feature, a cross-modal temporal visual feature corresponding to each audio feature, a homomodal temporal visual feature corresponding to each visual feature, and a cross-modal temporal audio feature corresponding to each visual feature, includes: Determine an audio-to-audio similarity between the audio feature of each video clip and the audio features of the N video clips, and determine a same-modal temporal audio feature corresponding to each of the audio features based on the audio-to-audio similarity; Determine the audio-visual similarity between the audio feature of each video clip and the visual features of the N video clips, and determine the cross-modal temporal visual feature corresponding to each of the audio features based on the audio-visual similarity; Determine a visual-to-visual similarity between a visual feature of each video clip and the visual features of the N video clips, and determine a same-modal temporal visual feature corresponding to each of the visual features based on the visual-to-visual similarity; Determine the visual and audio similarity between the visual feature of each video clip and the audio features of the N video clips, and determine the cross-modal temporal audio feature corresponding to each of the visual features based on the visual and audio similarity.

4. The earthquake survivor identification method based on audio and video fusion according to claim 3 is characterized in that: The step of determining the audio-to-audio similarity between the audio features of each video clip and the audio features of the N video clips includes: Based on the audio features of the N video clips and an audio-to-audio similarity calculation formula, determine the audio-to-audio similarity between each of the audio features and the audio features of the N video clips; The audio-to-audio similarity calculation formula is as follows: in, represents the audio features of the kth segment of the target video, where k is a positive integer less than N. represents the audio features of the jth segment of the target video, where j is a positive integer less than N, express and The audio-to-audio similarity, T represents the transposition operation, d represents the dimension of the feature, represents the audio features of the qth segment of the target video, q is a positive integer less than N, and exp represents an exponential function with the natural constant e as the base; The determining, based on the audio-to-audio similarity, a same-modal temporal audio feature corresponding to each of the audio features comprises: Based on the audio-to-audio similarity between the audio feature of each video clip and the audio features of the N video clips, using a homomodal temporal audio feature calculation formula, determine a homomodal temporal audio feature corresponding to each of the audio features; The same-modal temporal audio feature calculation formula is as follows: in, Representation and The corresponding same-modal temporal audio features, Represents the audio features of the sth segment of the target video, where s is a positive integer less than N. express and The audio-to-audio similarity.

5. The earthquake survivor identification method based on audio and video fusion according to claim 3 is characterized in that: The determining of the audio-visual similarity between the audio feature of each video clip and the visual features of the N video clips includes: Determining the audio-visual similarity between each of the audio features and the visual features of the N video clips based on the audio features of the N video clips and an audio-visual similarity calculation formula; The audio-visual similarity calculation formula is as follows: in, represents the audio features of the kth segment of the target video, where k is a positive integer less than N. represents the visual features of the jth segment of the target video, where j is a positive integer less than N. express and The audio visual similarity between them, T represents the transposition operation, d represents the dimension of the feature, represents the visual features of the qth segment of the target video, q is a positive integer less than N, and exp represents an exponential function with the natural constant e as the base; The determining, based on the audio-visual similarity, a cross-modal temporal visual feature corresponding to each of the audio features comprises: Based on the audio-to-audio similarity between the audio feature of each video clip and the visual features of the N video clips, a cross-modal temporal visual feature calculation formula is used to determine the cross-modal temporal visual feature corresponding to each of the audio features; The cross-modal temporal visual feature calculation formula is as follows: in, Representation and The corresponding cross-modal temporal visual features, represents the visual features of the sth segment of the target video, where s is a positive integer less than N. express and Audio-visual similarity.

6. An earthquake survivor identification device based on audio and video fusion, characterized in that: include: An input module, used to determine a target video, wherein the target video includes N video clips, where N is a positive integer; A feature extraction module, used to extract audio features and visual features corresponding to each of the N video clips; A cross-modal hybrid attention module, configured to determine temporal audio features and temporal visual features corresponding to each video clip by fusing homomodal features and cross-modal features of the audio features and visual features corresponding to each video clip; A classification module, used for determining an earthquake survivor recognition result of the target video based on the temporal audio features and the temporal visual features; The extracting audio features and visual features corresponding to each of the N video clips includes: Determining audio information and visual information corresponding to each of the N video clips; Extracting backbone features of the audio information and visual information, wherein the backbone features include audio backbone features, visual two-dimensional backbone features, and visual three-dimensional backbone features; The backbone features are mapped to audio features and visual features corresponding to each of the video clips.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the earthquake survivor identification method based on audio and video fusion as described in any one of claims 1 to 5 is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the earthquake survivor identification method based on audio and video fusion as described in any one of claims 1 to 5 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the earthquake survivor identification method based on audio and video fusion as described in any one of claims 1 to 5 is implemented.

Images