Video action classification method based on sound and related equipment

Abstract

The present application provides a sound-based video action classification method and related equipment, the method comprising: obtaining audio data of a video to be processed; constructing sound texture features of the audio data; inputting the sound texture features into a pre-trained first action classification network, outputting a first action classification result; obtaining a video frame of the video to be processed; inputting the video frame into a pre-trained second action classification network, outputting a second action classification result; and fusing the first action classification result and the second action classification result using a weighted average method to obtain an action classification result corresponding to the video to be processed. The method of the present application can improve the accuracy of action classification tasks by only adding a small number of parameters.

Description

Technical Field

[0001] The present application relates to the field of video analysis technology, and in particular to a sound-based video action classification method and related equipment. Background Art

[0002] At present, the amount of video data generated in the network every minute is huge, and video classification technology that can effectively identify human behavior and complex events is needed for application in scenarios such as autonomous driving, smart home, game interaction, video review, security, sports training, etc.

[0003] In the field of video action classification, existing network frameworks often only use video frames as input. When the objects interacting in the action are too small in proportion in the video and are not displayed in a prominent position, using only video frames cannot effectively distinguish the action category. Summary of the Invention

[0004] In view of this, the purpose of this application is to propose a sound-based video action classification method and related equipment to solve the above problems.

[0005] Based on the above objectives, the first aspect of the present application provides a method for classifying video actions based on sound, comprising:

[0006] Get the audio data of the video to be processed;

[0007] Constructing sound texture features of the audio data;

[0008] Inputting the sound texture feature into a pre-trained first action classification network and outputting a first action classification result;

[0009] Obtaining a video frame of the video to be processed;

[0010] Inputting the video frame into a pre-trained second action classification network and outputting a second action classification result;

[0011] The first action classification result and the second action classification result are fused by weighted averaging to obtain the action classification result corresponding to the video to be processed.

[0012] Furthermore, the sound texture feature is constructed in the following way:

[0013] Performing a fast Fourier transform on the audio data to obtain sound waveform data;

[0014] Filtering the sound waveform data using a first bandpass filter group to obtain first sub-band waveform data;

[0015] Obtain second sub-band waveform data by Hilbert transforming the first sub-band waveform data;

[0016] compressing the second sub-band waveform data to obtain third sub-band waveform data;

[0017] resampling the third sub-band waveform data at a target frequency to obtain fourth sub-band waveform data;

[0018] Acquire first statistical data and first correlation data of the fourth sub-band waveform;

[0019] Filtering the fourth sub-band waveform data using a second bandpass filter group to obtain fifth sub-band waveform data;

[0020] Acquire second statistical data and second correlation data of the fifth sub-band waveform;

[0021] The first statistical data, the first correlation data, the second statistical data and the second correlation data are fused to obtain the sound texture feature.

[0022] Furthermore, the first statistical data and the second statistical data each include: a mean value, a standard deviation, and a mean square response;

[0023] The first correlation data and the second correlation data are both Pearson correlation coefficients.

[0024] Furthermore, the center frequency range of the first band-pass filter group is 20 Hz to 10,000 Hz, and the center frequency range of the second band-pass filter group is 0.5 Hz to 200 Hz.

[0025] Furthermore, the action classification result is calculated by the following formula:

[0026] p=w1*p1+w2*p2

[0027] Among them, p represents the action classification result, p1 represents the first action type result, w1 represents the weight of the first action type result, p2 represents the second action type result, w2 represents the weight of the second action type result, and w1+w2=1.

[0028] Furthermore, the first action classification network includes: 5 hidden layers and 1 output layer, wherein the output dimensions of the 5 hidden layers are 128, 128, 64, 64 and 32 respectively, and the dimension of the output layer is N, where N represents the number of action recognition types.

[0029] Furthermore, the second action classification network is a P3D network.

[0030] Based on the same inventive concept, the second aspect of the present application provides a sound-based video action classification device, comprising:

[0031] A first acquisition module is configured to acquire audio data of a video to be processed;

[0032] A construction module, configured to construct a sound texture feature of the audio data;

[0033] a first classification module, configured to input the sound texture feature into a pre-trained first action classification network and output a first action classification result;

[0034] A second acquisition module is configured to acquire a video frame of the video to be processed;

[0035] a second classification module, configured to input the video frame into a pre-trained second action classification network and output a second action classification result;

[0036] The result fusion module is configured to fuse the first action classification result and the second action classification result by weighted averaging to obtain the action classification result corresponding to the video to be processed.

[0037] Based on the same inventive concept, the third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein the processor implements the method described in the first aspect when executing the program.

[0038] Based on the same inventive concept, the fourth aspect of this application provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute the method described in the first aspect.

[0039] As can be seen from the above, the sound-based video action classification method and related equipment provided by this application take into account the high correlation between audio data and images in the video, and therefore use audio data to assist in action classification. First, based on the audio data in the video, the corresponding first action classification result is obtained; then, based on the video frame of the video, the corresponding second action classification result is obtained, and finally, the first action classification result and the second action classification result are fused to obtain the final action classification result of the video. Only a small amount of parameters need to be added to improve the accuracy of the action classification task and avoid the impact of incomplete image information on the action classification results. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in this application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are merely embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0041] Figure 1 This is a flow chart of the sound-based video action classification method according to an embodiment of the present application;

[0042] Figure 2 This is a flow chart of the method for constructing sound texture features according to an embodiment of the present application;

[0043] Figure 3 This is a schematic diagram of the first action classification network structure of an embodiment of the present application;

[0044] Figure 4 This is a bar chart of the classification accuracy of seven action categories under three different networks in an embodiment of the present application;

[0045] Figure 5 This is a schematic diagram of the structure of a sound-based video action classification device according to an embodiment of the present application;

[0046] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the objectives, technical solutions and advantages of this application more clear, this application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0048] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should have the usual meanings understood by people with ordinary skills in the field to which this application belongs. The "first", "second" and similar words used in the embodiments of the present application do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0049] As described in the background technology section, the video action classification scheme in the related art is still difficult to meet the needs. Generally, deep neural networks are trained using video frames in the video, and then the trained deep neural networks are used to predict the action categories in the video. In the process of implementing this application, the applicant found that the video action classification scheme in the related art has at least the following problems: Although deep neural networks can automatically learn features from raw data and have demonstrated powerful performance in various fields, such as target detection, target recognition, image segmentation, etc., in the field of video action classification, due to the high complexity of video data, when the proportion of the object to be identified in the video is too small or not displayed in a prominent position, it is impossible to effectively classify the action of the target object.

[0050] Video data contains a huge amount of data and often includes data of multiple modalities, such as images, sounds, locations, time, etc. Therefore, it is possible to consider introducing other information in the video file to assist in classification, such as sound. Sound can convey important information about the environment, and sound can supplement visual information to a certain extent. For example, the images of the characters and the surrounding environment in the video are blurred, making it impossible to accurately judge the category of the characters' actions in the video. However, the video also records the voice information of the characters in the video, such as "walking" or "good steal", etc. The voice data can assist in judging whether the characters' actions in the video are layups or dribbling.

[0051] In view of this, an embodiment of the present application provides a sound-based video action classification method, which, based on the correlation between images and sounds in the video, utilizes the features provided by the video frames and audio to obtain corresponding action classification results respectively. By fusing the action classification results obtained based on the video frames and the action classification results based on the sound, a more accurate action classification result is finally obtained.

[0052] The technical solution of the present application is described in detail below through specific embodiments.

[0053] refer to Figure 1 An embodiment of the present application provides a method for classifying video actions based on sound, comprising the following steps:

[0054] Step S101: Acquire audio data of the video to be processed.

[0055] In this step, the video file includes a video stream and an audio stream, and the audio stream is separated from the video to obtain audio data. Schematically, the audio data of the video file can be obtained by changing the suffix of the video file to MP3 or WMA, or by using an audio extraction application to separate the audio data from the video file.

[0056] Step S102: constructing sound texture features of the audio data.

[0057] In this step, the usual practice for applying sound is to use the original waveform data of the sound as the input of the neural network or simply perform a two-dimensional conversion. This processing method does not take into account the characteristics and correlation of different frequencies in the sounds emitted by objects participating in action interactions in different action scenarios, that is, the sound texture characteristics.

[0058] Since the sound texture feature is a combination of multiple statistical data within a period of time, it is necessary to set a time window to ensure that more sound features are acquired and included in the calculation. The sound sampling time is set to more than 3.75 seconds.

[0059] Step S103: input the sound texture feature into a pre-trained first action classification network, and output a first action classification result.

[0060] In this step, the first action classification network, namely the sound neural network, can collect video files uploaded by users on video websites, and crop the collected video files to obtain clips containing the determined classified actions, and divide them into 80% training sets and 20% test sets. The first action classification network is trained to output the classification probability of each action, and the action with the highest probability value is selected as the classification result.

[0061] Step S104: obtaining a video frame of the video to be processed.

[0062] In this step, the selected video can be processed by the CV tool to extract 80 frames of images with a resolution of 224*224 from the video. If the entire video is less than 80 frames, the entire video will be looped to ensure that 80 frames of the selected video can be obtained.

[0063] Step S105: input the video frame into a pre-trained second action classification network, and output a second action classification result.

[0064] In this step, the second action classification network is a video neural network, specifically, a P3D network. The video frame is input into the P3D network, which can output the action classification result of the corresponding video frame.

[0065] The I3D network is a deep neural network that has proven effective in the field of video action. It can build deeper neural network structures with fewer training parameters. It is constructed by sequentially stacking multi-branch basic modules with similar structures. Each sub-branch of the basic module focuses on extracting different input data. The resulting trained I3D network model can achieve high classification accuracy.

[0066] Step S106: The first action classification result and the second action classification result are fused by weighted averaging to obtain an action classification result corresponding to the video to be processed.

[0067] In this step, since the structures of the two action classification networks mentioned above are different and the input data are also different, the features of the two action classification networks cannot be fused during the calculation process. Therefore, the classification results output by the two action classification networks are fused. The fusion method is weighted average, and corresponding weights are configured for the classification results output by each network. For example, the weight of the first action classification result is 0.5, and the weight of the second action classification result is 0.5. The weight can be set according to the classification accuracy of the network, or according to the actual situation, and no specific limitation is made here.

[0068] As can be seen, the sound-based video action classification method provided in this embodiment uses audio data to assist in action classification, considering the high correlation between audio data and images in the video. First, based on the audio data in the video, the corresponding first action classification result is obtained; then, based on the video frame of the video, the corresponding second action classification result is obtained. Finally, the first action classification result and the second action classification result are fused to obtain the final action classification result of the video. Only a small number of parameters need to be added to improve the accuracy of the action classification task and avoid the impact of incomplete image information on the action classification results.

[0069] In some embodiments, combined Figure 2 In the above-mentioned embodiment, step S102 may further include the following steps:

[0070] Step S1021: Perform fast Fourier transform on the audio data to obtain sound waveform data.

[0071] Step S1022: Filter the sound waveform data using a first bandpass filter group to obtain first sub-band waveform data.

[0072] In this step, the first bandpass filter bank includes 20 bandpass filters of different frequencies, with center frequencies ranging from 20 Hz to 10,000 Hz, distributed evenly on a logarithmic scale. This first bandpass filter bank simulates the sound filtering function (frequency selectivity) of the human cochlea. After passing through the first bandpass filter bank, 20 first subband waveform data are obtained.

[0073] Step S1023: Obtain second sub-band waveform data by performing Hilbert transform on the first sub-band waveform data.

[0074] In this step, the first sub-band waveform data is subjected to Hilbert transform to obtain the Hilbert element, ie, the second sub-band waveform data.

[0075] Step S1024: compress the second sub-band waveform data to obtain third sub-band waveform data.

[0076] In this step, the 0.3 power of the second sub-band waveform data is the third sub-band waveform data. By compressing the second sub-band waveform data, the sound compression function of the human cochlea can be simulated.

[0077] Step S1025 : resample the third sub-band waveform data at a target frequency to obtain fourth sub-band waveform data.

[0078] In this step, since the first filter group has 20 filters, the number of fourth sub-band waveform data is also 20, corresponding to 20 different sampling frequencies. Sampling the fourth sub-band waveform data at 400 Hz can also simulate the sound compression function of the human cochlea.

[0079] Step S1026: Acquire first statistical data and first correlation data of the fourth sub-band waveform.

[0080] In this step, the first statistical data includes: mean, standard deviation, and mean square response. The first correlation data is the Pearson correlation coefficient, specifically, for example: the Pearson correlation coefficient between the first waveform and the 20th waveform, the Pearson correlation coefficient between the 10th waveform and the 11th waveform, etc. The mean, standard deviation, mean square response of each sub-band waveform, and the Pearson correlation coefficient between sub-band waveforms are all part of the sound texture feature.

[0081] Step S1027 : Filter the fourth sub-band waveform data using a second band-pass filter group to obtain fifth sub-band waveform data.

[0082] In this step, the second filter group has 10 filters, and the center frequency range of the second bandpass filter group is 0.5 Hz to 200 Hz, also equally spaced on a logarithmic scale. Each of the 20 fourth sub-band waveform data items passes through the second filter group, resulting in a set of 10 sub-sub-band waveform data items, namely, the fifth sub-band waveform data items, for a total of 20 sets.

[0083] Step S1028: Acquire second statistical data and second correlation data of the fifth sub-band waveform.

[0084] In this step, the second statistical data includes the mean, standard deviation, and mean square response. The second correlation data is the Pearson correlation coefficient, specifically, for example, the Pearson correlation coefficient between the first and tenth waveforms, the Pearson correlation coefficient between the fifth and sixth waveforms, etc. The mean, standard deviation, mean square response of each sub-subband waveform, and the Pearson correlation coefficients between sub-subband waveforms are all part of the sound texture feature.

[0085] Step S1029: Fusing the first statistical data, the first correlation data, the second statistical data, and the second correlation data to obtain the sound texture feature.

[0086] In this step, the statistical data and correlation data of the fourth sub-band waveform data and the fifth sub-band waveform data are fused together to obtain a 1*320 matrix, namely, the sound texture feature.

[0087] In this embodiment, by simulating the process of sound processing by the human brain, the waveform within a fixed time is passed through different filter groups to simulate the filtering function of the cochlea on sound, so that the obtained sound texture features are more realistic, thereby ensuring the accuracy of action classification through audio data.

[0088] In some embodiments, the action classification result is calculated by the following formula:

[0089] p=w1*p1+w2*p2

[0090] Among them, p represents the action classification result, p1 represents the first action type result, w1 represents the weight of the first action type result, p2 represents the second action type result, w2 represents the weight of the second action type result, and w1+w2=1.

[0091] Specifically, p1 = AN(SoundTexture), where AN() represents the operation of the sound neural network, SoundTexture represents the input of the sound neural network, i.e., the sound texture features, and its dimension is [1, N1], where N1 represents the number of statistical data and correlation data. p1 = I3D(Frames), where I3D() represents the operation of the I3D network, Frames represents the input of the I3D network, i.e., the video frames, and its dimension is [N2, T, C, H, W], where N2 represents the number of videos in each group, T represents the number of video frames in each video, C represents the number of channels in each video frame, H represents the height of each video frame, and W represents the width of each video frame.

[0092] Schematically, different action categories correspond to different scores. For example, washing dishes is 1, brushing shoes is 2, the output of the first action classification network is 1, the output of the second action classification network is also 1, the weights of the first action category result and the second action category result are both 0.5, and the fused action classification result is also 1.

[0093] The weights of the first action type results and the second action type results can be set according to actual conditions, for example, matching weights according to the accuracy of the classification results of each network, which is not specifically limited here.

[0094] When the calculation result is a floating point number and the score corresponding to the action category is an integer, you can choose to round off the result to get the corresponding action classification result.

[0095] In some embodiments, as Figure 3 As shown, the first action classification network includes five hidden layers and one output layer. The output dimensions of the five hidden layers are 128, 128, 64, 64, and 32, respectively. The dimension of the output layer is N, where N represents the number of action recognition types. The dimensionality of each hidden layer can be varied as follows: [batchsize, 128] -> [batchsize, 128] -> [batchsize, 64] -> [batchsize, 64] -> [batchsize, 32] -> [batchsize, 7]. The calculation results of each hidden layer are processed by the ReLU activation function and batch normalization. To reduce the number of parameters that need to be trained in the network structure and avoid overfitting, the random dropout rate of hidden layer nodes in the network is set to 50%. It should be noted that the random dropout rate can be set according to actual conditions and is not specifically limited here.

[0096] Hereinafter, the effectiveness of the method of the present application will be illustrated through experimental data in combination with all the above embodiments.

[0097] The training and test data for the experiment were from the open-source video dataset Kinetics. Seven categories were randomly selected from the dataset: execising, parasailing, washing dishes, pull-ups, cleaning shoes, folding paper, and pumping fists. Videos without background sound were removed to create the training and test sets.

[0098] The action classification of the selected multiple videos is performed using the trained video neural network (I3D), sound neural network (AN) and two-way neural network (TwoStream), where the two-way neural network is a combination of the video neural network and the sound neural network, i.e. the method of this application. The action classification results are shown in Figure 2. Figure 4 As shown in the figure, it is easy to see that the accuracy of the classification results of the dual-path neural network in the classification of 7 categories is higher than the accuracy of the action classification results of the video neural network and the sound neural network, which can improve the accuracy of the video action classification task.

[0099] Table 1

[0100] network enter Accuracy Video Neural Networks Video Frame 40.1％ Sound Neural Network Sound texture features 37.5％ Two-way neural network Video frame + sound texture features 47.7％

[0101] As shown in Table 1, the dual neural network (i.e., I3D+AN) achieves a 7.6% improvement in action classification accuracy compared to two independent neural networks (video neural network and sound neural network). It should be noted that the data size for training the video neural network is 12.3MB, while the data size for training the sound neural network is 70KB. This shows that only a small increase in data size is required to improve the accuracy of the action classification task.

[0102] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied in a distributed scenario and performed by multiple devices working together. In such a distributed scenario, one of the multiple devices may only perform one or more steps of the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the method.

[0103] It should be noted that the above description is limited to some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in an order different from that described in the above embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0104] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a sound-based video action classification device.

[0105] refer to Figure 5 The sound-based video action classification device comprises:

[0106] The first acquisition module 501 is configured to acquire audio data of the video to be processed.

[0107] The construction module 502 is configured to construct the sound texture features of the audio data.

[0108] The first classification module 503 is configured to input the sound texture feature into a pre-trained first action classification network and output a first action classification result.

[0109] The second acquisition module 504 is configured to acquire the video frame of the video to be processed.

[0110] The second classification module 505 is configured to input the video frame into a pre-trained second action classification network and output a second action classification result.

[0111] The result fusion module 506 is configured to fuse the first action classification result and the second action classification result by weighted averaging to obtain the action classification result corresponding to the video to be processed.

[0112] As an optional embodiment, the construction module 502 is specifically configured to perform fast Fourier transform on the audio data to obtain sound waveform data; filter the sound waveform data using a first bandpass filter group to obtain first sub-band waveform data; perform Hilbert transform on the first sub-band waveform data to obtain second sub-band waveform data; compress the second sub-band waveform data to obtain third sub-band waveform data; resample the third sub-band waveform data at the target frequency to obtain fourth sub-band waveform data; obtain first statistical data and first correlation data of the fourth sub-band waveform; filter the fourth sub-band waveform data using a second bandpass filter group to obtain fifth sub-band waveform data; obtain second statistical data and second correlation data of the fifth sub-band waveform; and fuse the first statistical data, the first correlation data, the second statistical data and the second correlation data to obtain the sound texture feature.

[0113] As an optional embodiment, the first statistical data and the second statistical data both include: a mean value, a standard deviation, and a mean square response; and the first correlation data and the second correlation data are both Pearson correlation coefficients.

[0114] As an optional embodiment, the center frequency range of the first band-pass filter group is 20 Hz to 10,000 Hz, and the center frequency range of the second band-pass filter group is 0.5 Hz to 200 Hz.

[0115] As an optional embodiment, the action classification result is calculated by the following formula:

[0116] p=w1*p1+w2*p2

[0117] Among them, p represents the action classification result, p1 represents the first action type result, w1 represents the weight of the first action type result, p2 represents the second action type result, w2 represents the weight of the second action type result, and w1+w2=1.

[0118] As an optional embodiment, the first action classification network includes: 5 hidden layers and 1 output layer, wherein the output dimensions of the 5 hidden layers are 128, 128, 64, 64 and 32 respectively, and the dimension of the output layer is N, where N represents the number of action recognition types.

[0119] As an optional embodiment, the second action classification network is a P3D network.

[0120] For the convenience of description, the above devices are described as being divided into various modules according to their functions. Of course, when implementing this application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0121] The apparatus of the above embodiment is used to implement the corresponding sound-based video action classification method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described in detail here.

[0122] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor executes the program, the sound-based video action classification method described in any of the above embodiments is implemented.

[0123] Figure 6 10 is a schematic diagram showing a more specific hardware structure of an electronic device provided in this embodiment. The device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are communicatively connected to each other within the device via the bus 1050.

[0124] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0125] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, etc. The memory 1020 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0126] The input / output interface 1030 is used to connect input / output modules to implement information input and output. The input / output modules can be configured as components within the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and output devices may include a display, speaker, vibrator, indicator light, etc.

[0127] The communication interface 1040 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WiFi, Bluetooth, etc.).

[0128] The bus 1050 comprises a path for transmitting information between the various components of the device (eg, the processor 1010 , the memory 1020 , the input / output interface 1030 , and the communication interface 1040 ).

[0129] It should be noted that although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040, and the bus 1050, in a specific implementation, the device may also include other components necessary for normal operation. In addition, it will be understood by those skilled in the art that the above device may only include the components necessary to implement the embodiments of this specification, and does not necessarily include all the components shown in the figure.

[0130] The electronic device of the above embodiment is used to implement the corresponding sound-based video action classification method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0131] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the sound-based video action classification method described in any of the above embodiments.

[0132] The computer-readable media of this embodiment include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0133] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the sound-based video action classification method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0134] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. Within the scope of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0135] In addition, for simplicity of description and discussion, and in order not to make the embodiment of the application difficult to understand, the known power supply / ground connection with integrated circuit (IC) chip and other components may or may not be shown in the accompanying drawings provided. In addition, the device can be shown in the form of a block diagram to avoid making the embodiment of the application difficult to understand, and this also takes into account the following fact, that is, the details of the embodiment of these block diagram devices are highly dependent on the platform to be implemented in the embodiment of the application (that is, these details should be fully within the scope of understanding of those skilled in the art). When specific details (for example, circuit) are set forth to describe exemplary embodiments of the application, it will be apparent to those skilled in the art that the embodiment of the application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0136] Although the present invention has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may utilize the embodiments discussed.

[0137] The embodiments of the present application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of this application.

Claims

1. A method for video action classification based on sound, characterized in that: include: Get the audio data of the video to be processed; Constructing sound texture features of the audio data; Inputting the sound texture feature into a pre-trained first action classification network and outputting a first action classification result; Obtaining a video frame of the video to be processed; Inputting the video frame into a pre-trained second action classification network and outputting a second action classification result, where the second action classification network is a 13D network; The first action classification result and the second action classification result are fused by weighted averaging to obtain an action classification result corresponding to the video to be processed; The sound texture feature is constructed in the following way: Performing a fast Fourier transform on the audio data to obtain sound waveform data; Filtering the sound waveform data using a first bandpass filter group to obtain first sub-band waveform data, wherein the center frequency range of the first bandpass filter group is 20 Hz to 10000 Hz; Obtain second sub-band waveform data by Hilbert transforming the first sub-band waveform data; compressing the second sub-band waveform data to obtain third sub-band waveform data; resampling the third sub-band waveform data at a target frequency to obtain fourth sub-band waveform data; Acquire first statistical data and first correlation data of the fourth sub-band waveform; Filtering the fourth sub-band waveform data using a second band-pass filter group to obtain fifth sub-band waveform data, wherein the center frequency range of the second band-pass filter group is 0.5 Hz to 200 Hz; Acquire second statistical data and second correlation data of the fifth sub-band waveform; fusing the first statistical data, the first correlation data, the second statistical data, and the second correlation data to obtain the sound texture feature, wherein the first statistical data and the second statistical data both include: an average value, a standard deviation, and a mean square response, and the first correlation data and the second correlation data are both Pearson correlation coefficients; The first action classification network includes: 5 hidden layers and 1 output layer, wherein the output dimensions of the 5 hidden layers are 128, 128, 64, 64 and 32 respectively, the dimension of the output layer is N, N represents the number of action recognition types, and the random drop rate of the hidden layer nodes is set to 50%.

2. The classification method according to claim 1, characterized in that The action classification result is calculated by the following formula: p=w1*p1+w2*p2 Among them, p represents the action classification result, p1 represents the first action type result, w1 represents the weight of the first action type result, p2 represents the second action type result, w2 represents the weight of the second action type result, and w1+w2=1.

3. A video action classification device based on sound, characterized in that: include: A first acquisition module is configured to acquire audio data of a video to be processed; A construction module, configured to construct a sound texture feature of the audio data; a first classification module, configured to input the sound texture feature into a pre-trained first action classification network and output a first action classification result; A second acquisition module is configured to acquire a video frame of the video to be processed; a second classification module configured to input the video frame into a pre-trained second action classification network and output a second action classification result, wherein the second action classification network is a 13D network; A result fusion module is configured to fuse the first action classification result and the second action classification result by weighted averaging to obtain an action classification result corresponding to the video to be processed; The sound texture feature is constructed in the following way: Performing a fast Fourier transform on the audio data to obtain sound waveform data; Filtering the sound waveform data using a first bandpass filter group to obtain first sub-band waveform data, wherein the center frequency range of the first bandpass filter group is 20 Hz to 10000 Hz; Obtain second sub-band waveform data by Hilbert transforming the first sub-band waveform data; compressing the second sub-band waveform data to obtain third sub-band waveform data; resampling the third sub-band waveform data at a target frequency to obtain fourth sub-band waveform data; Acquire first statistical data and first correlation data of the fourth sub-band waveform; Filtering the fourth sub-band waveform data using a second band-pass filter group to obtain fifth sub-band waveform data, wherein the center frequency range of the second band-pass filter group is 0.5 Hz to 200 Hz; Acquire second statistical data and second correlation data of the fifth sub-band waveform; fusing the first statistical data, the first correlation data, the second statistical data, and the second correlation data to obtain the sound texture feature, wherein the first statistical data and the second statistical data both include: an average value, a standard deviation, and a mean square response, and the first correlation data and the second correlation data are both Pearson correlation coefficients; The first action classification network includes: 5 hidden layers and 1 output layer, wherein the output dimensions of the 5 hidden layers are 128, 128, 64, 64 and 32 respectively, the dimension of the output layer is N, N represents the number of action recognition types, and the random drop rate of the hidden layer nodes is set to 50%.

4. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 2 is implemented.

5. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 2.

Images