A fall action detection method and system based on multi-diffusion model

Abstract

The present application provides a fall action detection method and system based on a multinomial diffusion model, which belongs to the field of action detection technology, including: collecting sample fall action data and annotating the action to obtain a sample fall action data set and action annotation data; constructing a multinomial diffusion model, the multinomial diffusion model includes forward diffusion and backward denoising, the forward diffusion includes: using equal probability multinomial distribution noise to add noise to the action annotation data to obtain the intermediate distribution in the forward diffusion process; the backward denoising includes: constructing an audio-visual feature extraction network, substituting the sample fall action data set into the audio-visual feature extraction network to perform feature fusion of audio data and image data to obtain audio-visual features, constructing a denoising model, encoding the intermediate distribution, audio-visual features and diffusion steps of the forward diffusion into the denoising model to obtain a prediction result; decoding to obtain the final prediction result. The accuracy of fall action detection is improved and the false alarm rate of fall action detection is reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of motion detection, and in particular to a fall motion detection method and system based on a multi-layer diffusion model. Background Art

[0002] Current fall detection methods are mainly divided into two categories: wearable and non-wearable. Wearable devices are small and highly portable, but they are more likely to mistakenly identify certain activities as falls, leading to unnecessary concerns and emergency responses; non-wearable methods include those based on visual images, millimeter-wave radars, and wifi radio frequency signals; common visual image-based detection methods mainly use target detection and tracking, human posture key point detection, action recognition and other methods, and use a detection process in which multiple models are connected in series. The detection results are easily affected by factors such as body occlusion and personnel overlap; methods based on millimeter-wave radars and wifi radio frequency signals have a single data source and lack spatial semantic information. They are also prone to false alarms when personnel activities are frequent and there are many interference signals.

[0003] A 2D human posture estimation method based on a diffusion model is disclosed (patent application number: CN202310807926.0), which collects human posture images and annotation information, and uses the forward diffusion formula to add noise to the annotated joint coordinates to destroy the coordinate information, generating intermediate noise samples; then constructs a 2D human posture estimation diffusion model; then the model predicts the heat map of the target joint based on the input image and the generated noise sample, and then uses the mean square error between the prediction result and the annotation as the loss function to optimize the model parameters; finally, the joint heat map predicted by the model is sent to the reverse diffusion formula to complete the iterative denoising of Gaussian noise and obtain the accurate prediction after denoising. A sparse human motion capture system and model training method based on a diffusion probability model are disclosed (patent application number: CN202410340330.9), which introduces scene information by considering the correlation between the scene and human motion, thereby improving the quality of sparse human motion capture. A method and device for speech-driven gesture generation based on a diffusion model are disclosed (patent application number: CN202311007818.1), which includes: obtaining a human body gesture data set with speech annotations, preprocessing the human body gesture data set to obtain training data of gesture sequence fragments with speech information annotations, and adding noise to the gesture sequence in the training data to obtain noisy gesture sequence samples for training the diffusion model; constructing and training a diffusion model for speech-driven gesture generation, wherein the diffusion model regards the gesture generation task as a denoising process for noisy gesture sequences; using the trained diffusion model, according to a given speech input of any length, iterative denoising is performed starting from randomly sampled Gaussian noise to generate a gesture sequence.

[0004] The noise used in the diffusion process of the above-mentioned public scheme is Gaussian noise. Gaussian noise is commonly used in diffusion models and is only applicable to conventional diffusion models. When the modeling method of the diffusion model is adjusted according to the data type, Gaussian noise cannot achieve the optimal noise addition or denoising effect; and the above-mentioned scheme is the generation of actions or postures, and does not involve the recognition of actions. It cannot be used to detect whether there is a fall action, cannot detect the fall condition in a timely and accurate manner, and cannot reduce the false alarm rate of fall detection. Summary of the invention

[0005] The present application provides a fall action detection method and system based on a multi-diffusion model, aiming to solve the problem that the existing fall detection method cannot detect the fall condition in a timely and accurate manner, and cannot reduce the false alarm rate of fall detection.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A fall action detection method based on a multi-diffusion model, comprising:

[0008] Collect sample falling action data and perform action annotation to obtain a sample falling action data set and action annotation data;

[0009] Construct a multinomial diffusion model, which includes forward diffusion and backward denoising. Forward diffusion includes: adding noise to the action annotation data using equal probability multinomial distribution noise to obtain the intermediate distribution in the forward diffusion process;

[0010] Backward denoising includes: constructing an audio-visual feature extraction network, substituting the sample falling action data set into the audio-visual feature extraction network to fuse the features of audio data and image data to obtain audio-visual features, constructing a denoising model, encoding the intermediate distribution, audio-visual features, and the diffusion steps of forward diffusion into the denoising model to obtain the prediction result;

[0011] The prediction results are decoded to obtain the final prediction results of fall action detection.

[0012] As a preferred embodiment, sample falling action data is collected and action annotation is performed to obtain a sample falling action data set and action annotation data, specifically:

[0013] The sample falling action data are indoor and outdoor falling-related video data. The audio data and image data in the falling-related video data are extracted as a pair of data, the action category and the start and end time of the action are marked, and similar falling actions are added as negative samples for learning. The sound made by people when falling is added to the image data in the falling-related video data to increase the sample richness. After processing, the sample falling action dataset and action annotation data are obtained.

[0014] As a preferred method, the action annotation data is denoised using equal probability multinomial distribution noise to obtain the intermediate distribution in the forward diffusion process, specifically:

[0015] Continuously add multinomial distribution noise with equal probability to the true value data of the action annotation data until the action annotation data conforms to the multinomial distribution. In the forward diffusion process, the intermediate distribution in the diffusion process is obtained, where the action annotation data is spliced ​​by the matrix of action categories and the matrix of action start and end times.

[0016] Preferably, the sample falling action data set is substituted into the audio-visual feature extraction network to perform feature fusion of audio data and image data to obtain audio-visual features, specifically:

[0017] Perform 3D convolution processing based on a sample falling action dataset, where the sample falling action dataset includes a low frame rate image sequence, a high frame rate image sequence, and a high sampling rate audio spectrogram;

[0018] In the stem stage, the features of the low frame rate image sequence and the high frame rate image sequence are processed with a 3D convolution with a step size of 2 and a 3D maximum pooling with a step size of 2. The audio features of the high sampling rate audio spectrogram are only processed with a 3D convolution.

[0019] Each stage uses residual connection, first performs a 3D convolution with a stride of 2, and then stacks two 3D convolutions with a stride of 1;

[0020] The features of the high frame rate image sequence are fused with the features of the low frame rate image sequence through one 3D convolution, and the audio features of the high sampling rate audio spectrogram are fused with the features of the low frame rate image sequence through two 3D convolutions;

[0021] The three features are 3D mean pooled and then added and fused to obtain the audio-visual features.

[0022] Preferably, the denoising model is composed of three row-column feature encoding networks, and the row-column feature encoding networks are specifically:

[0023] First, the transformer network is used to query the column dimension of the input to restore the correlation information between categories, followed by two layers of MLP layers. Then, after time series convolution processing, the row dimension is queried to restore the time series features, followed by two layers of MLP layers to form a row-column feature encoding network.

[0024] Preferably, the intermediate distribution, audio-visual features and the diffusion step of the forward diffusion are encoded and input into the denoising model to obtain the prediction result, specifically:

[0025] The intermediate distribution, audio-visual features, and diffusion step encodings of the forward diffusion are respectively input into the corresponding row-column feature encoding network to obtain three row-column feature encodings. The three row-column feature encodings are stacked and linearly transformed to output the prediction results.

[0026] Preferably, the prediction result is decoded to obtain the final prediction result of the fall action detection, which is specifically:

[0027] The prediction results are screened according to the preset threshold to filter out the action starting points that meet the conditions;

[0028] According to the starting point of the action, the action interval where the falling action may occur is found through clustering as the candidate interval;

[0029] The falling action category with the highest number of occurrences in different candidate intervals is counted as the final action detection category, and the action detection category is suppressed by non-maximum values ​​to obtain the final prediction result.

[0030] A fall action detection system based on a multi-diffusion model, comprising:

[0031] Data collection module: used to collect sample fall action data and perform action annotation to obtain sample fall action data set and action annotation data;

[0032] Multinomial diffusion model construction module: used to construct a multinomial diffusion model, which includes forward diffusion and backward denoising. Forward diffusion includes: adding noise to the action annotation data using equal probability multinomial distribution noise to obtain the intermediate distribution in the forward diffusion process;

[0033] Backward denoising module: used to build an audio-visual feature extraction network, substitute the sample falling action data set into the audio-visual feature extraction network to fuse the features of audio data and image data, obtain audio-visual features, build a denoising model, encode the intermediate distribution, audio-visual features and the diffusion step of forward diffusion into the denoising model, and obtain the prediction result;

[0034] Prediction result decoding module: used to decode the prediction result to obtain the final prediction result of fall action detection.

[0035] An electronic device comprises a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement a fall action detection method based on a multi-number diffusion model as described above.

[0036] A computer-readable storage medium, wherein a computer program stored in the storage medium, when executed by a computer, implements a fall action detection method based on a multi-number diffusion model as described above.

[0037] The present invention has the following beneficial effects:

[0038] (1) This solution remodels the fall action detection problem as a generation problem of a diffusion model, generates prediction results under the guidance of image and audio features, and has a simple training and detection process. It can make full use of rich semantic features and temporal features, improve the accuracy of fall action detection, and reduce the false alarm rate of fall action detection.

[0039] (2) The action annotation data in this scheme is composed of a matrix of action categories and a matrix of action start and end times. Action category prediction is a C classification problem, while action start and end time prediction is a binary classification problem. The data are all discretely distributed, which is quite different from natural images. Conventional Gaussian diffusion and mixed Gaussian diffusion are not suitable. In order to learn the mapping relationship between highly uncertain noise and detection results, this scheme uses multinomial diffusion to add noise to the discrete action annotation data. During forward diffusion, multinomial distribution noise with equal probability is continuously added to the true value data until the discrete action annotation data conforms to the multinomial distribution. The use of equal probability multinomial distribution noise is based on the modeling method of fall action detection data proposed in this paper, that is, each row of data is category information. Only when each category is of equal probability, its uncertainty is the highest, which can achieve the best prediction effect and improve accuracy.

[0040] (3) In this scheme, image features and audio features are extracted simultaneously, the network coupling degree is higher, feature extraction and fusion are performed in the same network, the network structure design is more compact, and the features of different modes are learned and fused together in the same network. The fusion at different dimensional levels can fully learn the connection between different modes, which is efficient and has better feature fusion effect.

[0041] (4) The denoising model of this scheme is composed of three row-column feature encoding networks. The purpose of this design is mainly to take into account the particularity of the action annotation data form constructed by this scheme. The relationship between rows and columns is completely different. The rows represent the time sequence, and the columns represent the category information. Therefore, based on this consideration, the method of processing rows and columns separately is adopted. Finally, after stacking, the final prediction result is obtained through linear transformation, which improves the quality of the prediction result generation, so that the fall condition can be detected in a timely and accurate manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 The figure is a flow chart of a fall action detection method based on a multi-layer diffusion model in the present invention.

[0043] Figure 2 This is a flow chart of a specific implementation scheme in Example 1 of the present invention.

[0044] Figure 3Schematic diagram of the multi-diffusion process of fall detection in Example 1 of the present invention.

[0045] Figure 4 Schematic diagram of the structure of the audio-visual feature extraction network in Example 1 of the present invention.

[0046] Figure 5 Schematic diagram of the structure of the row-column feature encoding network in Example 1 of the present invention.

[0047] Figure 6 It is a structural schematic diagram of the fall action detection system in the present invention. DETAILED DESCRIPTION

[0048] Example 1

[0049] like Figure 1 As shown, a fall action detection method based on a multi-layer diffusion model includes the following steps:

[0050] S11, collecting sample falling action data and annotating the action to obtain a sample falling action data set and action annotation data;

[0051] S12, constructing a multinomial diffusion model, the multinomial diffusion model includes forward diffusion and backward denoising, the forward diffusion includes: adding noise to the action annotation data using equal probability multinomial distribution noise to obtain an intermediate distribution in the forward diffusion process;

[0052] S13, backward denoising includes: constructing an audio-visual feature extraction network, substituting the sample falling action data set into the audio-visual feature extraction network to perform feature fusion of audio data and image data to obtain audio-visual features, constructing a denoising model, encoding the intermediate distribution, audio-visual features and diffusion steps of forward diffusion into the denoising model to obtain a prediction result;

[0053] S14: Decode the prediction result to obtain the final prediction result of the fall action detection.

[0054] Recently, the diffusion model has demonstrated excellent high-quality image generation capabilities. By gradually adding noise to the image and continuously optimizing the loss between the noise generated by the model and the added noise, it can recover high-quality images from highly uncertain noisy data during inference. This embodiment proposes a fall action detection method based on a multi-layer diffusion model, which remodels the fall action detection problem as a generation problem of the diffusion model, generates prediction results under the guidance of image and audio features, and has a simple training and detection process that can make full use of rich semantic features and temporal features. The process of this embodiment is as follows: Figure 2 The specific contents are as follows.

[0055] 1. Collect and produce audio and video datasets related to fall action detection.

[0056] Collect indoor and outdoor video data related to falling (i.e., sample falling action data), extract the image and audio from the video data clips related to falling as a pair of data, annotate the action category and the start and end time of the action; add similar falling actions as negative samples for learning; and increase the sample richness of the image data of falling by adding the sound made by people when falling. After processing, we get the sample falling action data set and action annotation data.

[0057] 2. Modeling of multi-item diffusion model for fall detection.

[0058] The multi-diffusion of fall action detection mainly includes forward diffusion and backward denoising. This paper mainly solves the problem of converting fall action detection into image generation, proposes an audio-visual feature model as a denoising generation condition, and proposes a row-column feature encoding network for fall action detection.

[0059] 1. How to convert fall action detection into image generation problem.

[0060] The task of fall action detection is to determine whether a fall action has occurred within a given time series. This solution reconstructs the fall detection result and defines the detection result as , with a value range of [0,1], where S represents discrete action annotation data. The action category prediction uses a The matrix representation of , N represents the number of detection frames, C represents the number of detection action categories (in addition to the fall action, there is also a fall negative sample action); the prediction of the starting point (start time) and the end point (end time) of the action each uses one , 2 indicates whether it is the starting point of the action; at this time, S can be regarded as a matrix with a size of The diffusion model is used to generate prediction results of action categories and action start times.

[0061] The reason for converting the falling action into an image: after converting it into a data form similar to an image, it can be generated using a diffusion model, thus realizing the use of a diffusion model for falling action detection. This part is mainly to convert the falling action detection data into a form that can be processed using a diffusion model, which has significant advantages in image generation.

[0062] 2. Multiple diffusion denoising processes for fall action detection, such as Figure 3 shown.

[0063] 2.1 Forward diffusion.

[0064] The data S is composed of a matrix of action categories and a matrix of action starting points (i.e., a matrix of the time when the action starts and ends). Action category prediction is a C classification problem, while starting point prediction is a binary classification problem. The data are all discretely distributed, which is quite different from natural images. Conventional Gaussian diffusion and mixed Gaussian diffusion are not suitable. In order to learn the mapping relationship between highly uncertain noise and detection results, this paper uses multinomial diffusion to add noise to discrete data S. During forward diffusion, multinomial noise v with equal probability is continuously added to the true value data until S conforms to the multinomial distribution. At this time, the elements of each row of S are equally likely, and the uncertainty is the highest.

[0065] The noise added in the forward diffusion process of the image is usually Gaussian distributed noise. For more complex data types, a mixed Gaussian model (GMM) is used to fit the noise distribution. The equiprobable multinomial distribution used in this article is just a special processing method for the fall detection data proposed in this article. The mixed Gaussian model can theoretically fit a variety of complex data distributions by adjusting parameters. In most cases of the diffusion model, a single Gaussian distribution noise can be used. For complex scenes such as image denoising, the noise of the mixed Gaussian model is used. Gaussian noise and mixed Gaussian noise are two common ways to add noise. This article uses equiprobable multinomial distribution noise based on the modeling method of the fall action detection data proposed in this article, that is, the data in each row is category information. Only when each category is of equal probability, its uncertainty is the highest, which is an unconventional method of adding noise.

[0066] 2.2 Backward denoising.

[0067] Through the forward diffusion process, we can get the intermediate distribution st={s1, s2, ..., sT} of the forward diffusion process. , these intermediate distributions can be used as supervisory signals to train the denoising model First, extract the audio-visual features of the video , where N represents the number of video frames, C M represents the number of channels, and the generation process of each reverse step is When performing reverse diffusion, we first initialize a noise s that obeys a multinomial distribution. T , execute the denoising model , which can be achieved process, is the final prediction result.

[0068] ;

[0069] in, Represents the embedding of step t generated by S using the sine function, that is, the diffusion step encoding of the forward diffusion.

[0070] 2.2.1 Audiovisual Feature Extraction Network .

[0071] like Figure 4 As shown, in the audio-visual feature extraction network proposed in this scheme, image features and audio features are learned synchronously in the same network structure and are continuously integrated during the learning process. The network structure mainly adopts 3D convolution processing, and performs 3D convolution processing based on the sample falling action dataset, where the sample falling action dataset includes low frame rate image sequence, high frame rate image sequence and high sampling rate audio spectrogram; in the stem (initial layer) stage, the features of the low frame rate image sequence and the high frame rate image sequence are subjected to a 3D convolution processing with a step size of 2 and a 3D maximum pooling with a step size of 2, and the audio features of the high sampling rate audio spectrogram are only subjected to a 3D convolution processing once; each stage adopts residual connection, firstly performs a 3D convolution with a step size of 2, and then stacks two 3D convolutions with a step size of 1; the features of the high frame rate image sequence are fused with the features of the low frame rate image sequence through a 3D convolution, and the audio features of the high sampling rate audio spectrogram are fused with the features of the low frame rate image sequence through two 3D convolutions; the three features are subjected to 3D mean pooling and then added and fused to obtain the audiovisual features.

[0072] 2.2.2 Denoising Model .

[0073] like Figure 5 As shown, the denoising model The denoising model is composed of three row-column feature encoding networks. This feature encoding method is only for the data form mentioned in this article. Since the row and column attributes of data S are too different from those of natural images, the following row and column feature encoding network is used: first, the transformer network is used to query the column dimension of the input to restore the association information between categories, followed by two layers of MLP layers, and then after time series convolution processing, the row dimension is queried to restore the time series features, followed by two layers of MLP layers to form a row and column feature encoding network. The input of the row and column feature encoding network is , by Splicing composition, adding position encoding before Transformer encoding, the final output ; After stacking three row and column feature codes, the prediction result is output through linear transformation .

[0074] 3. Training the Audiovisual Feature Extraction Network .

[0075] Audio-visual feature extraction network The training of the proposed method takes the pre-annotated video and audio data pairs (i.e., the sample falling action dataset) as input. The input image part includes low frame rate image sequences and high frame rate image sequences. The low frame rate image sequence is used to extract spatial semantic information. At the same time, the lower frame rate can reduce the amount of calculation, and the more channels ensure that sufficient information can be extracted; the high frame rate image sequence can fully capture the temporal action information, and the low number of channels reduces the computational complexity; the audio feature input uses the two-dimensional (time and frequency) log-mel spectrum as input, and the sampling frame rate is faster than the fast channel of the image. Since the learning rate of audio data is faster than that of image data, the network is easier to converge. In order to maintain the learning consistency of the entire feature network, the audio channel is randomly discarded during the model training stage. When optimizing the multimodal feature extraction network through the annotated dataset training, a linear transformation classification head is added after the network, and the network parameters are reversely optimized by calculating the cross entropy loss between the classification results and the annotated categories.

[0076] 4. Training the Diffusion Denoising Model .

[0077] First, we use the audio-visual feature extraction network Extract audio-visual features of the corresponding video , while maintaining Freeze during the training of the denoising model; randomly initialize the denoising model , the intermediate distribution st generated by forward diffusion and the audio-visual features , diffusion step encoding Splicing composition , input denoising model, calculate output result Before forward diffusion The mean square error of the denoising model is updated inversely .

[0078] 5. Perform denoising prediction and decode the denoising model prediction results.

[0079] In the inference test phase, randomly generate multinomial distribution noise with equal probability , extract the audio-visual features of the video As the generation condition, and the step code of the iteration number t, it is input into the denoising model together. Output prediction results ={x a ,x s ,x e}.in ; Filter out the action starting points x that meet the conditions according to the defined threshold s , x e, the prediction results are filtered according to the preset threshold to filter out the action starting points that meet the conditions; based on the action starting points and through clustering, the action intervals where falling actions may occur are found as candidate intervals; the falling action categories with the most occurrences in different candidate intervals are counted as the final action detection categories, and the action detection categories are suppressed by non-maximum values ​​to obtain the final prediction results.

[0080] Note: The forward process of the diffusion model gradually adds noise to the original data. The step encoding of the iteration number t above is the number of steps added when adding noise forward. However, after parameterization, noise can be added directly at step t, such as Figure 3 As shown in , from the 0 state to the final T state, noise is added T times. This is one of the reasons why the diffusion model has improved image generation ability compared to the previous generation model. When training the denoising model, not only the final T state is used for training, but also the states between 0-T can be used for training. In order to distinguish the steps from different diffusions, the iterative steps are encoded during denoising and added to the denoising network to predict the 0 state.

[0081] Example 2

[0082] like Figure 6 As shown, a fall action detection system based on a multi-layer diffusion model includes:

[0083] Data collection module 10: used to collect sample fall action data and perform action annotation to obtain sample fall action data set and action annotation data;

[0084] The multinomial diffusion model construction module 20 is used to construct a multinomial diffusion model, which includes forward diffusion and backward denoising. The forward diffusion includes: adding noise to the action annotation data using equal probability multinomial distribution noise to obtain an intermediate distribution in the forward diffusion process;

[0085] Backward denoising module 30: used to construct an audio-visual feature extraction network, substitute the sample falling action data set into the audio-visual feature extraction network to fuse the features of the audio data and the image data, obtain the audio-visual features, build a denoising model, encode the intermediate distribution, the audio-visual features and the diffusion step of the forward diffusion into the denoising model, and obtain the prediction result;

[0086] Prediction result decoding module 40: used to decode the prediction result to obtain the final prediction result of the fall action detection.

[0087] One implementation of the above system is that in the data acquisition module 10, sample fall action data is collected and action annotation is performed to obtain a sample fall action data set and action annotation data; in the multinomial diffusion model construction module 20, a multinomial diffusion model is constructed, and the multinomial diffusion model includes forward diffusion and backward denoising. The forward diffusion includes: using equal probability multinomial distribution noise to add noise to the action annotation data to obtain the intermediate distribution in the forward diffusion process; in the backward denoising module 30, an audio-visual feature extraction network is constructed, and the sample fall action data set is substituted into the audio-visual feature extraction network to perform feature fusion of audio data and image data to obtain audio-visual features, and a denoising model is constructed. The intermediate distribution, audio-visual features, and diffusion steps of the forward diffusion are encoded and input into the denoising model to obtain a prediction result; in the prediction result decoding module 40, the prediction result is decoded to obtain the final prediction result of the fall action detection.

[0088] Example 3

[0089] Based on the above embodiments, this embodiment provides an electronic device.

[0090] Example 4

[0091] Based on the above embodiments, this embodiment provides a storage medium.

[0092] The above description is only a specific embodiment of the present invention, but the technical features of the present invention are not limited thereto. Any changes or modifications made by any technician in the field of the present invention are included in the patent scope of the present invention.

Claims

1. A fall action detection method based on a multi-layer diffusion model, characterized in that: include: Collect sample falling action data and perform action annotation to obtain a sample falling action data set and action annotation data; Construct a multinomial diffusion model, which includes forward diffusion and backward denoising. Forward diffusion includes: using equal probability multinomial distribution noise to add noise to the action annotation data to obtain the intermediate distribution in the forward diffusion process; substituting the sample fall action data set into the audio-visual feature extraction network to fuse the features of audio data and image data to obtain audio-visual features, specifically: Perform 3D convolution processing based on a sample falling action dataset, where the sample falling action dataset includes a low frame rate image sequence, a high frame rate image sequence, and a high sampling rate audio spectrogram; In the stem stage, the features of the low frame rate image sequence and the high frame rate image sequence are processed with a 3D convolution with a step size of 2 and a 3D maximum pooling with a step size of 2. The audio features of the high sampling rate audio spectrogram are only processed with a 3D convolution. Each stage uses residual connection, first performs a 3D convolution with a stride of 2, and then stacks two 3D convolutions with a stride of 1; The features of the high frame rate image sequence are fused with the features of the low frame rate image sequence through one 3D convolution, and the audio features of the high sampling rate audio spectrogram are fused with the features of the low frame rate image sequence through two 3D convolutions; The three features are 3D-mean pooled and then added together to obtain the audio-visual features; Backward denoising includes: constructing an audio-visual feature extraction network, substituting the sample falling action data set into the audio-visual feature extraction network to fuse the features of audio data and image data to obtain audio-visual features, constructing a denoising model, encoding the intermediate distribution, audio-visual features, and the diffusion steps of forward diffusion into the denoising model to obtain the prediction result; The denoising model consists of three row-column feature encoding networks, which are specifically: First, the transformer network is used to query the column dimension of the input to restore the association information between categories, followed by two layers of MLP layers. Then, after time series convolution processing, the row dimension is queried to restore the time series features, followed by two layers of MLP layers to form a row and column feature encoding network. The prediction results are decoded to obtain the final prediction results of fall action detection.

2. The method for detecting a fall action based on a multi-number diffusion model according to claim 1, characterized in that: Collect sample fall action data and perform action annotation to obtain sample fall action data set and action annotation data, specifically: The sample falling action data are indoor and outdoor falling-related video data. The audio data and image data in the falling-related video data are extracted as a pair of data, the action category and the start and end time of the action are marked, and similar falling actions are added as negative samples for learning. The sound made by people when falling is added to the image data in the falling-related video data to increase the sample richness. After processing, the sample falling action dataset and action annotation data are obtained.

3. The method for detecting a fall action based on a multi-number diffusion model according to claim 1, characterized in that: Use equal probability multinomial distribution noise to add noise to the action annotation data to obtain the intermediate distribution in the forward diffusion process, specifically: Continuously add multinomial distribution noise with equal probability to the true value data of the action annotation data until the action annotation data conforms to the multinomial distribution. In the forward diffusion process, the intermediate distribution in the diffusion process is obtained, where the action annotation data is spliced ​​by the matrix of action categories and the matrix of action start and end times.

4. The method for detecting a fall action based on a multi-layer diffusion model according to claim 1, characterized in that: The intermediate distribution, audio-visual features, and the diffusion steps of the forward diffusion are encoded and input into the denoising model to obtain the prediction results, specifically: The intermediate distribution, audio-visual features, and diffusion step encodings of the forward diffusion are respectively input into the corresponding row-column feature encoding network to obtain three row-column feature encodings. The three row-column feature encodings are stacked and linearly transformed to output the prediction results.

5. The method for detecting falling action based on a multi-number diffusion model according to claim 1, characterized in that: The prediction results are decoded to obtain the final prediction results of fall action detection, which are as follows: The prediction results are screened according to the preset threshold to filter out the action starting points that meet the conditions; According to the starting point of the action, the action interval where the falling action may occur is found through clustering as the candidate interval; The falling action categories with the highest number of occurrences in different candidate intervals are counted as the final action detection categories, and the action detection categories are subjected to non-maximum value suppression to obtain the final prediction results.

6. A fall action detection system based on a multi-number diffusion model, characterized in that: include: Data collection module: used to collect sample fall action data and perform action annotation to obtain sample fall action data set and action annotation data; Multinomial diffusion model construction module: used to construct a multinomial diffusion model, which includes forward diffusion and backward denoising. Forward diffusion includes: adding noise to the action annotation data using equal probability multinomial distribution noise to obtain the intermediate distribution in the forward diffusion process; Substitute the sample falling action dataset into the audio-visual feature extraction network to fuse the features of audio data and image data to obtain audio-visual features, specifically: Perform 3D convolution processing based on a sample falling action dataset, where the sample falling action dataset includes a low frame rate image sequence, a high frame rate image sequence, and a high sampling rate audio spectrogram; In the stem stage, the features of the low frame rate image sequence and the high frame rate image sequence are processed with a 3D convolution with a step size of 2 and a 3D maximum pooling with a step size of 2. The audio features of the high sampling rate audio spectrogram are only processed with a 3D convolution. Each stage uses residual connection, first performs a 3D convolution with a stride of 2, and then stacks two 3D convolutions with a stride of 1; The features of the high frame rate image sequence are fused with the features of the low frame rate image sequence through one 3D convolution, and the audio features of the high sampling rate audio spectrogram are fused with the features of the low frame rate image sequence through two 3D convolutions; The three features are 3D-mean pooled and then added together to obtain the audio-visual features; Backward denoising module: used to build an audio-visual feature extraction network, substitute the sample falling action data set into the audio-visual feature extraction network to fuse the features of audio data and image data, obtain audio-visual features, build a denoising model, encode the intermediate distribution, audio-visual features and the diffusion step of forward diffusion into the denoising model, and obtain the prediction result; The denoising model consists of three row-column feature encoding networks, which are specifically: First, the transformer network is used to query the column dimension of the input to restore the association information between categories, followed by two layers of MLP layers. Then, after time series convolution processing, the row dimension is queried to restore the time series features, followed by two layers of MLP layers to form a row and column feature encoding network. Prediction result decoding module: used to decode the prediction result to obtain the final prediction result of fall action detection.

7. An electronic device, characterized in that: It includes a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement a fall action detection method based on a multi-number diffusion model as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that: When the computer program stored in the storage medium is executed by a computer, a fall action detection method based on a multi-number diffusion model as described in any one of claims 1 to 5 is implemented.

Images