Human body fall prediction method, device, electronic device and storage medium

Abstract

The present invention provides a human fall prediction method, device, electronic device and storage medium, belonging to the technical field of fall prediction, and the method includes: obtaining a video stream to be detected, and determining multiple frames of images to be detected in the video stream to be detected; performing feature extraction on each frame of the image to be detected based on a first modal feature extraction model and a second modal feature extraction model, respectively, and correspondingly obtaining a human skeleton feature map and a human body compact feature map; inputting the human skeleton feature map and the human body compact feature map into a fall prediction model to obtain a fall probability, and determining a fall result based on the fall probability. The present invention improves the comprehensiveness of human posture information by predicting the fall probability based on the human skeleton feature map and the human body compact feature map, thereby improving the accuracy of the fall probability predicted by the prediction model.

Description

Technical Field

[0001] The present invention relates to the technical field of fall prediction and detection, and in particular to a human fall prediction method, device, electronic equipment and storage medium. Background Art

[0002] With the aging of society, falls are the number one factor affecting the healthy later life of the elderly and one of the main causes of fractures, brain injuries and other physical injuries. Secondly, falls are also a serious public health problem that will impose a huge burden on the health system and social care system, including medical resources such as medical expenses, rehabilitation costs and possible long-term care costs. At the same time, fall detection technology can also provide certain support for the independent living of people with disabilities, such as patients with sarcopenia (muscular dystrophy). Sarcopenia is a disease that affects muscle development and function. Patients are often at risk of falling, so fall detection becomes crucial.

[0003] There are two mainstream existing fall prediction methods, namely sensor-based and computer vision-based fall prediction methods. However, the sensor-based method has problems such as a narrow scope of application and poor user experience; the computer vision-based method has low accuracy due to factors such as lighting and viewing angle changes. In order to solve the above technical problems, CN117496664A proposes that after performing fall prediction on the surveillance video and obtaining the initial prediction results, the initial prediction results are further determined by obtaining the acceleration and heart rate changes of the monitored object in the direction of gravity. However, it ensures the accuracy of fall prediction through other auxiliary means, and still cannot solve the technical problem of inaccurate initial prediction results.

[0004] There is an urgent need to provide a human fall prediction method, device, electronic device and storage medium to achieve accurate prediction of falls through a single prediction model. Summary of the invention

[0005] In view of this, it is necessary to provide a human fall prediction method, device, electronic device and storage medium to solve the technical problem in the prior art that a single prediction model cannot accurately predict a fall and relies on other auxiliary judgment means.

[0006] On the one hand, in order to solve the above technical problems, the present invention provides a human fall prediction method, comprising:

[0007] Acquire a video stream to be detected, and determine multiple frames of images to be detected in the video stream to be detected;

[0008] Based on the first modality feature extraction model and the second modality feature extraction model, feature extraction is performed on each frame of the image to be detected, and a human skeleton feature map and a human body compact feature map are correspondingly obtained;

[0009] The human skeleton feature map and the human compact feature map are input into a fall prediction model to obtain a fall probability, and a fall result is determined based on the fall probability.

[0010] In a possible implementation, the first modal feature extraction model includes a skeleton key point extraction unit and a skeleton key point screening unit; then, based on the first modal feature extraction model, feature extraction is performed on each frame of the image to be detected to obtain a human skeleton feature map, including:

[0011] Based on the skeleton key point extraction unit, feature extraction is performed on each frame of the image to be detected to obtain an initial skeleton feature map, wherein the initial skeleton feature map includes a plurality of initial skeleton key points;

[0012] The importance of the multiple initial skeleton key points is determined based on the skeleton key point screening unit, and the multiple initial skeleton key points are screened based on the importance to obtain the human skeleton feature map composed of multiple target skeleton key points.

[0013] In one possible implementation, the target bone key points are a head key point, a left shoulder key point, a right shoulder key point, a left elbow key point, a right elbow key point, a left wrist key point, a right wrist key point, a left hip key point, a right hip key point, a left knee key point, a right knee key point, a left ankle key point and a right ankle key point.

[0014] In a possible implementation, the fall prediction model includes a convolutional gated loop module, an attention module, and a classification module;

[0015] The convolution gated loop module is used to extract features from multiple feature image pairs to obtain time-series correlation features; the feature image pairs include a human skeleton feature map and a human body compact feature map corresponding to the same frame of the image to be detected;

[0016] The attention module is used to determine the feature weights of the plurality of feature image pairs;

[0017] The classification module is used to perform full connection processing on the time series correlation features and the feature weights to obtain the fall probability.

[0018] In a possible implementation, the attention module includes a channel attention unit and a spatial attention unit connected in series;

[0019] The channel attention unit is used to extract channel attention weights of the multiple feature image pairs, and determine the channel attention based on the channel attention weights and the multiple feature image pairs;

[0020] The spatial attention unit is used to extract the spatial attention weight of the channel attention, and determine the feature weight based on the spatial attention weight and the multiple channel attentions.

[0021] In a possible implementation, the video stream to be detected includes multiple frames of initial images to be detected, and determining the multiple frames of images to be detected in the video stream to be detected includes:

[0022] Determine the average difference between the initial image to be detected in each frame and the initial image to be detected in the previous frame adjacent to the initial image to be detected;

[0023] Obtaining a difference value sequence based on the average difference value and the order of the multiple frames of initial images to be detected;

[0024] Smoothing the difference value sequence based on a preset window function to obtain a smoothed difference value sequence, and determining each local maximum value in the smoothed difference value sequence;

[0025] A frame index corresponding to the local maximum value is determined, and the initial image to be detected corresponding to the frame index is determined as the image to be detected.

[0026] In a possible implementation, determining the average difference between each frame of the initial image to be detected and the previous frame of the initial image to be detected adjacent to the initial image to be detected includes:

[0027] Acquire a plurality of first pixel values ​​in the initial image to be detected and a plurality of second pixel values ​​in the previous frame of the initial image to be detected that correspond one-to-one to the plurality of first pixel values;

[0028] determining a plurality of pixel difference values ​​based on the plurality of first pixel values ​​and the plurality of second pixel values;

[0029] The average difference value is determined based on the plurality of pixel difference values ​​and the area of ​​the initial image to be detected.

[0030] On the other hand, the present invention also provides a human fall prediction device, comprising:

[0031] A video stream acquisition unit, used for acquiring a video stream to be detected, and determining a plurality of frames of images to be detected in the video stream to be detected;

[0032] A feature extraction unit, used to extract features from each frame of the image to be detected based on the first modality feature extraction model and the second modality feature extraction model, and correspondingly obtain a human skeleton feature map and a human body compact feature map;

[0033] A fall probability prediction unit is used to input the human skeleton feature map and the human body compact feature map into a fall prediction model to obtain a fall probability, and determine a fall result based on the fall probability.

[0034] On the other hand, the present invention also provides an electronic device, including a memory and a processor, wherein:

[0035] The memory is used to store programs;

[0036] The processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps in the human fall prediction method described in any one of the possible implementations above.

[0037] On the other hand, the present invention also provides a computer-readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the human fall prediction method described in any possible implementation manner described above are implemented.

[0038] The beneficial effects of the present invention are as follows: the human fall prediction method provided by the present invention extracts features from each frame of the image to be detected based on the first modal feature extraction model and the second modal feature extraction model, respectively, and obtains a human skeleton feature map and a human compact feature map correspondingly, which can realize the extraction of two different modal feature maps, and then inputs the human skeleton feature map and the human compact feature map into the fall prediction model for prediction. Since different human posture information is contained in feature maps of different modalities, the comprehensiveness of human posture information can be improved, eliminating the technical problem of low recognition accuracy caused by only recognizing a single modal feature map, and improving the accuracy of the fall probability predicted by the prediction model. Moreover, the fall result can be predicted by the determined fall probability.

[0039] Furthermore, compared with the prior art in which the prediction results are only for the two situations of falling and non-falling, that is, the binary classification situation with probability of 0 or 1, the present invention obtains the probability of falling, which can provide a basis for subsequent fall judgment. For example, a fall can be judged when the probability of falling is greater than a threshold and the probability of falling gradually increases, thereby improving the accuracy of the fall results. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0041] Figure 1 A schematic diagram of a flow chart of an embodiment of a human fall prediction method provided by the present invention;

[0042] Figure 2A schematic diagram of an embodiment of a human skeleton characteristic map and a human compact characteristic map provided by the present invention;

[0043] Figure 3 For the present invention Figure 1 A schematic diagram of an embodiment of a flow chart of obtaining a human skeleton feature map in S102;

[0044] Figure 4 A schematic diagram of a flow chart of an embodiment of a fall prediction model provided by the present invention;

[0045] Figure 5 For the present invention Figure 1 A schematic flow chart of an embodiment of S101;

[0046] Figure 6 For the present invention Figure 5 A schematic flow chart of an embodiment of S501;

[0047] Figure 7 A schematic diagram of the structure of an embodiment of a human fall prediction device provided by the present invention;

[0048] Figure 8 A schematic structural diagram of an embodiment of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0050] It should be understood that the schematic drawings are not drawn to scale. The flowchart used in the present invention shows the operations implemented according to some embodiments of the present invention. It should be understood that the operations of the flowchart can be implemented out of order, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art, under the guidance of the content of the present invention, can add one or more other operations to the flowchart, and can also remove one or more operations from the flowchart. Some of the block diagrams shown in the accompanying drawings are functional entities, which do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor systems and / or microcontroller systems.

[0051] Reference to an "embodiment" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0052] The present invention provides a road detection device method, an apparatus, an electronic device and a storage medium, which are respectively described below.

[0053] Figure 1 A schematic diagram of an embodiment of a road detection device method provided by the present invention is shown in FIG. Figure 1 As shown, the road detection equipment method includes:

[0054] S101, obtaining a video stream to be detected, and determining a plurality of frames of images to be detected in the video stream to be detected;

[0055] S102, performing feature extraction on each frame of the image to be detected based on the first modal feature extraction model and the second modal feature extraction model, respectively, to obtain a human skeleton feature map and a human body compact feature map;

[0056] S103: input the human skeleton feature map and the human body compact feature map into a fall prediction model to obtain a fall probability, and determine a fall result based on the fall probability.

[0057] The method of obtaining the video stream to be detected in step S101 may be to obtain the video stream to be detected in real time based on a video shooting device, or to obtain the video stream from a storage medium storing the video stream to be detected.

[0058] It should be understood that the first modal feature extraction model and the second modal feature extraction model in step S102 refer to two models that extract different feature types.

[0059] In a specific embodiment of the present invention, the human skeleton feature map refers to a feature map including multiple human skeleton feature points, which is a two-dimensional feature map, and the human body compact feature map refers to a human body UV texture map, which is a three-dimensional feature map.

[0060] In a specific embodiment of the present invention, the first modality feature extraction model is a ViTPose++ model, and the second modality feature extraction model is a DensePose model. The ViTPose++ model is a human pose estimation model based on ViT, which achieves efficient and accurate pose estimation by dividing the image into tiles and using the self-attention mechanism of Transformer to capture global and local features in the image. DensePose is an algorithm for human pose estimation and surface density estimation, which aims to convert human pose information in an image into a dense description of the human body surface. The algorithm was developed by the Facebook AI Research (FAIR) team and released in 2018. The DensePose model is based on the Mask R-CNN architecture, and additional branches are added on the original basis for dense pose estimation. The goal of the DensePose model is to advance human pose estimation to a more detailed level, not only to detect the position of the joint points, but also to estimate the pose and surface density of each pixel. The pose estimation result is given in the form of UV texture map coordinates, where each pixel corresponds to a point on the human body surface, and each point has corresponding body part and pose information. The U and V in the UV texture mapping coordinates are the horizontal and vertical coordinates of the image on the display, that is, the U-th pixel in the horizontal direction (image width) and the V-th pixel in the vertical direction (image height). Each pixel corresponds to a texture coordinate.

[0061] Specifically, Figure 2 As shown, Figure 2 On the left is a diagram of human skeleton characteristics. Figure 2 On the right is a diagram of the compact features of the human body.

[0062] Since the video stream to be detected is data with strong temporal correlation, in some embodiments of the present invention, the basic model structure of the fall prediction model is a gated recurrent unit (GRU), which has a recurrent structure, so that information can be continuously transmitted within the network, can capture information in the time series, and consider the temporal correlation of different frame detection images. At the same time, compared with LSTM, the structure is simpler, which can improve the prediction efficiency of human falls.

[0063] It should be noted that before executing step S103, the fall prediction model needs to be trained, tested and verified to obtain a fall prediction model that can directly perform predictions.

[0064] It should also be noted that determining the fall result based on the fall probability in step S104 is specifically: judging whether the fall probability is greater than the probability threshold, and whether the fall probability gradually increases. If both are met, the fall result is a fall; if neither is met, the fall result is no fall.

[0065] Compared with the prior art, the human fall prediction method provided by the embodiment of the present invention performs feature extraction on each frame of the image to be detected based on the first modal feature extraction model and the second modal feature extraction model, respectively, and obtains a human skeleton feature map and a human compact feature map correspondingly, thereby realizing the extraction of two different modal feature maps, and then inputs the human skeleton feature map and the human compact feature map into the fall prediction model for prediction. Since different human posture information is contained in feature maps of different modalities, the comprehensiveness of the human posture information can be improved, eliminating the technical problem of low recognition accuracy caused by only recognizing a single modal feature map, thereby improving the accuracy of the fall probability predicted by the prediction model.

[0066] Furthermore, compared with the prior art in which the prediction results are only for the two situations of falling and non-falling, that is, the binary classification situation with probability of 0 or 1, the embodiment of the present invention obtains the probability of falling, which can provide a basis for subsequent fall judgment. For example, a fall can be judged when the probability of falling is greater than a threshold and the probability of falling gradually increases, thereby improving the accuracy of the fall result.

[0067] In a specific embodiment of the present invention, when the first modal feature extraction model is only the ViTPose++ model, the human skeleton feature map obtained is a feature map including 17 human skeleton key points, and the 17 human skeleton key points are shown in Table 1:

[0068] Table 1 Key points of human skeleton

[0069] serial number Key points of human body serial number Key points of human body 0 nose 9 left wrist 1 Left Eye 10 right wrist 2 Right eye 11 Left hip 3 left ear 12 Right hip 4 Right ear 13 Left knee 5 Left shoulder 14 right knee 6 Right shoulder 15 Left ankle 7 left elbow 16 Right ankle 8 right elbow

[0070] It can be seen from Table 1 that a large number of human skeleton key points will lead to a low efficiency of fall prediction. In order to improve the prediction efficiency while ensuring the accuracy of fall prediction, in some embodiments of the present invention, the first modal feature extraction model includes a skeleton key point extraction unit and a skeleton key point screening unit; Figure 3 As shown, in step S102, feature extraction is performed on each frame of the image to be detected based on the first modal feature extraction model to obtain a human skeleton feature map, including:

[0071] S301, extracting features from each frame of the image to be detected based on a skeleton key point extraction unit to obtain an initial skeleton feature map, where the initial skeleton feature map includes a plurality of initial skeleton key points;

[0072] S302. Determine the importance of multiple initial skeleton key points based on the skeleton key point screening unit, and screen the multiple initial skeleton key points based on the importance to obtain a human skeleton feature map composed of multiple target skeleton key points.

[0073] The embodiment of the present invention sets a skeleton key point screening unit to screen the initial skeleton key points based on the importance of the determined initial skeleton key points, thereby eliminating redundant initial skeleton key points with low importance, obtaining important target skeleton key points, and reducing the number of key points in the human skeleton feature map, thereby improving the efficiency of fall prediction while ensuring the accuracy of fall prediction.

[0074] Step S302 specifically includes: removing the initial skeleton key points whose importance is lower than the importance threshold. Specifically, the importance threshold is 0.02.

[0075] In a specific embodiment of the present invention, the model structure of the skeleton key point screening unit is a random forest. After screening, the target skeleton key points are 12 key points corresponding to the numbers 5-16 in Table 1.

[0076] Furthermore, since the human key points numbered 0-4 represent the movements of the human head, in order to further improve the comprehensiveness of the human skeleton key points, in some embodiments of the present invention, the human key points numbered 0-4 are combined to generate the head key points, and the importance of the head key points is calculated. After calculation, the importance is greater than 0.02, so the head key points are used as the target skeleton key points. That is, the target skeleton key points include the head key points, the left shoulder key points, the right shoulder key points, the left elbow key points, the right elbow key points, the left wrist key points, the right wrist key points, the left hip key points, the right hip key points, the left knee key points, the right knee key points, the left ankle key points and the right ankle key points.

[0077] Specifically, the key points of the head are the center points of the five key points: the left ear, the right ear, the left eye, the right eye and the nose.

[0078] In order to improve the prediction performance of the fall prediction model, in some embodiments of the present invention, such as Figure 4 As shown in the figure, the fall prediction model includes a convolutional gated loop module, an attention module, and a classification module:

[0079] The convolution gated loop module is used to extract features from multiple feature image pairs to obtain temporal correlation features; the feature image pairs include a human skeleton feature map and a human body compact feature map corresponding to the same frame of the image to be detected;

[0080] The attention module is used to determine the feature weights of multiple feature image pairs;

[0081] The classification module is used to perform full-connection processing on the time series correlation features and feature weights to obtain the fall probability.

[0082] Among them, the convolutional gated recurrent unit (ConvGRU) is an improved structure of the gated recurrent unit (GRU). Specifically, the fully connected operation in the activation function is changed to a convolution operation. The activation function is the sigmoid function. To reset the gate control the degree of preservation of the hidden state at the previous moment, To update the gate to control the information allocation between the previous hidden state and the candidate hidden state at the current moment. The specific formula is as follows:

[0083]

[0084]

[0085] in, and represents the corresponding tensor, Indicates the state at the previous moment. Indicates the current input. represents the convolution operation, Represents the sigmoid function, which can transform the data into the range of 0~1, thus acting as a gating signal. Used to choose to forget some of the fall information extracted in the past, It is used to select how much of the fall-related information extracted in the past should be copied and passed to the next moment, so as to focus on the change of fall posture between the previous and next sequences.

[0086] The embodiment of the present invention sets an iterative prediction model including a convolutional gated recurrent unit and an attention module, fully considers the importance of different visual areas in each feature image pair, fully considers the temporal and spatial characteristics of the video frame image, obtains feature weights with temporal information and spatial information, and improves the accuracy of the determined fall probability.

[0087] In some embodiments of the present invention, Figure 4 As shown, the attention module includes a channel attention unit and a spatial attention unit connected in series;

[0088] The channel attention unit is used to extract channel attention weights of multiple feature image pairs, and determine the channel attention based on the channel attention weights and the multiple feature image pairs;

[0089] The spatial attention unit is used to extract the spatial attention weights of channel attention and determine the feature weights based on the spatial attention weights and multiple channel attentions.

[0090] The channel attention unit is used to capture the relationship between different channels of the feature matrix. By calculating the importance of each channel, the feature matrix is ​​adaptively weighted to improve the information interaction and combination capabilities between different channels. First, the spatial dimension is compressed by average pooling, and the attention on finer channels is inferred by maximum pooling. Then, the channel attention feature weights are generated through two convolution operations. ,Right now:

[0091]

[0092] in, represents the activation function sigmoid, represents two convolution operations, represents average pooling, represents the maximum pooling. Then Input to the channel attention unit Multiply to get channel attention ,Right now:

[0093]

[0094] in, Represents element-wise multiplication.

[0095] The spatial attention unit aims to capture the relationship between different spatial positions in the feature matrix and determine the importance of different positions by calculating the spatial distribution information of the feature matrix, thereby enhancing the network's perception of spatial structure. In the same way, average pooling and maximum pooling operations are performed along the channel dimension, and then their outputs are concatenated into a feature descriptor, and then the spatial attention feature weights are generated through the convolution layer. ,Right now:

[0096]

[0097] in, Represents a convolution operation. Then the spatial attention feature weights Input to the spatial attention unit Multiply to get the final output of the CBAM module ,Right now:

[0098]

[0099] It indicated The weight or attention of each element in the

[0100] In some embodiments of the present invention, the video stream to be detected includes multiple frames of initial images to be detected, and the initial images to be detected with large dynamic changes between two adjacent frames are the images that need to be paid attention to. Therefore, in some embodiments of the present invention, Figure 5As shown, step S101 includes:

[0101] S501, determining an average difference value between each frame of the initial image to be detected and the previous frame of the initial image to be detected adjacent to it;

[0102] S502, obtaining a difference value sequence based on the average difference value and the order of multiple frames of initial images to be detected;

[0103] S503, smoothing the difference value sequence based on a preset window function to obtain a smoothed difference value sequence, and determining each local maximum value in the smoothed difference value sequence;

[0104] S504 , determining a frame index corresponding to the local maximum value, and determining an initial image to be detected corresponding to the frame index as the image to be detected.

[0105] The embodiment of the present invention processes multiple initial images to be detected by using an inter-frame difference method based on local maximum values, which can better capture important changes in the video, so that the algorithm pays more attention to changes in key points of the human body, thereby improving the accuracy and real-time performance of fall detection.

[0106] It should be understood that: before step S501, each frame of the initial image to be detected needs to be converted from the camera coordinate system to the LUV color space, and the specific conversion coordinates are not specifically limited here.

[0107] The window function is generated based on the Hanning signal filter so as to achieve a smoothing effect by performing a weighted average operation on the input difference value sequence.

[0108] In some embodiments of the present invention, Figure 6 As shown, step S501 includes:

[0109] S601, acquiring a plurality of first pixel values ​​in an initial image to be detected and a plurality of second pixel values ​​in a previous frame of the initial image to be detected that correspond one-to-one to the plurality of first pixel values;

[0110] S602, determining a plurality of pixel difference values ​​based on a plurality of first pixel values ​​and a plurality of second pixel values;

[0111] S603: Determine an average difference value based on the multiple pixel difference values ​​and the area of ​​the initial image to be detected.

[0112] Step S603 specifically includes: accumulating a plurality of pixel difference values, and taking the ratio of the accumulation result to the initial area of ​​the image to be detected as the average difference value.

[0113] In summary, the embodiment of the present invention extracts the image to be detected from the initial image to be detected by the inter-frame difference method, thereby reducing the interference of repeated and redundant data on the experiment; then, ViTPose++ is used to extract the key points of the human body, thereby avoiding the privacy leakage of others caused by the traditional image data set; finally, the human body key point data is trained with random forest and the feature importance is evaluated to screen out the most representative human body key points, thereby ensuring robustness while reducing the amount of data calculation and improving model performance.

[0114] Furthermore, the embodiments of the present invention solve the problems existing in the current fall detection algorithm from three dimensions: time (relative displacement tracking relationship between the time series of GRU), space (relative position relationship of key points of the human body), and channel (relationship between input feature maps). By combining the information of time, space, and channel through the attention module, the spatiotemporal features in the video sequence can be effectively captured, and the importance of each channel can be adaptively learned, thereby improving the model's detection performance for fall behavior.

[0115] Furthermore, the embodiment of the present invention adopts a dual-channel input method that combines DensePose to generate a dense human body feature map with ViTPose++ to extract a human body key point skeleton feature map, thereby improving the modal diversity of human posture information input and further improving the accuracy of fall prediction.

[0116] In order to better implement the human body fall prediction method in the embodiment of the present invention, based on the human body fall prediction method, the embodiment of the present invention also provides a human body fall prediction device, such as Figure 7 As shown, the human body fall prediction device 700 includes:

[0117] The video stream acquisition unit 701 is used to acquire the video stream to be detected and determine multiple frames of images to be detected in the video stream to be detected;

[0118] A feature extraction unit 702 is used to extract features from each frame of the image to be detected based on the first modality feature extraction model and the second modality feature extraction model, respectively, to obtain a human skeleton feature map and a human body compact feature map;

[0119] The fall probability prediction unit 703 is used to input the human skeleton feature map and the human body compact feature map into the fall prediction model to obtain the fall probability, and determine the fall result based on the fall probability.

[0120] The human fall prediction device 700 provided in the above embodiment can implement the technical solution described in the above human fall prediction method embodiment. The specific implementation principles of the above modules or units can refer to the corresponding contents in the above human fall prediction method embodiment, which will not be repeated here.

[0121] like Figure 8As shown, the present invention also provides an electronic device 800. The electronic device 800 includes a processor 801, a memory 802 and a display 803. Figure 8 Only some components of the electronic device 800 are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0122] In some embodiments, the processor 801 may be a central processing unit (CPU), a microprocessor or other data processing chip, and is used to run program codes or process data stored in the memory 802, such as the human fall prediction method of the present invention.

[0123] In some embodiments of the present invention, the processor 801 may be a single server or a server group. The server group may be centralized or distributed. In some embodiments, the processor 801 may be local or remote. In some embodiments, the processor 801 may be implemented in a cloud platform. In one embodiment, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, a multi-cloud, etc., or any combination thereof.

[0124] In some embodiments, the memory 802 may be an internal storage unit of the electronic device 800, such as a hard disk or memory of the electronic device 800. In other embodiments, the memory 802 may also be an external storage device of the electronic device 800, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the electronic device 800.

[0125] Furthermore, the memory 802 may include both an internal storage unit of the electronic device 800 and an external storage device. The memory 802 is used to store application software installed in the electronic device 800 and various data.

[0126] In some embodiments, the display 803 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 803 is used to display information of the electronic device 800 and to display a visual user interface. The components 801-803 of the electronic device 800 communicate with each other via a system bus.

[0127] In some embodiments of the present invention, when the processor 801 executes the human fall prediction program in the memory 802, the following steps may be implemented:

[0128] Acquire a video stream to be detected, and determine multiple frames of images to be detected in the video stream to be detected;

[0129] Extract features from each frame of the image to be detected based on the first modality feature extraction model and the second modality feature extraction model, respectively, and obtain a human skeleton feature map and a human body compact feature map accordingly;

[0130] The human skeleton feature map and the human compact feature map are input into the fall prediction model to obtain the fall probability, and the fall result is determined based on the fall probability.

[0131] It should be understood that: when the processor 801 executes the human fall prediction program in the memory 802, in addition to the above functions, other functions can also be implemented. For details, please refer to the description of the corresponding method embodiment above.

[0132] Furthermore, the embodiment of the present invention does not specifically limit the type of the electronic device 800 mentioned. The electronic device 800 may be a portable electronic device such as a mobile phone, a tablet computer, a personal digital assistant (PDA), a wearable device, a laptop computer, etc. Exemplary embodiments of portable electronic devices include but are not limited to portable electronic devices equipped with IOS, Android, Microsoft or other operating systems. The above-mentioned portable electronic devices may also be other portable electronic devices. It should also be understood that in some other embodiments of the present invention, the electronic device 800 may not be a portable electronic device, but a desktop computer with a touch-sensitive surface (such as a touch panel).

[0133] Accordingly, an embodiment of the present invention further provides a computer-readable storage medium, which is used to store computer-readable programs or instructions. When the program or instructions are executed by a processor, the steps or functions in the human fall prediction method provided in the above-mentioned method embodiments can be implemented.

[0134] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0135] The above is a detailed introduction to a human fall prediction method, device, electronic device and storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A human fall prediction method, characterized in that: include: Acquire a video stream to be detected, and determine multiple frames of images to be detected in the video stream to be detected; Based on the first modality feature extraction model and the second modality feature extraction model, feature extraction is performed on each frame of the image to be detected, and a human skeleton feature map and a human body compact feature map are correspondingly obtained; The first modal feature extraction model includes a skeleton key point extraction unit and a skeleton key point screening unit; the second modal feature extraction model performs posture estimation and surface density estimation for each pixel while detecting the position of the human body joint points, and the posture estimation result is given in the form of a UV texture map; Inputting the human skeleton feature map and the human compact feature map into a fall prediction model to obtain a fall probability, and determining a fall result based on the fall probability; The fall prediction model includes a convolutional gated loop module, an attention module and a classification module; The convolution gated loop module is used to extract features from multiple feature image pairs to obtain time-series correlation features; the feature image pairs include a human skeleton feature map and a human body compact feature map corresponding to the same frame of the image to be detected; The attention module is used to determine the feature weights of the plurality of feature image pairs; The classification module is used to perform full connection processing on the time series correlation feature and the feature weight to obtain the fall probability; The attention module includes a channel attention unit and a spatial attention unit connected in series; The channel attention unit is used to extract channel attention weights of the multiple feature image pairs, and determine the channel attention based on the channel attention weights and the multiple feature image pairs; The spatial attention unit is used to extract the spatial attention weight of the channel attention, and determine the feature weight based on the spatial attention weight and multiple channel attentions.

2. The human body fall prediction method according to claim 1, characterized in that: Based on the first modality feature extraction model, feature extraction is performed on each frame of the image to be detected to obtain a human skeleton feature map, including: Based on the skeleton key point extraction unit, feature extraction is performed on each frame of the image to be detected to obtain an initial skeleton feature map, wherein the initial skeleton feature map includes a plurality of initial skeleton key points; The importance of the multiple initial skeleton key points is determined based on the skeleton key point screening unit, and the multiple initial skeleton key points are screened based on the importance to obtain the human skeleton feature map composed of multiple target skeleton key points.

3. The human body fall prediction method according to claim 2, characterized in that: The target bone key points are the head key point, left shoulder key point, right shoulder key point, left elbow key point, right elbow key point, left wrist key point, right wrist key point, left hip key point, right hip key point, left knee key point, right knee key point, left ankle key point and right ankle key point.

4. The human body fall prediction method according to claim 1, characterized in that: The video stream to be detected includes multiple frames of initial images to be detected, and determining the multiple frames of images to be detected in the video stream to be detected includes: Determine the average difference between the initial image to be detected in each frame and the initial image to be detected in the previous frame adjacent to the initial image to be detected; Obtaining a difference value sequence based on the average difference value and the order of the multiple frames of initial images to be detected; Smoothing the difference value sequence based on a preset window function to obtain a smoothed difference value sequence, and determining each local maximum value in the smoothed difference value sequence; A frame index corresponding to the local maximum value is determined, and the initial image to be detected corresponding to the frame index is determined as the image to be detected.

5. The human body fall prediction method according to claim 4, characterized in that: The step of determining the average difference between each frame of the initial image to be detected and the previous frame of the initial image to be detected adjacent to the initial image to be detected comprises: Acquire a plurality of first pixel values ​​in the initial image to be detected and a plurality of second pixel values ​​in the previous frame of the initial image to be detected that correspond one-to-one to the plurality of first pixel values; determining a plurality of pixel difference values ​​based on the plurality of first pixel values ​​and the plurality of second pixel values; The average difference value is determined based on the plurality of pixel difference values ​​and the area of ​​the initial image to be detected.

6. A human fall prediction device, characterized in that: include: A video stream acquisition unit, used for acquiring a video stream to be detected, and determining a plurality of frames of images to be detected in the video stream to be detected; A feature extraction unit, used to extract features from each frame of the image to be detected based on the first modality feature extraction model and the second modality feature extraction model, and correspondingly obtain a human skeleton feature map and a human body compact feature map; The first modal feature extraction model includes a skeleton key point extraction unit and a skeleton key point screening unit; the second modal feature extraction model performs posture estimation and surface density estimation for each pixel while detecting the position of the human body joint points, and the posture estimation result is given in the form of a UV texture map; A fall probability prediction unit, used for inputting the human skeleton feature map and the human body compact feature map into a fall prediction model to obtain a fall probability, and determining a fall result based on the fall probability; The fall prediction model includes a convolutional gated loop module, an attention module and a classification module; The convolution gated loop module is used to extract features from multiple feature image pairs to obtain time-series correlation features; the feature image pairs include a human skeleton feature map and a human body compact feature map corresponding to the same frame of the image to be detected; The attention module is used to determine the feature weights of the plurality of feature image pairs; The classification module is used to perform full connection processing on the time series correlation feature and the feature weight to obtain the fall probability; The attention module includes a channel attention unit and a spatial attention unit connected in series; The channel attention unit is used to extract channel attention weights of the multiple feature image pairs, and determine the channel attention based on the channel attention weights and the multiple feature image pairs; The spatial attention unit is used to extract the spatial attention weight of the channel attention, and determine the feature weight based on the spatial attention weight and multiple channel attentions.

7. An electronic device, characterized in that: comprising a memory and a processor, wherein: The memory is used to store programs; The processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps in the human fall prediction method described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the human fall prediction method described in any one of claims 1 to 5 are implemented.

Images