A fall detection method, a fall detection device, a computer device and a medium

Abstract

The application discloses a fall detection method, a fall detection device, computer equipment and a medium, wherein the fall detection method of one embodiment comprises: using a fall detection model to detect a received RGB video to be detected and outputting a first detection result, comprising: using a key point detection model to detect key points of a first group of images and generating a key point heat map, the first group of images being images extracted from the RGB video to be detected at a first preset interval; generating key point data tensors according to the key point heat map, using a first three-dimensional convolution network to process data of the key point data tensors and outputting a first result tensor; using a full connection network to process data of the first result tensor and outputting a first detection result; comparing the first detection result with a pre-set fall detection threshold and outputting a fall detection result. The fall detection method provided by the application can accurately realize fall detection and has practical application value.

Description

Technical Field

[0001] The present invention relates to the technical field of human body detection, and in particular to a fall detection method, a fall detection device, a computer device, and a medium. Background Art

[0002] Every year, a large number of elderly and disabled people suffer serious physical injuries due to falls. Effective fall detection has positive significance for the treatment of fallers and is also beneficial to the subsequent medical care of fallers. Summary of the Invention

[0003] In order to solve at least one of the above problems, a first embodiment of the present invention provides a fall detection method, comprising:

[0004] Using a fall detection model to detect the received RGB video to be tested and outputting a first detection result, including: using a key point detection model to perform key point detection on a first group of images and generate a key point heat map, where the first group of images are images extracted from the RGB video to be tested according to a first preset interval; generating a key point data tensor according to the key point heat map, using a first three-dimensional convolutional network to perform data processing on the key point data tensor and output a first result tensor; using a fully connected network to perform data processing on the first result tensor and output a first detection result;

[0005] The first detection result is compared with a preset fall detection threshold and a fall detection result is output.

[0006] For example, in the fall detection method provided in some embodiments of the present application, before using the fully connected network to process the first tensor and output the first detection result, using the fall detection model to detect the received RGB video to be tested and outputting the first detection result further includes:

[0007] Performing data processing on a second set of images using a second three-dimensional convolutional network and outputting a second result tensor, wherein the second set of images is images extracted from a preset time period of the RGB video to be tested according to a second preset interval;

[0008] The first result tensor and the second result tensor are concatenated and the concatenated result is used as the updated first result tensor.

[0009] For example, in the fall detection method provided in some embodiments of the present application, before using the fall detection model to detect the received RGB video to be tested and outputting the first detection result, the fall detection method further includes:

[0010] Training the fall detection model using the training sample includes: inputting the RGB sample video into the fall detection model and outputting a second detection result, evaluating the second detection result using a loss function and adjusting parameters of the fall detection model.

[0011] For example, in the fall detection method provided in some embodiments of the present application, the loss function is:

[0012]

[0013] Wherein, l is the label of the RGB sample video, and x is the second detection result output by the fall detection model.

[0014] For example, in the fall detection method provided in some embodiments of the present application, before using the training samples to train the fall detection model, the fall detection method further includes:

[0015] Obtaining the training samples includes: generating a first group of sample images and a second group of sample images based on the RGB sample video, performing key point detection on the first group of sample images using the key point detection model and generating a key point sample heat map, generating a key point sample data tensor based on the key point sample heat map, performing data enhancement on the second group of sample images, and associating and enhancing the key point sample heat map and the enhanced second group of sample images.

[0016] For example, in the fall detection method provided in some embodiments of the present application, the using the key point detection model to perform key point detection on the first group of sample images and generate a key point sample heat map, and generating a key point sample data tensor based on the key point sample heat map further includes: performing key point detection on each sample image of the first group of sample images to output each key point sample, and deleting one key point in the key point sample with a preset probability to enhance the key point sample, generating the key point sample heat map based on the enhanced key point sample, and then generating the key point sample data tensor based on the key point sample heat map;

[0017] The performing data enhancement on the second set of sample images further comprises: performing image color enhancement on each sample image of the second set of sample images, and performing random excision processing on the second set of sample images with enhanced image colors to enhance the second set of sample images;

[0018] The performing key point associating and enhancing on the key point sample heat map and the enhanced second group of sample images further includes: performing random left-right flipping processing on the key point sample heat map and the enhanced second group of sample images.

[0019] For example, in the fall detection method provided in some embodiments of the present application,

[0020] The first group of images is images extracted from the received RGB video to be tested according to a first preset interval and further includes:

[0021] Dividing the RGB video to be tested into multiple videos to be tested at first intervals according to the first preset intervals;

[0022] Randomly extracting a frame of image from each first interval video to be tested to generate the first group of images;

[0023] and / or

[0024] The using of the key point detection model to perform key point detection on the first group of images and generate a key point heat map further includes: downsampling the key point heat map.

[0025] For example, in the fall detection method provided in some embodiments of the present application,

[0026] The key point detection model is openpose or hrnet;

[0027] and / or

[0028] The first three-dimensional convolutional network is one of I3D, TSM, and SlowOnly, the second three-dimensional convolutional network is one of I3D, TSM, and SlowOnly, the first three-dimensional convolutional network and the second three-dimensional convolutional network are the same network or different networks, and when the first three-dimensional convolutional network and the second three-dimensional convolutional network are the same network, the network weights are different.

[0029] A second embodiment of the present invention provides a fall detection device, comprising a communication unit, a fall detection model, and a controller, wherein the controller is configured to:

[0030] Using the communication unit to receive the RGB video to be tested;

[0031] Using the fall detection model to detect the RGB video to be tested and outputting a first detection result, including: using a key point detection model to perform key point detection on a first group of images and generate a key point heat map, where the first group of images are images extracted from the RGB video to be tested according to a first preset interval; generating a key point data tensor according to the key point heat map, using a first three-dimensional convolutional network to perform data processing on the key point data tensor and output a first result tensor; using a fully connected network to perform data processing on the first result tensor and output a first detection result;

[0032] The first detection result is compared with a preset fall detection threshold and a fall detection result is output.

[0033] For example, in the fall detection device provided in some embodiments of the present application, the fall detection model further includes a second three-dimensional convolutional network, and the controller uses the fall detection model to detect the RGB video to be tested and output a first detection result, which is further configured as follows:

[0034] Performing key point detection on a first set of images using a key point detection model and generating a key point heat map, wherein the first set of images is images extracted from the RGB video to be tested according to a first preset interval;

[0035] Generate a key point data tensor according to the key point heat map, perform data processing on the key point data tensor using a first three-dimensional convolutional network and output a first result tensor;

[0036] Performing data processing on a second set of images using a second three-dimensional convolutional network and outputting a second result tensor, wherein the second set of images is images extracted from a preset time period of the RGB video to be tested according to a second preset interval;

[0037] Concatenate the first result tensor and the second result tensor and use the concatenated result as the updated first result tensor;

[0038] Use a fully connected network to perform data processing on the first result tensor and output a first detection result.

[0039] A third embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method described in the first embodiment is implemented.

[0040] A fourth embodiment of the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first embodiment when executing the program.

[0041] The beneficial effects of the present invention are as follows:

[0042] In response to the current existing problems, the present invention develops a fall detection method, a fall detection device, a computer device and a medium. Based on the RGB video sequence, the fall detection model is used to detect and output the detection results, and the detection results are compared using a preset detection threshold and the fall detection results are output. Specifically, the key point detection model of the fall detection model is used to complete the extraction of all key points at one time and generate a key point heat map, the first three-dimensional convolutional network is used to realize the correlation analysis between multiple frames of the RGB video sequence, and the detection results are output through a fully connected network, thereby compensating for the problems existing in the existing technology, effectively improving the accuracy of fall detection, and having practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0044] Figure 1 A flow chart showing a fall detection method according to an embodiment of the present invention is shown;

[0045] Figure 2 A structural block diagram of a fall detection device according to an embodiment of the present invention is shown;

[0046] Figure 3a-3b Showing a single frame image and a corresponding heat map according to an embodiment of the present invention;

[0047] Figure 4 A schematic structural diagram of a computer device according to another embodiment of the present invention is shown. DETAILED DESCRIPTION

[0048] In order to more clearly illustrate the present invention, the present invention is further described below in conjunction with preferred embodiments and accompanying drawings. Similar components in the accompanying drawings are represented by the same reference numerals. It should be understood by those skilled in the art that the following detailed description is illustrative rather than restrictive and should not be used to limit the scope of protection of the present invention.

[0049] In the prior art, the commonly used fall detection methods are mainly divided into the following categories:

[0050] Wearable device-based. Wearable devices are not suitable for large-scale monitoring in public places, such as hospitals. In Chinese patent documents, a patent application numbered CN201611155006.1 discloses a wearable device-based fall detection algorithm and related equipment. This detection method is not conducive to fall detection in large public areas such as hospitals. In addition, the detection algorithm based on wearable devices is complex to operate and is not suitable for use by the elderly and disabled. At the same time, such wearable devices need to consider practical issues such as battery life.

[0051] Depth camera-based: As the distance increases, the accuracy of depth cameras decreases significantly. At the same time, most depth camera-based algorithms are based on some manually set rules and have poor generalization.

[0052] Based on RGB video sequences. Currently, most algorithms based on RGB video sequences directly use the original video as input. Since fall training data is difficult to collect, most rely on actors simulating in the laboratory. This method also suffers from the problem of poor generalization.

[0053] In view of the above situation, the inventor has proposed a fall detection method after extensive research and experiments, such as Figure 1 As shown, a fall detection method according to an embodiment of the present invention includes:

[0054] Using a fall detection model to detect the received RGB video to be tested and outputting a first detection result, including: using a key point detection model to perform key point detection on a first group of images and generate a key point heat map, where the first group of images are images extracted from the RGB video to be tested according to a first preset interval; generating a key point data tensor according to the key point heat map, using a first three-dimensional convolutional network to perform data processing on the key point data tensor and output a first result tensor; using a fully connected network to perform data processing on the first result tensor and output a first detection result;

[0055] The first detection result is compared with a preset fall detection threshold and a fall detection result is output.

[0056] This embodiment uses a fall detection model to detect and output detection results based on RGB video sequences, uses a preset detection threshold to compare the detection results, and outputs the fall detection results. Specifically, the key point detection model of the fall detection model completes the extraction of all key points and generates a key point heat map at one time, implements correlation analysis between multiple frames of the RGB video sequence through a first three-dimensional convolutional network, and outputs the detection results through a fully connected network, thereby remedying the problems existing in the existing technology, effectively improving the accuracy of fall detection, and having practical application value.

[0057] In a specific example, Figure 2 The figure shows a fall detection device, wherein the communication unit is used to receive an externally transmitted RGB video, the fall detection model is used to detect the RGB video and output the detection result to the controller, and the controller compares the detection result with a preset fall detection threshold and outputs the fall detection result. Figure 1 As shown, a specific detection example is used to illustrate:

[0058] In the first step, a fall detection model is used to detect the received RGB video to be tested and output a first detection result.

[0059] First, a key point detection model is used to perform key point detection on a first group of images and generate a key point heat map, where the first group of images is extracted from the RGB video to be tested according to a first preset interval.

[0060] In this embodiment, the RGB video to be tested is divided into multiple first-interval test videos at a first preset interval, and a frame of image is randomly extracted from each first-interval test video to generate the first set of images. Specifically, a 5 to 15 second RGB video to be tested is divided into 24 to 96 first-interval test videos. For example, a 10-second RGB video to be tested is divided into 48 equal-interval data sets at 30 frames per second, for a total of 300 frames. One image is randomly extracted from each data set to form the first set of images, thus obtaining 48 images.

[0061] It is worth noting that this embodiment does not specifically limit the length and number of segments of the RGB video to be tested. Those skilled in the art should select an appropriate length and an appropriate number of segments according to actual application requirements to achieve accurate extraction of data from the RGB video to be tested as a design criterion, which will not be repeated here.

[0062] In this embodiment, the key point detection model is openpose or hrnet, and the key point detection model is used to detect key points of 48 images, and a key point heat map is generated based on the detection results. Specifically, key point detection is performed on the human body in each image and 17 key points of the human body are output. For example, a single frame image is used as an example to illustrate: Figure 3a The figure shows an image with a person. The key point detection model generates 17 key points for human key point detection, as shown in Figure 3b As shown, a key point heat map is formed for 17 key points. This embodiment generates a heat map with 48*17 key points based on the key points detected in 48 images. In other words, a 10-second RGB video to be tested is represented by 48 images. The key point detection model extracts all the key points of the 48 images (i.e., the 10-second RGB video to be tested) at once and generates a key point heat map representing the video, effectively improving data processing efficiency and reducing data processing time.

[0063] Then, a key point data tensor is generated according to the key point heat map, and a first three-dimensional convolutional network is used to process the key point data tensor and output a first result tensor.

[0064] In this embodiment, a heat map formed by key point detection of 48 images is used as an example for explanation. The width of the heat map image is W, the height is H, the time length is T (48 in this embodiment), and the number of key points is K (17 in this embodiment). Then, all key point data of the 10-second RGB video to be tested are pieced together into a TxKxHxW key point data tensor. The key point data tensor is processed using a first three-dimensional convolutional network, that is, the three-dimensional convolutional network performs correlation analysis between multiple frames and outputs a first result tensor.

[0065] In this embodiment, the first three-dimensional convolutional network is one of I3D, TSM, and SlowOnly, which performs a three-dimensional convolution operation on the input data and outputs a first result tensor. To improve the running speed, the key point heat map is downsampled, for example, by a factor of 4, thereby reducing the data volume and effectively speeding up the data processing.

[0066] Finally, a fully connected network is used to perform data processing on the first result tensor and output a first detection result.

[0067] In this embodiment, the fully connected network performs classification according to the input first result tensor and outputs a first detection result.

[0068] In the second step, the first detection result is compared with a preset fall detection threshold and the fall detection result is output.

[0069] In this embodiment, the first detection result is a specific numerical value that can be converted into a probability through data processing, i.e., representing the probability of a fall; the preset fall detection threshold is threshold data corresponding to the first detection result and is used to evaluate the first detection result. For example, if the first detection result is a specific numerical value, the fall detection threshold is the specific numerical value used to evaluate the first detection result; if the first detection result is converted into a probability through data processing, the fall detection threshold is the probability threshold used to evaluate the probability. For example, if the probability threshold is 0.5, when the probability converted from the first detection result is 0.4, it is less than the probability threshold, indicating that the person in the image has not fallen, and the fall detection result is output; when the probability converted from the first detection result is 0.6, it is greater than the probability threshold, indicating that the person in the image has fallen, and the fall detection result is output. This embodiment can intuitively and clearly determine whether a person has fallen based on the fall detection result, thereby providing timely treatment to the person who has fallen, effectively improving the recovery rate and level of the person who has fallen.

[0070] At this point, based on an RGB video sequence, the fall detection model including the key point detection model and the first three-dimensional convolutional network can accurately determine whether a fall event has occurred, thus remedying the problems existing in the existing technology and having practical application value.

[0071] Considering the problem of easy confusion in image judgment in the prior art, in an optional embodiment, before using the fully connected network to process the first tensor and output the first detection result, using the fall detection model to detect the received RGB video to be tested and outputting the first detection result further includes:

[0072] Performing data processing on a second set of images using a second three-dimensional convolutional network and outputting a second result tensor, wherein the second set of images is images extracted from a preset time period of the RGB video to be tested according to a second preset interval;

[0073] The first result tensor and the second result tensor are concatenated and the concatenated result is used as the updated first result tensor.

[0074] In this embodiment, based on the above-mentioned key point detection and key point heat map generation of equally spaced images of the RGB video to be tested, the image data of the original video of the RGB video to be tested is added, and the key point heat map of the complete RGB video to be tested and the original video image are used to represent the RGB video to be tested, thereby further improving the detection accuracy of fall events and effectively avoiding the problem of detection confusion.

[0075] Specifically, this implementation divides the RGB video to be tested into 6 time periods, performs sampling in the last time period, and samples 8 frames. The 8 frames of image data are transmitted to the second 3D convolutional network. The second 3D convolutional network is one of I3D, TSM, and SlowOnly. The 8 frames of original image data are processed and the second result tensor is output.

[0076] It is worth noting that this application does not specifically limit the original video image. Taking into account the timeliness of the fall event, the original video image of the later stage of the video is used as the basis for judgment. In addition, the first three-dimensional convolutional network and the second three-dimensional convolutional network can be the same network or different networks. This application does not specifically limit this. Taking into account that the first three-dimensional convolutional network and the second three-dimensional convolutional network process different input data, the network weights are different when the first three-dimensional convolutional network and the second three-dimensional convolutional network are the same network. Those skilled in the art should select an appropriate three-dimensional convolutional network according to actual application requirements, which will not be described here.

[0077] The second result tensor obtained by the original video image data is then spliced ​​with the first result tensor representing the complete RGB video to be tested to update the first result tensor. The updated first result tensor has the characteristics of both.

[0078] Finally, a fully connected layer is used to classify the concatenated first result tensor, output the detection result, and judge the detection result according to the fall detection threshold to output the fall detection result of whether a fall event has occurred.

[0079] This embodiment can solve the confusion problem of fall event detection in the prior art by using a key point heat map representing a complete RGB video to be tested and a small amount of original video images, thereby effectively improving the detection accuracy of fall events in the RGB video to be tested.

[0080] In an optional embodiment, before using the fall detection model to detect the input RGB video to be tested, the fall detection method further includes obtaining training samples and using the training samples to train the fall detection model.

[0081] In a specific example, obtaining the training samples includes: generating a first group of sample images and a second group of sample images based on an RGB sample video, performing key point detection on the first group of sample images using the key point detection model and generating a key point sample heat map, generating a key point sample data tensor based on the key point sample heat map, performing data enhancement on the second group of sample images, and performing key point association and enhancement on the key point sample heat map and the enhanced second group of sample images.

[0082] In this embodiment, first, a 10-second RGB sample video is divided into 48 data portions at equal intervals of 30 frames per second, for a total of 300 frames. One image is randomly sampled from each data portion to obtain 48 images. The 48 image sets form the first group of sample images, and the last 8 images are selected from the 48 images as the second group of sample images.

[0083] Then, the key point detection model is used to perform key point detection on each sample image of the first group of sample images to output each key point sample. For example, human body key point detection is performed on the first sample image and 17 key points are output. The key points of the key point samples are deleted with a preset probability to enhance the key point samples. For example, a key point is deleted from the key point samples with a probability of 0.7, thereby performing data enhancement on the key point samples to obtain more key point samples. Specifically, key point detection is performed on each sample image and key point deletion is performed based on the obtained key point samples to enhance the key point samples. For example, one key point in the key point samples is deleted to enhance the key point samples. A key point sample heat map is generated using the key points of the enhanced key point samples, and a key point sample data tensor is obtained based on the key point sample heat map. This embodiment selectively deletes key points based on the obtained key points to achieve data enhancement operations, and then obtains a key point sample heat map and a key point sample data tensor based on the data enhancement, thereby achieving data enhancement of the training samples.

[0084] Then, data enhancement is performed on the second set of sample images. Specifically, each sample image in the second set of sample images is subjected to image color enhancement, such as randomly adjusting the brightness, saturation, and contrast of each sample image, thereby achieving data enhancement for the second set of sample images. Simultaneously, a random excision process is performed on the enhanced second set of sample images, such as randomly selecting 1 to 3 points in the image with a probability of 50%, and setting a 7x7 area near the selected points to a fixed value of 127, thereby enhancing the second set of sample images.

[0085] In an optional embodiment, the enhanced second set of sample images may be normalized, specifically as follows:

[0086]

[0087] Wherein, Iout represents the output image data, Iin is the enhanced second set of sample images, μ represents the mean of the image Iin, which is [123.675, 116.28, 103.53] in this embodiment, and σ represents the variance of the image Iin, which is [58.395, 57.12, 57.375] in this embodiment.

[0088] This embodiment accelerates the convergence of the fall detection model by normalizing the enhanced second set of sample images. It is worth noting that those skilled in the art should understand that, in addition to normalizing the training samples, when using the fall detection model to detect RGB test videos, the second set of images also needs to be normalized to ensure consistency with the training samples, thereby ensuring the accuracy of the fall detection results output by the fall detection model.

[0089] Finally, the key point sample heatmap and the enhanced second set of sample images are key point associated and enhanced. Specifically, the key point sample heatmap obtained from the first set of sample images and the enhanced second set of sample images are randomly flipped left and right, for example, with a probability of 50% to flip the heatmap and the second set of samples simultaneously, thereby further enhancing the training samples.

[0090] At this point, the acquisition of training samples is completed.

[0091] The fall detection model is trained based on the obtained training samples, specifically including:

[0092] First, the RGB sample video is input into the fall detection model and a second detection result is output.

[0093] In this embodiment, the fall detection model is trained based on the RGB sample video using the enhanced training samples obtained in the above embodiment. For example, the enhanced key point sample heat map, key point sample data tensor, enhanced second set of sample images, and enhanced data for key point association are input into the fall detection model for training. Specifically, a first three-dimensional convolutional network is used to process the key point sample data tensor to obtain a first training result tensor, and a second three-dimensional convolutional network is used to process the enhanced second set of sample images to obtain a second training result tensor. The first and second training result tensors are concatenated and input into a fully connected network, which then performs classification and outputs a second detection result.

[0094] Then, a loss function is used to evaluate the second detection result and adjust the parameters of the fall detection model.

[0095] In this embodiment, based on the labeling of the RGB sample video, cross entropy is used as the loss function of the fall detection model. The loss function is:

[0096]

[0097] Where l is the label of the RGB sample video, and x is the second detection result output by the fall detection model. The gradient of each weight in the fall detection model network is calculated based on the loss function, and the parameters of the fall detection model are updated using gradient descent. This means that the parameters of the fall detection model are adjusted using the loss function.

[0098] Corresponding to the fall detection method provided in the above embodiment, an embodiment of the present application also provides a fall detection device for implementing the above fall detection method. Since the fall detection device provided in the embodiment of the present application corresponds to the fall detection method provided in the above embodiments, the previous implementation is also applicable to the LCD screen testing method provided in this embodiment and will not be described in detail in this embodiment.

[0099] like Figure 2 As shown, an embodiment of the present application further provides a fall detection device for implementing the above-mentioned fall detection method, comprising a communication unit, a fall detection model, and a controller, wherein the controller is configured to:

[0100] Using the communication unit to receive the RGB video to be tested;

[0101] Using the fall detection model to detect the RGB video to be tested and outputting a first detection result, including: using a key point detection model to perform key point detection on a first group of images and generate a key point heat map, where the first group of images are images extracted from the RGB video to be tested according to a first preset interval; generating a key point data tensor according to the key point heat map, using a first three-dimensional convolutional network to perform data processing on the key point data tensor and output a first result tensor; using a fully connected network to perform data processing on the first result tensor and output a first detection result;

[0102] The first detection result is compared with a preset fall detection threshold and a fall detection result is output.

[0103] This embodiment uses a fall detection model to detect and output detection results based on RGB video sequences, uses a preset detection threshold to compare the detection results, and outputs the fall detection results. Specifically, the key point detection model of the fall detection model completes the extraction of all key points and generates a key point heat map at one time, implements correlation analysis between multiple frames of the RGB video sequence through a first three-dimensional convolutional network, and outputs the detection results through a fully connected network, thereby remedying the problems existing in the existing technology, effectively improving the accuracy of fall detection, and having practical application value.

[0104] In an optional embodiment, the fall detection model further includes a second three-dimensional convolutional network, and the controller uses the fall detection model to detect the RGB video to be tested and output a first detection result, which is further configured as follows:

[0105] Performing key point detection on a first set of images using a key point detection model and generating a key point heat map, wherein the first set of images is images extracted from the RGB video to be tested according to a first preset interval;

[0106] Generate a key point data tensor according to the key point heat map, perform data processing on the key point data tensor using a first three-dimensional convolutional network and output a first result tensor;

[0107] Performing data processing on a second set of images using a second three-dimensional convolutional network and outputting a second result tensor, wherein the second set of images is images extracted from a preset time period of the RGB video to be tested according to a second preset interval;

[0108] Concatenate the first result tensor and the second result tensor and use the concatenated result as the updated first result tensor;

[0109] Use a fully connected network to perform data processing on the first result tensor and output a first detection result.

[0110] In the embodiment, on the basis of the key point detection and the generation of the key point heat map of the equally spaced extracted images of the RGB video to be tested, the image data of the original video of the RGB video to be tested is added, the key point heat map and the original video image of the complete RGB video to be tested are used to represent the RGB video to be tested, the detection accuracy of the fall event is further improved, and the problem of detection confusion is effectively avoided.

[0111] Another embodiment of the application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement: detecting a received RGB video to be tested using a fall detection model and outputting a first detection result, including: using a key point detection model to detect key points of a first group of images and generate a key point heat map, the first group of images being images extracted from the RGB video to be tested at a first preset interval; generating a key point data tensor according to the key point heat map, using a first three-dimensional convolution network to process the key point data tensor and output a first result tensor; using a fully connected network to process the first result tensor and output a first detection result; comparing the first detection result with a pre-set fall detection threshold and outputting a fall detection result.

[0112] In actual application, the computer readable storage medium can adopt any combination of one or more computer readable media. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples (non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiment, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus.

[0113] The computer readable signal medium can include a data signal propagated in a baseband or as a part of a carrier wave, in which a computer readable program code is borne. Such a propagated data signal can take multiple forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination thereof. The computer readable signal medium can also be any computer readable medium that is not a computer readable storage medium and that can send, propagate or transmit a program for use by or in connection with an instruction execution system, device or apparatus.

[0114] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0115] Computer program code for performing the operations of the present invention may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, Python, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0116] like Figure 4 As shown, a structural diagram of a computer device provided by another embodiment of the present invention. Figure 4 The computer device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0117] like Figure 4 As shown, computer device 12 is implemented as a general-purpose computing device. Components of computer device 12 may include, but are not limited to, one or more processors or processing units 16, system memory 28, and a bus 18 that connects various system components (including system memory 28 and processing unit 16).

[0118] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. Examples of these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0119] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0120] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be configured to read and write non-removable, non-volatile magnetic media ( Figure 4 Not shown, often called a "hard drive"). Although Figure 4 Not shown, a magnetic disk drive for reading and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of the present invention.

[0121] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment. Program modules 42 generally implement the functions and / or methods of the embodiments described herein.

[0122] The computer device 12 may also communicate with one or more external devices 14 (e.g., a keyboard, a pointing device, a display 24, etc.), one or more devices that enable a user to interact with the computer device 12, and / or any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may occur through an input / output (I / O) interface 22. Furthermore, the computer device 12 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 20. Figure 4 As shown, the network adapter 20 communicates with the other modules of the computer device 12 via the bus 18. Figure 4 Not shown, other hardware and / or software modules may be used in conjunction with computer device 12, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0123] The processor unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing a fall detection method provided by an embodiment of the present invention.

[0124] In response to the current existing problems, the present invention develops a fall detection method, a fall detection device, a computer device and a medium. Based on the RGB video sequence, the fall detection model is used to detect and output the detection results, and the detection results are compared using a preset detection threshold and the fall detection results are output. Specifically, the key point detection model of the fall detection model is used to complete the extraction of all key points at one time and generate a key point heat map, the first three-dimensional convolutional network is used to realize the correlation analysis between multiple frames of the RGB video sequence, and the detection results are output through a fully connected network, thereby compensating for the problems existing in the existing technology, effectively improving the accuracy of fall detection, and having practical application value.

[0125] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.

Claims

1. A fall detection method, characterized in that: include: Using a fall detection model to detect the received RGB video to be tested and outputting a first detection result, including: using a key point detection model to perform key point detection on a first group of images and generate a key point heat map, the first group of images are images extracted from the RGB video to be tested according to a first preset interval, and the key point heat map is used to characterize the RGB video to be tested; generating a key point data tensor according to the key point heat map, using a first three-dimensional convolutional network to perform data processing on the key point data tensor and output a first result tensor, the first result tensor is the result of the first three-dimensional convolutional network performing correlation analysis on multiple frames of the RGB video to be tested and outputting the first result tensor; using a second three-dimensional convolutional network to perform data processing on a second group of images and output a second result tensor, the second group of images are images extracted from a preset time period of the RGB video to be tested according to a second preset interval; splicing the first result tensor and the second result tensor and using the splicing result as an updated first result tensor; using a fully connected network to perform data processing on the first result tensor and output a first detection result; The first detection result is compared with a preset fall detection threshold and a fall detection result is output.

2. The fall detection method according to claim 1, wherein: Before using the fall detection model to detect the received RGB video to be tested and outputting the first detection result, the fall detection method further includes: Training the fall detection model using the training sample includes: inputting the RGB sample video into the fall detection model and outputting a second detection result, evaluating the second detection result using a loss function and adjusting parameters of the fall detection model.

3. The fall detection method according to claim 2, wherein: The loss function is: Wherein, l is the label of the RGB sample video, and x is the second detection result output by the fall detection model.

4. The fall detection method according to claim 2, wherein: Before using the training samples to train the fall detection model, the fall detection method further includes: Obtaining the training samples includes: generating a first group of sample images and a second group of sample images based on the RGB sample video, performing key point detection on the first group of sample images using the key point detection model and generating a key point sample heat map, generating a key point sample data tensor based on the key point sample heat map, performing data enhancement on the second group of sample images, and associating and enhancing the key point sample heat map and the enhanced second group of sample images.

5. The fall detection method according to claim 4, characterized in that: The performing key point detection on the first group of sample images using the key point detection model and generating a key point sample heat map, and generating a key point sample data tensor based on the key point sample heat map further comprises: performing key point detection on each sample image of the first group of sample images to output each key point sample, deleting one key point in the key point sample with a preset probability to enhance the key point sample, generating the key point sample heat map based on the enhanced key point sample, and then generating the key point sample data tensor based on the key point sample heat map; The performing data enhancement on the second set of sample images further comprises: performing image color enhancement on each sample image of the second set of sample images, and performing random excision processing on the second set of sample images with enhanced image colors to enhance the second set of sample images; The performing key point associating and enhancing on the key point sample heat map and the enhanced second group of sample images further includes: performing random left-right flipping processing on the key point sample heat map and the enhanced second group of sample images.

6. The fall detection method according to claim 1, wherein: The first group of images is images extracted from the received RGB video to be tested according to a first preset interval and further includes: Dividing the RGB video to be tested into multiple videos to be tested at first intervals according to the first preset intervals; Randomly extracting a frame of image from each first interval video to be tested to generate the first group of images; and / or The using of the key point detection model to perform key point detection on the first group of images and generate a key point heat map further includes: downsampling the key point heat map.

7. The fall detection method according to claim 1, wherein: The key point detection model is openpose or hrnet; and / or The first three-dimensional convolutional network is one of I3D, TSM, and SlowOnly, the second three-dimensional convolutional network is one of I3D, TSM, and SlowOnly, the first three-dimensional convolutional network and the second three-dimensional convolutional network are the same network or different networks, and when the first three-dimensional convolutional network and the second three-dimensional convolutional network are the same network, the network weights are different.

8. A fall detection device, characterized in that: The system comprises a communication unit, a fall detection model and a controller, wherein the controller is configured as follows: Using the communication unit to receive the RGB video to be tested; Using the fall detection model to detect the RGB video to be tested and outputting a first detection result, including: using a key point detection model to perform key point detection on a first group of images and generate a key point heat map, the first group of images are images extracted from the RGB video to be tested according to a first preset interval, and the key point heat map is used to characterize the RGB video to be tested; generating a key point data tensor according to the key point heat map, using a first three-dimensional convolutional network to perform data processing on the key point data tensor and output a first result tensor, the first result tensor is the result of the first three-dimensional convolutional network performing correlation analysis on multiple frames of the RGB video to be tested and outputting the first result tensor; using a second three-dimensional convolutional network to perform data processing on a second group of images and output a second result tensor, the second group of images are images extracted from a preset time period of the RGB video to be tested according to a second preset interval; splicing the first result tensor and the second result tensor and using the splicing result as an updated first result tensor; using a fully connected network to perform data processing on the first result tensor and output a first detection result; The first detection result is compared with a preset fall detection threshold and a fall detection result is output.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

Images