Method for training a fall detection model, fall detection method and related devices

Abstract

The embodiment of the application relates to the technical field of intelligent monitoring, and discloses a method for training a fall detection model, a fall detection method and related devices, a plurality of action samples are acquired, the plurality of action samples are taken as a training set, difference calculation is performed on each training sample in the training set, and a plurality of difference samples are obtained. Each difference sample and the corresponding training sample are spliced in channels to obtain a plurality of spliced samples. Finally, the plurality of spliced samples are used to iteratively train a pre-set neural network until the neural network converges, and a fall detection model is obtained. In the above manner, the neural network can refer to difference information in a difference image and ignore the interference of static heat sources. On the other hand, the change of dynamic data in N frames of multi-channel images is focused on learning. Therefore, the fall detection model obtained through training has strong anti-interference capability, and fall detection is more accurate.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of intelligent monitoring, in particular to a method for training a fall detection model, a fall detection method and related devices. BACKGROUND

[0002] Human fall detection technology helps to discover children and the elderly falling in time, and avoid more serious consequences. Current fall detection methods are mostly for specific scenarios, such as sports scenarios, suitable for young people, or through wearable devices to determine whether a fall has occurred. On the one hand, these methods have poor detection effects for slow-moving elderly people. On the other hand, wearable devices need to be worn, which is not convenient and is not suitable for special scenarios such as bathing. In places involving privacy, such as bathrooms and bedrooms, it is necessary to balance the need to protect privacy and guardianship, and low-resolution infrared devices are often used instead of high-resolution RGB cameras as guardianship tools. Infrared thermal imaging cameras only collect temperature information of the target, and the information resolution is low, which can protect the privacy of users.

[0003] In some solutions known to the present inventors, high-frame-rate, low-resolution infrared thermal imaging devices are used to obtain temperature distribution data in a scene for capturing and analyzing whether a human body has fallen. For low-resolution infrared data, the background environment is an important factor affecting the accuracy of fall detection. When the temperature of a static interference heat source in the background and the target human body is similar, it will be difficult to distinguish between objects and human bodies, thereby enhancing the anti-interference ability of the fall detection model, which is crucial for improving detection accuracy. SUMMARY

[0004] Therefore, some embodiments of the present application provide a method for training a fall detection model, a fall detection method and related devices, so that the fall detection model obtained by training has strong anti-interference ability, and the fall detection is more accurate.

[0005] In a first aspect, some embodiments of the present application provide a method for training a fall detection model, comprising: obtaining a plurality of action samples, each action sample comprising N infrared images, and each action sample being labeled with a true class label reflecting whether a fall has occurred;

[0006] The plurality of action samples are used as a training set, and difference calculation is performed on each training sample in the training set to obtain a plurality of difference samples;

[0007] Each difference sample and the corresponding training sample are channel spliced to obtain a plurality of spliced samples;

[0008] The plurality of spliced samples are used to iteratively train a pre-set neural network until the neural network converges, and a fall detection model is obtained.

[0009] In some embodiments, the foregoing difference calculation is performed on each training sample in the training set to obtain a plurality of difference samples, including:

[0010] For the i-th frame of infrared image in the training sample, if i < 3, the i-th frame of infrared image is calculated with the first frame of infrared image, and if i ≥ 3, the i-th frame of infrared image is calculated with the i-2-th frame of infrared image, to obtain N difference images;

[0011] The negative difference in the N difference images is set to 0, and the positive difference is retained, to obtain N positive difference images; and the positive difference in the N difference images is set to 0, and the negative difference is retained, to obtain N negative difference images;

[0012] After the N positive difference images and the N negative difference images are normalized, channel splicing is performed to obtain difference samples.

[0013] In some embodiments, the foregoing channel splicing is performed on each difference sample and the corresponding training sample to obtain a plurality of spliced samples, including:

[0014] The training sample is normalized;

[0015] The normalized training sample and the difference sample are spliced in the channel to obtain a spliced sample.

[0016] In some embodiments, the foregoing neural network includes a feature extraction network, a recurrent neural network, and an Attention layer connected in sequence;

[0017] The feature extraction network is configured to extract features from the input spliced sample and output N first feature maps;

[0018] The recurrent neural network is configured to extract features frame by frame from the input N first feature maps and output N fusion feature maps;

[0019] The Attention layer is configured to calculate weights from the input N fusion feature maps and output an attention vector, which is input into a SoftMax layer to calculate the fall probability.

[0020] In some embodiments, the foregoing feature extraction network includes a plurality of reverse residual modules, at least one depth separable convolution module, and at least one convolution layer, and for the plurality of reverse residual modules, the at least one depth separable convolution module, or the at least one convolution layer, the number of output channels is configured to be no more than 128, the channel expansion multiple is configured to be no more than 3, and the number of residual connection layers is configured to be no more than 2.

[0021] In some embodiments, the method further includes:

[0022] Randomly insert a plurality of interference color blocks in the background of each frame of infrared image of the action sample to obtain an interference sample;

[0023] The plurality of action samples are taken as a training set, including:

[0024] The plurality of action samples and the plurality of interference samples are taken as a training set.

[0025] In some embodiments, before the step of randomly inserting a plurality of interference color blocks in the background of each frame of infrared image of the action sample to obtain an interference sample, the method further includes:

[0026] Data augmentation processing is performed on the plurality of action samples to obtain an expanded plurality of augmented samples;

[0027] Randomly inserting a plurality of interference color blocks in the background of each frame of infrared image of the action sample to obtain an interference sample includes:

[0028] Randomly inserting a plurality of interference color blocks in the background of each frame of infrared image of the augmented sample to obtain an interference sample.

[0029] In a second aspect, some embodiments of the present application provide a fall detection method, including:

[0030] Obtaining a test action sample;

[0031] Inputting the test action sample into the fall detection model to output a category of whether to fall, wherein the fall detection model is trained by the method for training a fall detection model according to any one of claims 1-7.

[0032] In a third aspect, some embodiments of the present application provide an electronic device, including:

[0033] At least one processor, and

[0034] A memory in communication connection with the at least one processor, wherein

[0035] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of the first aspect or the second aspect.

[0036] In a fourth aspect, some embodiments of the present application provide a computer readable storage medium, including a computer readable storage medium, the computer readable storage medium stores computer executable instructions, and the computer executable instructions are used to make a computer device execute the method of the first aspect or the second aspect.

[0037] The method for training a fall detection model provided by the embodiments of the present application is different from the prior art. The method comprises the following steps: acquiring a plurality of action samples, taking the plurality of action samples as a training set, performing difference calculation on each training sample in the training set to obtain a plurality of difference samples. Each difference sample and the corresponding training sample are spliced in a channel to obtain a plurality of spliced samples. Finally, the plurality of spliced samples are used to iteratively train a pre-set neural network until the neural network converges, and a fall detection model is obtained. Each action sample comprises N infrared images, and each action sample is labeled with a real class label reflecting whether a fall occurs.

[0038] In this embodiment, the difference sample and the training sample are combined into a multi-channel spliced sample, so that the spliced sample comprises N multi-channel images, and each image comprises an infrared image and a difference image. Thus, the spliced sample is input into the neural network for training, so that the neural network can refer to the difference information in the difference image and ignore the interference of static heat sources; on the other hand, the change of dynamic data in the N multi-channel images is focused on learning. Therefore, the fall detection model obtained by training has strong anti-interference ability, and the fall detection is more accurate. BRIEF DESCRIPTION OF DRAWINGS

[0039] One or more embodiments are illustrated by way of example with reference to the drawings, which are not limiting of the embodiments and which are merely meant to explain the embodiments. The reference numbers in the drawings indicate like reference numbers for like elements, unless otherwise indicated. The drawings in the figures are not necessarily to scale, with emphasis instead being placed upon illustrating the principles of the embodiments.

[0040] Figure 1 A schematic diagram of an infrared image in some embodiments of the present application;

[0041] Figure 2 A schematic diagram of an application scenario of a fall detection system in some embodiments of the present application;

[0042] Figure 3 A schematic diagram of the structure of an electronic device in some embodiments of the present application;

[0043] Figure 4 A flowchart of a method for training a fall detection model in some embodiments of the present application;

[0044] Figure 5 A schematic diagram of two infrared images before and after a fall in some embodiments of the present application;

[0045] Figure 6 A flowchart of a fall detection method in some embodiments of the present application. DETAILED DESCRIPTION

[0046] The application will be described in detail below with specific embodiments. The following examples will help those skilled in the art to further understand the application, but do not limit the application in any form. It should be noted that for those skilled in the art, without departing from the concept of the application, a number of modifications and improvements can be made. These are within the scope of protection of the application.

[0047] In order to make the purpose, technical scheme and advantages of the application more clear, the application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described here are only used to explain the application and do not limit the application.

[0048] It should be noted that the various features in the embodiments of the application can be combined with each other without conflict, and are within the scope of protection of the application. In addition, although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order from the module division in the device or the order in the flowchart. In addition, the "first", "second", "third" and the like used herein do not limit the data and execution order, but only distinguish the same items or similar items with basically the same function and effect.

[0049] Unless otherwise defined, all technical and scientific terms used in the specification have the same meaning as understood by those skilled in the art to which the application belongs. The terms used in the specification of the application are only for the purpose of describing the specific embodiments and are not used to limit the application. The term "and / or" used in the specification includes any and all combinations of one or more related listed items.

[0050] In addition, the technical features involved in each embodiment of the application described below can be combined with each other as long as there is no conflict.

[0051] In order to facilitate the understanding of the method provided by the embodiments of the application, first, the terms involved in the embodiments of the application are introduced:

[0052] (1) Infrared thermal imaging technology

[0053] Infrared thermal imaging technology uses photoelectric technology to detect the infrared signal of a specific wave band of the thermal radiation of an object, converts the signal into an infrared imaging image that can be distinguished by human vision, and can further calculate the temperature value. The value of each pixel point in the infrared imaging image is the temperature value of the object in the corresponding world coordinate system. Thus, the infrared thermal imaging technology enables humans to overcome visual barriers, so that people can "see" the temperature distribution of the surface of the object.

[0054] For example, an infrared camera is installed on the ceiling in the room to capture and collect infrared videos of the human body in the room. It can be understood that the infrared video includes a plurality of continuous infrared images. Please refer to Figure 1 , Figure 1 A frame of infrared image collected when an infrared camera with low resolution (24*32) is installed on the ceiling above the ground. Figure 1 The high-light area in the middle is the temperature distribution of the human body lying on the ground, and the temperature of the human body is higher than that of other background areas.

[0055] Before introducing the embodiments of the present application, the inventor of the present application will first introduce the fall detection method known to the inventor, so that the subsequent embodiments of the present application can be easily understood.

[0056] In some schemes, an infrared image fall detection method based on improved Alphapose is proposed, including the following steps: collecting an infrared video frame as input, and performing data preprocessing on the infrared video frame. The height of the human target box in the previous frame is taken as a reference object to calculate the relative moving speed of the hip joint key point and the transformation of the human width-height ratio, and compared with the key point information and the posture category to output a fall judgment signal. After receiving the fall judgment signal, the human posture in the subsequent frame is continuously detected in combination with the posture classification, and a fall judgment result is output.

[0057] In this scheme, the limbs nodes of the human body need to be clearly seen in the infrared image to label the human key points, that is, the resolution requirement of the infrared image data is high, and it cannot be adapted to places involving privacy (such as bathroom or bedroom, etc.). Secondly, it requires that no heat source other than human beings appears in the middle of the scene, which further narrows down the applicable scene. In addition, the model used has a large number of parameters and cannot run on devices with low computing power.

[0058] It can be seen that in order to apply fall detection to actual scenes, a fall detection model with anti-interference ability needs to be trained based on low-resolution infrared image data, so that the fall detection model can accurately detect falls on the basis of considering privacy. In addition, the fall detection model has the characteristics of light weight and can be applied to devices with low computing power.

[0059] To solve the above problems, the embodiment of the present application provides a method for training a fall detection model, a fall detection method and related devices. A plurality of action samples are obtained, the plurality of action samples are taken as a training set, differential calculation is performed on each training sample in the training set to obtain a plurality of differential samples. Each differential sample and the corresponding training sample are spliced in the channel to obtain a plurality of spliced samples. Finally, the plurality of spliced samples are used to iteratively train a pre-set neural network until the neural network converges, and a fall detection model is obtained. Each action sample includes N infrared images, and each action sample is labeled with a real class label reflecting whether a fall occurs.

[0060] In this embodiment, the differential sample and the training sample are combined into a multi-channel spliced sample, so that the spliced sample includes N multi-channel images, and each image includes an infrared image and a differential image. Thus, the spliced sample is input into the neural network for training, so that the neural network can refer to the differential information in the differential image and ignore the interference of static heat sources; on the other hand, the change of dynamic data in the N multi-channel images is focused on learning. Therefore, the fall detection model obtained by training has strong anti-interference ability, and the fall detection is more accurate.

[0061] The following describes an exemplary application of the electronic device provided by the embodiment of the present application for training a fall detection model or for fall detection. The electronic device provided by the embodiment of the present application can be a server, for example, a server deployed in the cloud. The electronic device provided by some embodiments of the present application can be a notebook computer, a desktop computer or a mobile device, etc. In some embodiments, the electronic device can be a monitoring device with an infrared camera.

[0062] As an example, refer to Figure 2 , Figure 2 is an application scenario diagram of the fall detection system provided by the embodiment of the present application. The terminal 10 is connected to the server 20 through a network, wherein the network can be a wide area network or a local area network, or a combination of the two.

[0063] The terminal 10 can be used to obtain training data and build a neural network. For example, a person skilled in the art downloads prepared training data on the terminal, and builds the network structure of the neural network. The training data includes a plurality of action samples. It can be understood that the terminal 10 can also be used to obtain test action samples. For example, the infrared camera sends the test action samples collected to the terminal 10, so that the terminal 10 obtains the test action samples. In some embodiments, the terminal 10 can be integrated with the infrared camera.

[0064] In some embodiments, the terminal 10 locally performs the method for training the fall detection model provided by the embodiments of the present application to complete the training of the designed neural network using the training data, determine the final model parameters, and thus the neural network is configured with the final model parameters, and the fall detection model is obtained. In some embodiments, the terminal 10 can also send the training data and the constructed neural network stored on the terminal by the person skilled in the art to the server 20 through the network, the server 20 receives the training data and the neural network, trains the neural network using the training data, determines the final model parameters, and then sends the final model parameters to the terminal 10, the terminal 10 saves the final model parameters, so that the neural network is configured with the final model parameters, and the fall detection model is obtained.

[0065] The structure of the electronic device in the embodiments of the present application is described below, Figure 3 is a structural schematic diagram of the electronic device 500 in the embodiments of the present application, and the electronic device 500 includes at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in the electronic device 500 are coupled together through a bus system 540. It can be understood that the bus system 540 is used to realize the connection and communication between the components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for the purpose of clear illustration, all the buses are marked as the bus system 540 in Figure 3 .

[0066] The processor 510 can be an integrated circuit chip with a signal processing capability, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0067] The user interface 530 includes one or more output devices 531 that enable the presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 530 also includes one or more input devices 532 that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0068] The memory 550 includes volatile memory or nonvolatile memory, and can include both volatile and nonvolatile memory. Among them, the nonvolatile memory can be a read-only memory (ROM), and the volatile memory can be a random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory. The memory 550 can optionally include one or more storage devices that are physically separate from the processor 510.

[0069] In some embodiments, the memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, which are exemplarily described below.

[0070] The operating system 551 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, and the like, for implementing various basic services and processing hardware-based tasks.

[0071] The network communication module 552 is used to communicate with other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including Bluetooth, wireless compatibility authentication (WiFi), and universal serial bus (USB), and the like.

[0072] The display module 553 is used to enable the presentation of information via one or more output devices 531 associated with the user interface 530 (e.g., a display screen, a speaker, and the like) (e.g., a user interface for operating peripheral devices and displaying content and information).

[0073] The input processing module 554 is used to detect and interpret one or more user inputs or interactions from one or more input devices 532.

[0074] According to the foregoing, it can be understood that the method for training a fall detection model provided by the embodiments of the present application can be implemented by various types of electronic devices with processing capabilities, such as by the processor of the electronic device or by other devices with computing processing capabilities. Other devices with computing processing capabilities can be smart terminals or servers in communication connection with the electronic device, and the like.

[0075] The method for training a fall detection model provided by the embodiments of the present application is described below in conjunction with exemplary applications and implementations of the electronic device provided by the embodiments of the present application. Please refer to Figure 4 , Figure 4is a flowchart of a method for training a fall detection model provided by an embodiment of the present application. It can be understood that the execution subject of the training method can be one or more processors of an electronic device.

[0076] Please refer again to Figure 4 The method S100 can specifically include the following steps.

[0077] S10: Obtain a plurality of action samples.

[0078] Each action sample includes N infrared images, and each action sample is labeled with a real class label reflecting whether a fall occurs.

[0079] It can be understood that the action sample is an infrared video obtained by an infrared camera collecting human action or a deformation thereof. In some embodiments, the N infrared images are consecutive images or interval images in the infrared video. Since the video shooting sampling rate is high, the action sample can reflect the human action feature and the feature of whether a fall occurs.

[0080] Each action sample is labeled with a real class label reflecting whether a fall occurs. In some embodiments, a hot encoding label can be used to label the label, for example, 1 represents a fall, and 0 represents a non-fall.

[0081] In some embodiments, the infrared camera is used to collect a plurality of initial video data, wherein some of the initial video data includes a video of human fall action, and some of the initial video data includes a video of human non-fall action. After the infrared camera collects the initial video data, the initial video data is sent to an electronic device (such as a computer or a server).

[0082] In some embodiments, the frame number M of the initial video data is much larger than the required N frames of the action sample. In the initial video data including human fall action, the frame number of the standing moment before the fall and the frame number of the lying moment are labeled. In the initial video data including human non-fall action, the starting frame number of the non-fall action is labeled.

[0083] In some embodiments, for the initial video data including human fall action, a sliding window N is used for sliding frame-by-frame sampling, sliding 1 frame each time, to obtain a plurality of samples. It can be understood that the samples including the standing moment before the fall frame and the lying moment frame are retained as fall action samples. If the i-th frame in the initial video data is the frame number of the standing moment before the fall, and the i+1-th frame is the frame number of the lying moment, then after sliding frame-by-frame sampling, the sample containing the i-th frame and the i+1-th frame are both fall action samples. Thus, one M-frame initial video data can collect N-1 fall action samples.

[0084] In this embodiment, the sliding window N is used for frame-by-frame sliding sampling, in order to retain the instantaneous changes of the falling action for subsequent neural network learning, and because the initial video data reflecting the falling is relatively small, frame-by-frame sampling can obtain more falling samples.

[0085] In some embodiments, for the initial video data including non-falling actions (other actions), the sliding window N is used for sliding interval frame sampling, and 2 frames or 3 frames are slid each time to obtain a plurality of samples. It can be understood that the samples including non-falling action frames are retained as non-falling action samples.

[0086] In this embodiment, considering that there are many types of non-falling actions, interval frame sliding sampling using the sliding window N can effectively reduce the sample amount of each non-falling action without affecting the continuity of the action, so that the non-falling action samples are not concentrated in a certain action and are more abundant.

[0087] In some embodiments, after obtaining a plurality of falling action samples and a plurality of non-falling action samples, 60% of the plurality of falling action samples and 30% of the plurality of non-falling action samples are randomly extracted as the aforementioned several action samples.

[0088] After the electronic device obtains the several action samples, the action samples can be labeled by manual or label tool, so that each action sample is labeled with a real class label.

[0089] S20: Use the several action samples as a training set, and perform difference calculation on each training sample in the training set to obtain several difference samples.

[0090] In this embodiment, the several action samples are used as a training set, and the training samples in the training set are action samples.

[0091] In some embodiments, the method S100 further includes:

[0092] S50: Randomly insert a plurality of interference color blocks in the background of each frame of infrared image of the action sample to obtain an interference sample.

[0093] The interference color block is an equivalent connected color block. For each frame of infrared image, the plurality of interference color blocks inserted randomly have different shapes, sizes and pixel values. For the plurality of infrared images in the action sample, the number of interference color blocks inserted in each frame of infrared image is also not completely the same.

[0094] Please refer to Figure 5 , Figure 5 (a) in FIG. 1 is a schematic diagram of the action sample in which two different interference color blocks are inserted in the standing moment frame before falling, Figure 5(b) is a schematic diagram of the lying down moment frame in the action sample after inserting 2 different interference color blocks. It can be understood that the 2 interference color blocks in (a) are the same as the 2 interference color blocks in (b).

[0095] In some embodiments, an existing hot spot segmentation algorithm can be used to determine the position of the human body in each frame of the infrared image of the action sample. In the non-human body area of each frame of the infrared image, a plurality of positions are randomly selected, and a plurality of interference color blocks are randomly inserted to obtain interference samples.

[0096] In this embodiment, the aforementioned "using a plurality of action samples as a training set" includes using the plurality of action samples and a plurality of interference samples as a training set.

[0097] Using the plurality of action samples and the interference samples with inserted color value blocks as a training set can enhance the training set, enable the model to focus on learning the data changes of the dynamic human body and ignore various interferences in the background during the training process, and enhance the anti-interference ability of the model.

[0098] In some embodiments, before the aforementioned step S50, the method S100 further includes:

[0099] S60: performing data enhancement processing on the plurality of action samples to obtain a plurality of expanded enhanced samples.

[0100] In some embodiments, data enhancement refers to processing more representations from original data without substantially increasing the data, improving the quantity and quality of the original data, and approaching the value generated by a larger amount of data. Thus, by performing data enhancement processing on the plurality of training samples and the plurality of action samples, the size of the enhanced data set is larger and more abundant.

[0101] In some embodiments, data enhancement includes rotation, X-axis mirroring, and / or Y-axis mirroring. In this embodiment, any one of the plurality of action samples is randomly processed by rotation, X-axis mirroring, and / or Y-axis mirroring, which can effectively obtain abundant samples.

[0102] In this embodiment, the aforementioned step S50 specifically includes randomly inserting a plurality of interference color blocks in the background of each frame of the infrared image of the enhanced sample to obtain interference samples.

[0103] It can be understood that by randomly inserting a plurality of interference color blocks in the background of each frame of the infrared image of the enhanced sample, the number of interference samples can be increased, and the interference samples can be more abundant. Thus, using the plurality of action samples and the interference samples with inserted color value blocks as a training set can enhance the anti-interference ability of the model.

[0104] Differential calculation is performed on each training sample in the training set to obtain a plurality of differential samples. The differential calculation refers to pixel value subtraction operation on corresponding pixel points of two images. The differential calculation can be performed by using the following formula:

[0105] D (i,j) =P (i,j) -H (i,j)

[0106] wherein P (i,j) is the pixel value of the pixel point at the i-th row and the j-th column in an infrared image in the training sample, H (i,j) is the pixel value of the pixel point at the i-th row and the j-th column in an infrared image in the training sample, and D (i,j) is the pixel value (i.e., differential value) of the pixel point at the i-th row and the j-th column in the differential image.

[0107] It can be understood that the differential value of the pixel region in the two images of the training sample which does not overlap is large, and the differential value of the pixel region which overlaps is close to 0. Since the position of the static heat source in each infrared image in the training sample does not change, the differential calculation can eliminate the static heat source in the training sample.

[0108] In some embodiments, the foregoing step S20 specifically comprises:

[0109] S21: for the i-th infrared image in the training sample, if i < 3, differential calculation is performed on the i-th infrared image and the first infrared image, and if i ≥ 3, differential calculation is performed on the i-th infrared image and the (i-2)-th infrared image to obtain N differential images.

[0110] S22: the negative differential in the N differential images is set to 0, and the positive differential is retained to obtain N positive differential images; and the positive differential in the N differential images is set to 0, and the negative differential is retained to obtain N negative differential images.

[0111] S23: after the N positive differential images and the N negative differential images are normalized respectively, channel splicing is performed to obtain differential samples.

[0112] In this embodiment, the infrared images with frame numbers less than 3 in the training sample are all subjected to differential calculation with the first infrared image, and the infrared images with frame numbers greater than or equal to 3 are subjected to equal-interval differential calculation with the second infrared image in front of them to obtain N differential images.

[0113] In an embodiment, the N differential images can be copied into two parts, the negative differential in the first part of the N differential images is set to 0, and the positive differential is retained to obtain N positive differential images D 0+. Set the positive difference in the second N difference images to 0, and keep the negative difference, to obtain N negative difference images D 0- . Wherein, the negative difference refers to the difference value that is negative in the difference image, and the positive difference refers to the difference value that is positive in the difference image.

[0114] Then, the N positive difference images D 0+ and the N negative difference images D 0- are respectively normalized, that is, the difference values in the difference images are unified to a fixed range, so that the neural network training can converge.

[0115] In some embodiments, for any one positive difference image or negative difference image, the difference value is normalized to the range of 0-255 by using the following formula:

[0116]

[0117] Wherein, V min is the minimum value in the image, V max is the maximum value in the image, old v is the v-th original value in the image, and new v is the v-th new value obtained after normalization processing in the image.

[0118] The N normalized positive difference images DP 0+ and the N normalized negative difference images DP 0- are channel spliced, for example, the data dimension of DP 0+ is N x 24 x 32, the data dimension of DP 0- is N x 24 x 32, and the channel splicing is performed to obtain the dimension of the difference sample DP as N x 24 x 32 x 2. That is, the difference sample includes N positive difference images and N negative difference images.

[0119] In this embodiment, by performing equal-interval difference calculation and channel splicing of positive difference images and negative difference images, on the one hand, the static heat source in the training sample can be eliminated, and on the other hand, for a dynamic human heat source data, the difference sample includes N positive difference images and N negative difference images, so that the subsequent neural network can learn the falling behavior of the human body by comprehensively using the positive difference images and the negative difference images, thereby making the falling detection model obtained by training more accurate.

[0120] S30: Channel splicing each difference sample and the corresponding training sample to obtain a plurality of spliced samples.

[0121] For example, the differential samples with a dimension of N x 24 x 32 x 2 and the corresponding training samples with a dimension of N x 24 x 32 x 1 are channel spliced to obtain a spliced sample with a dimension of N x 24 x 32 x 3. That is, the spliced sample includes the training sample and the differential sample, and the differential sample includes N positive differential images and N negative differential images.

[0122] In some embodiments, the foregoing step S30 specifically includes:

[0123] S31: performing normalization processing on the training sample.

[0124] S32: channel splicing the normalized training sample and the differential sample to obtain a spliced sample.

[0125] The training sample is normalized so that the pixel values in the training sample are unified to a fixed range, and the training sample will not be shocked due to the sudden appearance of unfamiliar or uncommon pixel values during training, which is conducive to the convergence of the neural network during training. In this embodiment, the above formula (1) can be used to perform normalization processing on each frame of infrared image in the training sample.

[0126] By channel splicing the normalized training sample and the differential sample, the spliced sample obtained will not be shocked due to the appearance of unfamiliar or uncommon values, which is conducive to accelerating the convergence of the neural network.

[0127] S40: iteratively training the pre-set neural network using a plurality of spliced samples until the neural network converges to obtain a fall detection model.

[0128] Here, the plurality of spliced samples are used as training data for inputting the neural network to train the pre-set neural network, and the parameters of the neural network are constantly adjusted. Under the constraint of the loss function, the predicted class label output by the neural network will be closer and closer to the true class label. When the loss calculated by the loss function fluctuates within a certain range or reaches a certain value, the neural network converges, and the parameters at the time of convergence are used as model parameters to obtain a fall detection model.

[0129] In this embodiment, by merging the differential sample and the training sample into a multi-channel spliced sample, the spliced sample includes N frames of multi-channel images, and each image includes an infrared image and a differential image. Thus, the spliced sample is input into the neural network for training, so that the neural network can refer to the differential information in the differential image and ignore the interference of static heat sources; on the other hand, the change of dynamic data in the N frames of multi-channel images is focused on learning. Therefore, the fall detection model trained has strong anti-interference ability, and the fall detection is more accurate.

[0130] In some embodiments, the neural network comprises a feature extraction network, a recurrent neural network and an Attention layer connected in sequence.

[0131] The N frames of images in the spliced sample are input into the neural network frame by frame, the images are calculated and processed by the feature extraction network, the output result is input into the recurrent neural network for calculation and processing, and the output result is input into the Attention layer for calculation and processing.

[0132] The feature extraction network is configured to extract features of the input spliced sample and output N first feature maps.

[0133] In some embodiments, the feature extraction network comprises a plurality of inverted residual modules, at least one depth separable convolution module and at least one convolution layer, and for the plurality of inverted residual modules, the at least one depth separable convolution module or the at least one convolution layer, the number of output channels is configured to be no more than 128, the channel expansion multiple is configured to be no more than 3, and the number of residual connection layers is configured to be no more than 2.

[0134] The inverted residual module comprises three convolution layers, the input is first expanded in channel by a 1*1 convolution layer, then a 3*3 depth separable convolution layer is used, and finally a 1*1 pointwise convolution layer is used to compress the channel. The inverted residual module is used to extract features.

[0135] The depth separable convolution module comprises a plurality of depth separable convolution layers, in which one convolution kernel is responsible for one channel, and one channel is only convolved by one convolution kernel.

[0136] It can be understood that the inverted residual module, the depth separable convolution layer and the convolution layer are common components in the field of machine learning, and will not be described in detail here.

[0137] In some embodiments, the structure of the feature extraction network is as shown in the following table:

[0138] Input Operator t c n s 24×32×3 Conv2d - 32 - 2 12×16×32 inverted_residual_block 1 16 1 1 12×16×16 inverted_residual_block 3 16 2 2 6×8×16 inverted_residual_block 3 24 2 1 6×8×32 inverted_residual_block 3 36 2 2 3×4×32 inverted_residual_block 3 64 1 1 3×4×32 Conv2d - 128 - 1 3×4×128 DepthwiseConv2D - - - 1 1×1×128 Conv2d - 56 - 1 1×1×56 Reshape - - - -

[0139] In the table, Input represents the data dimension of the network layer input into the Operator column, for example, 24*32*3 represents height* width*channel number. Operator represents the name of the network layer used, specifically, Conv2d represents a convolution layer, inverted_residual_block represents an inverted residual module, and DepthwiseConv2D represents a depth separable convolution module.

[0140] In the table, parameter t represents the channel expansion multiple, parameter c represents the number of output channels, parameter n represents the number of residual connection layers, and parameter s represents the convolution kernel sliding step number.

[0141] In this embodiment, the number of output channels of each network layer configuration does not exceed 128, the configured channel expansion multiple does not exceed 3, and the configured residual connection layer number does not exceed 2. That is, fewer parameters are used to effectively compress the model and obtain a lightweight fall detection model. Configuring 4 residual modules can further compress the model.

[0142] In addition, the feature extraction network finally outputs a first feature map with a small dimension, such as 1x1x56, which can reduce the input dimension of the subsequent network layer, thereby reducing the parameters.

[0143] The recurrent neural network is used to perform feature fusion extraction on the N first feature maps frame by frame, and outputs N fusion feature maps.

[0144] In some embodiments, the recurrent neural network can be a single-layer 48-unit LSTM network. The N first feature maps are first input into the recurrent neural network for feature extraction, and N one-dimensional fusion feature maps are output. It can be understood that the i-th fusion feature map fuses the feature information of 1 to i frames in the N first feature maps.

[0145] The Attention layer is used to calculate the weight of the input N fusion feature maps, and outputs an attention vector. The attention vector is used as input to the SoftMax layer to calculate the fall probability.

[0146] The N fusion feature maps are input into the Attention layer, and an attention vector that fuses the N fusion feature maps is output. In some embodiments, the attention vector is a 1-dimensional 56-value vector. Specifically, the calculation of the fusion feature map input into the Attention layer is as follows:

[0147]

[0148]

[0149]

[0150] where F 2, is the i-th frame in the N fusion feature maps, w i is the weight of the i-th frame, b i is the intercept, fw is the N weight, and fw i is the i-th weight. fa is the proportion of each weight in fw, and the sum of fa is 1, and f1 is the attention vector. The N fusion feature maps F2 are accumulated according to the weight fa to obtain the attention vector f1.

[0151] Finally, the attention vector is input into the SoftMax layer for classification calculation, and the fall probability is calculated and output.

[0152] In this embodiment, the neural network includes a feature extraction network, a recurrent neural network and an Attention layer connected in sequence, can effectively extract features of the input splicing sample, and improve the accuracy of model classification.

[0153] To sum up, the method for training a fall detection model provided in the embodiments of the present application combines the difference sample and the training sample into a multi-channel splicing sample, so that the splicing sample includes N frames of multi-channel images, and each image includes an infrared image and a difference image. Thus, the splicing sample is input into the neural network for training, so that the neural network can refer to the difference information in the difference image and ignore the interference of static heat sources; on the other hand, the change of dynamic data in the N frames of multi-channel images is focused on learning. Therefore, the fall detection model obtained by training has strong anti-interference ability, and the fall detection is more accurate.

[0154] After the fall detection model is trained by the method for training a fall detection model provided in the embodiments of the present application, the fall detection model can be applied to fall detection. The fall detection method provided in the embodiments of the present application can be implemented by various types of electronic devices with computing processing capability, such as a smart terminal, a server or a monitoring device with an infrared camera.

[0155] The fall detection method provided in the embodiments of the present application will be described below in combination with an exemplary application and implementation of a terminal provided in the embodiments of the present application. Referring to Figure 6 , Figure 6 is a flowchart of the fall detection method provided in the embodiments of the present application. The method S200 includes the following steps:

[0156] S201: Obtain a test action sample.

[0157] S202: Input the test action sample into the fall detection model, and output a category of whether to fall, wherein the fall detection model is trained by any one of the methods for training a fall detection model in the training method embodiments.

[0158] It can be understood that the test action sample is an infrared video collected by an infrared camera in an actual application scenario. The infrared camera sends the test action sample to the terminal, so that the terminal obtains the test action sample.

[0159] The terminal is built-in with a fall detection application program, and the fall detection model is encapsulated in the fall detection application program. The fall detection model is called to perform fall detection on the aforementioned test action sample, and after a series of calculation and processing, a category of whether to fall is output. It can be understood that the fall detection model is trained by the method for training a fall detection model in the above embodiments, and has the same structure and function as the fall detection model in the above embodiments, which will not be described here.

[0160] The embodiments of the present application further provide a computer readable storage medium, for example, a memory including program codes, which can be executed by a processor to complete the method for training the fall detection model or the fall detection method in the above embodiments. For example, the computer readable storage medium can be a Read-Only Memory (ROM), a Random Access Memory (RAM), a Compact Disc Read-Only Memory (CDROM), a magnetic tape, a floppy disk and an optical data storage device, etc.

[0161] The embodiments of the present application further provide a computer program product including one or more program codes stored in a computer readable storage medium. The processor of the electronic device reads the program codes from the computer readable storage medium, and the processor executes the program codes to complete the steps of the method for training the fall detection model or the steps of the fall detection method provided in the above embodiments.

[0162] It should be noted that the apparatus embodiments described above are only schematic, and the units described as separate units can or can not be physically separate, and the units displayed as units can or can not be physical units, i.e., they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments.

[0163] Through the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of software plus a general hardware platform, and of course, can also be implemented by hardware. Those skilled in the art can understand that all or part of the processes in the above embodiments can be completed by a computer program instructing related hardware, and the program can be stored in a computer readable storage medium. When the program is executed, it can include the processes of the above embodiments. The storage medium can be a magnetic disc, an optical disc, a Read-Only Memory (ROM) or a Random Access Memory (RAM), etc.

[0164] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, but not to limit them; under the idea of the present application, the technical features in the above examples or different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes of different aspects of the present application as described above, which are not provided in details for simplicity; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of training a fall detection model, the method comprising: The method comprises the following steps: a plurality of action samples are acquired, each of the action samples comprises N infrared images, and each of the action samples is labeled with a real class label reflecting whether falling occurs; each training sample in the training set is subjected to differential calculation to obtain a plurality of differential samples; each differential sample and a corresponding training sample are channel spliced to obtain a plurality of spliced samples; the pre-set neural network is iteratively trained by using the plurality of spliced samples until the neural network converges, and the falling detection model is obtained; wherein, the differential calculation of each training sample in the training set to obtain a plurality of differential samples comprises: for the i-th infrared image in the training sample, if i < 3, the i-th infrared image is subjected to differential calculation with the first infrared image, if i ≥ 3, the i-th infrared image is subjected to differential calculation with the i-2-th infrared image, and N differential images are obtained; the negative differential in the N differential images is set to 0, and the positive differential is retained to obtain N positive differential images; and the positive differential in the N differential images is set to 0, and the negative differential is retained to obtain N negative differential images; after the N positive differential images and the N negative differential images are subjected to normalization processing respectively, the channel splicing is performed to obtain the differential sample.

2. The method of claim 1, wherein, the channel splicing of each differential sample and a corresponding training sample to obtain a plurality of spliced samples comprises: the training sample is subjected to normalization processing; the training sample after normalization processing and the differential sample are channel spliced to obtain the spliced sample.

3. The method of claim 1, wherein, The neural network comprises a feature extraction network, a recurrent neural network and an Attention layer connected in sequence; wherein, the feature extraction network is used for feature extraction on the input spliced sample, and outputs N first feature maps; the recurrent neural network is used for feature fusion extraction on the N first feature maps frame by frame, and outputs N fusion feature maps; the Attention layer is used for weight calculation on the N fusion feature maps, and outputs an attention vector, which is used for inputting a SoftMax layer to calculate a falling probability.

4. The method of claim 3, wherein, The feature extraction network comprises a plurality of reverse residual modules, at least one depth separable convolution module and at least one convolution layer, for the plurality of reverse residual modules, the at least one depth separable convolution module or the at least one convolution layer, the number of output channels configured is not more than 128, the channel expansion multiple configured is not more than 3, and the number of residual connection layers configured is not more than 2.

5. The method of claim 1, wherein, The method further comprises: a plurality of interference color blocks are randomly inserted into the background of each infrared image of the action sample to obtain an interference sample; the training set comprises: the plurality of action samples and a plurality of interference samples are used as the training set.

6. The method of claim 5, wherein, Before the step of inserting a plurality of interference color blocks into the background of each infrared image of the action sample to obtain an interference sample, the method further comprises: The data augmentation processing is performed on the plurality of action samples to obtain a plurality of augmented samples after expansion; The plurality of interference color blocks are randomly inserted into the background of each frame of infrared image of the action sample to obtain an interference sample, including: The plurality of interference color blocks are randomly inserted into the background of each frame of infrared image of the augmented sample to obtain the interference sample.

7. A fall detection method characterized by, Including: An action sample is obtained; The test action sample is input into a fall detection model to output a category of whether falling, wherein the fall detection model is trained by the method for training a fall detection model according to any one of claims 1-6.

8. An electronic device, comprising: Including: At least one processor, and The memory is in communication with the at least one processor, wherein The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer executable instructions for causing a computer device to perform the method of any one of claims 1-7.

Images