A fall event detection method, device, apparatus and storage medium

Abstract

Embodiments of the present application disclose a fall event detection method, device and equipment, and a storage medium. In the fall event detection, the CSI data corresponding to an image in a preset area is mapped into three-dimensional space data of the image to obtain mapping data, and the fall event detection is performed according to the mapping data. That is, the fall event detection is performed by using wireless sensing technology and visual technology, without the need for the user to wear other devices, thereby providing convenience for the user. Meanwhile, the CSI data and the image are fused, the diversity of information is increased, and the accuracy of the detection result is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and particularly relates to a fall event detection method and device, equipment and a storage medium. BACKGROUND

[0002] In fall detection of a user, a traditional method is that the user wears a wearable device, and whether the user falls is determined according to data reported by the wearable device. This not only brings inconvenience to the user, but also the sensitivity of the wearable device is low, which affects the accuracy of a detection result.

[0003] Application Content

[0004] Embodiments of the present application provide a fall event detection method, device, equipment and storage medium, which can improve the accuracy of a detection result.

[0005] In a first aspect, the embodiments of the present application provide a fall event detection method, applied to an electronic device, the electronic device being connected with a wireless perception device, and the method comprises the following steps.

[0006] Obtaining an image sequence and channel state information (CSI) data in a preset area, the image sequence comprising a plurality of images;

[0007] Mapping the CSI data corresponding to the images into three-dimensional space data of the images to obtain mapping data;

[0008] Performing fall event detection according to the mapping data.

[0009] In a second aspect, the embodiments of the present application provide a fall event detection device, applied to an electronic device, the electronic device being connected with a wireless perception device, and the device comprises the following steps.

[0010] An obtaining module, configured to obtain an image sequence and channel state information (CSI) data in a preset area, the image sequence comprising a plurality of images;

[0011] A mapping module, configured to map the CSI data corresponding to the images into three-dimensional space data of the images to obtain mapping data;

[0012] A detection module, configured to perform fall event detection according to the mapping data.

[0013] In a third aspect, the embodiments of the present application provide an electronic device, comprising the following steps.

[0014] A camera module, configured to capture images;

[0015] A processor;

[0016] A memory, configured to store computer program instructions;

[0017] When the computer program instructions are executed by the processor, the method as described in the first aspect is implemented.

[0018] In a fourth aspect, the embodiments of the present application provide a computer readable storage medium having computer program instructions stored thereon, and when the computer program instructions are executed by a processor, the method as described in the first aspect is implemented.

[0019] The fall event detection method, device, equipment and storage medium provided by the embodiments of the present application are used to map the CSI data corresponding to the image in the preset area into the three-dimensional space data of the image to obtain mapping data, and then the fall event detection is performed according to the mapping data. That is, the wireless sensing technology and the visual technology are used to detect the fall event, and the user does not need to wear other equipment, which provides convenience for the user. Meanwhile, the CSI data and the image are fused, the diversity of information is increased, and the accuracy of the detection result is improved. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments of the present application will be briefly introduced. For those skilled in the art, other drawings can also be obtained without creative labor on the premise of these drawings.

[0021] Figure 1 A scene diagram of a fall event detection method provided by the embodiments of the present application;

[0022] Figure 2 A structure diagram of a wireless sensing device provided by the embodiments of the present application;

[0023] Figure 3 A flowchart of a fall event detection method provided by the embodiments of the present application;

[0024] Figure 4 A flowchart of another fall event detection method provided by the embodiments of the present application;

[0025] Figure 5 A schematic diagram of a user interface provided by the embodiments of the present application;

[0026] Figure 6 A schematic diagram of a parameter configuration interface provided by the embodiments of the present application;

[0027] Figure 7 A structure diagram of a fall event detection device provided by the embodiments of the present application;

[0028] Figure 8 A structure diagram of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0029] The features and exemplary embodiments of various aspects of the present application will be described below in detail, in order to make the purposes, technical solutions and advantages of the present application more clear and apparent, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only configured to explain the present application, and are not configured to limit the present application. The present application can be implemented without some of these specific details for those skilled in the art. The following description of the embodiments is only to provide a better understanding of the present application by showing examples of the present application.

[0030] It should be noted that, in this paper, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the elements defined by the statement "include" do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0031] In related technologies, when detecting a user's fall, the user needs to wear a wearable device, which not only brings inconvenience to the user, but also has low sensitivity of the wearable device, affecting the accuracy of the detection result.

[0032] To this end, the embodiments of the present application provide a fall event detection method, which does not require the user to wear a wearable device, and improves the accuracy of the detection result.

[0033] The fall event detection method provided by the embodiments of the present application can be applied to Figure 1 The scene shown in the figure can include an electronic device 10 and a wireless perception device 11, and the electronic device 10 and the wireless perception device 11 establish a communication connection based on a wireless network.

[0034] The electronic device 10 can be a device with wireless communication capability, for example, it can be a notebook computer, a desktop computer, a projector, etc. Figure 1 Taking the electronic device 10 as an example.

[0035] The wireless sensing device 11 can be a device with wireless sensing capability, for example, can be a router. The wireless sensing technology is a technology that can sense the surrounding environment by using wireless signals, for example, can sense the changes caused by the movement of objects, pets or people in the surrounding environment. The embodiments of the present application can detect whether the user falls by using the wireless sensing device 11, so that the user does not need to wear any device, and the user is provided with convenience.

[0036] Exemplarily, referring to Figure 2 The wireless sensing device 11 can include a station management entity (SME) 110 and a layer management entity (LME) 111.

[0037] The LME 111 can include a MAC layer management entity (MLME) 1110 and a physical layer management entity (PLME) 1111.

[0038] Specifically, the SME 110 can control the LME 111 to perform a wireless sensing process according to the start instruction sent by the electronic device 10, and obtain channel state information (CSI) data of the surrounding environment. The embodiments of the present application take the SME 110 controlling the MLME 1110 to perform the wireless sensing process as an example.

[0039] The electronic device 10 can determine whether there is a falling event in the preset area, that is, whether the user in the preset area falls, according to the CSI data sent by the SME 110 in combination with the image sequence in the preset area, without the user wearing any device, and the user is provided with convenience.

[0040] According to the above application scenario, the falling event detection method provided by the embodiments of the present application will be described in combination with specific embodiments. The method can be performed by the electronic device 10 as shown in the figure. Figure 1

[0041] Figure 3 A flowchart of a falling event detection method provided by the embodiments of the present application.

[0042] As Figure 3 shown, the falling event detection method can include the following steps:

[0043] S310, obtaining an image sequence and channel state information (CSI) data in a preset area. ​

[0044] The image sequence includes a plurality of images.

[0045] S320, mapping the CSI data corresponding to the image into three-dimensional space data of the image to obtain mapping data.

[0046] S330, performing fall event detection according to the mapping data.

[0047] In the embodiments of the present application, when performing fall event detection, the CSI data corresponding to the image in the preset area is mapped into the three-dimensional space data of the image to obtain mapping data, and the fall event detection is performed according to the mapping data. That is, the present application utilizes wireless sensing technology and visual technology to perform fall event detection, without the need for the user to wear other devices, thereby providing convenience for the user. Meanwhile, the CSI data and the image are fused, the diversity of information is increased, and the accuracy of the detection result is improved.

[0048] The above steps will be described in detail as follows:

[0049] In S310, the preset area can be a certain area in a room, for example, a living room, a bedroom, etc. The image sequence can be a collection of a plurality of images obtained by a camera module collecting the preset area, wherein the camera module can be a device capable of collecting images or videos, such as a camera.

[0050] Exemplarily, the video taken by the camera module can be sampled at a certain frame rate to obtain the image sequence. In some embodiments, in order to improve the accuracy of the detection result, the image sequence can be preprocessed, for example, the images in the image sequence can be denoised to eliminate the noise of the images.

[0051] In the embodiments of the present application, the camera module can be integrated on the electronic device, or can be independent of the electronic device. When the camera module is independent of the electronic device, the camera module establishes a communication connection with the electronic device, so that the electronic device can obtain the images in the preset area in real time.

[0052] The CSI data is data obtained by a wireless sensing device sensing the environment in the preset area. Exemplarily, the CSI data here can be 5*30*3*3 wireless sensing signal corresponding CSI data, wherein 5 represents 5 groups of wireless sensing signals, 30 represents the size of each group of CSI data, and 3*3 represents the number of antennas corresponding to the wireless sensing signal. In some embodiments, the electronic device can obtain the CSI data in the following manner:

[0053] sending a start instruction to the wireless sensing device to enable the wireless sensing device to execute a wireless sensing process according to the start instruction to obtain the CSI data of the preset area;

[0054] receive the CSI data sent by the wireless sensing device.

[0055] In the embodiments of the present application, the start instruction can include but is not limited to Event ID and period, wherein Event ID is the identification of the detection event, for example, for the fall detection event, Event ID can be fall down testing, of course, Event ID can also adopt other representation manners, and the embodiments of the present application are not limited. The period is the detection duration, which can be set as T1, and exemplarily, T1=0 represents always detecting.

[0056] After receiving the start instruction, the SME can execute the wireless sensing process to obtain the CSI data in the preset area.

[0057] In some embodiments, the SME executes the wireless sensing process in the following process:

[0058] 1. The SME sends a wireless sensing request message to the MLME based on the received start instruction.

[0059] The wireless sensing request message can include but is not limited to parameters PeerSTAAddress and Measurementconfiguration, wherein PeerSTAAddress is used to represent the address of the device participating in the wireless sensing, and Measurementconfiguration is used to represent the measurement parameter of the wireless sensing.

[0060] Exemplarily, Measurement configuration can include but is not limited to control parameter Measurementcontrol, periodic measurement parameter Periodical profile, and event measurement parameter Event profile.

[0061] The control parameter Measurement control can include but is not limited to Measurement ID, Measurementtype and M-band. Wherein, Measurement ID is used to identify the current wireless sensing measurement process. M-band is used to represent the measurement frequency band of the wireless sensing, for example, it can be 5Ghz, 6Ghz, 60Ghz.

[0062] The Measurement type can be set as periodical or event-based. Illustratively, when the Measurement type is set as periodical, the Measurement configuration can include a periodical measurement parameter Periodical profile. The periodical measurement parameter Periodical profile can include, but is not limited to, a time interval for the MLME to send a measurement report to the SME, a number of measurement reports, and a number of reported items in the measurement report, which can include, but is not limited to, a measured distance, a field of view angle, a position coordinate, whether a user exists, a heart rate of the user, a breathing rate, and the like.

[0063] Illustratively, when the Measurement type is set as event-based, the Measurement configuration can include an event measurement parameter Event profile. The event measurement parameter Event profile can include an event identification of one or more events.

[0064] In some embodiments, the event identification of different events and the judgment condition can be set locally in advance in the form of Table 1. The local setting form of each event in Table 1

[0065] Event ID Threshold Event ID Threshold

[0066] The Threshold can be determined according to the definition of the event. Illustratively, the definition of the event Event A1 (fall down) is whether a user exists in a region with a measured distance within D1 and a field of view angle range of (H1, V1), and whether the user falls down. The local setting form of the event can be shown in Table 2.

[0067] Table 2: Local setting form of Event A1 (fall down)

[0068] Event A1 (D1, H1, V1) Figure 4 Figure 4

[0069] Thus, when the Measurement type is set as event-based, the MLME can find Table 1 according to the event measurement parameter Event profile in the wireless perception request message sent by the SME, determine the judgment condition of the corresponding event, and then measure and report the CSI data in the corresponding region according to the judgment condition, so that the electronic device combines the CSI data to detect the fall down event.

[0070] 2. The MLME receives the wireless sensing request message, and judges whether the value of the PeerSTAAddress is the address of the device.

[0071] The device here is the wireless sensing device where the MLME is located. The embodiment of the application takes the Measurement type set as event-based as an example.

[0072] 2.1. When the value of the PeerSTAAddress is the address of the device, the MLME can find the locally stored Table 1 according to the event identifier in the Event profile, determine the Threshold, then send the wireless sensing signal according to the Threshold, and obtain the CSI data based on the received echo signal.

[0073] 2.2. When the value of the PeerSTAAddress is the address of another wireless sensing device, the MLME can act as the initiator of the wireless sensing process, and send the sensing request frame to the wireless sensing device corresponding to the PeerSTAAddress.

[0074] Exemplarily, the sensing request frame can include but is not limited to the frame type (which can be set as sensing request here), the role of the device participating in the wireless sensing, the Measurement configuration, and the sensing service identifier Sensing SID.

[0075] The role of the device participating in the wireless sensing process can include at least one of the following: the initiator of the wireless sensing process (the address or identifier of the device initiating the wireless sensing process), the responder of the wireless sensing process (the address or identifier of the device responding to the initiator of the wireless sensing process), the sender of the wireless sensing process (the address or identifier of the device sending the wireless sensing data packet in the wireless sensing process), and the receiver of the wireless sensing process (the address or identifier of the device receiving the sensing data packet and performing measurement or calculation in the wireless sensing process).

[0076] 2.3. The responder of the wireless sensing process (the wireless sensing device corresponding to the PeerSTAAddress) receives the sensing request frame, judges whether the wireless sensing service corresponding to the Sensing SID is supported by itself according to the Sensing SID, and sends the sensing response frame to the initiator of the wireless sensing process.

[0077] Exemplarily, the sensing response frame can include but is not limited to a frame type (which can be set as sensing response here), a status code, and a role of the device participating in the wireless sensing process.

[0078] The status code is used to indicate whether the responder of the wireless sensing process supports the wireless sensing service corresponding to the Sensing SID. Exemplarily, if the responder of the wireless sensing process supports the wireless sensing service corresponding to the Sensing SID, the value of the status code can be SUCCESS, otherwise, the value of the status code is REFUSED.

[0079] 2.4, after receiving the sensing response frame, the initiator (MLME) of the wireless sensing process sends an acknowledgement message to the SME and sends a first indication message to the responder of the wireless sensing process to instruct the responder of the wireless sensing process to measure the preset area to obtain the CSI data.

[0080] The acknowledgement message can include but is not limited to the value of the PeerSTAAddress and the status code.

[0081] 2.5, the initiator (MLME) of the wireless sensing process receives the CSI data in the preset area sent by the responder of the wireless sensing process and sends it to the SME. In this way, the SME can obtain the CSI data in the preset area to provide a basis for subsequent fall event detection.

[0082] In some embodiments, the responder of the wireless sensing process can also generate a measurement report according to the CSI data after obtaining the CSI data, and send a sensing report frame to the initiator (MLME) of the wireless sensing process.

[0083] Exemplarily, the sensing report frame can include but is not limited to a frame type (which can be set as sensing report here) and a Sens-report, and the Sens-report is used to indicate the measurement report.

[0084] Exemplarily, the Sens-report can include but is not limited to a Measurement control, a Periodical report, and an Event report. The related content of the Measurement control can be referred to the above embodiments, and will not be described here for brevity.

[0085] Periodical report is a measurement report generated by the responder of the wireless sensing process based on the CSI data when the Measurement type is set as periodical; and Event report is a measurement report generated by the responder of the wireless sensing process based on the CSI data when the Measurement type is set as event-based.

[0086] Exemplarily, the Periodical report can include but is not limited to the result of the report item and the number of times of reporting, wherein the result of the report item is also the specific value of the report item, and the related description of the report item can refer to the above embodiments, which will not be described herein for brevity.

[0087] The Event report can include but is not limited to the event identifier triggering the report and the time when the event is detected, wherein the event identifier triggering the report is also the event identifier of the detected event.

[0088] In this way, when the Measurement type is set as event-based, the SME can also preliminarily determine whether a fall event occurs in the preset area according to the sensing report frame sent by the initiator (MLME) of the wireless sensing process.

[0089] In order to ensure the synchronization of the image sequence and the CSI data, exemplarily, the video obtained by the camera module can be sampled at a frequency of 20 frames per second, and the CSI data can be sampled at a frequency of 100 Hz per second, so that one video frame corresponds to five CSI data.

[0090] In S320, the three-dimensional space data is the representation data of the image in the three-dimensional space, and exemplarily, the above two-dimensional image can be converted into the three-dimensional space data through a neural network model. For example, the image can be input into a pre-trained 3D convolutional neural network model, and the three-dimensional space data of the image can be output by the 3D convolutional neural network model. Of course, other ways can also be used to determine the three-dimensional space data of the image, which is not limited in the embodiments of the present application.

[0091] Exemplarily, the above 3D convolutional neural network model can include 8 convolutional layers (filter: 3x3x3, stride: 1x1x1), 5 pooling layers (filter: 2x2x2, stride: 2x2x2, except that the first filter: 1x2x2, stride: 1x2x2), 2 fully connected layers (4096), and 1 softmax classification layer. The structure is simple, and thus the three-dimensional space data of the image can be quickly obtained, improving the efficiency.

[0092] The filter numbers of the convolution layers are 64, 128, 256, 256 and 256 respectively, and the time domain depth of each convolution kernel is d, and the optimal architecture of the network model can be determined by changing d. The size of the pooling kernel is 2*2*2 (except for the first layer), the first layer is 1*2*2, and the step length is 1. The two fully connected layers have 2048 outputs respectively.

[0093] In the embodiment of the present application, the input of the 3D convolutional neural network model is c*l*h*w, where c is the number of channels, l is the length of the frame number, h and w are the height and width of the frame respectively, the size of the 3D convolution kernel is d*k*k, where d is the time depth of the kernel and k is the spatial size of the kernel. In the embodiment of the present application, the height and width of the image are 128*171, the frame number is set to 16, the number of channels is 3, and the size of the 3D convolution kernel is 3*3*3.

[0094] It should be noted that the architecture of the 3D convolutional neural network model described above is only an example, and in actual application, the architecture of the 3D convolutional neural network model described above can be adjusted as needed, or other architectures can be used, as long as the three-dimensional spatial data of the image can be obtained.

[0095] In the embodiment of the present application, the CSI data corresponding to the image is mapped into the three-dimensional spatial data of the image, that is, the mapping relationship between the three-dimensional spatial data of the image and the CSI data in the same time period is established, and the CSI data is mapped into the three-dimensional spatial data of the image based on the mapping relationship, so as to realize the fusion of the CSI data and the three-dimensional spatial data. Compared with single image or CSI data, the fused data (mapped data) increases the comprehensiveness of information, so that the accuracy of the detection result can be improved when detecting the fall event.

[0096] In some embodiments, the CSI data corresponding to the image can be mapped into the three-dimensional spatial data of the image by using a support vector machine (SVM), so as to realize the fusion of the three-dimensional spatial data and the CSI data. Of course, other methods can also be used, and the embodiment of the present application is not limited.

[0097] In S330, according to the mapped data, the fall event detection is performed, that is, the fall event detection is performed according to the fused data of the CSI data and the three-dimensional spatial data, so that the accuracy of the detection result can be improved, and the user does not need to wear any device.

[0098] In order to detect whether a fall event occurs, in some embodiments, the following is referred to Figure 3 The above S330 can include S410-S430 as shown below.

[0099] S410, performing semantic segmentation on the mapped data to determine a detection object.

[0100] S420. Extract the contour of the detected object to obtain the first image.

[0101] S430. Based on the first confidence level of the first image and the contour of the detected object, perform fall event detection on the detected object.

[0102] The first confidence level is used to represent the confidence level of the fall event detection result obtained based on the first image.

[0103] Figure 5 Other steps of the method shown can be found in [reference needed]. Figure 5 The descriptions of the embodiments shown are for succinct purposes and will not be repeated here.

[0104] The above steps are explained in detail below:

[0105] In S410, in some embodiments, a semantic segmentation network can be used to perform semantic parsing on the mapped data to determine the detection object. For example, the semantic segmentation network can be a U-Net fully convolutional network, but other networks or methods can also be used; this application embodiment does not limit the scope of the application.

[0106] In S420, the first image is an image containing the contour of the detected object, also known as a human segmentation mask image. In some embodiments, the contour of the detected object can be extracted using a Mask R-CNN network to obtain the first image. For example, the size of the first image is 1*46*82.

[0107] In S430, the first confidence level is used to represent the reliability of the fall event detection result obtained based on the first image, that is, the confidence level of the first image. The higher the confidence level of the first image (first confidence level), the higher the confidence level of the detection result when performing fall event detection based on the first image. The confidence level of the first image (first confidence level) can be determined in advance based on experience or historical detection results.

[0108] For example, the contour of the detected object can be matched with a feature template to obtain the matching degree between the contour and the feature template. Based on the weighted result of the first confidence level and the matching degree, it can be determined whether the detected object has fallen. For example, if the weighted result is greater than a set threshold, it can be determined that the detected object has fallen; otherwise, it can be determined that the detected object has not fallen.

[0109] For example, the contour of the object to be detected can also be input into a pre-trained neural network, which determines the probability of the object falling, and determines whether the object has fallen based on a weighted result of a first confidence level and the probability of falling.

[0110] Exemplarily, the neural network can adopt a simplified convolutional neural network (CNN). Exemplarily, the simplified CNN is implemented based on an architecture Pytorch-0.4.1. The simplified CNN can be trained by using the first sample image. Compared with a conventional training by directly using the original image, the training by using the first sample image can simplify the training process and save the training time. The first sample image is an image containing the contour of the sample detection object.

[0111] Embodiments of the present application do not limit the specific structure of the simplified CNN. For example, the simplified CNN can include a convolution layer with a convolution kernel of 7x7x64, an activation layer, a max pooling layer with a convolution kernel of 3x3, and two sub-modules, each of which includes two convolution layers with a convolution kernel of 3x3, a BatchNorm layer, and a fully connected layer. In each sub-module, the two convolution layers are connected through an activation layer. Of course, other structures can also be adopted, and embodiments of the present application do not limit them as long as the fall probability of the detection object can be determined based on the contour of the detection object.

[0112] Embodiments of the present application perform fall event detection on the detection object according to the confidence of the first image and the contour of the detection object. Not only the influence degree of the first image itself on the detection result is considered, but also the influence degree of the contour of the detection object on the detection result is considered, that is, multiple factors are considered comprehensively, so that the accuracy of the detection result is improved.

[0113] In order to improve the accuracy of the detection result, in some embodiments, the above S430 can include S4301 and S4302 as shown below.

[0114] S4301, determining the key points of the detection object according to the contour, and marking the key points in the first image to obtain a second image.

[0115] The key points of the detection object can be the joint points of the human joints. The positions of the joint points are different when the human body is in different states, such as a fall state or a standing state.

[0116] In some embodiments, the contour and the template can be matched to determine the key points of the detection object; or the first image can be input into a Body-25 model, and the Body-25 model can input an image (second image) containing the key points. Exemplarily, the second image can include 25 key points (joint points of human joints).

[0117] S4302, performing fall event detection on the detection object according to the first confidence of the first image, the second confidence of the second image, the contour, and the position information of the key points.

[0118] wherein the second confidence is used to represent the confidence of the fall event detection result based on the second image, that is, the higher the confidence of the second image (the second confidence) is, the higher the confidence of the detection result is when the fall event detection is performed based on the second image.

[0119] To improve the accuracy of the detection result, S4302 may, for example, include the following steps:

[0120] determining a first probability of the detection object falling according to the contour;

[0121] determining a second probability of the detection object falling according to the position information of the key points;

[0122] performing the fall event detection on the detection object according to a weighted sum of the first probability and the first confidence, and the second probability and the second confidence.

[0123] In some embodiments, the contour of the detection object can be input into the improved CNN described above, and the first probability of the detection object falling can be determined by the improved CNN.

[0124] Similarly, the second image containing the position information of the key points can also be input into the pre-trained neural network, and the second probability of the detection object falling can be determined by the pre-trained neural network. Here, the structure of the pre-trained neural network can be similar to the improved CNN described above.

[0125] In some embodiments, a first weighted result of the first probability and the first confidence, and a second weighted result of the second probability and the second confidence can be determined, and whether the detection object falls can be determined according to the first weighted result and the second weighted result.

[0126] For example, in the case that the cumulative sum of the first weighted result and the second weighted result is greater than a set threshold, it can be determined that the detection object falls, otherwise, it can be determined that the detection object does not fall.

[0127] In the embodiments of the present application, whether the detection object falls is determined based on the contour of the human body and in combination with the body joint points, thereby improving the accuracy of the detection result.

[0128] To improve the accuracy of the detection result, S4302 may, for example, include the following steps:

[0129] determining limb information corresponding to the key points according to the association relationship between the key points, and connecting the key points according to the limb information to obtain a third image;

[0130] determining a first probability of the detection object falling according to the contour;

[0131] determining a second probability of the detection object falling according to the position information of the key points;

[0132] determine a third probability of the detection object falling according to the limb information and position information of the key points corresponding to the limb information;

[0133] perform fall event detection on the detection object according to a weighted sum of the first probability and the first confidence, the second probability and the second confidence, and the third probability and the third confidence, the third confidence being used to represent the confidence of the fall event detection result obtained based on the third image.

[0134] The limb information corresponding to the key points is the limb to which the key points belong. The third image is an image containing position coordinates (x, y coordinates) of each key point and connection relationships of each key point.

[0135] The determination process of the first probability and the second probability can refer to the above embodiments, which will not be described here again.

[0136] In some embodiments, the image data of the third image can be classified by using an SVM to determine the state of the image data, where the state can include a fall state, a platform state, or a normal state (for example, a standing state), and different states can correspond to different weights, that is, different probabilities. Therefore, the third probability, that is, the weight of the fall state, can be determined according to the weight output by the SVM.

[0137] Exemplarily, a first weighted result of the first probability and the first confidence, a second weighted result of the second probability and the second confidence, and a third weighted result of the third probability and the third confidence can be determined, and then whether the detection object falls can be determined according to an accumulated sum of the first weighted result, the second weighted result, and the third weighted result.

[0138] Exemplarily, P = λ1P1 + λ2P2 + λ3P3, where P is the accumulated sum of the first weighted result, the second weighted result, and the third weighted result, λ1, λ2, and λ3 are the first confidence, the second confidence, and the third confidence respectively, and P1, P2, and P3 are the first probability, the second probability, and the third probability respectively. Exemplarily, λ1 = λ2 = 0.3, and λ3 = 0.4.

[0139] In the embodiments of the present application, the fall probability of the detection object is determined based on the contour of the detection object, the key points, and the coordinates of the key points and the connection relationships between the key points, and then whether the detection object falls is determined according to the probability and the confidence of the corresponding image, thereby improving the accuracy of the detection result.

[0140] In some embodiments, after S330, the method can further include the following steps:

[0141] In the case of a fall event, an alarm is performed;

[0142] extracts the falling picture from the image sequence and sends the falling picture to the registered terminal of the electronic device.

[0143] The alarm manner is not limited in the embodiments of the present application. For example, the alarm can be performed through the sound of the electronic device, the alarm can be performed through the flickering of the indicator light, and the alarm can be performed through the voice manner.

[0144] In the embodiments of the present application, when it is determined that the detection object falls, the alarm can remind other users in the room to timely understand the situation, and the falling picture of the detection object can be sent to the registered terminal of the electronic device to timely inform the user of the registered terminal, so that serious consequences can be avoided.

[0145] In some embodiments, before S310, the method can further include the following steps:

[0146] displaying a user interface, the user interface including a detection control corresponding to a detection event;

[0147] in response to a first input of the user to the target detection control, entering a parameter configuration interface, the target detection control being a detection control corresponding to a falling detection event, the parameter configuration interface including a region type;

[0148] in response to a second input of the user to the region type, determining a preset region.

[0149] Exemplarily, refer to Figure 5 , Figure 6 A schematic diagram of a user interface provided by the embodiments of the present application is shown. The user interface exemplarily includes a perception switch 50, a detection control 51, a detection control 52, a detection control 53, a detection control 54, and a detection control 55. One detection control can correspond to one detection event. For example, the detection control 51 corresponds to a person number detection event, the detection control 52 corresponds to a falling detection event, the detection control 53 corresponds to a device detection event, the detection control 54 corresponds to a sleep detection event, and the detection control 55 corresponds to an environment detection event.

[0150] When the perception switch 50 is turned on, a certain detection control can be triggered to perform the setting of the corresponding detection event. The detection control 53 is used to determine whether the wireless perception device and the electronic device are connected and normally powered on. Exemplarily, the user clicks the detection control 53. If the wireless perception device and the electronic device are connected and normally powered on, the Figure 6 The controls shown in the figure display preset colors, for example, green, otherwise the user is prompted that the wireless perception device is abnormal. In this way, the user can conveniently know the situation of the wireless perception device and the electronic device.

[0151] The target detection control is a detection control triggered by a first input. The first input can be a click, touch or other operation on the detection control. As an example, when it is detected that the detection control 52 is triggered, the detection control 52 can be determined as the target detection control.

[0152] At this time, the electronic device displays a parameter configuration interface as shown in Figure 5 The user can configure parameters in the parameter configuration interface. As shown in Figure 5 The parameter configuration interface can include, but is not limited to, the electronic device model, software version, hardware version, area type (living room or bedroom, etc.), child (yes or no), old person (yes or no), and device switch button. The second input can be an input for determining the preset area, for example, in the case of inputting a living room at the area type, it indicates that the preset area is a living room.

[0153] When the user clicks the device switch button, the electronic device can be closed or started. In the case of closing the electronic device, the device model, software version and other parameter fields are grayed out, and the corresponding parameters are no longer displayed. The device switch button is closed by default.

[0154] In some embodiments, before S310, it can be confirmed whether the electronic device is connected to a network. For example, the electronic device can determine whether it is connected to a network according to the state of the detection control as shown in Figure 5 For example, if the electronic device is not connected to a network, the detection controls as shown in Figure 1 are all grayed out and cannot be clicked. At this time, the electronic device can prompt the user to connect to the network. Of course, the user can also determine whether the electronic device is connected to the network.

[0155] In the case of connecting the electronic device to the network, the user can click the detection control 52 as shown in Figure 7 At this time, the electronic device sends a ping command to the gateway to determine the strength of the network signal according to the ping command. As an example, if the electronic device sends a ping command to the gateway and does not receive a reply from the gateway within a preset time period, the electronic device can prompt the user that "the current network signal is weak and cannot be detected", indicating that the current network cannot be used for sensing detection. The size of the preset time period can be set according to actual needs, for example, it can be set to 1s.

[0156] In the embodiments of the present application, the electronic device and the wireless sensing device are combined to sense the surrounding environment by using the wireless network signal, and the image collected by the camera module is combined to analyze from multiple dimensions to determine whether the detection object falls. Without the need for the user to wear any device, the accuracy of the detection result is improved.

[0157] Based on the same inventive concept, the embodiments of the present application also provide a fall event detection device. The device can be applied to Figure 7The electronic device is shown to be connected with a wireless sensing device. The following will be described in combination with Figure 7 The fall event detection device provided by the embodiment of the present application is described in detail.

[0158] As shown in the figure, the fall event detection device can include: Figures 1-6

[0159] The acquisition module 71 is configured to acquire image sequences and channel state information (CSI) data in a preset area, and the image sequences include a plurality of images.

[0160] The mapping module 72 is configured to map the CSI data corresponding to the images into three-dimensional space data of the images to obtain mapping data.

[0161] The detection module 73 is configured to perform fall event detection according to the mapping data.

[0162] The fall event detection method, device, equipment and storage medium provided by the embodiment of the present application are used to map the CSI data corresponding to the images in a preset area into three-dimensional space data of the images to obtain mapping data, and perform fall event detection according to the mapping data. That is, the present application uses wireless sensing technology and visual technology to perform fall event detection, without the need for users to wear other devices, thereby providing convenience for users. Meanwhile, the present application fuses the CSI data and the images, increases the diversity of information, and improves the accuracy of the detection result.

[0163] In one embodiment, the detection module 73 includes:

[0164] The semantic segmentation unit is configured to perform semantic segmentation on the mapping data to determine a detection object.

[0165] The extraction unit is configured to extract an outline of the detection object to obtain a first image.

[0166] The detection unit is configured to perform fall event detection on the detection object according to a first confidence of the first image and the outline of the detection object, and the first confidence is used to represent the credibility of the fall event detection result based on the first image.

[0167] In one embodiment, the detection unit includes:

[0168] The determination subunit is configured to determine key points of the detection object according to the outline, and mark the key points in the first image to obtain a second image.

[0169] The detection subunit is configured to perform fall event detection on the detection object according to a first confidence of the first image, a second confidence of the second image, the outline, and position information of the key points, and the second confidence is used to represent the credibility of the fall event detection result based on the second image. ​

[0170] In one embodiment, the detecting subunit is specifically configured to:

[0171] determine a first probability of the detection object falling according to the contour;

[0172] determine a second probability of the detection object falling according to the position information of the key points;

[0173] perform fall event detection on the detection object according to a weighted sum of the first probability and the first confidence and the second probability and the second confidence.

[0174] In one embodiment, the detecting subunit is specifically configured to:

[0175] determine limb information corresponding to the key points according to the association relationship between the key points, and connect the key points according to the limb information to obtain a third image;

[0176] determine a first probability of the detection object falling according to the contour;

[0177] determine a second probability of the detection object falling according to the position information of the key points;

[0178] determine a third probability of the detection object falling according to the limb information and the position information of the key points corresponding to the limb information;

[0179] perform fall event detection on the detection object according to a weighted sum of the first probability and the first confidence, the second probability and the second confidence, and the third probability and a third confidence, the third confidence being used to represent a confidence of a fall event detection result obtained based on the third image.

[0180] In one embodiment, the fall event detection apparatus can further include:

[0181] a display module configured to display a user interface before the acquisition module 71 acquires the image sequence and the channel state information (CSI) data in the preset area, the user interface including a detection control corresponding to a detection event;

[0182] a first response module configured to enter a parameter configuration interface in response to a first input of a target detection control by a user, the target detection control being a detection control corresponding to a fall detection event, the parameter configuration interface including an area type;

[0183] a second response module configured to determine a preset area in response to a second input of the area type by the user.

[0184] In one embodiment, the acquisition module 71 is specifically configured to:

[0185] The wireless sensing device is sent a starting instruction, so that the wireless sensing device performs a wireless sensing process according to the starting instruction to obtain CSI data of a preset area.

[0186] The CSI data sent by the wireless sensing device is received.

[0187] Figure 8 Each module in the device has the function of implementing Figure 8 The functions of each embodiment of the fall event detection method can achieve the corresponding technical effects, and for brevity, will not be described here.

[0188] Based on the same inventive concept, the embodiments of the present application also provide an electronic device, which can be a physical terminal or device with wireless communication function. The following will be described in combination with Figures 1-6 The electronic device provided by the embodiments of the present application will be described in detail.

[0189] As Figures 1-6 The electronic device can include a camera module 80, a processor 81, and a memory 82 for storing computer program instructions.

[0190] The camera module 80 is used to collect images of a preset area, which can be a living room or a bedroom.

[0191] The processor 81 can include a central processing unit (CPU) or a specific integrated circuit (ASIC), or can be configured to implement one or more integrated circuits of the embodiments of the present application.

[0192] The memory 82 can include a large-capacity memory for data or instructions. For example, but not limited to, the memory 82 can include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disc, a magneto-optical disc, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of the above. In one example, the memory 82 can include a removable or non-removable (or fixed) medium, or the memory 82 is a non-volatile solid-state memory. In one example, the memory 82 can be a read-only memory (ROM). In one example, the ROM can be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM), or a flash memory, or a combination of two or more of the above.

[0193] The processor 81 implements the functions of the electronic device by reading and executing computer program instructions stored in the memory 82. Figure 8 In the embodiment shown, the processor 81 implements the functions of the electronic device by reading and executing computer program instructions stored in the memory 82. Figures 1-6 The corresponding technical effects achieved by the embodiment described above are not repeated here for brevity.

[0194] In one example, the electronic device can further include a communication interface 83 and a bus 84. As shown, the camera module 80, the processor 81, the memory 82, and the communication interface 83 are connected by the bus 84 and complete communication with each other. Figure 7

[0195] The communication interface 83 is mainly used to realize the communication between the modules, devices, and / or equipment in the embodiments of the present application.

[0196] The bus 84 includes hardware, software, or both, which couples the components of the electronic device to each other. By way of example, and not limitation, the bus 84 can include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable bus or combination of two or more of these. Where appropriate, the bus 84 can include one or more buses. Although the embodiments of the present application describe and show a particular bus, the present application contemplates any suitable bus or interconnect.

[0197] After the electronic device obtains the image sequence and channel state information (CSI) data in the preset area, it can execute the fall event detection method in the embodiments of the present application, thereby realizing the fall event detection method described above and the fall event detection device described above. ​ The fall event detection method described above and the fall event detection device described above. ​

[0198] ​​In addition, in combination with the fall event detection method in the above embodiments, the embodiments of the present application can provide a computer storage medium for implementation. The computer storage medium has computer program instructions stored thereon; the computer program instructions are executed by a processor to implement any one of the fall event detection methods in the above embodiments.

[0199] It should be noted that the present application is not limited to the specific configurations and processes described above and shown in the drawings. For the sake of brevity, detailed descriptions of well-known methods are omitted. In the above embodiments, several specific steps are described and shown as examples. However, the method processes of the present application are not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order of the steps, after understanding the spirit of the present application.

[0200] The functional blocks shown in the structural block diagrams described above can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a functional card, etc. When implemented in software, the elements of the present application are program or code segments used to perform the required tasks. The program or code segments can be stored in a machine-readable medium or transmitted through a data signal carried in a carrier wave over a transmission medium or communication link. The "machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segments can be downloaded via a computer network such as the Internet, an intranet, etc.

[0201] It should also be noted that the exemplary embodiments mentioned in the present application describe some methods or systems based on a series of steps or devices. However, the present application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiments, or in an order different from the embodiments, or several steps can be performed simultaneously.

[0202] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other processing device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other processing device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0203] The above describes only specific implementation of the present application. For the convenience and brevity of description, the specific working process of the system, module and unit described above can refer to the corresponding process in the foregoing method embodiments, which will not be described herein. It should be understood that the protection scope of the present application is not limited in this way. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed in the present application, and these modifications or replacements should be covered within the protection scope of the present application.

Claims

1. A fall event detection method, characterized by, The method is applied to an electronic device connected with a wireless sensing device, and comprises: obtaining an image sequence and channel state information (CSI) data in a preset area, the image sequence comprising a plurality of images; mapping CSI data corresponding to the images into three-dimensional space data of the images to obtain mapping data; performing semantic segmentation on the mapping data to determine a detected object; extracting a contour of the detected object to obtain a first image; determining key points of the detected object according to the contour and marking the key points in the first image to obtain a second image, the key points being joint points of human joints, the positions of the joint points being different when the human body is in different states; performing fall event detection on the detected object according to a first confidence of the first image, a second confidence of the second image, the contour and position information of the key points, the first confidence being used to represent a confidence of a fall event detection result obtained based on the first image, and the second confidence being used to represent a confidence of a fall event detection result obtained based on the second image.

2. The method of claim 1, wherein, The fall event detection on the detected object according to the first confidence of the first image, the second confidence of the second image, the contour and the position information of the key points comprises: determining a first probability of the detected object falling according to the contour; determining a second probability of the detected object falling according to the position information of the key points; performing fall event detection on the detected object according to a weighted sum of the first probability and the first confidence and the second probability and the second confidence.

3. The method of claim 1, wherein the fall event detection on the detected object according to the first confidence of the first image, the second confidence of the second image, the contour and the position information of the key points comprises: determining limb information corresponding to the key points according to a correlation between the key points, and connecting the key points according to the limb information to obtain a third image; determining a first probability of the detected object falling according to the contour; determining a second probability of the detected object falling according to the position information of the key points; determining a third probability of the detected object falling according to the limb information and position information of the key points corresponding to the limb information; performing fall event detection on the detected object according to a weighted sum of the first probability and the first confidence, the second probability and the second confidence and the third probability and a third confidence, the third confidence being used to represent a confidence of a fall event detection result obtained based on the third image.

4. The method according to any one of claims 1 to 3, characterized in that, Before obtaining the image sequence and the CSI data in the preset area, the method further comprises: displaying a user interface, the user interface comprising a detection control corresponding to a detection event; in response to a first input of a target detection control by a user, entering a parameter configuration interface, the target detection control being a detection control corresponding to a fall detection event, and the parameter configuration interface comprising an area type. In response to a second input of the user on the area type, the preset area is determined.

5. The method according to any one of claims 1 to 3, characterized in that, Obtaining channel state information (CSI) data, including: Sending a start instruction to the wireless sensing device to enable the wireless sensing device to perform a wireless sensing process according to the start instruction to obtain the CSI data of the preset area; Receiving the CSI data sent by the wireless sensing device.

6. A fall event detection apparatus characterized by comprising: An electronic device is applied to, and the device comprises: An obtaining module configured to obtain an image sequence and channel state information (CSI) data in a preset area, the image sequence comprising a plurality of images; A mapping module configured to map the CSI data corresponding to the image data into three-dimensional spatial data of the images to obtain mapping data; A semantic segmentation unit configured to perform semantic segmentation on the mapping data to determine a detection object; An extraction unit configured to extract an outline of the detection object to obtain a first image; A determination subunit configured to determine key points of the detection object according to the outline, and mark the key points in the first image to obtain a second image, the key points being joint points of human joints, and the positions of the joint points being different when the human body is in different states; A detection subunit configured to perform fall event detection on the detection object according to a first confidence of the first image, a second confidence of the second image, the outline, and position information of the key points, the first confidence being used to represent a confidence of a fall event detection result obtained based on the first image, and the second confidence being used to represent a confidence of a fall event detection result obtained based on the second image.

7. An electronic device, comprising: Comprise: A camera module configured to capture images; A processor; A memory configured to store computer program instructions; When the computer program instructions are executed by the processor, the method of any one of claims 1-5 is implemented.

8. A computer-readable storage medium having stored thereon computer program instructions, wherein, When the computer program instructions are executed by the processor, the method of any one of claims 1-5 is implemented.

Images