Human motion recognition method, neural network training method and related devices and equipment

Abstract

The present application provides a method for recognizing human motion, a neural network training method and related devices. It relates to the field of artificial intelligence, and specifically to the field of wireless recognition technology. The method comprises: an execution device acquires an image to be recognized, wherein the image to be recognized is an image obtained by encoding the signal characteristics of M frames of wireless signals in the dimensions of time and channel, and the M frames of wireless signals are wireless signals transmitted by a first wireless transmitting device and received by a first wireless receiving device; then, the execution device inputs the image to be recognized into a target human motion recognition model to obtain the type of human motion corresponding to the image to be recognized. The present application converts the signal characteristics of M frames of wireless signals into images, so as to use a neural network model to recognize human motion classification more accurately.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method for recognizing human motion, a method for training a neural network, and related devices and equipment. Background Art

[0002] Wireless signals can be used not only to transmit data, but also to sense the environment. In an indoor environment, the radio waves (also referred to as wireless signals in this application) generated by the wireless transmitting device are propagated through multiple paths such as direct radiation, reflection, and scattering, forming a multipath superposition signal at the wireless receiving device, and the wireless receiving device can receive the multipath propagation superposition signal. The multipath superposition signal is affected by the physical space in which it propagates, and carries information reflecting the physical space in which the signal propagates, including both human factors (whether there are people and their position, characteristics, posture, movements, etc.), as well as other external factors.

[0003] By analyzing the signal characteristics of the received signal and "decoding" the environmental information during the propagation process, human movements in the environment can be identified. How to more accurately identify human movements based on the signal characteristics of wireless signals is a technical problem that needs to be solved urgently. Summary of the invention

[0004] The present application provides a method for recognizing human motion, a method for processing wireless signals, a neural network training method and related devices and equipment, which can recognize human motion based on the signal characteristics of wireless signals, and then monitor human motion occurring in the environment.

[0005] In a first aspect, a method for recognizing human body movements is provided, comprising:

[0006] An execution device obtains an image to be identified, wherein the image to be identified is an image obtained by encoding signal features of M frames of wireless signals in the dimensions of time and channel, the M frames of wireless signals are wireless signals transmitted by a first wireless transmitting device and received by a first wireless receiving device, a distance between the first wireless receiving device and the first wireless transmitting device is greater than a first threshold, and M is an integer greater than 1; further, the image to be identified is input into a target human motion recognition model to obtain a type of human motion corresponding to the image to be identified.

[0007] Optionally, the target human action recognition model is a trained neural network used to recognize the category of human actions in an input image.

[0008] It should be understood that the execution device may be a first wireless receiving device, or a first wireless transmitting device, or a terminal such as a mobile phone, a tablet computer, a personal digital assistant, a cloud, a server, etc. Among them, the first wireless receiving device or the first wireless transmitting device may be a device having the function of receiving wireless signals and extracting signal characteristics of wireless signals, such as a lamp, a refrigerator, a washing machine, a wardrobe, a water heater, a curtain, a lock, a wireless router, and other smart home devices.

[0009] The above method converts the signal features of M frames of wireless signals into images so as to utilize a neural network model to identify and classify human movements, thereby achieving simplicity and accuracy.

[0010] Optionally, in scenario a of the embodiment of the present application, the target human motion recognition model may be a human fall recognition model, and the types of human motions that can be recognized by the human fall recognition model may include falls and non-falls, and may also include falls in one of multiple areas, non-falls in one of multiple areas, and not in one of multiple areas, etc. The execution device may also send prompt information indicating the recognition result of the human fall recognition model to the guardian's terminal when a fall event is detected.

[0011] Optionally, in scenario b of the embodiment of the present application, the target human motion recognition model may be an intruder recognition model, and the types of human motions that can be recognized by the intruder recognition model may include intrusion by a person and intrusion by no one, and may also include intrusion into one of multiple areas, non-intrusion into one of multiple areas, etc. The execution device may also send a prompt message indicating the recognition result of the human fall recognition model to the terminal of the homeowner or the owner when a fall event is detected.

[0012] Optionally, in scenario c of the embodiment of the present application, the target human motion recognition model may be a motion recognition model, and the types of human motions that can be recognized by the motion recognition model may include multiple human motions or multiple classifications of human motions. After detecting the target human motion or the classification of the target human motion, the execution device may also execute the operation corresponding to the target human motion according to the correspondence between the multiple human motions and the multiple operations, or execute the operation corresponding to the classification of the target human motion according to the correspondence between the multiple human motion classifications and the multiple operations.

[0013] Optionally, the first wireless receiving device may extract signal characteristics of the M-frame wireless signal through a network card, and the signal characteristics may include but are not limited to at least one of CSI amplitude, CSI phase, RSSI, and the like.

[0014] In a possible implementation of the embodiment itself, the signal feature includes channel state information (CSI); CSI includes CSI amplitude and CSI phase, and the image to be identified includes a first image and a second image, wherein any frame of the wireless signal in the M frames of wireless signals includes N channels, and the pixel value of the i-th row and j-th column in the first image represents the CSI amplitude of the j-th frame of the wireless signal on the i-th channel among the N channels arranged in the first order; the pixel value of the i-th row and j-th column in the second image represents the CSI phase of the j-th frame of the wireless signal on the i-th channel among the N channels arranged in the first order, i, j, and N are positive integers, i≤N, j≤M.

[0015] In a possible implementation of the embodiment itself, the image to be identified includes a third image and a fourth image, wherein the pixel value of the i-th row and j-th column in the third image represents the CSI amplitude of the j-th frame wireless signal on the i-th channel among N channels arranged in a second order; the pixel value of the i-th row and j-th column in the fourth image represents the CSI phase of the j-th frame wireless signal on the i-th channel among N channels arranged in a second order, and the first order is different from the second order.

[0016] In the above method, the image to be recognized of the input target human motion model includes images obtained by channel coding in two orders, which can extract richer feature information and further improve the accuracy of recognition.

[0017] Optionally, the frequencies of the N channels arranged in the first order are increasing, and the frequencies of the N channels arranged in the second order are non-increasing.

[0018] Optionally, the frequencies of the N channels arranged in the first order are decreasing, and the frequencies of the N channels arranged in the second order are non-decreasing.

[0019] Optionally, any two adjacent channels in the N channels arranged in the first order are not adjacent in the N channels arranged in the second order.

[0020] In a possible implementation of the embodiment itself, the signal feature includes a received signal strength indication (RSSI), the image to be identified also includes a fifth image, and the pixel value of the jth column in the fifth image represents the RSSI of the jth frame wireless signal.

[0021] In the above method, since RSSI is the superposition of the amplitudes of all subcarriers, the overall amplitude of the wireless signal is taken into consideration and used as input data for human motion recognition, thereby improving accuracy.

[0022] In a possible implementation of the embodiment itself, the execution device and the first wireless receiving device are the same device, and one implementation of the execution device obtaining the image to be identified may be: the execution device receives M frames of wireless signals transmitted by the first wireless transmitting device; extracts the signal features of each frame of wireless signals in the M frames of wireless signals; encodes the signal features of the M frames of wireless signals in the time and channel dimensions to obtain the image to be identified.

[0023] In a possible implementation of an embodiment of the present application, another implementation method for the execution device to obtain the image to be identified may be: the execution device receives signal characteristics of M frames of wireless signals sent by a first wireless receiving device; encodes the signal characteristics of the M frames of wireless signals in time and channel dimensions to obtain the image to be identified.

[0024] In a possible implementation of the embodiment itself, before the image to be recognized is input into the human motion recognition model and the recognition result is obtained, the method may also include: the execution device searches for the target human motion recognition model corresponding to the combination of the first wireless transmitting device and the first wireless receiving device according to the correspondence between the combination of the wireless transmitting device and the wireless receiving device and the human motion recognition model.

[0025] In a possible implementation of the embodiment, the execution device and the training device are the same device, and the method may further include:

[0026] Collect training samples, where the training samples include sample images and categories of human motions corresponding to the sample images, where the sample images are obtained by encoding signal features of M frames of first wireless signals in time and channel dimensions, where the M frames of first wireless signals are wireless signals transmitted by the first wireless transmitting device and received by the first wireless receiving device when a first human motion occurs between the first wireless receiving device and the first wireless transmitting device, and where the first human motion belongs to the category of human motions corresponding to the sample images;

[0027] A neural network is trained according to sample images and the categories of human actions corresponding to the sample images to obtain a target human action recognition model, wherein the sample images are training inputs and the categories of human actions corresponding to the sample images are training labels.

[0028] In a possible implementation of the embodiment of the present application, the sample image includes at least one of a first sample image, a second sample image, a third sample image, a fourth sample image and a fifth sample image, wherein the pixel value of the i-th row and j-th column in the first sample image represents the CSI amplitude of the first wireless signal of the j-th frame on the i-th channel among N channels arranged in a first order; the pixel value of the i-th row and j-th column in the second sample image represents the CSI phase of the first wireless signal of the j-th frame on the i-th channel among N channels arranged in a first order; the pixel value of the i-th row and j-th column in the third sample image represents the CSI amplitude of the first wireless signal of the j-th frame on the i-th channel among N channels arranged in a second order; the pixel value of the i-th row and j-th column in the fourth sample image represents the CSI phase of the first wireless signal of the j-th frame on the i-th channel among N channels arranged in a second order; the pixel value of the j-th column in the fifth image represents the RSSI of the first wireless signal of the j-th frame; the first order is different from the second order, the first wireless signal includes N channels, i, j, N are positive integers, i≤N, j≤M.

[0029] In the second aspect, an embodiment of the present application also provides a method for processing wireless signals, the method comprising: a wireless receiving device receives M frames of wireless signals transmitted by a wireless transmitting device, extracts signal features of each frame of the wireless signals in the M frames, and then encodes the signal features of the M frames of wireless signals in time and channel dimensions to obtain an image to be identified.

[0030] Optionally, the wireless receiving device may also send the image to be recognized to the execution device, so that the execution device recognizes the category of the human body movement according to the image to be recognized.

[0031] Among them, the implementation of the image to be recognized and the category of the execution device to recognize human body movements can refer to the relevant description of the image to be recognized in the first aspect above, which will not be repeated here.

[0032] The wireless receiving device can be an electronic device with a wireless signal receiving function, such as a lamp, refrigerator, washing machine, wardrobe, water heater, curtain, lock, wireless router and other smart home devices.

[0033] In a third aspect, an embodiment of the present application also provides a neural network training method, the method comprising: a training device receives signal characteristics of M frames of first wireless signals sent by a first wireless receiving device, wherein the M frames of first wireless signals are wireless signals transmitted by the first wireless transmitting device and received by the first wireless receiving device when a first human motion occurs between the first wireless receiving device and the first wireless transmitting device, and M is an integer greater than 1; then, the training device encodes the signal characteristics of the M frames of first wireless signals in time and channel dimensions to obtain a sample image; further, the training device trains the neural network according to the sample image and the category of the human motion corresponding to the sample image to obtain a target human motion recognition model, wherein the sample image is a training input, the category of the human motion corresponding to the sample image is a training label, and the first human motion belongs to the category of the human motion corresponding to the sample image.

[0034] It should be understood that the training device can be a first wireless transmitting device, a first wireless receiving device, or a terminal such as a mobile phone, a tablet computer, a personal digital assistant, a cloud, a server, etc. It can also be the same device as the execution device, which is not limited here.

[0035] The above method uses the above neural network training method to encode the signal features of the collected M frames of wireless signals into sample images, and trains the neural network with the sample images to obtain a target human motion recognition model, so as to realize the recognition of human motion based on wireless signals more accurately.

[0036] It should be understood that in another possible implementation, the training device may also obtain training samples, which include the sample images and the categories of human body movements corresponding to the sample images. In this case, the training device does not need to perform the signal characteristics and encoding operations of receiving the M frames of the first wireless signal.

[0037] Optionally, in scenario a of the embodiment of the present application, the above-mentioned first human motion may be a falling motion or a non-falling motion, and the above-mentioned target human motion recognition model obtained through training may specifically be a human fall recognition model, which may recognize that types of human motions include falling and non-falling; in another implementation, the above-mentioned first human motion may be a falling motion in one of multiple areas, a non-falling motion in one of multiple areas, or any motion that is not in one of multiple areas, and the above-mentioned target human motion recognition model obtained through training may specifically be a human fall recognition model, and the classification of human motions that may be recognized by the model may include falling in one of multiple areas, not falling in one of multiple areas, and not being in one of multiple areas, etc.

[0038] Optionally, in scenario b of the embodiment of the present application, the first human motion may be a human intrusion or an unintrusion, and the target human motion recognition model obtained through training may specifically be an intruder recognition model, and the types of human motions that can be recognized by the model may include human intrusion and unintrusion. In another implementation, the first human motion may also be an intrusion into one of multiple areas or an intrusion into one of multiple areas, and the classification of human motions that can be recognized by the model may include intrusion into one of multiple areas, unintrusion into one of multiple areas, and the like.

[0039] Optionally, in scenario c of an embodiment of the present application, the first human motion may be any human motion, and the target human motion recognition model obtained through the training may specifically be a motion recognition model, which may be able to recognize human motion types including multiple human motions or multiple categories of human motions.

[0040] It should be understood that the training device can be a first wireless transmitting device, a first wireless receiving device, or a terminal such as a mobile phone, a tablet computer, a personal digital assistant, a cloud, a server, etc. It can also be the same device as the execution device, which is not limited here.

[0041] In a possible implementation of an embodiment of the present application, one implementation of a training device collecting training samples may be: the training device receives signal characteristics of M frames of first wireless signals sent by a first wireless receiving device; encodes the signal characteristics of the M frames of first wireless signals in time and channel dimensions to obtain a sample image.

[0042] In a possible implementation of the embodiment of the present application, the signal feature includes channel state information (CSI); the CSI includes a CSI amplitude and a CSI phase, any one of the M frames of the first wireless signal includes N channels, and the training device encodes the signal features of the M frames of the first wireless signal in the time and channel dimensions to obtain a sample image, which may include:

[0043] The training device encodes the CSI amplitude of the first wireless signal of the jth frame on the i-th channel among N channels arranged in a first order as the pixel value of the i-th row and j-th column in the first sample image; and the training device encodes the CSI phase of the first wireless signal of the j-th frame on the i-th channel among N channels arranged in a first order as the pixel value of the i-th row and j-th column in the second sample image; i and j are positive integers, i≤N, j≤M, and the sample images include a first sample image and a second sample image.

[0044] In a possible implementation of the embodiment of the present application, the training device encodes the signal features of the M frames of the first wireless signal in the time and channel dimensions to obtain a sample image, and may also include:

[0045] The training device encodes the CSI amplitude of the first wireless signal of the jth frame on the i-th channel among N channels arranged in a second order as the pixel value of the i-th row and j-th column in the third sample image; and the training device encodes the CSI phase of the first wireless signal of the j-th frame on the i-th channel among N channels arranged in a second order as the pixel value of the i-th row and j-th column in the fourth sample image, i and j are positive integers, i≤N, j≤M, the sample images include a third sample image and a fourth sample image, and the first order is different from the second order.

[0046] In the above method, the sample images used for training include multiple images obtained by channel coding in different orders, which provide differentiated input data for model training, reduce the difficulty of model training, and enable the trained target human action recognition model to better extract richer feature information, further improving the recognition accuracy.

[0047] In a possible implementation of an embodiment of the present application, the signal characteristics include a received signal strength indication (RSSI), and the training device encodes the signal characteristics of the first wireless signal of the M frames in the time and channel dimensions to obtain a sample image. It may also include: the training device encodes the RSSI of the first wireless signal of the jth frame into the pixel value of the jth column in the fifth sample image, and the sample image also includes the fifth sample image.

[0048] In the above method, since RSSI is the superposition of the amplitudes of all subcarriers, the overall amplitude of the wireless signal is taken into consideration and used as input data for human motion recognition, thereby improving accuracy.

[0049] Optionally, the frequencies of the N channels arranged in the first order are increasing, and the frequencies of the N channels arranged in the second order are non-increasing.

[0050] Optionally, the frequencies of the N channels arranged in the first order are decreasing, and the frequencies of the N channels arranged in the second order are non-decreasing.

[0051] Optionally, any two adjacent channels in the N channels arranged in the first order are not adjacent in the N channels arranged in the second order.

[0052] In a fourth aspect, the present application also provides a human motion recognition device, including:

[0053] an acquisition unit, configured to acquire an image to be identified, wherein the image to be identified is an image obtained by encoding signal features of M frames of wireless signals in dimensions of time and channel, the M frames of wireless signals are wireless signals transmitted by a first wireless transmitting device and received by a first wireless receiving device, a distance between the first wireless receiving device and the first wireless transmitting device is greater than a first threshold, and M is an integer greater than 1;

[0054] The recognition unit is used to input the image to be recognized into a target human action recognition model, and the type of human action corresponding to the image to be recognized.

[0055] Optionally, the device may further include a functional unit for implementing the method described in the first aspect. For details, please refer to the relevant description of the first aspect above, which will not be repeated here.

[0056] In a fifth aspect, an embodiment of the present application further provides a wireless signal processing device, including:

[0057] A receiving unit, used to receive M frames of wireless signals transmitted by a wireless transmitting device;

[0058] An extraction unit, used to extract signal features of each frame of wireless signals in the M frames of wireless signals;

[0059] The encoding unit is used to encode the signal characteristics of the M frames of wireless signals in time and channel dimensions to obtain an image to be identified.

[0060] Optionally, the wireless signal processing device may further include a sending unit, configured to send the image to be recognized to an execution device, so that the execution device recognizes the category of the human body movement according to the image to be recognized.

[0061] Optionally, the device may further include a functional unit for implementing the method described in the second aspect. For details, please refer to the relevant description of the second aspect above, which will not be repeated here.

[0062] In a sixth aspect, an embodiment of the present application further provides a neural network training device, characterized in that the method comprises:

[0063] a receiving unit, configured to receive signal characteristics of M frames of first wireless signals sent by the first wireless receiving device, wherein the M frames of first wireless signals are wireless signals transmitted by the first wireless transmitting device and received by the first wireless receiving device when a first human body motion occurs between the first wireless receiving device and the first wireless transmitting device, and M is an integer greater than 1;

[0064] An encoding unit, configured to encode signal features of the M frames of the first wireless signal in time and channel dimensions to obtain a sample image;

[0065] A training unit is used to train a neural network according to the sample image and the category of human motion corresponding to the sample image to obtain the target human motion recognition model, wherein the sample image is a training input, the category of human motion corresponding to the sample image is a training label, and the first human motion belongs to the category of human motion corresponding to the sample image.

[0066] Optionally, the device may further include a functional unit for implementing the method described in the second aspect. For details, please refer to the relevant description of the second aspect above, which will not be repeated here.

[0067] In the seventh aspect, an embodiment of the present application also provides a human motion recognition device, including a processor and a memory, wherein the memory is used to store programs, and the processor executes the programs stored in the memory. When the program stored in the memory is executed, the processor implements any one of the methods shown in the first aspect.

[0068] In the eighth aspect, an embodiment of the present application also provides a wireless signal processing device, including a processor, a wireless communication module and a memory, the memory is used to store programs, the wireless communication module is used to receive wireless signals sent by a wireless transmitting device and extract signal characteristics of the M-frame wireless signals, and the processor executes the program stored in the memory. When the program stored in the memory is executed, the processor implements any one of the methods shown in the second aspect.

[0069] In the ninth aspect, an embodiment of the present application further provides a neural network training device, comprising a processor and a memory, wherein the memory is used to store programs, and the processor executes the programs stored in the memory. When the programs stored in the memory are executed, the processor implements any one of the methods shown in the third aspect.

[0070] In the tenth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable medium is used to store program code, wherein the program code includes code for executing any one of the methods shown in the first aspect.

[0071] In the eleventh aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable medium is used to store program code, wherein the program code includes a method for executing any one of the methods shown in the second aspect.

[0072] In the twelfth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable medium is used to store program code, wherein the program code includes code for executing any one of the methods shown in the third aspect.

[0073] In a thirteenth aspect, an embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method in the first aspect.

[0074] In a fourteenth aspect, an embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method in the above-mentioned second aspect.

[0075] In a fifteenth aspect, an embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any one of the methods in the third aspect described above.

[0076] In the sixteenth aspect, a chip is provided, comprising a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface to execute the method in the first aspect, the second aspect or the third aspect.

[0077] Optionally, as an implementation method, the chip may also include a memory, in which instructions are stored, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to execute the method in the first aspect, the second aspect or the third aspect.

[0078] In a seventeenth aspect, an electronic device is provided, which includes the human body motion recognition device of any one of the fourth to sixth aspects.

[0079] In an eighteenth aspect, an electronic device is provided, comprising the wireless signal processing device of any one of the fifth to seventh aspects.

[0080] In a nineteenth aspect, an electronic device is provided, which includes the neural network training device of any one of the sixth to eighth aspects above. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0082] Figure 1 is a schematic diagram of a wireless sensing system provided in an embodiment of the present application;

[0083] Figure 2A is a schematic diagram illustrating a distribution mode of a wireless transmitting device and a wireless receiving device for identifying a human fall provided in an embodiment of the present application;

[0084] Figure 2B A schematic diagram of a flow chart of a method for identifying a human fall provided in an embodiment of the present application;

[0085] Figure 2C is a schematic diagram illustrating another distribution mode of wireless transmitting devices and wireless receiving devices for identifying a human fall provided in an embodiment of the present application;

[0086] Figure 2D is a schematic diagram illustrating a method for identifying a human fall provided in an embodiment of the present application;

[0087] Figure 3A is a schematic diagram illustrating a distribution mode of a wireless transmitting device and a wireless receiving device in a method for identifying an intruder intrusion provided by an embodiment of the present application;

[0088] Figure 3B A flowchart of a method for identifying an intruder intrusion provided by an embodiment of the present application;

[0089] Figure 4 It is a system architecture diagram provided by an embodiment of the present application;

[0090] Figure 5 is a schematic diagram of human action recognition based on a CNN model provided in an embodiment of the present application;

[0091] Figure 6 It is a schematic diagram of a chip hardware structure provided in an embodiment of the present application;

[0092] Fig. 7A is a schematic flow chart of a neural network training method provided in an embodiment of the present application;

[0093] Figure 7B It is a flowchart of a method for implementing a training device to obtain training samples provided in an embodiment of the present application;

[0094] Figure 7C It is a flowchart of another implementation method of the training device obtaining training samples provided in an embodiment of the present application;

[0095] Fig.7D is an example diagram of a first sample image and a second sample image provided in an embodiment of the present application;

[0096] Fig. 7E is an example diagram of a third sample image and a fourth sample image provided in an embodiment of the present application;

[0097] Figure 7F is an example diagram of a fifth sample image provided in an embodiment of the present application;

[0098] Figure 7Gis an example diagram of an example of a convolutional neural network architecture provided in an embodiment of the present application;

[0099] Fig. 8A It is a flow chart of a method for recognizing human body movements provided in an embodiment of the present application;

[0100] Figure 8B is a flow chart of another method for recognizing human body movements provided in an embodiment of the present application;

[0101] Figure 8C is a flow chart of another method for recognizing human body movements provided in an embodiment of the present application;

[0102] Fig. 9 is a schematic block diagram of a neural network training device provided in an embodiment of the present application;

[0103] Fig.10 is a schematic block diagram of a human motion recognition device provided in an embodiment of the present application;

[0104] Fig.11 is a schematic block diagram of a wireless signal processing device provided in an embodiment of the present application;

[0105] Fig.12 is a schematic block diagram of a neural network training device provided in an embodiment of the present application;

[0106] Fig.13 is a schematic block diagram of another human motion recognition device provided in an embodiment of the present application;

[0107] Fig.14 It is a schematic block diagram of another wireless signal processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0108] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0109] Before introducing the embodiments of the present application, the principle of motion recognition based on wireless signals involved in the embodiments of the present application is first introduced.

[0110] See also Figure 1The schematic diagram of the wireless sensing system shown in the figure includes at least one wireless transmitting device and at least one wireless receiving device, and the wireless transmitting device and the wireless receiving device are respectively set at different positions in the room. The wireless transmitting device (Tx) transmits wireless signals to the surroundings, and the wireless signals are mainly propagated to the receiving device (Rx) through three paths. The first is the line-of-sight (LOS) path, that is, after the wireless signal is transmitted by the wireless transmitting device, it reaches the wireless receiving device directly without being reflected by any obstacles; the second is the static object reflection path, that is, after being reflected by obstacles in the environment (such as ceilings, floors, tables and other indoor static objects), it reaches the wireless receiving device; the third is the dynamic object reflection path, that is, after being reflected by dynamic objects such as human bodies and animals, it reaches the wireless receiving device. Among the three paths, the first two propagation paths are relatively stable; however, under the third propagation path, due to the different positions and movements of the human body, the propagation paths of the wireless signals will be different, which will lead to different signal characteristics such as RSSI and CSI of the wireless signals reflected by the human body received by the wireless receiving device. Therefore, human body movements can be identified by analyzing changes in signal characteristics such as RSSI and CSI of the received wireless signal.

[0111] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0112] The human motion recognition method provided in the embodiment of the present application can be applied to motion monitoring, indoor security, automatic control or other scenarios requiring human motion recognition. Specifically, the human motion recognition method in the embodiment of the present application can be applied to scenarios of human fall detection, indoor security and human motion control, etc. The following briefly introduces the scenarios of human fall detection, indoor security and human motion control.

[0113] Scenario a (human fall detection):

[0114] Usually, when an elderly person is alone at home or lives alone, or a child is alone at home, for the safety of the elderly / child, a wireless transmitting device and a wireless receiving device can be set up in the room, such as Figure 2A As shown. Figure 2B The figure is a flow chart of a method for identifying a human fall provided in an embodiment of the present application, and human fall detection is implemented through some or all of the following steps:

[0115] (a1) Collecting training samples

[0116] S202: The wireless receiving device can receive M frames of wireless signals S1 transmitted by the wireless transmitter when the human body falls, and obtain signal characteristics of the M frames of wireless signals S1 through the network card (specifically, the CSI and RSSI reading interface on the network card), such as RSSI and / or CSI, etc., where M is a positive integer greater than 1.

[0117] S204: Encode the signal features of the received M frames of wireless signal S1 in the dimensions of time and channel to obtain a sample image labeled “fall”.

[0118] S206: The wireless receiving device may receive M frames of wireless signals S2 transmitted by the wireless transmitter when the human body performs actions other than falling, and obtain signal characteristics of the M frames of wireless signals S2, such as RSSI and / or CSI, through the network card.

[0119] S208: Encode the signal features of the M frames of wireless signal S2 to obtain a sample image labeled as “non-fall”.

[0120] It should be understood that the above S104 or 108 can be performed by a wireless receiving device or a training device, and the steps in S202-S204 and the steps in S106-S108 can be performed in any order. The above S101-S108 is only an implementation of an embodiment of the present application. Multiple sample images containing labels can be obtained through multiple acquisitions to obtain a training sample set.

[0121] It should also be understood that in order to meet the user's requirements for the accuracy of human fall recognition and obtain a human fall recognition model with finer recognition granularity, the sample image can be annotated with labels of different granularities. In another implementation of the embodiment of the present application, in order to identify the room where the fall occurred, the label can also be "fallen in room A", "fallen in room B", "fallen in room C" and "did not fall", etc. The wireless transmitting device can collect and obtain the sample images corresponding to the above labels respectively.

[0122] Assuming that the wireless signals S1 and S2 include N channels, N is a positive integer, the obtained sample image can be an image of size M×N; further, if the signal characteristics include RSSI, CSI amplitude and CSI phase, the sample image can include 3 images of size M×N, which need to be trained using a neural network including 3 channels (such as a convolutional neural network); further, if the channels in the image whose signal characteristics are CSI amplitude and CSI phase are reordered, 2 more images can be obtained, and in this case, a convolutional neural network including 5 channels needs to be used for training.

[0123] The wireless receiving device may be a smart home device such as a lamp, refrigerator, washing machine, wardrobe, water heater, curtain, lock, wireless router, etc. in the room that has the function of receiving wireless signals and extracting signal characteristics of wireless signals. The wireless transmitting device may be a smart home device with the function of transmitting wireless signals, such as a WiFi router, etc. In the next wireless signal transmission and reception process, the wireless transmitting device may also be used as a wireless receiving device, and the wireless receiving device may also be used as a wireless transmitting device.

[0124] In order to meet the user's requirements for the accuracy of human fall recognition, different label granularities can be used. For example, in order to identify in which room the fall occurred, the classification of human actions (i.e., labels) can include falling in room A, not falling in room A, falling in room B, not falling in room B, falling in room C, not falling in room C, etc. The wireless acquisition device can collect sample images corresponding to each of the above labels.

[0125] (a2) Training a human fall recognition model

[0126] S210: The training device obtains a human fall recognition model through training with the collected image samples.

[0127] After obtaining the training sample set, the training device can use the training sample set to train the initialized neural network to obtain a human fall recognition model. The training device can be the wireless receiving device itself; the training device can also be a wireless transmitting device, such as a WiFi router, etc.; it can also be a terminal such as a mobile phone, a tablet computer, a personal digital assistant (PDA), etc.; it can also be a server or a cloud, etc. In the case where the training device is not a wireless receiving device, the wireless receiving device can send the signal features of the collected M-frame wireless signals to the training device, and the training device encodes them to obtain a sample image; or, the wireless receiving device encodes the signal features of the collected M-frame wireless signals to obtain a sample image. It should also be understood that the wireless receiving device can receive the label input by the user for the sample image, and the wireless receiving device can also send the label corresponding to the sample image to the training device; or, after receiving the signal features of the sample image or the M-frame wireless signal, the training device can receive the label input by the user for the signal features of the received sample image or the M-frame wireless signal to achieve labeling of the sample image.

[0128] (a3) Online recognition of human falls

[0129] S212: The wireless receiving device receives M frames of wireless signals transmitted by the wireless transmitter in real time, and extracts signal features of the M frames of wireless signals S6.

[0130] S214: The wireless receiving device may send the signal characteristics of the M-frame wireless signal to the executing device.

[0131] S216: The execution device may encode the received signal features in the dimensions of time and channel to obtain an image to be identified.

[0132] S218: The execution device inputs the recognition image into a human fall recognition model, and the human fall recognition model processes the input image to be recognized to obtain a recognition result.

[0133] The wireless receiving device encodes the signal features of the M frames of wireless signals to obtain the image to be identified, and the wireless receiving device can send the image to be identified to the execution device. The execution device can be the same as the training device or a different device. The execution device can be the wireless receiving device itself; it can also be a wireless transmitting device, such as a WiFi router; it can also be a terminal such as a mobile phone, a tablet computer, a personal digital assistant (PDA); it can also be a server or a cloud.

[0134] It should be understood that the execution device can also send the recognition result to a terminal, such as a mobile phone of the guardian of the elderly or child; when the recognition result is a fall, the execution device can send the recognition result to the guardian's mobile phone, tablet computer and other terminals, so that the guardian can know the fall of the elderly or child and take emergency measures in time; it can also call "120" to send a distress message to the emergency center, etc.

[0135] In another scenario of human fall recognition, a wireless receiving device can be set up in different areas, such as Figure 2C As shown, a wireless receiving device is respectively set in different areas such as Room A, Room B, and Room C, including a lamp 201 in Room A, a water heater 202 in Room B, and a refrigerator 203 in Room C. It should be understood that the first wireless receiving device (e.g., lamp 201) set in the first area is used to detect a human fall event in the first area (Room A), and the first wireless receiving device is any wireless receiving device. One or more wireless transmitting devices can be set. In the embodiment of the present application, a wireless transmitting device is used as an example, for example, a WiFi router 204 is set in Room C.

[0136] For different combinations of wireless transmitting devices and wireless receiving devices, since the locations of each wireless transmitting device and wireless receiving device are different, the transmission paths of their wireless signals are different. Therefore, a human fall recognition model needs to be trained for each pair of wireless transmitting devices and wireless receiving devices (that is, a combination of a wireless transmitting device and a wireless receiving device), that is, one pair of wireless transmitting devices and wireless receiving devices corresponds to one human fall recognition model.

[0137] For example, the lamp 201 in room A collects M frames of wireless signals when a person falls in room A, and extracts the signal features of the M frames of wireless signals; then, the signal features of the received M frames of wireless signals are encoded in the dimensions of time and channel, and a sample image labeled "falling in room A" is obtained. Similarly, the lamp 201 in room A collects M frames of wireless signals when a person is in room A but does not fall, and extracts and encodes the M frames of wireless signals to obtain a sample image labeled "not falling in room A", and the lamp 201 in room A collects M frames of wireless signals when a person is not in room A, and extracts and encodes the M frames of wireless signals to obtain a sample image labeled "not in room A". Furthermore, the training device trains an initialized neural network (such as a convolutional neural network) through the sample images labeled as "falling in room A", the sample images of "not falling in room A" and the sample images of "not in room A" obtained above to obtain a first human fall recognition model, wherein the training device can be a lamp 201, a WiFi router 204, a mobile phone or tablet computer, a server, etc.

[0138] Similarly, the water heater 202 in room B collects the wireless signals transmitted by the WiFi router 203, and obtains sample images with labels of "falling in room B", "not falling in room B" and "no longer in room B", and then trains the initialized convolutional neural network based on the sample images obtained by the water heater 202 to obtain a second human fall recognition model. Similarly, the refrigerator 203 in room C can collect the wireless signals transmitted by the WiFi router 203, and obtain sample images with labels of "falling in room C", "not falling in room C" and "not in room C", and then trains the initialized convolutional neural network based on the sample images obtained by the refrigerator 203 to obtain a third human fall recognition model. Regarding the specific implementation of the water heater 202 and the refrigerator 203 collecting wireless signals and obtaining sample images, please refer to the relevant description in the specific implementation of the lamp 201 collecting wireless signals to obtain sample images, which will not be repeated here.

[0139] After obtaining the above-mentioned first human fall recognition model, second human fall recognition model, and third human fall recognition model, the training device can send the trained model to the execution device. Then, the execution device can use the received model to identify whether a human fall event occurs in the room to be monitored. For example, the first wireless receiving device receives M frames of wireless signals transmitted by the first wireless transmitting device in real time, and then, the first wireless receiving device can extract signal features in the M frames of wireless signals, and then send the extracted signal features, the identifier of the first wireless transmitting device and the identifier of the second wireless receiving device to the execution device; the execution device can encode the received signal features in the dimensions of time and channel to obtain an image to be recognized; then, according to the correspondence between the combination of the first wireless transmitting device and the first wireless receiving device and the human fall recognition model, determine the human fall recognition model corresponding to the combination of the first wireless transmitting device and the first wireless receiving device, and input the image to be recognized into the human fall recognition model corresponding to the combination of the first wireless transmitting device and the first wireless receiving device to obtain the recognition result of the image to be recognized. Figure 2D As shown, the image encoded by the signal characteristics of the M frames of wireless signals transmitted by the WiFi router 204 received by the lamp 201 can be input into the first human fall recognition model, and then, one of the recognition results "fall in room A", "not fall in room A" or "not in room A" is obtained. Similarly, the fall events in room B and room C can be identified through the wireless signals collected by the water heater 204 and the refrigerator 203 respectively.

[0140] In another implementation of the embodiment, the execution device may be the wireless receiving device itself, and in this case, the wireless receiving device may store one or more human fall recognition models corresponding to the wireless reception. It should be understood that when there are multiple wireless transmitting devices, one wireless receiving device may be combined with multiple wireless transmitting devices, and in this case, one wireless receiving device may correspond to multiple wireless transmitting devices.

[0141] For example, the lamp 201 in room A receives the M-frame wireless signal transmitted by the WiFi router 204, and the lamp 201 extracts the signal features of the M-frame wireless signal, encodes the signal features to obtain the image to be identified, and then the lamp 201 inputs the image to be identified into the first human fall model to obtain the recognition result. When the recognition result is a fall, the lamp 201 can send a prompt message to the user's terminal to inform the user that someone in room A has fallen.

[0142] In an embodiment of the present application, by encoding the signal features of multiple frames of wireless signals into images, a neural network can be used for training to obtain a human fall recognition model, and then whether a person falls can be identified based on the multiple frames of wireless signals, thereby improving the accuracy of the recognition results.

[0143] Scenario b (indoor security):

[0144] Usually, for indoor security considerations, users need to monitor whether there is someone breaking into their home. In this case, one or more wireless transmitting devices and one or more wireless receiving devices can be set up in the room, such as Figure 3A The figure shows a schematic diagram of the arrangement of a wireless transmitting device and a wireless receiving device. Figure 3B The intruder detection method shown in the figure can realize intruder detection by some or all of the following steps:

[0145] (b1) Collect training samples;

[0146] S302: The wireless receiving device may receive the M frames of wireless signals S3 transmitted by the wireless transmitting device when someone breaks into the room, and then obtain the signal characteristics of the M frames of wireless signals S3, such as RSSI and / or CSI, through the network card (specifically, the CSI and RSSI reading interface on the network card), where M is a positive integer greater than 1. The action of someone breaking into the room may be someone opening the door, or someone entering from the door, etc.

[0147] S304: Encode the signal features of the M frames of wireless signal S3 in the dimensions of time and channel to obtain a sample image labeled “someone intrudes”.

[0148] S306: The wireless receiving device may receive the M frames of wireless signals S4 transmitted by the wireless transmitting device when no one breaks into the room (i.e., other actions or no actions other than the action of someone breaking into the room, such as someone is in the room but no one opens the door, or no one is in the room and no one opens the door, etc.). The action of someone breaking into the room may be someone opening the door, or someone entering from the door, etc.

[0149] S308: Encode the signal features of the M frames of wireless signal S4 in the dimensions of time and channel to obtain a sample image labeled “no intrusion”.

[0150] It should be understood that multiple sample images containing labels can be obtained through multiple acquisitions to obtain a training sample set. To improve the accuracy of recognition, the wireless transmitting device and the wireless receiving device can be set at the indoor door or outdoor door, and one can be set indoors.

[0151] It should also be understood that the above-mentioned sample images labeled “someone intruding” and sample images labeled “no one intruding” may also include other acquisition methods. For example, the wireless receiving device may also acquire M frames of wireless signals received when someone appears at the door, and obtain a sample image labeled “someone intruding”; the wireless receiving device may also acquire M frames of wireless signals when there is no one at the door and human body movements occur at other locations or when no human body movements occur, and obtain a sample image labeled “no one intruding”.

[0152] Similar to the above-mentioned wireless signals S1 and S2, the signal characteristics of the wireless signals S3 and S4 including N channels can be encoded to obtain 1 image of size M×N, 3 images of size M×N, 5 images of size M×N or other images, respectively. For details, please refer to the relevant description in the above-mentioned scenario 1, which will not be repeated here.

[0153] Similar to the above scenario 1, the wireless receiving device can be a smart home device in the room that has the function of receiving wireless signals and extracting signal characteristics of wireless signals. The wireless transmitting device can be a smart home device that has the function of transmitting wireless signals, etc. For details, please refer to the relevant description in the above scenario 1, which will not be repeated here.

[0154] In order to meet the user's requirements for the accuracy of indoor intrusion identification, different label granularities can be used. For example, in order to identify the room that a person breaks into, the classification of human actions (i.e., labels) can include breaking into room A, breaking into room B, breaking into room C, no one breaking into the room, etc. The wireless acquisition device can collect sample images corresponding to each of the above labels.

[0155] (b2) training the intruder identification model;

[0156] S310: The training device obtains an intruder recognition model through training with the collected image samples.

[0157] After obtaining the training sample set, the training device can use the training sample set to train the initialized neural network to obtain an intruder recognition model. The specific implementation of the training device and how the training device labels the image samples is similar to that in scenario 1. For details, please refer to the relevant description in step (b) in the above scenario 1, and the present embodiment will not be repeated.

[0158] (b3) Online detection

[0159] S312: The wireless receiving device may receive M frames of wireless signals transmitted by the wireless transmitter in real time, and extract signal features of the M frames of wireless signals.

[0160] S314: The wireless receiving device may send the signal characteristics of the M-frame wireless signal to the executing device.

[0161] S316: The execution device may encode the received signal features in the dimensions of time and channel to obtain an image to be identified.

[0162] S318: The execution device inputs the recognition image to the intruder recognition model, and the intruder recognition model processes the input image to be recognized to obtain a recognition result.

[0163] The specific implementation of the execution device is similar to that in Scenario 1. For details, please refer to the relevant description in step (c) in the above Scenario 1, and the embodiment of this application will not be repeated here.

[0164] In another implementation of the embodiment of the present application, similar to the above scenario 1, the above S216 can also be executed by a wireless receiving device, and the specific embodiment of the present application will not be repeated. It should be understood that the execution device can also send the recognition result to a terminal, such as the owner's mobile phone; when the recognition result is "someone broke in", the execution device can send the recognition result to the owner's mobile phone, tablet computer and other terminals, so that the owner can take emergency measures in time to understand the indoor safety situation; it can also call "110" to send an alarm message to the public security department, etc.

[0165] In an embodiment of the present application, by encoding the signal features of multiple frames of wireless signals into images, a neural network can be used for training to obtain an intruder recognition model, and then whether someone has broken into the room can be identified based on the multiple frames of wireless signals, thereby improving the accuracy of the recognition results.

[0166] Scene C (Human Motion Control)

[0167] To meet the personalized needs of users, human body movements can be used to control terminals such as mobile phones, tablets, PDAs, smart watches, VR / AR devices, and smart home devices such as TVs, water heaters, refrigerators, lights, locks, curtains, etc.

[0168] The user can design human body movements and set the operations corresponding to the human body movements. In one implementation (1) of the embodiment of the present application, the above operation can be an operation on a device, and the device to be operated is a mobile phone as an example, as shown in Table 1 below:

[0169] Serial number Human body movements operate 1 Open your arms Open the music player 2 Raise your hands Hang up the phone 3 Raise one hand Answering a call 4 applaud Take a photo after 2 seconds …… …… ……

[0170] Table 1

[0171] In a specific application scenario, when the mobile phone detects a current human motion, the target operation corresponding to the current human motion can be determined through the correspondence between the human motion and the operation as described in Table 1, and then the mobile phone executes the target operation.

[0172] The embodiment of the present application is illustrated using Table 1 as an example. In other implementations of the embodiment of the present application, other correspondences between human body movements and operations may also be set, and multiple operations may also act on multiple devices respectively, which is not limited in the embodiment of the present application.

[0173] To achieve the above functions, it is first necessary to identify human movements, collect training samples, and obtain a movement recognition model through training samples. The movement recognition model can identify the category of human movements, and the category of human movements can include multiple human movements.

[0174] (c1) Collect training samples

[0175] S402: The wireless receiver can receive the M frames of wireless signals S5 transmitted by the wireless transmitter when the first human body action occurs, and obtain the signal characteristics of the M frames of wireless signals S5 through the network card (specifically, the CSI and RSSI reading interface on the network card), such as RSSI and / or CSI, etc., where M is a positive integer greater than 1. The wireless receiving device and the wireless transmitting device can be distributed at any two different locations in the room. For example, in scene a Figure 2A and Figure 2C For the distribution method in , please refer to the relevant description in the above scenario a, which will not be repeated here.

[0176] S404: Encode the signal features of the received M frames of wireless signal S5 in the dimensions of time and channel to obtain a sample image labeled as “first human body motion”.

[0177] It can be understood that, through the above acquisition method, sample images with labels of various human body movements can be obtained. The above multiple training samples constitute a sample training set. Among them, the first human body movement is any one of the multiple human body movements.

[0178] In another implementation of the embodiment of the present application, the human body motions may be further classified according to the positions. In this case, the categories of the human body motions may include human body motion A occurring at the first position, human body motion B occurring at the first position, human body motion A occurring at the second position, human body motion B occurring at the second position, etc. The embodiment of the present application is described by taking the classification of human body motions into multiple human body motions as an example.

[0179] (c2) training action recognition model;

[0180] S406: The training device obtains an action recognition model through training with the collected image samples.

[0181] After obtaining the training sample set, the training device can use the training sample set to train the initialized neural network to obtain the action recognition model. The specific implementation of the training device and how the training device labels the image samples is similar to that in scenario 1. For details, please refer to the relevant description in step (a2) in the above scenario a, and the present embodiment will not be repeated.

[0182] (c3) Online detection

[0183] S408: The wireless receiving device may receive M frames of wireless signals transmitted by the wireless transmitter in real time, and extract signal features of the M frames of wireless signals.

[0184] S410: The wireless receiving device may send the signal characteristics of the M-frame wireless signal to the executing device.

[0185] S412: The execution device may encode the received signal features in the dimensions of time and channel to obtain an image to be recognized.

[0186] S414: The execution device inputs the recognition image into the action recognition model, and the action recognition model processes the input image to be recognized to obtain a recognition result. The recognition result indicates the category of the human body action.

[0187] The specific implementation of the execution device is similar to that in scenario one. For details, please refer to the relevant description in step (a3) ​​in the above scenario a, and the embodiment of this application will not be repeated here.

[0188] In another implementation of the embodiment of the present application, similar to the above scenario 1, the above S412 can also be executed by a wireless receiving device. At this time, the wireless receiving device can send the image to be identified to the execution device, and the specific embodiment of the present application will not be repeated.

[0189] It should be understood that after the execution device obtains the recognition result, that is, the human body action or category of human body action corresponding to the image to be identified, also referred to as the target human body action in the embodiment of the present application, it can execute or implement the operation corresponding to the target human body action based on the correspondence between the human body action and the operation.

[0190] For example, the execution device is a mobile phone, and when the target human body motion is "open arms", according to the correspondence between human body motions and operations shown in Table 1, the mobile phone can open a music player at this time.

[0191] For another example, the execution device is a mobile phone. When the operation corresponding to the target human body action is to turn on the water heater, the mobile phone can send an opening instruction to the water heater through the client, so that the water heater turns on after receiving the opening instruction. At this time, terminals such as mobile phones and tablets can include the function of managing smart home devices in the home, for example, controlling and managing devices under the same WiFi.

[0192] It should be noted that the specific implementation of scene a, scene b, and scene c can also refer to the relevant description in the embodiment of the training method of the following model and the embodiment of the human motion recognition method, which will not be repeated here.

[0193] It should be noted that although M frames of wireless signals, sample images, training sample sets, initialized neural networks, images to be identified, etc. are used in various scenarios of this application, it should be understood that in different scenarios, the meanings of the above-mentioned M frames of wireless signals, initialized neural networks, and images to be identified are independent of each other.

[0194] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and related concepts such as neural networks involved in the embodiments of the present application are first introduced below.

[0195] (1) Human motion recognition based on wireless signals

[0196] See also Figure 1 The wireless sensing system shown in the figure includes at least one wireless transmitting device and at least one wireless receiving device.

[0197] When the wireless signal transmitted by the wireless transmitting device reaches the wireless receiving device after being reflected by dynamic objects such as human bodies and animals, the different positions and movements of the human bodies will lead to different propagation paths of the wireless signal, which in turn leads to different signal characteristics such as RSSI and CSI of the wireless signal reflected by the human body received by the wireless receiving device. Therefore, human movements can be identified by analyzing the changes in signal characteristics such as RSSI and CSI of the received wireless signal.

[0198] In an embodiment of the present application, the signal features of multiple frames of wireless signals are encoded into an image in the channel and time dimensions, and then related methods such as image processing, machine learning, and computer graphics are used to identify human body movements or categories of human body movements corresponding to the image data.

[0199] (2) Neural Network

[0200] A neural network can be composed of neural units. A neural unit can be a s The output of the operation unit with the intercept 1 as input can be:

[0201]

[0202] Where s=1, 2, ...n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolution layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the characteristics of the local receptive field. The local receptive field can be an area composed of several neural units.

[0203] (3) Deep Neural Networks

[0204] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with many hidden layers. There is no special metric for "many" here. From the position of different layers of DNN, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Although DNN looks complicated, the work of each layer is actually not complicated. Simply put, it is the following linear relationship expression: in, is the input vector, is the output vector, b is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since there are many DNN layers, the number of coefficients W and offset vectors b is also large. The definitions of these parameters in DNN are as follows: Take coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the kth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as It should be noted that the input layer does not have a W parameter. In a deep neural network, more hidden layers allow the network to better describe complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by many layers of vector W).

[0205] (4) Convolutional Neural Network

[0206] Convolutional neural network (CNN) is a deep neural network with a convolutional structure. Convolutional neural network contains a feature extractor consisting of a convolution layer and a subsampling layer. The feature extractor can be regarded as a filter, and the convolution process can be regarded as using a trainable filter to convolve with an input image or convolution feature plane (feature map). Convolution layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal. In the convolution layer of the convolutional neural network, a neuron can only be connected to some neurons in the adjacent layer. A convolution layer usually contains several feature planes, and each feature plane can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as the way to extract image information is independent of position. The implicit principle is that the statistical information of a part of the image is the same as that of other parts. This means that the image information learned in a part can also be used in another part. Therefore, the same learned image information can be used for all positions on the image. In the same convolution layer, multiple convolution kernels can be used to extract different image information. Generally speaking, the more convolution kernels there are, the richer the image information reflected by the convolution operation.

[0207] The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0208] (5) Loss Function

[0209] In the process of training a deep neural network, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually a process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible.

[0210] (6) Back propagation algorithm

[0211] Convolutional neural networks can use the error back propagation (BP) algorithm to correct the size of the parameters in the convolutional neural network during the training process, so that the reconstruction error loss of the convolutional neural network becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the parameters in the convolutional neural network are updated by back propagating the error loss information, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, which aims to obtain the optimal parameters of the convolutional neural network, such as the weight matrix.

[0212] (7) Pixel value

[0213] The pixel value of an image can be a red, green, and blue (RGB) color value, and the pixel value can be a long integer representing the color. For example, the pixel value is 256*Red+100*Green+76Blue, where Blue represents the blue component, Green represents the green component, and Red represents the red component. In each color component, the smaller the value, the lower the brightness, and the larger the value, the higher the brightness. For a grayscale image, the pixel value can be a grayscale value.

[0214] In the embodiment of the present application, the pixel value of the image to be identified is related to the signal characteristics of the M frames of wireless signals, and the pixel value of the sample image is related to the signal characteristics of the M frames of wireless signals from which the sample image is acquired. For example, the pixel value of the i-th row and j-th column in the image to be identified may include one or more of the CSI amplitude, CSI phase, RSSI, etc. of the j-th frame of wireless signals on the i-th channel of N channels arranged in a specific order after normalization or unprocessing.

[0215] The following describes the system architecture provided by the embodiment of the present application. Figure 4 , an embodiment of the present invention provides a system architecture 100. As shown in the system architecture 100, the system architecture may include one or more wireless transmitting devices 110, one or more wireless receiving devices 120, a training device 130 and an execution device 140, wherein:

[0216] The wireless transmitting device 110 is a device capable of transmitting wireless signals, and establishes a communication connection with the wireless receiving device 120 to send wireless signals to the wireless receiving device. The wireless receiving device 120 can receive multiple frames of wireless signals transmitted by the wireless transmitting device 110, and extract the signal characteristics of each frame of wireless signals, and the signal characteristics may include RSSI and / or CSI. In one implementation, the wireless receiving device 120 may include a network card, and extract the RSSI, CSI phase and CSI amplitude of each frame of wireless signals through the network card.

[0217] The wireless transmitting device 110 or the wireless receiving device 120 may be a smart home device with a short-range wireless communication function, such as a lamp, a refrigerator, a washing machine, a wardrobe, a water heater, a curtain, a lock, a television, a wireless router, or a terminal such as a mobile phone, a tablet computer, or a desktop computer at a fixed location. The wireless transmitting device 110 and the wireless receiving device 120 may communicate via Bluetooth, WiFi, IrDA, ZigBee, UWB, etc. For example, the wireless transmitting device 110 is a WiFi router and the wireless receiving device 120 is a smart TV, and the two communicate via WiFi.

[0218] In one implementation of the embodiments of the present application, the wireless receiving device 120 can send the extracted signal features to the training device 130 or the execution device 140. At this time, the training device 130 can encode the signal features of the received multi-frame wireless signals in the dimensions of time and channel to obtain the sample images in various embodiments of the present application. It should be understood that the wireless receiving device 120 can receive the labels input by the user for the multi-frame wireless signals, and the wireless receiving device 120 can also send the labels corresponding to the multi-frame wireless signals to the training device 130, which are the labels of the sample images; or, after the training device 130 extracts the signal features of the multi-frame wireless signals, it can receive the labels input by the user for the signal features of the multi-frame wireless signals to label the sample images.

[0219] Similarly, after receiving the signal characteristics of the multi-frame wireless signal sent by the wireless transmitting device 110, the execution device 140 can encode the signal characteristics of the received multi-frame wireless signal in the dimensions of time and channel to obtain the image to be identified in each embodiment of the present application.

[0220] In another implementation of the embodiment of the present application, the wireless receiving device 120 can encode the signal features of multiple frames of wireless signals in the dimensions of time and channel to obtain the image to be identified or the sample image in the present application. Then, the wireless receiving device 120 sends the obtained sample image to the database 150 or the training device 130, or sends the obtained image to be identified to the execution device 140. It should be understood that the wireless receiving device 120 can receive the label input by the user for the sample image, and the wireless receiving device 120 can also send the label corresponding to the sample image to the training device 130; or, after receiving the sample image, the training device 130 can receive the label input by the user for the sample image to achieve labeling of the sample image.

[0221] Optionally, the user may manually provide label data, and the manual provision may be performed through an interface provided by the wireless receiving device 120 or an I / O interface of the training device.

[0222] The training device 130 or the execution device 140 can be the wireless receiving device 120 itself, or the wireless transmitting device 110, such as a WiFi router, etc., or a terminal such as a mobile phone, a tablet computer, a personal digital assistant (PDA), or a server or the cloud.

[0223] A sample image and the human action category of the sample image constitute a training sample, and multiple training samples constitute a training sample set (also called training data) in the embodiment of the present application. After obtaining the training sample set, the training device 130 can use the training sample set to train the initialized neural network 131 to obtain a human action recognition model 132. The human action recognition model 132 can be applied to different systems or devices, such as Figure 4 The execution device 140 shown can input the image to be recognized into the target human action recognition model to obtain a recognition result, which indicates the type of human action occurring on the transmission path of the multi-frame wireless signal of the image to be recognized.

[0224] As attached Figure 4 As shown, the execution device 140 may be configured with an I / O interface 143 for data exchange with an external device, such as the wireless receiving device 120. The wireless receiving device 120 may send the image to be identified obtained by encoding the signal features of the multi-frame wireless signal received from the wireless transmitting device 110 or the signal features of the multi-frame wireless signal to the execution device 140. For another example, the execution device 140 may send the recognition result of the image to be identified to the user device 170 to inform the user of the recognition result of the image to be identified.

[0225] Optionally, the execution device 140 may include an image encoding module 141, which may encode signal features of multiple frames of wireless signals in time and channel dimensions to obtain an image to be identified.

[0226] When the execution device 140 encodes the signal characteristics of multiple frames of wireless signals, or when the calculation module 142 of the execution device 140 performs calculations and other related processing, the execution device 140 can call the data, code, etc. in the data storage system 150 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing in the data storage system 150.

[0227] Finally, the I / O interface 143 sends the processing result, such as the recognition result obtained above, to other devices or performs corresponding operations according to the recognition result.

[0228] It is worth noting that the training device 130 can generate a corresponding human motion recognition model 132 based on different training samples for different goals or different tasks, and the corresponding human motion recognition model 134 can be used to achieve the above goals or complete the above tasks, thereby providing the user with the desired results.

[0229] It can be understood that the wireless receiving device 120 can store the training sample in the database 160, and the training device 130 can obtain the human action recognition model 132 by training based on the training data maintained in the database 160. The following will describe in more detail how the training device 120 obtains the target human action recognition model 132 based on the training sample with the first embodiment. The target human action recognition model 132 can be used to implement the human action recognition method provided in the second embodiment of the present application.

[0230] It is worth noting that Figure 4 This is only a schematic diagram of a system architecture provided by an embodiment of the present invention. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. Figure 1 In the embodiment, the data storage system 150 is an external memory relative to the execution device 140. In other cases, the data storage system 150 may also be placed in the execution device 140.

[0231] The human action recognition method and apparatus provided in the embodiment of the present application can also be used to expand the training data in the database 160, such as Figure 4 The I / O interface 143 of the execution device 140 shown can send the image processed by the execution device (such as the image to be recognized) and the category of the human body movement input by the user as a training data pair to the database 160, so that the training data maintained by the database 160 is richer, thereby providing richer training data for the training work of the training device 130.

[0232] In some embodiments of the present application, the wireless receiving device 120 may also serve as the execution device 140. In some embodiments of the present application, the training device 130 and the execution device 140 may be the same device. For details, please refer to the relevant descriptions in the embodiment of the training method of the neural network of the present application (i.e., embodiment 1) and the embodiment of the human motion recognition method (i.e., embodiment 2), which will not be repeated here.

[0233] As mentioned in the previous basic concept introduction, convolutional neural network is a deep neural network with convolution structure, which is a deep learning architecture. Deep learning architecture refers to multiple levels of learning at different abstract levels through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network, in which each neuron can respond to the image input into it.

[0234] like Figure 5 As shown, the convolutional neural network (CNN) 200 may include an input layer 210 , a convolutional layer / pooling layer 220 (wherein the pooling layer is optional), and a neural network layer 230 .

[0235] Convolutional layer / pooling layer 220:

[0236] Convolutional Layer:

[0237] like Figure 5 The convolution layer / pooling layer 220 shown may include layers 221-226, for example: in one implementation, layer 221 is a convolution layer, layer 222 is a pooling layer, layer 223 is a convolution layer, layer 224 is a pooling layer, layer 225 is a convolution layer, and layer 226 is a pooling layer; in another implementation, layers 221 and 222 are convolution layers, layer 223 is a pooling layer, layer 224 and 225 are convolution layers, and layer 226 is a pooling layer. That is, the output of a convolution layer can be used as the input of a subsequent pooling layer, or as the input of another convolution layer to continue the convolution operation.

[0238] The following will take the convolution layer 221 as an example to introduce the internal working principle of a convolution layer.

[0239] The convolution layer 221 may include a plurality of convolution operators, which are also called kernels. The convolution operator is equivalent to a filter that extracts specific information from the input image matrix in image processing. The convolution operator can be essentially a weight matrix, which is usually predefined. In the process of performing convolution operations on the image, the weight matrix is ​​usually processed one pixel after another (or two pixels after two pixels... depending on the value of the step length stride) in the horizontal direction on the input image, thereby completing the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image. In the process of performing convolution operations, the weight matrix will extend to the entire depth of the input image. Therefore, convolution with a single weight matrix will produce a convolution output with a single depth dimension, but in most cases, a single weight matrix is ​​not used, but multiple weight matrices of the same size (row × column), that is, multiple isotype matrices, are applied. The output of each weight matrix is ​​stacked to form the depth dimension of the convolution image, and the dimension here can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features in the image, for example, one weight matrix is ​​used to extract image edge information, another weight matrix is ​​used to extract specific colors of the image, and another weight matrix is ​​used to blur unwanted noise in the image, etc. The multiple weight matrices have the same size (rows × columns), and the feature maps extracted by the multiple weight matrices of the same size are also the same size. The extracted feature maps of the same size are then merged to form the output of the convolution operation.

[0240] The weight values ​​in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, so that the convolutional neural network 200 can make correct predictions.

[0241] When the convolutional neural network 200 has multiple convolutional layers, the initial convolutional layer (for example, 221) often extracts more general features, which can also be called low-level features. As the depth of the convolutional neural network 200 increases, the features extracted by the later convolutional layers (for example, 226) become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

[0242] Pooling layer:

[0243] Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer. Figure 5 Each layer 221-226 illustrated in 220 may be a convolution layer followed by a pooling layer, or may be multiple convolution layers followed by one or more pooling layers. In the image processing process, the only purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer may include an average pooling operator and / or a maximum pooling operator for sampling the input image to obtain an image of smaller size. The average pooling operator may calculate the pixel values ​​in the image within a specific range to generate an average value as the result of average pooling. The maximum pooling operator may take the pixel with the largest value in the range within a specific range as the result of maximum pooling. In addition, just as the size of the weight matrix used in the convolution layer should be related to the image size, the operator in the pooling layer should also be related to the image size. The size of the image output after processing by the pooling layer may be smaller than the size of the image input to the pooling layer, and each pixel in the image output by the pooling layer represents the average value or maximum value of the corresponding sub-region of the image input to the pooling layer.

[0244] Neural Network Layer 230:

[0245] After being processed by the convolution layer / pooling layer 220, the convolution neural network 200 is not sufficient to output the required output information. As mentioned above, the convolution layer / pooling layer 220 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolution neural network 200 needs to use the neural network layer 230 to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer 230 may include multiple hidden layers (such as Figure 5231, 232 to 23n) and output layer 240 shown, the parameters contained in the multi-layer hidden layer can be pre-trained according to relevant training data of a specific task type, for example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc. ...

[0246] After the multiple hidden layers in the neural network layer 230, that is, the last layer of the entire convolutional neural network 200 is the output layer 240, which has a loss function similar to the classification cross entropy, specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 200 (such as Figure 5 The propagation from 210 to 240 is forward propagation), and the reverse propagation (such as Figure 5 The propagation from 240 to 210 is called back propagation) and then starts to update the weight values ​​and biases of the aforementioned layers to reduce the loss of the convolutional neural network 200 and the error between the result output by the convolutional neural network 200 through the output layer and the ideal result.

[0247] It should be noted that if Figure 5 The convolutional neural network 200 shown is only an example of a convolutional neural network. In specific applications, the convolutional neural network may also exist in the form of other network models.

[0248] The following introduces a chip hardware structure provided by an embodiment of the present application.

[0249] Figure 6 A chip hardware structure provided in an embodiment of the present invention includes a neural network processor 30. The chip can be arranged as follows: Figure 4 The execution device 140 shown in FIG. 1 is used to complete the calculation work of the calculation module 142. The chip can also be set in a Figure 4 The training device 130 shown in FIG. 1 is used to complete the training work of the training device 130 and output the target model / rule 132. Figure 6 The algorithms of each layer in the convolutional neural network shown in the figure can be used in Figure 6 is implemented in the chip shown.

[0250] The neural network processor 30 can be any processor suitable for large-scale XOR operation processing, such as NPU, TPU, or GPU. Take NPU as an example: NPU can be mounted on the host CPU as a coprocessor, and the host CPU assigns tasks to it. The core part of NPU is the operation circuit 303, which is controlled by the controller 304 to extract the matrix data in the memory (301 and 302) and perform multiplication and addition operations.

[0251] In some implementations, the operation circuit 303 includes multiple processing units (Process Engine, PE) inside. In some implementations, the operation circuit 303 is a two-dimensional systolic array. The operation circuit 303 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 303 is a general-purpose matrix processor.

[0252] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit 303 takes the weight data of the matrix B from the weight memory 302 and caches it on each PE in the operation circuit 303. The operation circuit 303 takes the input data of the matrix A from the input memory 301, performs matrix operations based on the input data of the matrix A and the weight data of the matrix B, and stores the partial results or the final results of the matrix in the accumulator 308.

[0253] The unified memory 306 is used to store input data and output data. The weight data is directly transferred to the weight memory 302 through the memory unit access controller (DMAC, Direct Memory Access Controller) 305. The input data is also transferred to the unified memory 306 through the DMAC.

[0254] The bus interface unit (BIU) 310 is used for the interaction between the DMAC and the instruction fetch buffer 309; the bus interface unit 301 is also used for the instruction fetch buffer 309 to obtain instructions from the external memory; the bus interface unit 301 is also used for the storage unit access controller 305 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0255] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 306, or to transfer weight data to the weight memory 302, or to transfer input data to the input memory 301.

[0256] The vector calculation unit 307 includes multiple operation processing units, and when necessary, further processes the output of the operation circuit 303, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. The vector calculation unit 307 is mainly used for the calculation of non-convolutional layers or fully connected layers (FC) in neural networks, and can specifically process the calculations of Pooling, Normalization, etc. For example, the vector calculation unit 307 can apply a nonlinear function to the output of the operation circuit 303, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 307 generates a normalized value, a merged value, or both.

[0257] In some implementations, the vector calculation unit 307 stores the processed vectors in the unified memory 306. In some implementations, the vectors processed by the vector calculation unit 307 can be used as activation inputs of the operation circuit 303, for example, for use in subsequent layers in a neural network, such as Figure 5 As shown, if the current processing layer is hidden layer 1 (231), the vector processed by the vector calculation unit 307 can also be used in the calculation of hidden layer 2 (232).

[0258] The controller 304 is connected to an instruction fetch buffer 309 for storing instructions used by the controller 304 .

[0259] The unified memory 306, the input memory 301, the weight memory 302 and the instruction fetch memory 309 are all on-chip memories. The external memory is independent of the NPU hardware architecture.

[0260] in, Figure 5 The operations of each layer in the convolutional neural network shown can be performed by the operation circuit 303 or the vector calculation unit 142.

[0261] The following describes the method provided by this application from the perspectives of model training and model application:

[0262] The training method of the human motion recognition model provided in the embodiment of the present application involves the processing of computer vision. Specifically, data processing methods such as data training, machine learning, and deep learning can be applied to the training data (such as the sample image in the present application) to perform symbolic and formalized intelligent information modeling, extraction, preprocessing, training, etc., and finally obtain a trained target human motion recognition model (specifically a human fall recognition model in scene a, specifically an intruder recognition model in scene b, and specifically an action recognition model in scene c); and, the human motion recognition method provided in the embodiment of the present application can use the above-mentioned trained human motion recognition model, input the input data (such as the image to be recognized in the present application) into the trained human motion recognition model, and obtain output data (such as the recognition result in the present application). It should be noted that the neural network training method and human motion recognition method provided in the embodiment of the present application are inventions based on the same concept, and can also be understood as two parts in a system, or two stages of an overall process: such as a model training stage and a model application stage.

[0263] Embodiment 1:

[0264] It should be noted that the embodiment of the present application can be applied to a system consisting of at least one wireless transmitting device and at least one wireless receiving device. The embodiment of the present application is described by taking a first wireless transmitting device and a first wireless receiving device as an example.

[0265] like Fig. 7A As shown, a neural network training method is provided in Example 1 of the present application. The method 70 can be specifically performed as follows: Figure 4 The training device 130 shown in FIG. 1 is executed, and the training samples in the method 70 may be as follows: Figure 4 The training data maintained in the database 160 shown in FIG. 1 may be used by the training device 130 to obtain training samples from the database 160. In another implementation, the sample image in the method 70 may also be sent to the training device 130 via the wireless receiving device 120, such as Figure 7B The flowchart of a method for implementing a training device to obtain training samples is shown; it can also be obtained by encoding the signal features of the received multi-frame wireless signals by the training device 130 in the time and signal dimensions, such as Figure 7C The flowchart shown is a schematic diagram of another implementation method for a training device to obtain training samples.

[0266] Optionally, the method 70 may be processed by a CPU, or by a CPU and a GPU together, or may not use a GPU but use other processors suitable for neural network calculations, for example, Figure 6 The neural network processor 30 shown is not limited here.

[0267] It should be noted that the embodiments of the present application can be applied to a system consisting of at least one wireless transmitting device and at least one wireless receiving device. The embodiments of the present application are described by taking a first wireless transmitting device and a first wireless receiving device as examples. As mentioned above, the training method of a neural network provided in the embodiments of the present application may include some or all of the following steps:

[0268] S72: Obtain a training sample, where the training sample includes a sample image and a category of a human body action corresponding to the sample image.

[0269] Among them, the sample image is obtained by encoding the signal characteristics of M frames of first wireless signals in time and channel dimensions, and the M frames of first wireless signals are wireless signals transmitted by the first wireless transmitting device and received by the first wireless receiving device when the first human motion occurs between the first wireless receiving device and the first wireless transmitting device. The first human motion belongs to the category of human motion corresponding to the sample image, the distance between the first wireless receiving device and the first wireless transmitting device is greater than the first threshold, and M is an integer greater than 1.

[0270] Among them, the first wireless receiving device can establish a communication connection with the first wireless transmitting device, and the first wireless transmitting device can transmit a wireless signal to the first wireless receiving device, wherein the wireless signal can be a signal carrying data or a handshake signal transmitted by the wireless transmitting device at a certain period, and there is no limitation on this.

[0271] S74: Using the sample image as training input and the category of the human action corresponding to the sample image as training label, the initialized neural network is trained to obtain a target human action recognition model.

[0272] For example, the sample image is input into an initialized neural network (or an initialized convolutional neural network) to obtain a predicted category. Then, according to the error between the predicted category and the true category (i.e., the label of the sample image), the model parameters of the initialized neural network are adjusted through the back propagation algorithm. The model parameters of the neural network are adjusted iteratively to make this error smaller and smaller. The smaller the error, the closer the predicted category is to the true category, and the more accurately the network can predict the category of human body movements.

[0273] In one implementation of an embodiment of the present application, corresponding to the above-mentioned scenario a, when the first wireless signal of the M frame is S1, the first human body motion is a human body falling motion; when the first wireless signal of the M frame is S2, the first human body motion is other motions or no motion except the human body falling motion. The categories of human body motions may include falling and non-falling. The training samples may include sample images labeled "falling" and sample images labeled "non-falling". The target human body motion recognition model obtained by training is a human body falling recognition model.

[0274] In another implementation of the embodiment of the present application, corresponding to the above-mentioned scenario b, when the first wireless signal of the M frame is S3, the first human body motion is the action of someone breaking into the room; when the first wireless signal of the M frame is S4, the first human body motion is other actions or no action except the action of someone breaking into the room. The categories of human body motions may include the action of someone breaking into the room and the action of no one breaking into the room. The training samples may include sample images labeled "someone breaking in" and sample images labeled "no one breaking in". The target human body motion recognition model obtained by training is the intruder recognition model.

[0275] In another implementation of the embodiment of the present application, corresponding to the above scenario c, the category of human motion may include multiple human motions. The M-frame first wireless signal is S5. The training sample may include a sample image labeled "first human motion", and the first human motion may be any one of multiple human motions. The target human motion recognition model obtained by training is the motion recognition model.

[0276] like Figure 7B As shown, in one implementation of S72: the training device and the first wireless receiving device are different devices, see Figure 7B The steps shown in steps S7202-S7210 in FIG. are as follows:

[0277] S7202: The first wireless receiving device receives M frames of first wireless signals when the first human body motion occurs, and extracts signal features of the M frames of first wireless signals.

[0278] In some embodiments of the present application, the M frames of first wireless signals are wireless signals received by the first wireless receiving device when a first human body motion is transmitted between the first wireless receiving device and the first wireless receiving device. The M frames of first wireless signals are sorted in the time sequence when they are received, and M is an integer greater than 1.

[0279] S7204: The first wireless receiving device encodes signal features of the M frames of the first wireless signal in time and signal dimensions to obtain a sample image.

[0280] S7206: The first wireless receiving device receives the category of the human body action input for the sample image, and obtains a training sample, where the training sample includes the sample image and the category of the human body action input for the sample image.

[0281] In the embodiment of the present application, the category of the human body motion input for the sample image is also referred to as the category of the human body motion corresponding to the sample image, that is, the category of the human body motion to which the first human body motion belongs.

[0282] The first human motion may be any motion. In different application scenarios, human motion may have different classification methods. For example, in an implementation of scenario one, the categories of human motion may include falling and not falling; for another example, in an implementation of scenario two, the categories of human motion may be divided into someone breaking in and no one breaking in; for another example, in an implementation of scenario three, the categories of human motion may be divided into first motion, second motion, third motion, etc. The user may identify the category to which the first human motion belongs, and then input the category to which the first human motion belongs into the wireless receiving device for the image sample. Then, the wireless receiving device may receive the category of human motion input by the user for the sample image, and then obtain a training sample. The first human motion is set to different human motions, and then a set of training samples (also called a training sample set) may be obtained.

[0283] S7208: The first wireless receiving device sends the training sample to the training device.

[0284] S7210: The training device receives training samples.

[0285] It should be understood that the training device can also obtain the training samples generated by the first wireless receiving device in other ways. For example, the first wireless receiving device can send the training samples to a database, and the training device can obtain the training samples from the database. Then, the training device can train the target human motion recognition model based on the obtained training samples.

[0286] In some embodiments of the present application, the above step S7206 may also be performed by the training device after S7210.

[0287] like Figure 7C As shown, in the second implementation of S72, S7204 may be executed by a training device. After the above S7202, the first wireless receiving device may send the signal features of the M frames of the first wireless signal to the training device, and the training device may encode the signal features of the M frames of the first wireless signal to obtain a sample image. In this implementation, the following steps may be included:

[0288] S7212: The first wireless receiving device receives M frames of first wireless signals when the first human body motion occurs, and extracts signal features of the M frames of first wireless signals.

[0289] S7214: The first wireless receiving device sends signal characteristics of M frames of the first wireless signal to the training device.

[0290] S7216: The training device receives signal characteristics of M frames of the first wireless signal.

[0291] S7218: The training device encodes the signal features of the M frames of the first wireless signal in time and signal dimensions to obtain a sample image.

[0292] S7220: Training receives the category of human body movements inputted for the sample image, and obtains a training sample, where the training sample includes the sample image and a label for the sample image (ie, the category of the input human body movement).

[0293] In some embodiments of the present application, after S7216 and before S7218, the training device may receive a category of a human motion input for a signal feature of the first wireless signal of the M frames, and then use the input category of the human motion as a label of a sample image obtained by encoding the signal feature of the first wireless signal of the M frames.

[0294] In some embodiments of the present application, after S7212 and before S7214, the first wireless receiving device may receive the signal characteristics of the first wireless signal of the M frames / the category of the human body motion input by the first wireless signal of the M frames, and S7214 may also include sending the category of the input human body motion to the training device. At this time, the training device may not execute S7220, and the training device may use the category of the input human body motion as the label of the sample image encoded by the signal characteristics of the first wireless signal of the M frames.

[0295] In another implementation of S72, if the first wireless receiving device and the training device are the same device, then it is not necessary to execute the above S7208-S7210, S7214-S7216.

[0296] In an embodiment of the present application, a first wireless receiving device or a training device encodes the signal features of the M-frame first wireless signal in the time and signal dimensions to obtain a sample image. The signal features may include CSI amplitude, CSI phase, and optionally, RSSI. At this time, the sample image may include a first sample image and / or a third sample image obtained by encoding the CSI amplitude of the M-frame first wireless signal in the time and channel dimensions, and a second sample image and / or a fourth sample image obtained by encoding the CSI phase of the M-frame first wireless signal in the time and channel dimensions. Optionally, the sample image may also include a fifth sample image obtained by encoding the RSSI of the M-frame first wireless signal in the time and channel dimensions. The embodiment of the present application takes M frames of first wireless signals, each frame of the wireless signal includes N channels (N is an integer greater than 1) as an example to illustrate the encoding method of the first sample image, the second sample image, the third sample image, the fourth sample image, and the fifth sample image:

[0297] (1) First sample image

[0298] Among them, for the signal characteristics of each frame of the first wireless signal, the CSI information is composed of subcarriers of multiple channels (the embodiment of the present application takes 30 channels as an example), and each subcarrier contains amplitude and phase information, so the amplitude and phase need to be encoded separately. For the CSI amplitude of a frame of the first wireless signal, it is first normalized to limit the CSI amplitude of each channel to the range of [0,1]. Among them, the CSI amplitude of the first wireless signal of the jth frame on the i-th channel of the N channels arranged in the first order is normalized to A ij ,Right now:

[0299]

[0300] Among them, A i,j represents the original CSI amplitude of the first wireless signal of the jth frame on the i-th channel among the N channels arranged in the first order, O A j represents the set of original CSI amplitudes on N channels in the first wireless signal of the jth frame, O A j,min express O A j The minimum value of CSI amplitude, O A j,max express O A j The maximum value of the CSI amplitude.

[0301] The first order may be an order of increasing frequencies of the channels, an order with the greatest difference from the order of increasing frequencies of the channels, or any order.

[0302] In the embodiment of the present application, the first order is the order in which the frequencies of the channels are increased, and the obtained sequence of the channels is channel 1, channel 2, channel 3, ..., channel N.

[0303] Then, the CSI amplitude of the first wireless signal of the M frames is encoded in the channel and time dimensions to construct a 2D picture and obtain a first sample image. Specifically, the CSI amplitude of the wireless signal of the jth frame on the i-th channel of the N channels arranged in the first order is encoded as the pixel value of the i-th row and j-th column in the first image, where i, j, and N are positive integers, i=1, 2, ..., N, and j=1, 2, ..., M. Fig.7D As shown, the pixel value of the i-th row and j-th column in the first sample image represents the CSI amplitude of the first wireless signal of the j-th frame on the i-th channel of the N channels arranged in the first order. Fig.7D Taking N=30 and M=30 as an example, a first sample image with a resolution of 30*30 pixels can be obtained.

[0304] (2) Second sample image

[0305] Similar to the CSI amplitude, the CSI phase of each frame of the first wireless signal is first normalized to limit the CSI phase of each channel to the range of [0,1]. The normalization method can refer to the normalization method of the CSI amplitude in the first sample image above, which will not be repeated here.

[0306] Then, the CSI phase of the first wireless signal of the M frames is encoded in the channel and time dimensions to construct a 2D picture and obtain a second sample image. Specifically, the CSI phase of the wireless signal of the jth frame on the i-th channel of the N channels arranged in the first order is encoded as the pixel value of the i-th row and j-th column in the second image; i = 1, 2, ..., N, j = 1, 2, ..., M. Fig.7D As shown, the pixel value of the i-th row and the j-th column in the second sample image represents the CSI phase of the first wireless signal of the j-th frame on the i-th channel among the N channels arranged in the first order.

[0307] (3) The third sample image

[0308] It can be understood that when N channels are arranged in the order of increasing or decreasing channel frequencies, two adjacent subcarriers are also two adjacent columns in the 2D picture of the first sample image or the second sample image obtained by encoding. Usually, the effects of different objects and different actions on signal characteristics are quite different at different frequencies. Therefore, the recognition of these objects and actions requires the integration of signal characteristics of multiple different frequencies. The local perception mechanism of CNN itself determines that each convolution window can perceive the information of adjacent positions in the 2D picture. Although the fusion of different frequency features can be achieved by the combination of multi-layer networks, the intermediate pooling processing and different convolution kernel operations will cause the loss of some information. Therefore, in order to better extract the characteristic information of the signal features, in some embodiments of the present application, the input layer can allow CNN to perceive channel features with large frequency differences. At this time, the original order of increasing or decreasing channel frequencies is changed. In one implementation, the first order may be an order different from the order in which the frequencies of the channels are increasing or decreasing; in another implementation, the first order may be an order in which the frequencies of the channels are increasing or decreasing, and a third sample image and a fourth sample image may also be constructed, in which the channels are arranged in an order different from the order in which the frequencies of the channels are increasing or decreasing, i.e., a second order.

[0309] Further, any two adjacent channels in the N channels arranged in the first order are not adjacent in the N channels arranged in the second order, so as to ensure frequency differentiation.

[0310] In one implementation of the embodiment of the present application, the first order may be an order in which the frequencies of the channels are increased or decreased. The channels are numbered in an order in which the frequencies of the channels are increased or decreased. The 30 channels arranged in the first order may be expressed as follows:

[0311] Vector V == [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30] T

[0312] In one implementation of the embodiment of the present application, the 30 channels arranged in the second order can be expressed as:

[0313] Vector V′ = [1,11,21,2,12,22,3,13,23,4,14,24,5,15,25,6,16,26,7,17,27,8,18,28,9,19,29,10,20,30] T ,At this time, the frequencies of any two adjacent channels in the second order have a large difference.

[0314] Similar to the first sample image, the normalized CSI amplitude of each frame of the first wireless signal is used to encode the CSI amplitude of each frame of the first wireless signal in the channel and time dimensions to construct a 2D image and obtain the third sample image. The CSI amplitude of the j-th frame of the wireless signal on the i-th channel of the N channels arranged in the second order is encoded as the pixel value of the i-th row and j-th column in the third image, i = 1, 2, ..., N, j = 1, 2, ..., M. Fig. 7E As shown, the pixel value of the i-th row and j-th column in the third sample image represents the CSI amplitude of the first wireless signal of the j-th frame on the i-th channel among the N channels arranged in the second order.

[0315] (4) Fourth sample image

[0316] Similar to the second sample image, the normalized CSI phase of each frame of the first wireless signal is used to encode the CSI phase of each frame of the first wireless signal in the channel and time dimensions to construct a 2D image, thereby obtaining a fourth sample image, such as Fig. 7E As shown, the pixel value of the i-th row and the j-th column in the fourth sample image represents the CSI phase of the first wireless signal of the j-th frame on the i-th channel among the N channels arranged in the second order.

[0317] (5) Fifth sample image

[0318] Since the CSI information includes the frequency responses of multiple subcarriers measured simultaneously from the data of a frame of wireless signal, rather than the overall amplitude response of all subcarriers superimposed. Taking into account the overall amplitude of the wireless signal, the RSSI of the first wireless signal in each frame can be encoded in the channel and time dimensions to obtain the fifth sample image. Specifically, the CSI phase of the j-th frame wireless signal on the i-th channel of the N channels arranged in the second order is encoded as the pixel value i=1, 2, ..., N, j=1, 2, ..., M in the i-th row and j-th column of the fourth image. As Figure 7F As shown, the pixel value of the jth column in the fifth sample image represents the RSSI of the first wireless signal of the jth frame.

[0319] In one implementation of the embodiment of the present application, multiple of the above-mentioned first sample image, second sample image, third sample image, fourth sample image and fifth sample image can be fused into one image to obtain an image to be identified. At this time, the pixel value of the i-th row and j-th column of the image to be identified includes: the CSI amplitude and CSI phase of the first wireless signal of the j-th frame on the i-th channel of the N channels arranged in the first order, the CSI amplitude and CSI phase of the first wireless signal of the j-th frame on the i-th channel of the N channels arranged in the second order, the RSSI of the first wireless signal of the j-th frame, etc.

[0320] It should be noted that one or more of the first sample image, the second sample image, the third sample image, the fourth sample image, and the fifth sample image can be simultaneously input into different channels of the initialized neural network, and the convolution layers of each channel can share weights. The neural network in another implementation of the embodiment of the present application may also include more channels, which is not limited here.

[0321] like Figure 7G The figure shows a structural schematic diagram of a convolutional neural network provided by an embodiment of the present application. The convolutional neural network may include an input layer, a convolutional layer, a pooling layer, a fully connected layer, and an output layer. Among them, the feature matrix in the signal feature is extracted by the convolutional layer, and the feature matrix indicates the feature information of the input multi-frame wireless signal; the pooling layer compresses the input feature matrix, and ensures the high compactness of the feature matrix while reducing the dimension; the fully connected layer connects each unit of the input and output, so that the output can integrate the information of each input; the output layer can use softmax to normalize the output.

[0322] In some embodiments, the size of the input layer can be "30×30×5", where "30×30" corresponds to the signal features of 30 consecutive frames of wireless signals on 30 channels respectively; the depth is 5, corresponding to 5 images that can be encoded for every 30 frames of wireless signals, namely the first sample image, the second sample image, the third sample image, the fourth sample image and the fifth sample image mentioned above. Each convolution layer is followed by an activation function, and some embodiments of the present application can use a rectified linear unit (ReLU) as the activation function.

[0323] In some embodiments, the convolution kernel size of the first convolution layer may be "3×3×5×20", where the depth 5 corresponds to the size of the input layer. The stride is 1, and padding is used, so after this convolution layer, the image size becomes "30×30×20".

[0324] In some embodiments, the output image of the first convolutional layer is processed by the maximum pooling layer (Max pooling), and its size is halved to "15×15×20"; then it passes through the second convolutional layer, whose convolution kernel size can be "3×3×40", so the image size becomes "15×15×40".

[0325] In some embodiments, the output image of the second convolutional layer is halved in size after maximum pooling, becoming "8×8×20"; then after passing through the third convolutional layer, the convolution kernel size is "3×3×80", and the output image size becomes "8×8×80".

[0326] In some embodiments, the output image of the upper layer passes through the first fully connected layer, and the output size is compressed to "1×1×160", and then passes through the second fully connected layer, and the output size is compressed to "1×1×100", corresponding to 100 output categories, where the size of the final output layer can be flexibly adjusted according to the category of human body movements required by the task.

[0327] In some embodiments, the fully connected layer passes through an output layer, for example, the output layer may be a sofamax layer, and normalizes the data output by the fully connected layer to obtain a probability distribution of categories belonging to different human body movements.

[0328] It should be noted that the above Figure 7G What is shown is an example of the architecture of a convolutional neural network of an embodiment of the present application. In other embodiments of the present application, the above-mentioned input layer, convolution layer, pooling layer, fully connected layer and output layer can also be of other sizes, and the convolution kernel can also be of other sizes, step sizes and filling methods, etc., which are not limited here.

[0329] Among them, in a neural network training process, one training sample can be used, or multiple training samples or all training samples can be used, which is not limited in the embodiments of the present application.

[0330] The embodiment of the present application uses Q training samples as an example to illustrate a training process. The training device can determine the losses corresponding to the Q training samples according to the error between the predicted result and the true result of each training sample. According to the losses corresponding to the Q training samples, the parameters of the neural network are updated through the optimization algorithm to make the losses smaller and smaller. The optimization algorithm can be a gradient descent method, an Adam algorithm or other optimization algorithms, which are not limited in the embodiment of the present application. Among them, the loss function is used to calculate the losses corresponding to the Q training samples, and the error can be the mean absolute error (MAE), mean squared error (MSE) or root mean squared error (RMSE) between the predicted result and the true result, etc. It can also have other forms, which are not limited in the present application.

[0331] It should be noted that the embodiment of the present application is described by taking the first wireless transmitting device and the first wireless receiving device as an example. When the system includes multiple wireless transmitting devices and / or multiple wireless receiving devices, the combination of the above-mentioned one more wireless transmitting device and one wireless receiving device corresponds to a human motion recognition model, and the training method of the model can refer to the implementation of the training method of the target human motion recognition model corresponding to the combination of the above-mentioned first wireless transmitting device and the first wireless receiving device, which will not be repeated here.

[0332] Through the above-mentioned neural network training method, by encoding the signal features of the collected M frames of wireless signals into sample images, the target human motion recognition model obtained by training the neural network with the sample images can realize the recognition of human motion based on wireless signals more accurately.

[0333] Furthermore, the sample images used in training include multiple images obtained by channel coding in different orders, which provide differentiated input data for model training, reduce the difficulty of model training, and enable the trained target human motion recognition model to better extract richer feature information, further improving recognition accuracy.

[0334] Embodiment 2:

[0335] like Fig. 8A 1 is a flow chart of a method for recognizing human actions provided in the second embodiment of the present invention. The method 80 is implemented by the human action recognition model obtained by training in the first embodiment. The method 80 can be specifically implemented as follows: Figure 4The wireless receiving device 120 shown is executed, or implemented by the wireless receiving device 120 and the execution device 140. For example, the wireless receiving device 120 can execute step S802, the image encoding module 141 in the execution device 140 can be used to execute step S804, and the calculation module 142 can be used to execute step S806; for another example, the wireless receiving device 120 can execute steps S802-S804, and the calculation module 142 in the execution device 140 can be used to execute step S806, etc.

[0336] Optionally, the method 80 may be processed by a CPU disposed in the wireless receiving device 120 and / or a CPU disposed in the execution device 140, or may be processed by the above CPU and a processor suitable for neural network calculation (e.g., Figure 6 The neural network processor 30 shown in the figure is not limited in this application.

[0337] The method 80 may include but is not limited to some or all of the following steps:

[0338] S802: Receive M frames of wireless signals, and extract signal features of the M frames of wireless signals.

[0339] The M frames of wireless signals are wireless signals transmitted by the first wireless transmitting device and received by the first wireless receiving device, the distance between the first wireless receiving device and the first wireless transmitting device is greater than the first threshold, and M is an integer greater than 1. The M frames of wireless signals are sorted in chronological order. Signal characteristics may include CSI amplitude, CSI phase, RSSI, etc.

[0340] S804: Encode the signal features of the M frames of wireless signals in time and channel dimensions to obtain an image to be identified.

[0341] In one implementation of an embodiment of the present application, the wireless signal may include N channels, and the image to be identified may include a first image and a second image, wherein the pixel value of the i-th row and j-th column in the first image represents the CSI amplitude of the j-th frame wireless signal on the i-th channel among the N channels arranged in the first order; the pixel value of the i-th row and j-th column in the second image represents the CSI phase of the j-th frame wireless signal on the i-th channel among the N channels arranged in the first order, i and j are positive integers, i≤N, j≤M.

[0342] In another implementation of the embodiment of the present application, the image to be identified may also include a third image and a fourth image, wherein the pixel value of the i-th row and j-th column in the third image represents the CSI amplitude of the j-th frame wireless signal on the i-th channel among N channels arranged in a second order; the pixel value of the i-th row and j-th column in the fourth image represents the CSI phase of the j-th frame wireless signal on the i-th channel among N channels arranged in the second order, and the first order is different from the second order.

[0343] Optionally, the first order is an order of increasing or decreasing frequencies of the channels, the second order is different from the first order, and any two channels with adjacent frequencies are not adjacent in the N channels arranged in the second order.

[0344] In another implementation of the embodiment of the present application, the signal feature may also include a received signal strength indication (RSSI), and the image to be identified may also include a fifth image, wherein the pixel value of the jth column in the fifth image represents the RSSI of the jth frame wireless signal.

[0345] It should be noted that the methods for generating the first image, the second image, the third image, the fourth image, and the fifth image can refer to the related descriptions of generating the first sample image, the second sample image, the third sample image, the fourth sample image, and the fifth sample image in the above-mentioned embodiment 1, respectively, and will not be repeated here. The first sequence and the second sequence can refer to the related descriptions of the first sequence and the second sequence in the above-mentioned embodiment 1, respectively, and will not be repeated here.

[0346] It should also be understood that, corresponding to the training method of the above-mentioned neural network, the first sequence in the second embodiment is the same as the first sequence in the above-mentioned first embodiment.

[0347] S806: Input the image to be recognized into the target human action recognition model to obtain a recognition result, that is, the type of human action corresponding to the image to be recognized.

[0348] Optionally, the target human motion recognition model is a model obtained by the neural network training method in the above-mentioned embodiment 1. For details, please refer to the relevant description in the above-mentioned embodiment 1, which will not be repeated here.

[0349] In some embodiments of the present application, after the images to be recognized are respectively input into the target human action recognition model, the probability of each human action category can be obtained, and the human action category corresponding to the maximum probability is selected as the recognition result.

[0350] It should be noted that the sample images of the target human motion model obtained by training have the same format as the image to be recognized, that is, when the sample image includes the first sample image, the image to be recognized includes the first image; when the sample image includes the second sample image, the image to be recognized includes the second image; when the sample image includes the third sample image, the image to be recognized includes the third image; when the sample image includes the fourth sample image, the image to be recognized includes the fourth image; when the sample image includes the fifth sample image, the image to be recognized includes the fifth image. Moreover, the first order and the second order used when training the target human motion model are respectively the same as the first order and the second order in the embodiment of the human motion recognition method.

[0351] like Fig. 8A As shown, in the first implementation of the present application, the above steps S802-S806 can be executed by a receiving device of M frames of wireless signals, that is, a first wireless receiving device. It can be understood that at this time, the first wireless receiving device and the executing device are the same device.

[0352] like Figure 8B As shown, in the second implementation of the present application, the above step S802 can be executed by the first wireless receiving device, and the above S804-S806 are executed by the execution device. In this case, the first wireless receiving device and the execution device that execute S802 are different devices. After S802 and before S804, the method can also include steps S8032-S8034:

[0353] S8032: The first wireless receiving device sends the signal characteristics of the M-frame wireless signal to the execution device.

[0354] S8034: Execute the signal characteristics of the M-frame wireless signal received by the device.

[0355] like Figure 8C As shown, in the third implementation of the embodiment of the present application, the above steps S802-S804 can be executed by the first wireless receiving device, and S806 can be executed by the execution device. In this case, the first wireless receiving device and the execution device that execute S802 are different devices. After S804 and before S806, the method can also include steps S8052-S8054:

[0356] S8052: The first wireless transmitting device sends the image to be recognized to the executing device.

[0357] S8054: The execution device receives the image to be recognized.

[0358] In some embodiments of the present application, after obtaining the recognition result, the first wireless receiving device or the executing device may also send the recognition result to the first terminal. The first terminal may be Figure 4The user terminal 170 may also be the terminal of the guardian in scene a, or the terminal of the owner in scene b, etc.

[0359] In some embodiments of the present application, after the first wireless receiving device or the execution device obtains the recognition result, the first wireless receiving device or the execution device can also determine the target operation corresponding to the identified target human motion category based on the correspondence between the human motion category and the operation, and then control the first wireless receiving device or the execution device to execute or realize the target operation.

[0360] For example, the executing device is a mobile phone. Referring to the correspondence between human body movements and operations in scene c, when a call comes in, the user can make a "raise both hands" gesture. The mobile phone recognizes this action through the signal characteristics of the M frames of wireless signals received or the image to be identified, and then controls the mobile phone to answer the call without the user having to pick up the phone to operate.

[0361] In some embodiments of the present application, in a system consisting of at least one wireless receiving device and at least one wireless receiving device, each wireless receiving device can send signal features of M frames of wireless signals or images to be recognized to the execution device. At this time, the combination of a wireless transmitting device and a wireless receiving device corresponds to a human action recognition model.

[0362] At this time, when the first wireless receiving device sends the signal characteristics of M frames of wireless signals or the image to be identified to the execution device, it can also send the identifier of the first wireless transmitting device and the identifier of the first wireless receiving device to the execution device, and then process the received image to be identified according to the human motion recognition model (i.e., the target human motion recognition model) corresponding to the combination of the first wireless transmitting device and the first wireless receiving device to obtain a recognition result.

[0363] In some embodiments of the present application, for a system consisting of a wireless transmitting device and multiple wireless receiving devices, each wireless receiving device may send its identifier to the executing device when sending signal features or images to be identified to the executing device, and the executing device may determine the human motion recognition model corresponding to the combination of the wireless transmitting device and the wireless receiving device based on the identifier of the received wireless receiving device; similarly, for a system consisting of multiple wireless transmitting devices and one wireless receiving device, the wireless receiving device may send the identifier of the wireless transmitting device to the executing device, and the executing device may determine the human motion recognition model corresponding to the combination of the wireless transmitting device and the wireless receiving device based on the identifier of the received wireless transmitting device.

[0364] As shown in Table 2 below, an example of the correspondence between a combination of a wireless transmitting device and a wireless receiving device and a human motion recognition model.

[0365]

[0366] Table 2

[0367] It should be noted that the above Table 2 is only an exemplary description. The embodiments of the present application may also include other corresponding methods.

[0368] In one implementation of the embodiment of the present application, corresponding to the above scenario a, the categories of human motion may include falling and non-falling. The target human motion recognition model is a human fall recognition model; corresponding to the above scenario b, the categories of human motion may include the action of someone breaking into the room and the action of no one breaking into the room, and the target human motion recognition model is an intruder recognition model; corresponding to the above scenario c, the categories of human motion may include multiple human motions. The target human motion recognition model is an action recognition model.

[0369] In the embodiment of the present application, the signal features of M-frame wireless signals are encoded into an image, i.e., the image to be identified in the embodiment of the present application, and then the category of human motion on the transmission path of the M-frame wireless signals is identified more accurately through a trained neural network, i.e., a target human motion recognition model.

[0370] It should be noted that the present application also provides a method for processing wireless signals, which can be specifically performed as follows: Figure 4 The wireless receiving device 120 shown is executed. The method may include steps S802 and S804 in the above-mentioned embodiment 2. For details, please refer to the relevant description of the above-mentioned embodiment 2, which will not be repeated here. Optionally, the method can be applied to the human body motion recognition method described in the above-mentioned embodiment 2, and can also be applied to other fields or methods that require image encoding of signal characteristics of wireless signals, which are not limited here.

[0371] The following is an introduction to the device involved in the embodiments of the present application.

[0372] like Fig. 9 The figure shows a neural network training device provided by an embodiment of the present application. The device 900 may be Figure 4 The training device 130 in the system 100 shown, the device 90 may include but is not limited to the following functional units:

[0373] The receiving unit 901 is configured to receive signal characteristics of M frames of first wireless signals sent by the first wireless receiving device, wherein the M frames of first wireless signals are wireless signals transmitted by the first wireless transmitting device and received by the first wireless receiving device when a first human motion occurs between the first wireless receiving device and the first wireless transmitting device, and M is an integer greater than 1;

[0374] An encoding unit 902 is used to encode the signal characteristics of the M frames of the first wireless signal in time and channel dimensions to obtain a sample image;

[0375] The training unit 903 is used to train a neural network according to the sample image and the category of the human motion corresponding to the sample image to obtain the target human motion recognition model, wherein the sample image is a training input, the category of the human motion corresponding to the sample image is a training label, and the first human motion belongs to the category of the human motion corresponding to the sample image.

[0376] Optionally, the first wireless receiving device may extract signal characteristics of the M-frame wireless signal through a network card, and the signal characteristics may include but are not limited to at least one of CSI amplitude, CSI phase, RSSI, and the like.

[0377] Optionally, the sample image may include at least one sample image of a first sample image, a second sample image, a third sample image, a fourth sample image, and a fifth sample image. The encoding unit 902 may be specifically described in the above-mentioned embodiment 1, and will not be described in detail in this embodiment of the application.

[0378] The specific implementation of each of the above units can refer to the related description of the above embodiment, which will not be repeated here.

[0379] like Fig.10 The figure shows a human body motion recognition device provided by an embodiment of the present application. The device 1000 can be Figure 4 The execution device 140, the wireless transmitting device 110 or the wireless receiving device 120 in the system 100 shown, the apparatus 1000 may include but is not limited to the following functional units:

[0380] An acquisition unit 1001 is configured to acquire an image to be identified, wherein the image to be identified is an image obtained by encoding signal features of M frames of wireless signals in dimensions of time and channel, the M frames of wireless signals are wireless signals transmitted by a first wireless transmitting device and received by a first wireless receiving device, a distance between the first wireless receiving device and the first wireless transmitting device is greater than a first threshold, and M is an integer greater than 1;

[0381] The recognition unit 1002 is used to input the image to be recognized into a target human action recognition model to obtain the type of human action corresponding to the image to be recognized.

[0382] like Fig.11 The figure shows a wireless signal processing device provided by an embodiment of the present application. The device 1100 may be Figure 4 The wireless receiving device 120 in the system 100 shown in the figure, the apparatus 1100 may include but is not limited to the following functional units:

[0383] The receiving unit 1101 is configured to receive M frames of wireless signals transmitted by a wireless transmitting device;

[0384] An extraction unit 1102 is used to extract a signal feature of each frame of the wireless signal in the M frames of wireless signal;

[0385] The encoding unit 1103 is used to encode the signal characteristics of the M frames of wireless signals in time and channel dimensions to obtain an image to be identified.

[0386] Optionally, the apparatus 1100 may further include a sending unit 1104, configured to send the image to be recognized to an execution device, so that the execution device recognizes the category of the human body motion according to the image to be recognized.

[0387] Optionally, the above Fig.10 or Fig.11 In the device shown, device 1000 or device 1100 can extract signal characteristics of M-frame wireless signals through a network card, and the signal characteristics may include but are not limited to at least one of CSI amplitude, CSI phase, and RSSI.

[0388] Optionally, the above Fig.10 or Fig.11 In the device shown, the image to be identified may include at least one of the first image, the second image, the third image, the fourth image, and the fifth image. For details, please refer to the relevant description in the above-mentioned embodiment 2, and the embodiment of the present application will not be repeated.

[0389] In one possible implementation, the device 1000 may also include: a search unit 1003, which is used to search for a target human motion recognition model corresponding to the combination of the first wireless transmitting device and the first wireless receiving device according to the correspondence between the combination of the wireless transmitting device and the wireless receiving device and the human motion recognition model before the image to be recognized in the recognition unit 1002 is input into the human motion recognition model and a recognition result is obtained.

[0390] In one possible implementation, the device 1000 or the device 1100 may also include a collection unit and a training unit, wherein the specific implementation of the functions of the collection unit in the device 1000 or the device 1100 can refer to the implementation of the collection unit 901 in the above-mentioned device 900, and the specific implementation of the functions of the training unit in the device 1000 or the device 1100 can refer to the implementation of the training unit 902 in the above-mentioned device 900, which will not be repeated here.

[0391] The specific implementation of each unit in the above-mentioned device 1000 or device 1100 can refer to the relevant description of the above-mentioned embodiment 2, and will not be repeated here.

[0392] Fig.12 It is a schematic diagram of the hardware structure of a neural network training device provided in an embodiment of the present application. Fig.12 The neural network training device 1200 shown (the device 1200 may be a computer device) may include a memory 1201, a processor 1202, a communication interface 1203, and a bus 1204. The memory 1201, the processor 1202, and the communication interface 1203 are connected to each other through the bus 1204.

[0393] The memory 1201 may be a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 1201 may store a program. When the program stored in the memory 1201 is executed by the processor 1202, the processor 1202 and the communication interface 1203 are used to execute the various steps of the neural network training method of the embodiment of the present application.

[0394] Processor 1202 can adopt a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the functions required to be performed by the units in the neural network training device of the embodiment of the present application, or to execute the neural network training method in the method embodiment 1 of the present application.

[0395] The processor 1202 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the training method of the neural network of the present application may be completed by an integrated logic circuit of hardware or software instructions in the processor 1202. The above-mentioned processor 1202 may also be a general-purpose processor, a digital signal processor (Digital Signal Processing, DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application may be directly embodied as a hardware decoding processor for execution, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1201, and the processor 1202 reads the information in the memory 1201, and combines its hardware to complete the functions required to be performed by the units included in the neural network training device of the embodiment of the present application, or executes the neural network training method of the method embodiment of the present application.

[0396] The communication interface 1203 uses a transceiver such as but not limited to a transceiver to implement communication between the device 1200 and other devices or communication networks. For example, training data (such as the sample image described in the first embodiment of the present application) can be obtained through the communication interface 1203.

[0397] The bus 1204 may include a path for transmitting information between various components of the device 1200 (eg, the memory 1201 , the processor 1202 , and the communication interface 1203 ).

[0398] It should be understood that the acquisition unit 901 in the neural network training device 900 is equivalent to the communication interface 1203 in the neural network training device 1200, and the training unit 902 can be equivalent to the processor 1202.

[0399] Fig.13 It is a schematic diagram of the hardware structure of the human body motion recognition device provided in an embodiment of the present application. Fig.13The human body motion recognition device 130 shown (the device 130 may be a computer device) includes a memory 1301, a processor 1302, a communication interface 1303 and a bus 1304. The memory 1301, the processor 1302 and the communication interface 1303 are connected to each other through the bus 1304.

[0400] The memory 1301 may be a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 1301 may store a program. When the program stored in the memory 1301 is executed by the processor 1302, the processor 1302 and the communication interface 1303 are used to execute each step of the human body motion recognition method of the embodiment of the present application.

[0401] Processor 1302 can adopt a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to realize the functions required to be performed by the unit in the human body motion recognition of the embodiment of the present application, or to execute the human body motion recognition method of the method embodiment of the present application.

[0402] The processor 1302 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the human body motion recognition method of the present application may be completed by the hardware integrated logic circuit or software instructions in the processor 1302. The above-mentioned processor 1302 may also be a general-purpose processor, a digital signal processor (Digital Signal Processing, DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as a hardware decoding processor for execution, or may be executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1301, and the processor 1302 reads the information in the memory 1301, and combines its hardware to complete the functions required to be performed by the units included in the human body motion recognition device of the embodiment of the present application, or executes the human body motion recognition method of the method embodiment of the present application.

[0403] The communication interface 1303 uses a transceiver such as but not limited to a transceiver to implement communication between the device 1300 and other devices or communication networks. For example, training data (such as the image to be recognized described in the second embodiment of the present application) can be obtained through the communication interface 1303.

[0404] The bus 1304 may include a path for transmitting information between various components of the device 1300 (eg, the memory 1301 , the processor 1302 , and the communication interface 1303 ).

[0405] It should be understood that the acquisition unit 1001 in the human motion recognition device 1000 is equivalent to the communication interface 1303 in the human motion recognition device 1300 ; the recognition unit 1002 and the search unit 1003 in the human motion recognition device 1000 can be equivalent to the processor 1302 .

[0406] Fig.141400 is a hardware structure diagram of a wireless signal processing device provided in an embodiment of the present application. The wireless signal processing device 1400 may include a memory 1401, a processor 1402, a wireless communication module 1403 and a bus 1404. The memory 1401, the processor 1402, and the wireless communication module 1403 are connected to each other through the bus 1404.

[0407] The memory 1401 may be a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 1401 may store a program. When the program stored in the memory 1401 is executed by the processor 1402, the processor 1402 and the wireless communication module 1403 are used to execute some or all steps of the human body motion recognition method of the embodiment of the present application.

[0408] Processor 1402 can adopt a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to realize the functions required to be performed by the final unit in the human body motion recognition embodiment of the present application, or to execute the human body motion recognition method of the method embodiment of the present application.

[0409] The wireless communication module 1403 is used to realize communication with a wireless transmitting device, an execution device, a training device, etc. For example, the wireless communication module 1403 receives M frames of wireless signals transmitted by the wireless transmitting device and extracts the signal characteristics of each frame of the wireless signals. The wireless communication module 1403 can be a WiFi module, a Bluetooth module, an IrDA module, a ZigBee module, a UWB module, etc.

[0410] Optionally, the device 1400 may further include other communication modules, such as a mobile communication module or a communication interface, etc., which is not limited in the embodiment of the present application.

[0411] The processor 1402 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the human body motion recognition method of the present application may be completed by the hardware integrated logic circuit or software instructions in the processor 1402. The above-mentioned processor 1402 may also be a general-purpose processor, a digital signal processor (Digital Signal Processing, DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1401, and the processor 1402 reads the information in the memory 1401, and combines its hardware to complete the functions required to be performed by the units included in the human body motion recognition device of the embodiment of the present application, or partially or completely executes the human body motion recognition method of the method embodiment of the present application. For example,

[0412] Receive M frames of wireless signals transmitted by a wireless transmitting device through the wireless communication module 1403;

[0413] Extracting signal features of each frame of wireless signal in the M frames of wireless signal by the wireless communication module 1403;

[0414] Encode the signal features of the M frames of wireless signals in the time and channel dimensions to obtain an image to be identified;

[0415] The image to be recognized is sent to the execution device through the wireless communication module 1403, so that the execution device recognizes the category of the human body movement according to the image to be recognized.

[0416] It should be noted that the device 1400 can be a smart home device. For different smart home devices, the device 1400 can have different hardware structures. For example, for lamps, the device 1400 can also include a light-emitting device to achieve light emission; for another example, for a refrigerator, the device 1400 can also include a freezing device to achieve the freezing function of the refrigerator, etc., which will not be repeated here.

[0417] It should be understood that the receiving unit 1101, the extracting unit 1102 and the sending unit 1104 in the human motion recognition device 1100 are equivalent to the wireless communication module 1403 in the human motion recognition device 1400; the encoding unit 1103 in the human motion recognition device 1000 can be equivalent to the processor 1402.

[0418] It should be noted that although Fig.12 , Fig.13 and Fig.14 The devices 1200, 1300, and 1400 shown in the figure only show the memory, processor, communication interface, or wireless communication module. However, in the specific implementation process, those skilled in the art should understand that the devices 1200, 1300, and 1400 also include other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the devices 1200, 1300, and 1400 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the devices 1200, 1300, and 1400 may also only include the devices necessary for implementing the embodiments of the present application, without having to include the devices shown in the figure. Fig.12 , Fig.13 or Fig.14 All devices shown in .

[0419] It can be understood that the apparatus 1200 is equivalent to the training device 130 in 1, and the apparatus 1300 is equivalent to Figure 1 The execution device 140 in the embodiment may be equivalent to Figure 1 The wireless receiving device in the embodiment 1 or the first receiving device in the embodiment 2. A person of ordinary skill in the art can appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0420] In some embodiments of the present application, the above-mentioned apparatus 900, 1200 may also be a wireless receiving device 120, such as the first wireless receiving device in embodiment one or embodiment two, or may be a wireless transmitting device 110, such as the first wireless transmitting device in embodiment one or embodiment two.

[0421] In some embodiments of the present application, the above-mentioned devices 1100 and 1300 can also be a wireless receiving device 120, such as the first wireless receiving device in embodiment one or embodiment two, or a wireless transmitting device 110, such as the first wireless transmitting device in embodiment one or embodiment two.

[0422] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0423] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0424] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0425] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0426] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0427] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for recognizing human body movements, characterized in that: include: The execution device obtains an image to be identified, wherein the image to be identified is an image obtained by encoding signal features of M frames of wireless signals in the dimensions of time and channel, the M frames of wireless signals are wireless signals transmitted by a first wireless transmitting device and received by a first wireless receiving device, and M is an integer greater than 1; the signal features include channel state information CSI; The execution device inputs the image to be recognized into a target human action recognition model to obtain the type of human action corresponding to the image to be recognized; the type of human action corresponding to the image to be recognized is a human action occurring between the first wireless receiving device and the first wireless transmitting device; The CSI includes CSI amplitude and CSI phase, and the image to be identified includes a first image and a second image, wherein any one frame of the M frames of wireless signals includes N channels, and the pixel value of the i-th row and j-th column in the first image represents the CSI amplitude of the j-th frame of the wireless signal on the i-th channel among the N channels arranged in a first order; the pixel value of the i-th row and j-th column in the second image represents the CSI phase of the j-th frame of the wireless signal on the i-th channel among the N channels arranged in the first order, i, j, and N are positive integers, i≤N, j≤M.

2. The method according to claim 1, characterized in that The image to be identified also includes a third image and a fourth image, wherein the pixel value of the i-th row and j-th column in the third image represents the CSI amplitude of the j-th frame wireless signal on the i-th channel among N channels arranged in a second order; the pixel value of the i-th row and j-th column in the fourth image represents the CSI phase of the j-th frame wireless signal on the i-th channel among N channels arranged in the second order, and the first order is different from the second order.

3. The method according to claim 2, characterized in that The frequencies of the N channels arranged in the first order are increasing, and the frequencies of the N channels arranged in the second order are non-increasing.

4. The method according to claim 2, characterized in that: The frequencies of the N channels arranged in the first order are decreasing, and the frequencies of the N channels arranged in the second order are non-decreasing.

5. The method according to any one of claims 1 to 4, characterized in that: The signal feature also includes a received signal strength indication (RSSI), and the image to be identified also includes a fifth image, and the pixel value of the jth column in the fifth image represents the RSSI of the jth frame of the wireless signal.

6. The method according to any one of claims 1 to 4, characterized in that: The execution device is the first wireless receiving device, and the execution device acquires the image to be recognized including: The execution device receives M frames of wireless signals transmitted by the first wireless transmitting device; The execution device extracts the signal feature of each frame of the wireless signal in the M frames of wireless signals; The execution device encodes the signal features of the M frames of wireless signals in time and channel dimensions to obtain an image to be identified.

7. The method according to any one of claims 1 to 4, characterized in that: The execution device acquires the image to be recognized, comprising: receiving signal characteristics of M frames of wireless signals sent by the first wireless receiving device; The execution device encodes the signal features of the M frames of wireless signals in time and channel dimensions to obtain an image to be identified.

8. The method according to any one of claims 1 to 4, characterized in that: Before inputting the image to be recognized into the human action recognition model to obtain the recognition result, the method further includes: The execution device searches for a target human motion recognition model corresponding to the combination of the first wireless transmitting device and the first wireless receiving device according to the correspondence between the combination of the wireless transmitting device and the wireless receiving device and the human motion recognition model.

9. A method for processing a wireless signal, characterized in that: The method comprises: The wireless receiving device receives the M frames of wireless signals transmitted by the wireless transmitting device; The wireless receiving device extracts the signal characteristics of each frame of the wireless signal in the M frames of wireless signals; the signal characteristics include channel state information CSI; The wireless receiving device encodes the signal features of the M frames of wireless signals in time and channel dimensions to obtain an image to be identified, and the image to be identified is used to identify human body movements between the wireless receiving device and the wireless transmitting device; the signal features include channel state information (CSI); the CSI includes CSI amplitude and CSI phase, any one frame of the M frames of wireless signals includes N channels, and the wireless receiving device encodes the signal features of the M frames of wireless signals in time and channel dimensions to obtain an image to be identified, including: The wireless receiving device encodes the CSI amplitude of the j-th frame wireless signal on the i-th channel among the N channels arranged in the first order into a pixel value of the i-th row and j-th column in the first image; The wireless receiving device encodes the CSI phase of the j-th frame wireless signal on the i-th channel among N channels arranged in the first order as a pixel value of the i-th row and j-th column in the second image; i, j, N are positive integers, i≤N, j≤M, and the image to be identified includes the first image and the second image.

10. The method according to claim 9, characterized in that The wireless receiving device encodes the signal features of the M frames of wireless signals in time and channel dimensions to obtain an image to be identified, and further includes: The wireless receiving device encodes the CSI amplitude of the j-th frame wireless signal on the i-th channel among the N channels arranged in the second order into a pixel value of the i-th row and j-th column in the third image; The wireless receiving device encodes the CSI phase of the j-th frame wireless signal on the i-th channel among N channels arranged in the second order as the pixel value of the i-th row and j-th column in the fourth image, and the image to be identified also includes the third image and the fourth image, and the first order is different from the second order.

11. The method according to claim 10, characterized in that The frequencies of the N channels arranged in the first order are increasing, and the frequencies of the N channels arranged in the second order are non-increasing.

12. The method according to claim 10, characterized in that The frequencies of the N channels arranged in the first order are decreasing, and the frequencies of the N channels arranged in the second order are non-decreasing.

13. The method according to any one of claims 9 to 12, characterized in that: The signal feature includes a received signal strength indication (RSSI), and the wireless receiving device encodes the signal features of the M frames of wireless signals in time and channel dimensions to obtain an image to be identified, further comprising: The wireless receiving device encodes the RSSI of the j-th frame of the wireless signal into the pixel value of the j-th column in the fifth image, and the image to be identified also includes the fifth image.

14. A method for training a neural network, characterized in that: The method comprises: The training device receives signal characteristics of M frames of first wireless signals sent by the first wireless receiving device, wherein the M frames of first wireless signals are wireless signals transmitted by the first wireless transmitting device and received by the first wireless receiving device when a first human body motion occurs between the first wireless receiving device and the first wireless transmitting device, and M is an integer greater than 1; the signal characteristics include channel state information CSI; The training device encodes the signal features of the M frames of the first wireless signal in time and channel dimensions to obtain a sample image; The training device trains a neural network according to the sample image and the category of the human action corresponding to the sample image to obtain a target human action recognition model, wherein the sample image is a training input, the category of the human action corresponding to the sample image is a training label, and the first human action belongs to the category of the human action corresponding to the sample image; the signal feature includes channel state information (CSI); the CSI includes a CSI amplitude and a CSI phase, any one of the M frames of the first wireless signal includes N channels, and the training device encodes the signal features of the M frames of the first wireless signal in time and channel dimensions to obtain a sample image, including: The training device encodes the CSI amplitude of the first wireless signal of the jth frame on the i-th channel among the N channels arranged in the first order into a pixel value of the i-th row and the j-th column in the first sample image; The training device encodes the CSI phase of the first wireless signal of the jth frame on the i-th channel among N channels arranged in the first order as the pixel value of the i-th row and j-th column in the second sample image; i and j are positive integers, i≤N, j≤M, and the sample images include the first sample image and the second sample image.

15. The method according to claim 14, characterized in that The training device encodes the signal features of the M frames of the first wireless signal in the time and channel dimensions to obtain a sample image, and further includes: The training device encodes the CSI amplitude of the first wireless signal of the jth frame on the i-th channel among the N channels arranged in the second order into a pixel value of the i-th row and j-th column in the third sample image; The training device encodes the CSI phase of the first wireless signal of the jth frame on the i-th channel among N channels arranged in the second order as the pixel value of the i-th row and j-th column in the fourth sample image, where i and j are positive integers, i≤N, j≤M, and the sample images include the third sample image and the fourth sample image, and the first order is different from the second order.

16. The method according to any one of claims 14-15, characterized in that: The signal feature includes a received signal strength indication (RSSI), and the training device encodes the signal features of the M frames of the first wireless signal in time and channel dimensions to obtain a sample image, and further includes: The training device encodes the RSSI of the first wireless signal of the j-th frame into the pixel value of the j-th column in the fifth sample image, and the sample image also includes the fifth sample image.

17. A human motion recognition device, characterized in that: include: an acquisition unit, configured to acquire an image to be identified, wherein the image to be identified is an image obtained by encoding signal features of M frames of wireless signals in the dimensions of time and channel, the M frames of wireless signals are wireless signals transmitted by a first wireless transmitting device and received by a first wireless receiving device, and M is an integer greater than 1; the signal features include channel state information CSI; An identification unit is used to input the image to be identified into a target human motion recognition model to obtain the type of human motion corresponding to the image to be identified; the type of human motion corresponding to the image to be identified is a human motion occurring between the first wireless receiving device and the first wireless transmitting device; the CSI includes a CSI amplitude and a CSI phase, any one of the M frames of wireless signals includes N channels, and the image to be identified includes a first image and a second image, wherein the wireless signal includes N channels, and the pixel value of the i-th row and j-th column in the first image represents the CSI amplitude of the j-th frame wireless signal on the i-th channel among the N channels arranged in a first order; the pixel value of the i-th row and j-th column in the second image represents the CSI phase of the j-th frame wireless signal on the i-th channel among the N channels arranged in the first order, and i, j, and N are positive integers, i≤N, j≤M.

18. The device according to claim 17, characterized in that The image to be identified includes a third image and a fourth image, wherein the pixel value of the i-th row and j-th column in the third image represents the CSI amplitude of the j-th frame wireless signal on the i-th channel among N channels arranged in a second order; the pixel value of the i-th row and j-th column in the fourth image represents the CSI phase of the j-th frame wireless signal on the i-th channel among N channels arranged in the second order, and the first order is different from the second order.

19. The device according to claim 18, characterized in that The frequencies of the N channels arranged in the first order are increasing, and the frequencies of the N channels arranged in the second order are non-increasing.

20. The device according to claim 18, characterized in that The frequencies of the N channels arranged in the first order are decreasing, and the frequencies of the N channels arranged in the second order are non-decreasing.

21. The device according to any one of claims 17 to 20, characterized in that The signal feature includes a received signal strength indication (RSSI), and the image to be identified also includes a fifth image, and the pixel value of the jth column in the fifth image represents the RSSI of the jth frame of the wireless signal.

22. The device according to any one of claims 17 to 20, characterized in that The acquisition unit is specifically used for: Receiving M frames of wireless signals transmitted by the first wireless transmitting device; Extracting signal features of each frame of wireless signals in the M frames of wireless signals; The signal features of the M frames of wireless signals are encoded in time and channel dimensions to obtain an image to be identified.

23. The device according to any one of claims 17 to 20, characterized in that The acquisition unit is specifically used for: receiving signal characteristics of M frames of wireless signals sent by the first wireless receiving device; The signal features of the M frames of wireless signals are encoded in time and channel dimensions to obtain an image to be identified.

24. The device according to any one of claims 17 to 20, characterized in that The device also includes: A search unit is used to search for a target human motion recognition model corresponding to the combination of the first wireless transmitting device and the first wireless receiving device according to the correspondence between the combination of the wireless transmitting device and the wireless receiving device and the human motion recognition model before the recognition unit inputs the image to be recognized into the human motion recognition model to obtain a recognition result.

25. A wireless signal processing device, characterized in that: The device comprises: A receiving unit, used to receive M frames of wireless signals transmitted by a wireless transmitting device; An extraction unit, configured to extract a signal feature of each frame of the wireless signal in the M frames of wireless signal; the signal feature includes channel state information CSI; A coding unit is used to encode the signal characteristics of the M frames of wireless signals in time and channel dimensions to obtain an image to be identified, wherein the image to be identified is used to identify human body movements between a wireless receiving device and a wireless transmitting device; the signal characteristics include channel state information (CSI); the CSI includes a CSI amplitude and a CSI phase, and any one frame of the M frames of wireless signals includes N channels. The coding unit is specifically used to: Encode the CSI amplitude of the j-th frame wireless signal on the i-th channel among the N channels arranged in the first order as a pixel value of the i-th row and j-th column in the first image; The CSI phase of the j-th frame wireless signal on the i-th channel among the N channels arranged in the first order is encoded as the pixel value of the i-th row and j-th column in the second image; i, j, N are positive integers, i≤N, j≤M, and the image to be identified includes the first image and the second image.

26. The device according to claim 25, characterized in that The encoding unit is further specifically used for: Encode the CSI amplitude of the j-th frame wireless signal on the i-th channel among the N channels arranged in the second order as a pixel value of the i-th row and j-th column in the third image; The CSI phase of the j-th frame wireless signal on the i-th channel among N channels arranged in the second order is encoded as the pixel value of the i-th row and j-th column in the fourth image, and the image to be identified also includes the third image and the fourth image, and the first order is different from the second order.

27. The device according to claim 26, characterized in that The frequencies of the N channels arranged in the first order are increasing, and the frequencies of the N channels arranged in the second order are non-increasing.

28. The device according to claim 26, characterized in that The frequencies of the N channels arranged in the first order are decreasing, and the frequencies of the N channels arranged in the second order are non-decreasing.

29. The device according to any one of claims 25 to 28, characterized in that The signal feature includes a received signal strength indication (RSSI), and the encoding unit is further configured to: The RSSI of the j-th frame of the wireless signal is encoded as the pixel value of the j-th column in the fifth image, and the image to be identified also includes the fifth image.

30. A neural network training device, characterized in that: The device comprises: a receiving unit, configured to receive signal characteristics of M frames of first wireless signals sent by a first wireless receiving device, wherein the M frames of first wireless signals are wireless signals transmitted by the first wireless transmitting device and received by the first wireless receiving device when a first human body motion occurs between the first wireless receiving device and the first wireless transmitting device, and M is an integer greater than 1; the signal characteristics include channel state information CSI; An encoding unit, configured to encode signal features of the M frames of the first wireless signal in time and channel dimensions to obtain a sample image; A training unit, configured to train a neural network according to the sample image and the category of the human action corresponding to the sample image, to obtain a target human action recognition model, wherein the sample image is a training input, the category of the human action corresponding to the sample image is a training label, and the first human action belongs to the category of the human action corresponding to the sample image; the signal feature includes channel state information (CSI); the CSI includes a CSI amplitude and a CSI phase, any one of the M frames of the first wireless signal includes N channels, and the encoding unit is specifically configured to: Encode the CSI amplitude of the first wireless signal of the jth frame on the i-th channel among the N channels arranged in the first order as a pixel value of the i-th row and j-th column in the first sample image; The CSI phase of the first wireless signal of the jth frame on the i-th channel among N channels arranged in the first order is encoded as the pixel value of the i-th row and j-th column in the second sample image; i and j are positive integers, i≤N, j≤M, and the sample image includes the first sample image and the second sample image.

31. The device according to claim 30, characterized in that The encoding unit is further specifically used for: Encode the CSI amplitude of the first wireless signal of the jth frame on the i-th channel among the N channels arranged in the second order as a pixel value of the i-th row and j-th column in the third sample image; The CSI phase of the first wireless signal of the jth frame on the i-th channel among N channels arranged in the second order is encoded as the pixel value of the i-th row and j-th column in the fourth sample image, where i and j are positive integers, i≤N, j≤M, and the sample image includes the third sample image and the fourth sample image, and the first order is different from the second order.

32. The device according to claim 30 or 31, characterized in that The signal feature includes a received signal strength indication (RSSI), and the encoding unit is specifically used to: encode the RSSI of the first wireless signal of the jth frame into a pixel value of the jth column in the fifth sample image, and the sample image also includes a fifth sample image.

33. A human motion recognition device, characterized in that: The invention comprises a processor and a memory, wherein the memory is used to store a program, and the processor executes the program stored in the memory. When the program stored in the memory is executed, the processor implements the method according to any one of claims 1 to 8.

34. A wireless signal processing device, characterized in that: It includes a processor, a wireless communication module and a memory, the memory is used to store programs, the wireless communication module is used to receive M-frame wireless signals sent by a wireless transmitting device and extract signal characteristics of the M-frame wireless signals, and the processor executes the program stored in the memory. When the program stored in the memory is executed, the processor implements the method described in any one of claims 9 to 13.

35. A neural network training device, characterized in that: It comprises a processor and a memory, wherein the memory is used to store a program, and the processor executes the program stored in the memory. When the program stored in the memory is executed, the processor implements the method according to any one of claims 14 to 16.

36. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program codes, and the program codes include codes for executing the method according to any one of claims 1 to 8.

37. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program codes, and the program codes include codes for executing the method according to any one of claims 9 to 13.

38. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program codes, and the program codes include codes for executing the method according to any one of claims 14 to 16.

Images