Behavior Recognition Method, Apparatus, Electronic Device and Readable Storage Medium

Abstract

This application relates to the field of terminal technologies, and provides a behavior recognition method, apparatus, electronic device, and readable storage medium. The method includes: obtaining environmental data collected by sensors of the electronic device, and obtaining communication data of the electronic device, where the environmental data includes at least one of acceleration information, geomagnetic information, and angular velocity information; extracting data features from the environmental data and the communication data to obtain forward trajectory data features and reverse trajectory data features within a preset time; respectively performing behavior prediction on the forward trajectory data features and the reverse trajectory data features to obtain a forward trajectory behavior prediction result and a reverse trajectory behavior prediction result; fusing the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result to obtain a behavior recognition result; improving the accuracy of recognizing user behavior based on induction data and communication data of sensors.

Description

Technical Field

[0001] The present application relates to the technical field of behavior recognition, and in particular, to a behavior recognition method, apparatus, electronic device, and readable storage medium. Background Art

[0002] Currently, the functions of electronic devices are becoming more and more abundant, and there are also more and more sensors integrated on the electronic devices. The data collected by various sensors are induction data of different dimensions. When analyzing the induction data of different dimensions, it is possible to identify the states of the electronic device such as positioning and posture. For some electronic devices carried by users, through these induction data, it is also possible to identify the states of the user's location (trajectory) or behavior, etc., and thus realize the monitoring of the user.

[0003] In the related art, generally, the following method is used to identify a person's location or behavior: collecting induction data through various sensors, then putting the extracted features extracted from the various induction data into a machine learning model for training, and then using the trained machine learning model to identify the behavior state of the user. The accuracy of this recognition method is not high. Especially when relying on a large number of dimensions of induction data, if the induction data of one dimension is lost, the recognition error will occur. Summary of the Invention

[0004] The present application provides a behavior recognition method, apparatus, electronic device, and readable storage medium, aiming to improve the accuracy of identifying user behavior through the induction data and communication data of sensors and enhance the user experience.

[0005] In a first aspect, the present application provides a behavior recognition method, which includes:

[0006] Obtaining the environmental data collected by the sensors of the electronic device, and obtaining the communication data of the electronic device, where the environmental data includes at least one of acceleration information, geomagnetic information, and angular velocity information;

[0007] Performing data feature extraction on the environmental data and the communication data to obtain the forward trajectory data features and the reverse trajectory data features within a preset time;

[0008] Performing behavior prediction on the forward trajectory data features and the reverse trajectory data features respectively to obtain a forward trajectory behavior prediction result and a reverse trajectory behavior prediction result;

[0009] Fusing the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result to obtain a behavior recognition result.

[0010] In the embodiments of the present application, by extracting features from communication data and environmental data collected by sensors, and then reversing and transposing the data features, reverse data features opposite to the data features of the forward trajectory within a preset time are obtained for the electronic device. Then, based on the forward trajectory data features and the reverse trajectory data, behavior prediction is performed to obtain the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result. Based on the method of fusing and proofreading the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result, the prediction result of behavior prediction based on data features is corrected by forward and reverse trajectories, improving the accuracy of identifying user behavior based on sensor sensing data and communication data and enhancing the user experience.

[0011] In a possible implementation manner, the step of fusing the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result to obtain a behavior recognition result includes:

[0012] Obtain the intersection samples in the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result;

[0013] Determine other samples except the intersection samples;

[0014] Modify the other samples according to the samples whose prediction results are flat-layer stationary or walking;

[0015] Obtain the behavior recognition result according to the modified other samples and the intersection samples.

[0016] In another possible implementation manner, the step of obtaining the behavior recognition result according to the modified other samples and the intersection samples includes:

[0017] Perform abnormal sample filtering processing on the modified other samples and / or the intersection samples, and obtain the behavior recognition result with the filtered other samples and the intersection samples.

[0018] In another possible implementation manner, after the step of respectively performing behavior prediction on the forward trajectory data features and the reverse trajectory data features to obtain the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result, it further includes:

[0019] Perform behavior prediction according to the environmental data and the communication data to obtain the forward trajectory behavior reference result and the reverse trajectory behavior reference result for the preset time;

[0020] Correct the forward trajectory behavior prediction result based on the forward trajectory behavior reference result, and correct the reverse trajectory behavior prediction result based on the reverse trajectory behavior reference result;

[0021] The step of fusing the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result to obtain a behavior recognition result includes:

[0022] Fuse the corrected forward trajectory behavior prediction result and the corrected reverse behavior prediction result to obtain a behavior recognition result.

[0023] In another possible implementation, the step of correcting the forward trajectory behavior prediction result based on the forward trajectory behavior reference result includes:

[0024] Compare each sample in the forward trajectory behavior reference result and the forward trajectory behavior prediction result in sequence;

[0025] Correct the sample corresponding to the forward trajectory behavior prediction result according to the sample of the forward trajectory behavior reference result.

[0026] In another possible implementation, the step of correcting the sample corresponding to the forward trajectory behavior prediction result according to the sample of the forward trajectory behavior reference result includes:

[0027] If the prediction result of the first sample of the forward trajectory behavior reference result is walking, correct the second sample corresponding to the first sample in the forward trajectory behavior prediction result and with a confidence level less than the second threshold, where the forward trajectory behavior prediction result is obtained through a preset behavior recognition algorithm model, and the confidence level is the accuracy of the preset behavior recognition algorithm model in recognizing the second sample;

[0028] If the prediction result of the first sample of the forward trajectory behavior reference result is not walking, determine the second sample corresponding to the first sample in the forward trajectory behavior prediction result;

[0029] Correct the forward trajectory behavior prediction result according to the matching degree of the prediction results of the first sample and the second sample.

[0030] In another possible implementation, the step of correcting the forward trajectory behavior prediction result according to the matching degree of the prediction results of the first sample and the second sample includes at least one of the following:

[0031] If the prediction result of the first sample of the forward trajectory behavior reference result is taking the escalator, and the prediction result of the second sample of the forward trajectory behavior prediction result is walking or stationary on the flat floor, then correct the second sample with reference to the first sample, where the second sample is the sample corresponding to the first sample in the forward trajectory behavior prediction result;

[0032] If the prediction result of the first sample of the forward trajectory behavior reference result is taking the escalator, and the prediction result of the second sample of the forward trajectory behavior prediction result is taking the elevator, then the second sample with a confidence level less than the first threshold is corrected with reference to the first sample;

[0033] If the prediction result of the first sample of the forward trajectory behavior reference result is stationary on the flat floor, and the prediction result of the second sample of the forward trajectory behavior prediction result is taking the escalator, then the second sample with a confidence level less than the second threshold is corrected with reference to the first sample.

[0034] In another possible implementation, the step of performing behavior prediction based on the environmental data and the communication data to obtain the forward trajectory behavior reference result and the reverse trajectory behavior reference result for the preset time includes:

[0035] Determine the number of steps corresponding to each sample according to the acceleration information of each sample;

[0036] If the number of steps is greater than zero, determine that the behavior of the sample is walking and predict that the sample is walking;

[0037] If the number of steps is equal to zero, determine that the behavior of the sample is non-walking, and determine the prediction result of the sample according to at least one of the number of non-walking samples, the geomagnetic information, and the communication data.

[0038] In another possible implementation, the step of determining the prediction result of the sample according to at least one of the number of non-walking samples, the geomagnetic information, and the communication data includes:

[0039] If the number of non-walking samples is outside the preset range, predict that the sample is stationary on the flat floor;

[0040] If the number of non-walking samples is within the preset range, then predict that the sample is taking the escalator.

[0041] In another possible implementation, the step of if the number of non-walking samples is within the preset range, then predict that the sample is taking the escalator includes:

[0042] If the number of non-walking samples is within the preset range, obtain the geomagnetic variance of the sample corresponding to the number of samples;

[0043] If the geomagnetic variance is greater than the preset variance, predict that the sample is taking the escalator;

[0044] If the geomagnetic variance is less than the preset variance, predict that the sample is stationary on the flat floor.

[0045] In another possible implementation, the step of fusing the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result to obtain a behavior recognition result includes:

[0046] Fuse the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result to obtain a fusion result;

[0047] Determine the sample position of a preset landmark point according to the fusion result;

[0048] Adjust the sample position of the preset landmark point, and obtain a behavior recognition result according to the fusion result after the adjustment of the sample position of the preset landmark point.

[0049] In another possible implementation, the step of adjusting the sample position of the preset landmark point includes:

[0050] Determine the number of steps corresponding to each sample according to the acceleration information of each sample;

[0051] Determine the turning sample position where the number of steps switches between zero and non-zero according to the number of steps of each sample;

[0052] Align the sample position of the preset landmark point in the fusion result with the turning sample position.

[0053] In another possible implementation, after the step of determining the sample position of the preset landmark point according to the fusion result, it further includes:

[0054] Determine the number of samples between two adjacent preset landmark points according to the sample position of the preset landmark point;

[0055] When the number of samples is greater than a preset number, modify the behavior corresponding to the samples between the preset landmark points to flat;

[0056] When the number of samples is less than the preset number, execute the step of adjusting the sample position of the preset landmark point.

[0057] In another possible implementation, the step of fusing the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result to obtain a behavior recognition result includes:

[0058] Fuse the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result to obtain a fusion result;

[0059] Predict a second behavior reference result of the trajectory within the preset time according to the acceleration information;

[0060] Correct the fusion result based on the second behavior reference result to obtain a behavior recognition result.

[0061] In another possible implementation, the step of correcting the fusion result based on the second behavior as the reference result includes:

[0062] Compare each sample in the second behavior as the reference result and the fusion result;

[0063] If the prediction result of the first sample in the second behavior as the reference result is flat or elevator, modify the second sample corresponding to the first sample in the fusion result according to the second behavior as the reference result;

[0064] If the prediction result of the first sample in the second behavior as the reference result is incorrect, and the prediction result of the second sample corresponding to the first sample in the fusion result is flat, correct the second sample according to the forward trajectory behavior prediction result and / or the reverse trajectory behavior prediction result.

[0065] In another possible implementation, the step of predicting the second behavior as the reference result of the trajectory within the preset time according to the acceleration information includes:

[0066] Determine the continuous superweight and weightlessness ratio corresponding to each sample according to the acceleration information of each sample;

[0067] If there are target samples with a continuous superweight and weightlessness ratio greater than a preset value, and the number of target samples continuously exceeds a preset number, determine that the prediction result of the target samples is taking the elevator;

[0068] If there are no target samples with a continuous superweight and weightlessness ratio greater than a preset value, or the number of target samples does not continuously exceed a preset number, determine that the prediction result of the target samples is flat.

[0069] In another possible implementation, both the forward trajectory data feature and the reverse trajectory data feature include time domain feature, statistical feature and behavior semantic feature.

[0070] In another possible implementation, the preset behavior recognition algorithm model is trained based on the Extreme Gradient Boosting (XGBoost) algorithm.

[0071] In a second aspect, the present application further provides a behavior recognition device, including: an acquisition module, a feature extraction module, a prediction module and a fusion module;

[0072] The acquisition module acquires the environmental data collected by the sensors of the electronic device and acquires the communication data of the electronic device, where the environmental data includes at least one of acceleration information, geomagnetic information and angular velocity information;

[0073] The feature extraction module is configured to extract data features from the environmental data and the communication data, and obtain forward trajectory data features and reverse trajectory data features within a preset time;

[0074] The prediction module is configured to perform behavior prediction on the forward trajectory data features and the reverse trajectory data features respectively, and obtain a forward trajectory behavior prediction result and a reverse trajectory behavior prediction result;

[0075] The fusion module is configured to fuse the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result to obtain a behavior recognition result.

[0076] In a third aspect, the present application provides an electronic device, including: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the electronic device, cause the electronic device to execute the behavior recognition method according to any one of the above first aspect or possible implementations of the first aspect.

[0077] In a fourth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the processor is caused to execute the behavior recognition method according to any one of the above first aspect or possible implementations of the first aspect.

[0078] In a fifth aspect, the present application provides a chip, the chip includes a processor and a data interface, and the processor reads instructions stored on a memory through the data interface and executes the behavior recognition method according to any one of the first aspect or possible implementations of the first aspect.

[0079] Optionally, as a possible implementation, the chip may further include a memory, and instructions are stored in the memory, and the processor is configured to execute the instructions stored on the memory, and when the instructions are executed, the processor is configured to execute the behavior recognition method according to any one of the first aspect or possible implementations of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 is a schematic structural diagram of an electronic device provided by an embodiment of the present application;

[0081] Figure 2 is a software structural block diagram of an electronic device provided by an embodiment of the present application;

[0082] Figure 3 is a flowchart of a behavior recognition method provided by an embodiment of the present application;

[0083] Figure 4 Module interaction diagram of the behavior recognition method provided by an embodiment of the present application;

[0084] Figure 5 Refined flowchart of the behavior recognition method provided by an embodiment of the present application;

[0085] Figure 6 Flowchart of the behavior recognition method provided by another embodiment of the present application;

[0086] Figure 7 Refined flowchart of the behavior recognition method provided by another embodiment of the present application;

[0087] Figure 8 Another refined flowchart of the behavior recognition method provided by another embodiment of the present application;

[0088] Figure 9 Flowchart of the behavior recognition method provided by yet another embodiment of the present application;

[0089] Figure 10 Flowchart of the behavior recognition method provided by still another embodiment of the present application;

[0090] Figure 11 Flowchart of the behavior recognition method provided by yet another embodiment of the present application;

[0091] Figure 12 Structural schematic diagram of a behavior recognition device provided by the present application;

[0092] Figure 13 Module schematic diagram of the behavior recognition method provided by an embodiment of the present application;

[0093] Figure 14 For Figure 6 Flow schematic diagram of step S605 in

[0094] Figure 15 For Figure 6 Flow schematic diagram of step S604 in

[0095] Figure 16 For Figure 9 Flow schematic diagram of step S906 in

[0096] Figure 17 For Figure 10 Flow schematic diagram of step S1006 in

[0097] Figure 18 For Figure 10 Flow schematic diagram of step S1005 in Detailed implementation manners

[0098] The terms "first", "second", "third", etc. in the description, claims and drawings of this application are used to distinguish different objects, rather than to limit a specific order.

[0099] In the embodiments of this application, words such as "exemplary" or "for example" are used to give examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as more preferred or more advantageous than other embodiments or design solutions. Rather, the use of words such as "exemplary" or "for example" is intended to present relevant concepts in a specific manner.

[0100] For the sake of clear and concise description of the following embodiments, first, a brief introduction to an implementation solution for behavior recognition is given:

[0101] Identifying the behavior of a user can enable the monitoring of the user. Especially for the elderly or children at home, it can enable the monitoring of the health and safety of the elderly or children. Currently, the identification of a user's behavior generally relies on an electronic device carried by the user, and the behavior of the user can be identified based on satellite positioning. However, when the user is indoors, satellite positioning is interfered with, and the positioning accuracy is not high, and the method of satellite positioning has high costs and high power consumption. Based on the fact that the functions of current electronic devices are becoming more and more abundant, there are also more and more sensors integrated on the electronic devices. The data collected by various sensors are sensing data in different dimensions. When analyzing the sensing data in different dimensions, it is possible to identify the states such as the positioning and posture of the electronic device. Therefore, using sensor information to identify user behavior is low-power, efficient and accurate.

[0102] Before satellite positioning, there was also a related technology that identified a person's position or behavior through the sensing data of sensors. And this method generally requires putting the sensing data of a large number of different types of sensors into a machine learning model for training, and then identifying the behavior state of the user through the trained machine learning model. The accuracy of this identification method is not high. Especially when relying on a large number of dimensions of sensing data, if the sensing data of one dimension is lost, the situation of misidentification will occur.

[0103] Based on the problems existing in the above technical solutions, the present application provides a behavior recognition method, which can accurately recognize human behaviors based on nine-axis sensor data and base station information, and has the advantages of low power consumption, high efficiency, and less dependence on data sources. Based on the fusion and correction of forward and reverse trajectory data, the accuracy of recognizing user behaviors through the sensing data and communication data of sensors can be improved, and the user experience can be enhanced. The provided behavior recognition method can be applied to electronic devices such as mobile phones, tablet computers, desktop computers, laptop computers, notebook computers, ultra-mobile personal computers (UMPCs), handheld computers, netbooks, personal digital assistants (PDAs), wearable electronic devices, and smart watches. Among them, for the electronic devices applicable to this behavior recognition method, their structures can be as Figure 2 shown.

[0104] As Figure 1 shown, Figure 1 FIG. is a composition example diagram of an electronic device provided by the present application. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0105] It can be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 100. In other embodiments, the electronic device 100 may include more or fewer components than shown, or combine certain components, or split certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0106] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0107] Among them, the controller may be the nerve center and command center of the electronic device 100. The controller may generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0108] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory may save the instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can directly call it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0109] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0110] The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be respectively coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 180K through the I2C interface, enabling the processor 110 and the touch sensor 180K to communicate through the I2C bus interface to implement the touch function of the electronic device 100.

[0111] The I2S interface can be used for audio communication. In some embodiments, the processor 110 may include multiple groups of I2S buses. The processor 110 may be coupled to the audio module 170 through the I2S bus to implement communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 may transmit an audio signal to the wireless communication module 160 through the I2S interface to implement the function of answering a call through a Bluetooth headset.

[0112] The PCM interface can also be used for audio communication to sample, quantize, and encode analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 may be coupled through the PCM bus interface. In some embodiments, the audio module 170 may also transmit an audio signal to the wireless communication module 160 through the PCM interface to implement the function of answering a call through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0113] The UART interface is a general-purpose serial data bus for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 may transmit an audio signal to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.

[0114] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate through the CSI interface to implement the shooting function of the electronic device 100. The processor 110 and the display screen 194 communicate through the DSI interface to implement the display function of the electronic device 100.

[0115] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to the camera 193, the display screen 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0116] The USB interface 130 is an interface that complies with the USB standard specification, and can specifically be a Mini USB interface, a Micro USB interface, a USB Type-C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. This interface can also be used to connect other electronic devices, such as AR devices, etc.

[0117] It can be understood that the interface connection relationship between the modules illustrated in this embodiment is only for illustrative purposes and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 can also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0118] The charging management module 140 is used to receive a charging input from a charger. Among them, the charger can be a wireless charger or a wired charger. In some embodiments of wired charging, the charging management module 140 can receive the charging input from a wired charger through the USB interface 130. In some embodiments of wireless charging, the charging management module 140 can receive the wireless charging input through the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device 100 through the power management module 141.

[0119] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives inputs from the battery 142 and / or the charging management module 140 and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, the wireless communication module 160, etc. The power management module 141 can also be used to monitor parameters such as the battery capacity, the number of battery cycles, and the battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be disposed in the processor 110. In some other embodiments, the power management module 141 and the charging management module 140 can also be disposed in the same device.

[0120] The wireless communication function of the electronic device 100 can be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modulation and demodulation processor, and the baseband processor, etc.

[0121] The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example: The antenna 1 can be multiplexed as the diversity antenna of the wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0122] The mobile communication module 150 can provide solutions for wireless communications such as 2G / 3G / 4G / 5G applied to the electronic device 100. The mobile communication module 150 can include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves by the antenna 1, filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor and convert it into electromagnetic waves through the antenna 1 for radiation. In some embodiments, at least some functional modules of the mobile communication module 150 can be disposed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 can be disposed in the same device.

[0123] The modulation and demodulation processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. Subsequently, the demodulator transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.), or displays an image or video through the display screen 194. In some embodiments, the modulation and demodulation processor may be an independent device. In other embodiments, the modulation and demodulation processor may be independent of the processor 110 and be provided in the same device as the mobile communication module 150 or other functional modules.

[0124] The wireless communication module 160 may provide solutions for wireless communications applied to the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite systems (GNSSs), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, performs frequency modulation and filtering processing on the electromagnetic wave signals, and transmits the processed signals to the processor 110. The wireless communication module 160 may also receive the signal to be transmitted from the processor 110, perform frequency modulation and amplification on it, and convert it into electromagnetic waves through the antenna 2 and radiate them out.

[0125] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, such that electronic device 100 can communicate with a network and other devices through wireless communication technologies. The wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS), and / or satellite based augmentation systems (SBAS).

[0126] Electronic device 100 implements a display function through a GPU, display screen 194, and an application processor, etc. The GPU is a microprocessor for image processing, and is connected to display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0127] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0128] A series of graphical user interfaces (GUIs) can be displayed on the display screen 194 of the electronic device 100, and these GUIs are the main screens of the electronic device 100. Generally, the size of the display screen 294 of the electronic device 100 is fixed, and only a limited number of controls can be displayed on the display screen 194 of the electronic device 100. A control is a GUI element, which is a software component included in an application program and controls all the data processed by the application program and the interaction operations related to these data. Users can interact with the control through direct manipulation to read or edit relevant information of the application program. Generally speaking, controls can include visible interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, Widgets, etc. For example, in the embodiments of the present application, the display screen 191 can display virtual buttons (one-key choreography, start choreography, scene choreography).

[0129] The electronic device 100 can implement the shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, an application processor, etc.

[0130] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and the light passes through the lens and is transmitted to the camera photosensitive element, where the optical signal is converted into an electrical signal. The camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize parameters such as the exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0131] Camera 193 is used to capture static images or videos. An object generates an optical image through a lens and projects it onto a photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transfers the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard format such as RGB or YUV. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0132] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0133] The video codec is used to compress or decompress digital videos. The electronic device 100 can support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0134] The NPU is a neural-network (NN) computing processor. By learning from the structure of biological neural networks, such as the transmission pattern between human brain neurons, it can quickly process input information and can also continuously self-learn. Through the NPU, applications such as intelligent cognition of the electronic device 100 can be realized, such as image recognition, face recognition, speech recognition, text understanding, etc. In the embodiments of this application, the NPU can also identify human behaviors by recognizing environmental data and communication data collected by the electronic device.

[0135] The external memory interface 120 can be used to connect an external memory card, such as a MicroSD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 210 through the external memory interface 120 to implement the data storage function. For example, files such as music and videos are saved in the external memory card.

[0136] The internal memory 121 can be used to store computer-executable program codes, and the executable program codes include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. For example, in this embodiment, the processor 110 can perform scenario arrangement by executing the instructions stored in the internal memory 121. The internal memory 121 can include a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.). The data storage area can store data created during the use of the electronic device 100 (such as audio data, phone book, etc.). In addition, the internal memory 121 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121, and / or the instructions stored in the memory provided in the processor.

[0137] The electronic device 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor, etc. For example, music playback, recording, etc.

[0138] The audio module 170 is used to convert digital audio information into an analog audio signal for output, and is also used to convert an analog audio input into a digital audio signal. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0139] The speaker 170A, also called a "loudspeaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or a hands-free call through the speaker 170A.

[0140] The receiver 170B, also called a "handset", is used to convert an audio electrical signal into a sound signal. When the electronic device 100 answers a call or a voice message, the voice can be listened to by bringing the receiver 170B close to the human ear.

[0141] The microphone 170C, also known as a "microphone" or "transmitter", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak close to the microphone 170C with their mouth to input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In some other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also implement a noise reduction function. In some other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, and implement functions such as directional recording.

[0142] The headphone jack 170D is used to connect a wired headphone. The headphone jack 170D can be a USB interface 130, or a 3.5mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0143] The pressure sensor 180A is used to sense pressure signals and can convert the pressure signals into electrical signals. In some embodiments, the pressure sensor 180A can be disposed on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can include at least two parallel plates with conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the intensity of the touch operation according to the pressure sensor 180A. The electronic device 100 can also calculate the position of the touch according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, the instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, the instruction to create a new short message is executed.

[0144] The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., the x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. Exemplarily, when the shutter is pressed, the gyroscope sensor 180B detects the angle of jitter of the electronic device 100, calculates the distance that the lens module needs to compensate according to the angle, and enables the lens to cancel the jitter of the electronic device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used for navigation and somatosensory game scenarios.

[0145] The barometric pressure sensor 180C is used to measure the barometric pressure. In some embodiments, the electronic device 100 calculates the altitude based on the barometric pressure value measured by the barometric pressure sensor 180C to assist in positioning and navigation.

[0146] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 can detect the opening and closing of the flip according to the magnetic sensor 180D. Furthermore, according to the detected opening and closing state of the leather case or the flip, features such as automatic flip unlocking can be set.

[0147] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device 100 and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0148] The distance sensor 180F is used to measure distance. The electronic device 100 can measure distance through infrared or laser. In some embodiments, in a shooting scene, the electronic device 100 can use the distance sensor 180F to measure distance to achieve rapid focusing.

[0149] The proximity light sensor 180G can include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The light-emitting diode can be an infrared light-emitting diode. The electronic device 100 emits infrared light outward through the light-emitting diode. The electronic device 100 uses the photodiode to detect the infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The electronic device 100 can use the proximity light sensor 180G to detect when the user holds the electronic device 100 close to the ear for a call, so as to automatically turn off the screen to achieve the purpose of power saving. The proximity light sensor 180G can also be used for automatic unlocking and locking of the leather case mode and pocket mode.

[0150] The ambient light sensor 180L is used to sense the ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the sensed ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance during photography. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touches.

[0151] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to achieve fingerprint unlocking, access application locks, fingerprint photography, fingerprint answering of incoming calls, etc.

[0152] The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 utilizes the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the electronic device 100 reduces the performance of the processor near the temperature sensor 180J in order to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is below another threshold, the electronic device 100 heats the battery 142 to prevent abnormal shutdown of the electronic device 100 caused by low temperature. In some other embodiments, when the temperature is below yet another threshold, the electronic device 100 boosts the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

[0153] The touch sensor 180K, also known as the "touch control device". The touch sensor 180K can be disposed on the display screen 194, and together with the display screen 194, it forms a touch screen, also known as the "touch control screen". The touch sensor 180K is used to detect touch operations acting on it or nearby. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual outputs related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100, at a different position from the display screen 194.

[0154] The bone conduction sensor 180M can obtain vibration signals. In some embodiments, the bone conduction sensor 180M can obtain the vibration signals of the vibrating bone mass of the human vocal part. The bone conduction sensor 180M can also contact the human pulse to receive blood pressure pulsation signals. In some embodiments, the bone conduction sensor 180M can also be disposed in the earphone to form a bone conduction earphone.

[0155] The audio module 170 can parse out voice signals based on the vibration signals of the vibrating bone mass of the human vocal part obtained by the bone conduction sensor 180M to implement voice functions. The application processor can parse out heart rate information based on the blood pressure pulsation signals obtained by the bone conduction sensor 180M to implement heart rate detection functions.

[0156] The button 190 includes a power-on button, volume buttons, etc. The button 190 can be a mechanical button or a touch button. The electronic device 100 can receive button inputs and generate key signal inputs related to the user settings and function controls of the electronic device 100.

[0157] The motor 191 can generate vibration prompts. The motor 191 can be used for incoming call vibration prompts and also for touch vibration feedback. For example, touch operations for different applications (such as taking pictures, audio playing, etc.) can correspond to different vibration feedback effects. Touch operations on different areas of the display screen 194 can also correspond to different vibration feedback effects for the motor 191. Different application scenarios (such as time reminder, receiving messages, alarm clock, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0158] The indicator 192 can be an indicator light and can be used to indicate the charging state, power change, and can also be used to indicate messages, missed calls, notifications, etc.

[0159] The SIM card interface 195 is used to connect to the SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to achieve contact and separation from the electronic device 100. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to achieve functions such as calls and data communication. In some embodiments, the electronic device 100 uses an eSIM, that is, an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0160] In addition, an operating system runs on the above components. For example, HarmonyOS, iOS operating system, Android open-source operating system, Windows operating system, etc. Application programs can be installed and run on this operating system.

[0161] Figure 2 It is a software structure block diagram of the electronic device provided by the embodiment of this application.

[0162] The layered architecture divides software into several layers, each layer having a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments of the present application, the software architecture of the electronic device at least includes an algorithm architecture layer, wherein the algorithm architecture layer includes an output layer, a core algorithm layer, a data processing layer, and a data layer.

[0163] For ease of understanding, the following embodiments of the present application will be described by taking the electronic device 200 having the Figure 1 and Figure 2 shown structure as an example, and in combination with the accompanying drawings and application scenarios, the behavior recognition method provided by the embodiments of the present application will be specifically described.

[0164] Figure 3 is a flowchart of the behavior recognition method provided by an embodiment of the present application, Figure 5 is a module interaction diagram of the behavior recognition method provided by an embodiment of the present application.

[0165] As Figure 3 and Figure 5 shown, the above-mentioned behavior recognition method may include:

[0166] Step S301: Obtain the environmental data collected by the sensors of the electronic device, and obtain the communication data of the electronic device, where the environmental data includes at least one of acceleration information, geomagnetic information, and angular velocity information;

[0167] In this embodiment, the electronic device is taken as a mobile terminal for illustration. The electronic device includes a nine-axis sensor, and the nine-axis sensor is a combination of three sensors: a 3-axis acceleration sensor, a 3-axis gyroscope, and a 3-axis electronic compass (geomagnetic sensor). The nine-axis sensor collects acceleration information, geomagnetic information, and angular velocity information. Optionally, the angular velocity information is collected by the gyroscope.

[0168] The behavior recognition method can be applied to the electronic device, or can also be applied to the terminal device communicating with the electronic device. For example, if the electronic device is a mobile phone carried by user A, user A can obtain the current behavior based on the electronic device. Or, if the electronic device is a mobile phone carried by user A and the terminal device is a mobile phone carried by user B connected to user A, user B can monitor the behavior status of user A. In this case, the environmental data and communication data (such as WiFi data) are uploaded by the electronic device of the monitored user, that is, the terminal device of user B recognizes and monitors the behavior of user A based on the environmental data and communication data uploaded by the electronic device of user A. The following takes the recognition method applied to the electronic device as an example for illustration.

[0169] Step S302: Extract data features from the environmental data and the communication data to obtain positive trajectory data features and reverse trajectory data features within a preset time;

[0170] In this embodiment, the user's behavior recognition is not obtained through single data analysis, but through analyzing the user's trajectory within a preset time. Therefore, when analyzing the behavior in this embodiment, it is based on the trajectory data of the user within the preset time period for analysis to more accurately determine the user's behavior. Optionally, the trajectory refers to the trajectory data formed by sorting the collected environmental data and communication data based on time series within the preset time, and the trajectory data arranged in the order of first collected and then collected is the forward trajectory data.

[0171] Optionally, based on this embodiment, a method of fusing the features of forward and reverse trajectory data is adopted to identify the user's behavior. After the electronic device obtains the environmental data collected by the sensor and the communication data of the electronic device, the process of preprocessing the environmental data and the communication data includes: feature extraction and reverse arrangement of the forward trajectory. Optionally, the electronic device can first extract features and then perform reverse arrangement of the forward trajectory; it can also first perform reverse arrangement of the forward trajectory and then extract features. Optionally, after extracting features first and then performing reverse arrangement of the forward trajectory, it can avoid extracting features from the forward and reverse trajectories respectively after reverse arrangement of the forward trajectory, increasing the number of feature extraction times.

[0172] Optionally, feature extraction refers to extracting the features of various data from the environmental data and communication data to form the input parameters of the preset behavior recognition algorithm model. In one possible implementation, the extracted data features include time domain features and statistical features. In another possible implementation, the extracted data features include time domain features, statistical features, and behavior semantic features. Optionally, the behavior semantic feature is another dimension of data features relative to the data extraction. Based on the combination of the behavior semantic feature and other features of the data to identify the behavior, the recognition accuracy of the preset behavior recognition algorithm model is further improved.

[0173] The reverse arrangement of the forward trajectory refers to arranging the forward trajectory data in reverse order according to the time series to obtain the reverse trajectory data, and the reverse trajectory data is opposite to the forward trajectory data. For example, if the forward trajectory data is 00001111100, the reverse trajectory data is 00111110000. In this embodiment, the process of feature extraction for the environmental data and communication data includes first extracting features from the forward trajectory data, and after extracting the forward trajectory data features, obtaining the reverse trajectory data features by reversing the forward trajectory data features.

[0174] Step S303: Respectively perform behavior prediction on the forward trajectory data features and the reverse trajectory data features to obtain a forward trajectory behavior prediction result and a reverse trajectory behavior prediction result;

[0175] In the embodiments of the present application, a preset behavior recognition algorithm model is used to perform behavior prediction on the forward trajectory data features and perform behavior prediction on the reverse trajectory data features. For example, the forward trajectory data features are input into the preset behavior recognition algorithm model, and the preset behavior recognition algorithm model processes the forward trajectory data features based on the algorithm architecture and outputs a forward trajectory behavior prediction result; the reverse trajectory data features are input into the preset behavior recognition algorithm model, and the preset behavior recognition algorithm model processes the reverse trajectory data features based on the algorithm architecture and inputs a reverse trajectory behavior prediction result.

[0176] Optionally, other methods can also be used to perform behavior prediction on the forward trajectory data features and the reverse trajectory data features respectively. The preset behavior recognition algorithm model is one of the ways of behavior prediction, which is not limited here.

[0177] Step S304: Fuse the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result to obtain a behavior recognition result.

[0178] The forward trajectory data features and the reverse trajectory data features are reverse-inverted. Therefore, if the prediction results corresponding to the forward trajectory data features and the prediction results corresponding to the reverse trajectory data features are the same, it means that the prediction result at this position is relatively accurate. Therefore, by fusing the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result, a more accurate behavior recognition result can be obtained.

[0179] Optionally, the behavior recognition result includes walking, staying still on a flat floor, taking an escalator, taking an elevator, etc. Or, in a possible implementation manner, the behavior recognition result includes flat floor and cross-floor. For example, it is recognized that a person is moving on a flat floor without performing a cross-floor action, or it is recognized that a person is performing a cross-floor action without moving on a flat floor.

[0180] In a possible implementation manner, the preset behavior recognition algorithm model is trained based on the Extreme Gradient Boosting (XGBoost) algorithm.

[0181] Optionally, the Extreme Gradient Boosting (XGBoost) algorithm is also called the XGBoost classifier. In this embodiment, the training process of the XGBoost classifier is as follows:

[0182] In the training stage of the model, the feature list extracted from the original data under various behaviors is combined with its corresponding behavior label to form a data set, and then passed into the classifier as the training set data and label in the supervised learning method, and trained based on this training set. The input of the XGBoost classifier is: the feature matrix under various behavior patterns; the output is: the prediction result data of each trajectory and the confidence level (accuracy) of the behavior data of each sample.

[0183] In a possible implementation, the XGBoost classifier outputs the corresponding initial behavior prediction result according to the specific classification.

[0184] In another possible implementation, in order to further improve the accuracy of the prediction result output by the XGBoost classifier, it is set that the XGBoost classifier outputs the prediction result in the way of converting multi-class to binary-class. For example, this application involves at least four behaviors: stationary, walking, going up and down the elevator, going up and down the escalator (in some embodiments, it also involves going up and down the stairs). The XGBoost classifier first separately identifies each category, and then performs post-processing according to requirements to merge the identification results. For example, stationary and walking are classified as flat-floor behaviors, and going up and down the stairs, going up and down the elevator, and going up and down the escalator are classified as cross-floor behaviors. Or the way of converting multi-class to three-class, for example: stationary and walking are in one category and belong to flat-floor movement behaviors, and going up and down the escalator and going up and down the elevator are separate categories respectively.

[0185] In a possible implementation, please refer to Figure 5 that the step of fusing the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result to obtain the behavior recognition result includes:

[0186] Step S501: Obtain the intersection samples in the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result;

[0187] Step S502: Determine other samples except the intersection samples;

[0188] Step S503: Modify the other samples according to the samples whose prediction results are flat-floor stationary or walking;

[0189] Step S504: Obtain the behavior recognition result according to the modified other samples and the intersection samples.

[0190] During the process of predicting the results of the XGBoost algorithm model, since the characteristics of walking and staying still on a flat floor are relatively obvious, the accuracy of behavior recognition is higher compared to the behavior of taking an elevator or an escalator. Therefore, during the fusion process of the forward trajectory behavior prediction results and the reverse trajectory behavior prediction results, the intersection samples with the prediction results of staying still on a flat floor and walking can be extracted as the prediction results of staying still on a flat floor or walking after fusion. Alternatively, the intersection samples with the prediction results of staying still on a flat floor and walking can be not taken, and the original data before fusion can be directly used as the prediction results of staying still on a flat floor or walking. Due to the prediction of the behavior of taking an elevator or an escalator, there are special cases, such as errors that may occur in the data collected at the moment of getting on the elevator, or errors that may occur during calculation. Therefore, the intersection samples with the prediction results of taking an elevator or an escalator are extracted as the prediction results after fusion, that is, only when both the forward trajectory behavior prediction results and the reverse trajectory behavior prediction results indicate the behavior of taking an elevator or an escalator, the prediction results of the intersection samples are used as the prediction results of the behavior of taking an elevator or an escalator.

[0191] That is to say, in some ways, the prediction results of all intersection samples in the forward trajectory behavior prediction results and the reverse trajectory behavior prediction results are retained as the first prediction results after fusion; while for other samples except the intersection samples, they are all modified to be the same as the samples of staying still on a flat floor or walking, and the modified prediction results are used as the second prediction results after fusion. The first prediction results and the second prediction results are combined to obtain the behavior recognition results. Optionally, for other samples except the intersection samples, if they are samples of staying still on a flat floor, there is no need to modify (or modify them to samples of staying still on a flat floor), and the samples of staying still on a flat floor are directly output. If they are walking samples, there is no need to modify (or modify them to walking samples). If they are neither walking nor staying still on a flat floor samples, they are modified to walking or staying still on a flat floor samples.

[0192] For example, if the forward trajectory behavior prediction result is: 00001111100, and the reverse trajectory behavior prediction result is 00111110000, the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result are fused to obtain 00XX111XX00. Among them, the intersection sample is "0011100", and the "XX" in the middle is other samples except the intersection samples. These other samples are corrected, and the prediction result after fusion is: 00001110000. Among them, based on the characteristics of the forward and reverse trajectory data, it is predicted that "111" is the prediction result of taking an escalator or an elevator, indicating that the accuracy of this prediction result is relatively high. Therefore, the accuracy of the behavior recognized by the prediction result after fusion is relatively high.

[0193] Alternatively, in some other ways, keep the prediction results of the intersection samples where the prediction results in the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result are taking the elevator or escalator as the first prediction result after fusion; and modify the other samples taking the elevator or escalator except the intersection samples to samples of staying still at the flat floor or walking as the second prediction result after fusion, and the prediction result of the samples of staying still at the flat floor or walking as the third prediction result. The first prediction result, the second prediction result and the third prediction result are fused to obtain the behavior recognition result.

[0194] In a possible implementation manner, the step of fusing the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result to obtain the behavior recognition result includes: comparing the prediction results of each sample in the forward initial behavior prediction result and the reverse initial behavior prediction result and the confidence level of the prediction result of each sample, and taking the prediction result with a higher confidence level as the prediction result after fusion. That is to say, select the result with a higher confidence level from the forward initial behavior prediction result and the reverse initial behavior prediction result as the behavior recognition result after fusion to improve the recognition accuracy.

[0195] In a possible implementation manner, the step of obtaining the behavior recognition result according to the modified other samples and the intersection samples includes:

[0196] Perform abnormal sample filtering processing on the modified other samples and / or the intersection samples, and obtain the behavior recognition result with the filtered other samples and the intersection samples.

[0197] The abnormal samples refer to the flat floor fluctuations and cross-floor fluctuations. Since the prediction results of the flat floor and the cross-floor predictions are both concentrated distributions, especially more obvious in the prediction result after fusion. For example, the prediction result is: 00001110000, indicating that the user first walks, then takes the escalator, and then walks again. If the prediction result is: 01001110010, and there is an escalator sample between two walking samples, it means that the escalator sample is an abnormal sample, and perform filtering processing on it to reduce the interference of abnormal samples.

[0198] Optionally, in this embodiment, before fusing the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result, abnormal filtering processing is respectively performed on the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result to improve the accuracy of fusion.

[0199] Optionally, the filtering method is to correct the abnormal samples.

[0200] Based on the verification tests of a large amount of data in this embodiment, in terms of behavior recognition, the recognition accuracy of binary classification can reach about 95% on average, and the recognition accuracy of ternary classification can reach about 94% on average. It can be seen that the accuracy of the recognition results of user behavior in this embodiment is high.

[0201] In this embodiment, by extracting the feature of the communication data and the environmental data collected by the sensor, and then reversing and inverting the data features, the reverse data features of the electronic device that are opposite to the data features of the forward trajectory within the preset time are obtained. Then, based on the forward trajectory data features and the reverse trajectory data, behavior prediction is performed to obtain the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result. Based on the method of fusing and proofreading the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result, the prediction result of behavior prediction based on data features is corrected by forward and reverse trajectory proofreading, which improves the accuracy of identifying user behavior based on the sensing data and communication data of the sensor and improves the user experience.

[0202] Figure 6 It is a flowchart of the behavior recognition method provided by another embodiment of the present application, as Figure 6 described, the method includes:

[0203] Step S601: Obtain the environmental data collected by the sensor of the electronic device, and obtain the communication data of the electronic device, where the environmental data includes at least one of acceleration information, geomagnetic information, and angular velocity information;

[0204] Step S602: Extract the data features of the environmental data and the communication data to obtain the forward trajectory data features and the reverse trajectory data features within the preset time;

[0205] Step S603: Perform behavior prediction on the forward trajectory data features and the reverse trajectory data features respectively to obtain the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result;

[0206] Step S604: Perform behavior prediction according to the environmental data and the communication data to obtain the forward trajectory behavior reference result and the reverse trajectory behavior reference result of the preset time;

[0207] Step S605: Correct the forward trajectory behavior prediction result based on the forward trajectory behavior reference result, and correct the reverse trajectory behavior prediction result based on the reverse trajectory behavior reference result;

[0208] Step S606: Fuse the corrected forward trajectory behavior prediction result and the corrected reverse behavior prediction result to obtain the behavior recognition result.

[0209] In this embodiment, the feature extraction methods for environmental data and communication data, the prediction methods for the forward trajectory behavior prediction results and the reverse trajectory behavior prediction results, and the fusion method for the forward trajectory behavior prediction results and the reverse trajectory behavior prediction results can adopt the above Figure 3 described embodiments, and will not be repeatedly defined herein.

[0210] In this embodiment, in order to further improve the accuracy of the prediction results, before fusing the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result, the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result are corrected to improve the recognition accuracy.

[0211] Optionally, since the accuracy of the prediction result of taking an elevator or escalator in the prediction results output by the preset behavior recognition algorithm model is relatively low, therefore, the forward trajectory behavior reference result can be the correction of the prediction result of taking an escalator, or the correction of the prediction result of taking an elevator. The following takes the correction of the prediction result of taking an escalator as an example for illustration.

[0212] In this embodiment, the correction principle of the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result is to identify the user's behavior through a method with higher accuracy in identifying a certain prediction result, and then compare this prediction result with the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result output by the preset behavior recognition algorithm model, and use the prediction result with higher accuracy to modify the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result, so that the corrected forward trajectory behavior prediction result and the reverse trajectory behavior prediction result have higher accuracy, making up for the defects of the algorithm model. For example, a method with higher accuracy in predicting the result of taking an escalator is used to predict the user's behavior, such as the escalator riding prediction rules introduced below.

[0213] It can be understood that the method for the forward trajectory behavior reference result to correct the forward trajectory behavior prediction result is the same as the method for the reverse trajectory behavior reference result to correct the reverse trajectory behavior prediction result. Therefore, the following takes one of the correction processes as an example for illustration. Please refer to Figure 7 , taking the modification of the forward trajectory behavior prediction result as an example. The steps for correcting the forward trajectory behavior prediction result based on the forward trajectory behavior reference result include:

[0214] Step S701: Compare each sample in the forward trajectory behavior reference result and the forward trajectory behavior prediction result in sequence;

[0215] Step S702: Correct the sample corresponding to the forward trajectory behavior prediction result according to the sample in the forward trajectory behavior reference result.

[0216] In this embodiment, a sample refers to a record of a preset length in the prediction result. For example, the prediction result within 2.56 s is one sample. The sample comparison method is one-to-one comparison. For example, the reference result of the forward trajectory behavior and the prediction result of the forward trajectory behavior are predicted from the environmental data and communication data corresponding to the same trajectory. The samples of the reference result of the forward trajectory behavior and the prediction result of the forward trajectory behavior are compared one-to-one based on time sequence. Similarly, the reference result of the reverse trajectory behavior and the prediction result of the reverse trajectory behavior are predicted from the environmental data and communication data corresponding to the same trajectory. The samples of the reference result of the reverse trajectory behavior and the prediction result of the reverse trajectory behavior are compared one-to-one based on time sequence.

[0217] Modify the sample corresponding to the prediction result of the forward trajectory behavior according to the sample of the reference result of the forward trajectory behavior. One possible implementation method is: determine the first sample whose prediction result in the reference result of the forward trajectory behavior is taking the escalator, and then modify the prediction result of the second sample corresponding to the first sample in the prediction result of the forward trajectory behavior with reference to the prediction result of taking the escalator. That is, only modify the prediction result of taking the escalator. For example, modify the prediction result that needs to be modified in the prediction result of the forward trajectory behavior to the prediction result of taking the escalator.

[0218] In another possible implementation method, in combination with reference to Figure 14 , the step of modifying the sample corresponding to the prediction result of the forward trajectory behavior according to the sample of the reference result of the forward trajectory behavior includes:

[0219] If the prediction result of the first sample of the reference result of the forward trajectory behavior is walking, modify the second sample corresponding to the first sample in the prediction result of the forward trajectory behavior and with a confidence level less than the second threshold, where the prediction result of the forward trajectory behavior is obtained through a preset behavior recognition algorithm model, and the confidence level is the accuracy of the preset behavior recognition algorithm model in recognizing the second sample. Figure 14 In

[0220] If the prediction result of the first sample of the reference result of the forward trajectory behavior is not walking, determine the second sample corresponding to the first sample in the prediction result of the forward trajectory behavior;

[0221] Modify the prediction result of the forward trajectory behavior according to the matching degree of the prediction results of the first sample and the second sample. [[ID=W22]]

[0222] In this implementation, based on the division of the prediction results of walking and non-walking, if the prediction result of the sample in the forward trajectory behavior reference result is walking and the prediction accuracy based on walking is very high, there is no need to determine whether the prediction result of the second sample in the forward trajectory behavior prediction result is walking or taking the escalator. Instead, directly modify the prediction result corresponding to the forward trajectory behavior prediction result, or modify the prediction result of the sample with a lower confidence level. Optionally, the modification method is to modify the second sample of the forward trajectory behavior prediction result to the prediction result of walking.

[0223] If the prediction result of the first sample in the forward trajectory behavior reference result is not walking, that is, it may be taking the escalator or elevator. Although the accuracy of the forward trajectory behavior reference result is relatively high, other interferences cannot be excluded. Therefore, the initial behavior prediction result is corrected according to the prediction result matching degree between the forward trajectory behavior reference result and the forward trajectory behavior prediction result to improve the accuracy of the correction.

[0224] In a possible implementation manner, the step of correcting the forward trajectory behavior prediction result according to the prediction result matching degree of the first sample and the second sample includes at least one of the following:

[0225] If the prediction result of the first sample in the forward trajectory behavior reference result is taking the escalator, and the prediction result of the second sample in the forward trajectory behavior prediction result is walking or stationary on the same floor, then correct the second sample with reference to the first sample. The second sample is the sample in the forward trajectory behavior prediction result corresponding to the first sample, so that the forward trajectory behavior prediction result is consistent with the forward trajectory behavior prediction result.

[0226] That is to say, if the forward trajectory behavior reference result is inconsistent with the forward trajectory behavior prediction result, and the forward trajectory behavior reference result is taking the escalator, it means that there is an error in the forward trajectory behavior prediction result. Modify the prediction result of the second sample of the forward trajectory behavior prediction result to taking the escalator.

[0227] If the prediction result of the first sample in the forward trajectory behavior reference result is taking the escalator, and the prediction result of the second sample in the forward trajectory behavior prediction result is taking the elevator, then correct the second sample with a confidence level less than the first threshold with reference to the first sample, so that the forward trajectory behavior prediction result is consistent with the forward trajectory behavior prediction result.

[0228] That is to say, if the reference result of the forward trajectory behavior is inconsistent with the predicted result of the forward trajectory behavior, and the reference result of the forward trajectory behavior is taking the escalator while the predicted result of the forward trajectory behavior is taking the elevator, it indicates that the predicted results of the reference result of the forward trajectory behavior and the predicted result of the forward trajectory behavior are relatively close. There may be an error in the reference result of the forward trajectory behavior or an error in the predicted result of the forward trajectory behavior. At this time, the predicted result of the second sample with a lower confidence level in the predicted result of the forward trajectory behavior is modified to taking the escalator, and the sample with a higher confidence level remains unchanged. In this way, a situation where the error becomes larger after correction may occur.

[0229] If the predicted result of the first sample of the reference result of the forward trajectory behavior is flat floor still, and the predicted result of the second sample of the predicted result of the forward trajectory behavior is taking the escalator, then the second sample with a confidence level less than the second threshold is corrected with reference to the first sample, so that the predicted result of the forward trajectory behavior is consistent with the predicted result of the forward trajectory behavior.

[0230] That is to say, if the results of the reference result of the forward trajectory behavior and the predicted result of the forward trajectory behavior are inconsistent, and the reference result of the forward trajectory behavior is flat floor still while the initial predicted result is taking the escalator, it indicates that the reference result of the forward trajectory behavior and the predicted result of the forward trajectory behavior are relatively close. There may be an error in the reference result of the forward trajectory behavior or an error in the predicted result of the forward trajectory behavior. At this time, the second sample with a lower confidence level in the predicted result of the forward trajectory behavior is modified to the predicted result of flat floor still, and the sample with a higher confidence level remains unchanged. In this way, a situation where the error becomes larger after correction may occur.

[0231] If the reference result of the forward trajectory behavior is consistent with the predicted result of the forward trajectory behavior, there is no need to correct the predicted result of the forward trajectory behavior.

[0232] Optionally, the first threshold is less than the second threshold. For example, the first threshold is 0.8 and the second threshold is 0.9.

[0233] The first sample refers to the sample in the reference result of the forward trajectory behavior, and the second sample refers to the sample corresponding to the first sample in the predicted result of the forward trajectory behavior. The first sample and the second sample are only used to distinguish whether the sample is from the reference result of the forward trajectory behavior or the initial predicted result, and are not limited to the first sample or the second sample.

[0234] In this embodiment, the correction of the sample refers to the correction of the predicted result corresponding to the sample. For example, when the pre-behavior of the first sample is taking the escalator, the correction of the first sample means modifying the predicted result to taking the elevator or flat floor still. Or, the modification of the sample can also be the modification of the sample data. After the sample data is modified, the corresponding output result is the corrected predicted result.

[0235] In this embodiment, the forward trajectory behavior prediction result is corrected by using the forward trajectory behavior reference result. Since the forward trajectory behavior reference result is identified by an identification method with relatively accurate escalator identification results, it has a very positive effect during the correction process. For example, when the forward trajectory behavior reference result is walking, samples with a prediction probability (confidence level) less than 0.9 in the forward trajectory behavior prediction result will be modified; if the forward trajectory behavior prediction result is stationary or walking, and the forward trajectory behavior reference result is an escalator, the forward trajectory behavior prediction result will be modified; if the forward trajectory behavior prediction result is an elevator, and the forward trajectory behavior reference result is an escalator, the prediction result with a prediction probability less than 0.8 in the forward trajectory behavior prediction result will be modified; if the forward trajectory behavior prediction result is an escalator, and the forward trajectory behavior reference result is a flat floor, the prediction result with a prediction probability less than 0.9 in the forward trajectory behavior prediction result will be modified.

[0236] In a possible implementation manner, a method for predicting the forward trajectory behavior reference result and the reverse trajectory behavior reference result is provided, and this identification method has a relatively high accuracy for escalator identification results. The following specifically describes the prediction method, that is, the prediction rule for taking an escalator set in the embodiments of the present application.

[0237] Please refer to Figure 8 , the step of performing behavior prediction based on the environmental data and the communication data to obtain the forward trajectory behavior reference result and the reverse trajectory behavior reference result at the preset time includes:

[0238] Step S801: Determine the number of steps corresponding to each sample according to the acceleration information of each sample;

[0239] Step S802: Determine whether the number of steps of the sample is greater than zero;

[0240] If so, that is, if the number of steps is greater than zero, execute Step S803: Determine that the behavior of the sample is walking and predict that the sample is walking;

[0241] If not, that is, if the number of steps is equal to zero, execute Step S804: Determine that the behavior of the sample is non-walking, and determine the prediction result of the sample according to at least one of the number of non-walking samples, the geomagnetic information, and the communication data.

[0242] When detecting the human behavior of getting on and off an escalator, the following characteristics are found through the analysis of the data under this behavior: (1) People are mostly in a stationary state when getting on and off an escalator; (2) The geomagnetic change is greater compared to when people are stationary; (3) The change in the WIFI signal strength is greater compared to when people are stationary. Therefore, based on the above characteristics, the prediction process of the forward trajectory behavior reference result in this embodiment is set.

[0243] If the behavior of the sample is not walking, it may be flat-level stillness or taking an escalator. For flat-level stillness, the number of non-walking samples is either very large or very small. When taking an escalator, the stop time of the electronic device on the escalator is determined based on the length of the escalator. Therefore, the number of samples on the escalator is fixed. When going up or down the escalator, the behavior of the sample will change. Therefore, the length of the escalator can be determined by the number of non-walking samples, and then it can be determined whether the user is in flat-level stillness or taking an escalator. Or, for flat-level stillness, the geomagnetic information of the electronic device does not change much, while when taking an escalator, the geomagnetic information of the electronic device fluctuates. Therefore, it can be predicted whether the user is in flat-level stillness or taking an escalator through the geomagnetic information. Or for flat-level stillness, the communication data strength (such as WiFi strength) of the electronic device changes little, while when taking an escalator, the communication data strength of the electronic device changes greatly. Therefore, it can be predicted whether the user is in flat-level stillness or taking an escalator through the communication data strength.

[0244] For example, in a possible implementation manner, the step of determining the prediction result of the sample according to at least one of the number of non-walking samples, the geomagnetic information, and the communication data includes:

[0245] If the number of non-walking samples is outside the preset range, it is determined that the prediction result of the sample is flat-level stillness;

[0246] If the number of non-walking samples is within the preset range, it is determined that the prediction result of the sample is taking an escalator.

[0247] For example, in a possible implementation manner, the step of determining the prediction result of the sample according to at least one of the number of non-walking samples, the geomagnetic information, and the communication data includes:

[0248] Obtain the geomagnetic variance of the sample corresponding to the number of non-walking samples;

[0249] If the geomagnetic variance is greater than the preset variance, predict that the sample is taking an escalator;

[0250] If the geomagnetic variance is less than the preset variance, predict that the sample is flat-level stillness.

[0251] For example, in a possible implementation manner, the step of determining the preset result of the sample according to at least one of the number of non-walking samples, the geomagnetic information, and the communication data includes:

[0252] Obtain the number of samples with large changes in communication data signal strength in the samples corresponding to the number of non-walking samples;

[0253] If the sample size with large variations in the communication data signal strength is greater than or equal to a preset sample size, predict that the sample is escalator-riding data;

[0254] If the sample size with large variations in the communication data signal strength is less than the preset sample size, predict that the sample is flat-floor stationary data.

[0255] In a possible implementation manner, the step of determining the prediction result of the sample according to at least one of the number of non-walking samples, the geomagnetic information, and the communication data includes:

[0256] If the number of non-walking samples is outside a preset range, predict that the sample is flat-floor stationary;

[0257] If the number of non-walking samples is within the preset range, obtain the geomagnetic variance of the sample corresponding to the sample number;

[0258] If the geomagnetic variance is greater than a preset variance, predict that the sample is escalator-riding;

[0259] If the geomagnetic variance is less than the preset variance, predict that the sample is flat-floor stationary.

[0260] In this implementation manner, by combining the number of non-walking samples and the geomagnetic variance to predict the reference result of the forward trajectory behavior, the accuracy of the reference result of the forward trajectory behavior is further improved.

[0261] In a possible implementation manner, the step of determining the prediction result of the sample according to at least one of the number of non-walking samples, the geomagnetic information, and the communication data includes:

[0262] If the number of non-walking samples is outside a preset range, predict that the sample is flat-floor stationary;

[0263] If the number of non-walking samples is within the preset range, obtain the geomagnetic variance of the sample corresponding to the sample number;

[0264] If the geomagnetic variance is less than the preset variance, predict that the sample is flat-floor stationary.

[0265] If the geomagnetic variance is greater than the preset variance, obtain the sample size of the samples with large variations in the communication data signal strength corresponding to the number of non-walking samples;

[0266] If the sample size with large variations in the communication data signal strength is greater than or equal to the preset sample size, predict that the sample is escalator-riding;

[0267] If the sample size with large variations in the communication data signal strength is less than the preset sample size, predict that the sample is flat-floor stationary.

[0268] In this implementation manner, the reference result of the forward trajectory behavior is predicted by combining the number of non-walking samples, the geomagnetic variance, and the change in the WiFi signal strength, further improving the accuracy of the reference result of the forward trajectory behavior.

[0269] Combine Figure 15, in this embodiment, acceleration, geomagnetism, and communication data are mainly used to determine the behavior of each sample in the entire trajectory (i.e., the prediction result). In the first stage of prediction, the acceleration (ACC data) is put into a pedometer for judgment to obtain the number of steps walked within each sample (2.56 s) in the entire trajectory. The underlying layer of this pedometer is implemented using a state machine. By continuously putting the data of the entire trajectory, the number of steps in the current state can be obtained. After obtaining the number of steps, according to the size of the number of steps, if it is greater than 0, it means that the person is in a walking state at this time. If it is 0, a special state needs to be considered. When a person is waving their hand or the mobile phone is in their pocket, the number of steps may not be counted (when the pedometer counts steps, the acceleration needs to exceed the upper threshold and be lower than the lower threshold within a walking cycle, and a person waving their hand and the mobile phone in the pocket will cause the mobile phone to be in a continuously large overweight state). At this time, the acceleration will be very large. By calculating the proportion of the acceleration greater than a certain value, this situation is filtered. When the number of steps is 0, except for the above special situation, it can be judged that the person is in a stationary state at this time, but the stationary state is divided into flat floor stationary and escalator stationary. At this time, the geomagnetic variance and / or WiFi signal strength are combined to identify. Considering that the general length of an escalator is between 3 and 25 samples, so, when judging, if the number of non-walking samples is less than 3 samples or greater than 25 samples, the behavior of the sample data can be directly identified as flat floor stationary. If the number of non-walking samples is between 3 samples and 25 samples, it means that it may be stationary while taking the escalator. To improve the recognition accuracy, it is judged by combining geomagnetism and / or communication data (such as WiFi). Since the geomagnetism is easily disturbed and some abnormal values appear, this embodiment uses geomagnetic filtering. Its main principle is to use the geomagnetic data of the previous and next 50 moments for weighted summation to filter the current moment. Among them, the previous and next 50 moments together account for a weight of 0.5, and the current moment accounts for a weight of 0.5. After filtering, the geomagnetic variance of the samples during the stationary time is calculated. However, considering that when a person is stationary in front of the elevator, a very large geomagnetic fluctuation will occur at a certain moment due to the opening of the elevator door. Therefore, the entire stationary stage (all consecutive non-walking samples) is divided into four segments. When the time of geomagnetic stability is greater than 3 / 4 of the entire stationary stage, the data of this stationary stage is identified as flat floor stationary. If the time of geomagnetic stability is less than 3 / 4 of the entire stationary stage, it is identified by combining communication data. Taking WiFi as an example. During the WIFI recognition process, if the number of WiFi signals with large changes in WiFi strength is large, the behavior of this sample is identified as escalator stationary. Optionally, the process of WiFi strength recognition is applied under the condition that the length of the stationary stage is greater than 10. If it is less than 10, the WIFI judgment will not be performed, and the judgment result of the previous stage will be maintained. In the stationary segment greater than 10, if the average value of the number of the first ten WiFi signals with large changes in WiFi strength is greater than 6, it is considered to be escalator stationary, otherwise it is considered to be flat floor stationary.After all samples are judged, the prediction result of the entire trajectory can be obtained, that is, the forward trajectory behavior reference result.

[0270] By predicting the prediction result of the escalator in the above manner, a forward trajectory behavior reference result with a relatively high escalator recognition accuracy can be obtained. Therefore, by correcting the forward trajectory behavior prediction result with the forward trajectory behavior reference result, the accuracy of the prediction result of the escalator can be improved, and further the recognition accuracy of the escalator can be improved.

[0271] Similarly, the reverse trajectory behavior reference result is obtained by the above method, and then the reverse trajectory behavior prediction result is corrected with the reverse trajectory behavior reference result by the above method.

[0272] Figure 9 It is a flowchart of the behavior recognition method provided by another embodiment of the present application. As Figure 9 described, the method includes:

[0273] Step S901: Obtain the environmental data collected by the sensors of the electronic device, and obtain the communication data of the electronic device. The environmental data includes at least one of acceleration information, geomagnetic information, and angular velocity information;

[0274] Step S902: Extract data features from the environmental data and the communication data to obtain forward trajectory data features and reverse trajectory data features within a preset time;

[0275] Step S903: Respectively perform behavior prediction on the forward trajectory data features and the reverse trajectory data features to obtain a forward trajectory behavior prediction result and a reverse trajectory behavior prediction result;

[0276] Step S904: Fuse the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result to obtain a fusion result;

[0277] Step S905: Determine the sample position of the preset landmark point according to the fusion result;

[0278] Step S906: Adjust the sample position of the preset landmark point, and obtain the behavior recognition result according to the fusion result adjusted by the sample position of the preset landmark point.

[0279] In this embodiment, the preprocessing method of the environmental data and the communication data, the method for obtaining the initial behavior prediction result, and the fusion method of the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result can adopt the above Figure 3 described embodiment.

[0280] In this embodiment, the recognition accuracy of the preset behavior recognition algorithm model is calculated by dividing the number of correctly recognized samples in the entire trajectory by all the samples in the entire trajectory. The recognition accuracy of the landmark recognition (Landmark point) at the start point and the end point of the escalator or elevator is calculated based on whether the start and end of the cross-floor behavior are recognized in the three samples (or five samples) before and after the junction where the flat-floor behavior is converted to the cross-floor behavior. That is to say, when the behavior switches, the preset behavior recognition algorithm model should be able to recognize the landmark point in a timely and accurate manner and determine that the behavior has switched. However, in order to improve the recognition accuracy, in this embodiment, during the data feature extraction, in the feature extraction stage, in order to take into account the semantic features before and after, the data used is not only the data at the current moment, but all the original data within multi-scale sliding windows of sizes 5, 10, and 15. This results in a large amount of original data of the behavior before the switch being included at the switch point of the prediction result, which makes the model unable to promptly reflect the behavior switch during prediction because the behavior represented by the features at this time is more inclined to the behavior before the switch. That is to say, in this embodiment, the recognition accuracy of the preset behavior recognition algorithm model has been improved, but the recognition accuracy of the landmark recognition (Landmark point) at the start point and the end point of the escalator or elevator is not high.

[0281] Based on this, when fusing the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result, the sample position of the prediction result predicted as the preset landmark point is adjusted to expand the position of the preset landmark point forward or backward to solve the delay problem.

[0282] In this embodiment, the preset landmark point includes the start point of taking the escalator or elevator and the end point of taking the escalator or elevator.

[0283] In a possible implementation manner, the method for adjusting the sample position of the preset landmark point can be: according to the experimental test results, after determining the sample position of the preset landmark point based on the fused prediction result, directly modify the landmark point to a preset number of samples forward or backward from the sample position, such as 3 to 5. Optionally, modify the start point of taking the escalator or elevator to 3 samples forward from the sample position, and modify the end point of taking the escalator or elevator to 2 samples backward from the sample position, where the front and back of the sample position are determined based on the time sequence. In this way, the problem of delayed landmark point recognition can be solved.

[0284] In a possible implementation manner, the method for adjusting the sample position of the preset landmark point can also be: determining the number of steps corresponding to each sample according to the acceleration information of each sample; then determining the turning sample position where the number of steps switches between zero and non-zero according to the number of steps of each sample; aligning the sample position of the preset landmark point in the fusion result with the turning sample position.

[0285] In this implementation manner, the sample positions of the preset landmark points of the fused prediction result can be directly aligned with the turning sample position. Or, according to the turning sample position, the sample positions of the preset landmark points of the fused prediction result are appropriately moved forward or backward by a preset number of samples, where the number of samples is determined according to the difference between the turning sample position and the sample position of the preset landmark point, and the number of samples is less than 3. For example, if the sample position of the preset landmark point is 5 samples different from the turning sample position, the sample position of the preset landmark point of the fused prediction result is moved forward or backward by 1 - 3 samples; if the difference is 2, the sample position of the preset landmark point of the fused prediction result is moved forward or backward by 2 samples.

[0286] Among them, the turning sample position refers to the position of the sample when it does not switch from zero to greater than zero, or from greater than zero to zero. Before a user gets on an escalator, the user walks first and stops at the escalator before getting on it. Therefore, when getting on the escalator, there is a process of the number of steps switching from greater than zero to zero; before getting off the escalator, the user stops on the escalator and needs to walk after getting off the escalator. Therefore, when getting off the escalator, there is a process of the number of steps switching from zero to greater than zero. Therefore, the turning sample position can be accurately predicted through the number of steps, and then based on this turning sample position, the position of the landmark point of the fused prediction result is adjusted to improve the recognition accuracy of the landmark point and solve the problem of landmark point delay.

[0287] In a possible implementation manner, in combination with Figure 16 , after the step of determining the sample position of the preset landmark point according to the fusion result, the following steps are further included:

[0288] Determine the number of samples between two adjacent landmark points according to the sample position of the preset landmark point;

[0289] When the number of samples is greater than the preset number, modify the behavior corresponding to the samples between the landmark points to a flat floor;

[0290] When the number of samples is less than the preset number, execute the step of adjusting the sample position of the preset landmark point.

[0291] Among them, some special situations are also judged. If the number of samples between the start and end of the escalator exceeds 25, it is corrected to a flat floor. If the number of samples between the start point and the end point does not exceed 25, it indicates a cross - floor behavior. During the cross - floor behavior, it includes the start point of the cross - floor behavior and the end point of the cross - floor behavior. The start point of the cross - floor behavior is the start point of the escalator or elevator, and the end point of the cross - floor behavior is the end point of the escalator or elevator. The position of the preset landmark point is adjusted based on the above method.

[0292] After identifying the preset landmark points, since it is impossible to determine whether the identified landmark points are the starting or ending points of the escalator or elevator, therefore, the samples between two landmark points may be samples of flat floor stillness or escalator samples. Therefore, to avoid misidentification, a preset quantity is set based on the length of the escalator. If the number of samples between two landmark points is greater than the preset quantity, it indicates that the behavior corresponding to the samples between the two landmark points is flat floor stillness rather than taking the escalator. If taking the escalator, after determining the starting and ending points of the escalator, the positions of the preset landmark points are adjusted, thereby improving the accuracy of the adjustment.

[0293] In this embodiment, in terms of Landmark point recognition, the average delay time between the start time and end time of all trajectory predicted Landmark points and the true Landmark points can be controlled within 4s. The recognition accuracy rate of all trajectory predicted Landmark points (the proportion of Landmark points with recognition error ≤ 3 samples in the predicted Landmark points) can reach about 90%, and the recognition accuracy rate of all trajectory predicted Landmark points (the proportion of Landmark points with recognition error ≤ 5 samples in the predicted Landmark points) can reach about 95%.

[0294] Figure 10 It is a flowchart of the behavior recognition method provided by another embodiment of this application, as Figure 10 described, the method includes:

[0295] Step S1001: Obtain the environmental data collected by the sensors of the electronic device, and obtain the communication data of the electronic device. The environmental data includes at least one of acceleration information, geomagnetic information, and angular velocity information;

[0296] Step S1002: Extract data features from the environmental data and the communication data to obtain positive trajectory data features and negative trajectory data features within a preset time;

[0297] Step S1003: Perform behavior prediction on the positive trajectory data features and the negative trajectory data features respectively to obtain a positive trajectory behavior prediction result and a negative trajectory behavior prediction result;

[0298] Step S1004: Fuse the positive trajectory behavior prediction result and the negative trajectory behavior prediction result to obtain a fusion result;

[0299] Step S1005: Predict a second behavior reference result of the trajectory within the preset time according to the acceleration information;

[0300] Step S1006: Correct the fusion result based on the second behavior reference result to obtain a behavior recognition result.

[0301] In this embodiment, the feature extraction methods for environmental data and communication data, the prediction methods for the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result, and the fusion method for the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result may adopt the above Figure 3 described embodiments and will not be repeatedly defined herein.

[0302] In this embodiment, in order to further improve the accuracy of the behavior recognition result, after fusing the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result, the fused prediction result is modified to improve the accuracy of the output target behavior.

[0303] Optionally, among the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result output by the preset behavior recognition algorithm model, the prediction result of taking an elevator or escalator has relatively low accuracy. Therefore, by obtaining a second behavior reference result, the prediction result of the elevator in the prediction results is corrected to improve the accuracy of the elevator prediction result. Since the environmental data features are obvious and easy to identify when taking an elevator, in order to reduce the amount of data correction processing, the correction of the prediction result of taking an elevator is set, that is, only the fused result after fusion needs to be corrected for the elevator. In some alternative embodiments, the correction of the prediction result of taking an escalator can also be performed after the forward and reverse trajectory behavior prediction results are fused.

[0304] In this embodiment, the principle of correcting the fused result is to identify the user's behavior through a method with higher accuracy in identifying a certain prediction result, and then compare this prediction result with the fused result, and modify the fused result with the prediction result with higher accuracy, so that the accuracy of the corrected behavior recognition result is higher, making up for the defects of the fused result. For example, the following listed prediction rules for taking an elevator.

[0305] The embodiment of the present application provides a prediction method for a second behavior reference result, and this prediction method has high accuracy in identifying an elevator. The following specifically describes the prediction method, that is, the prediction rules for taking an elevator set in the embodiment of the present application.

[0306] Combined with Figure 18 , the step of predicting the second behavior reference result of the trajectory within the preset time according to the acceleration information includes:

[0307] Determine the continuous overweight and weightlessness ratio corresponding to each sample according to the acceleration information of each sample;

[0308] If there is a target sample with a continuous proportion of overweight and weightlessness greater than a preset value, and the number of target samples continuously exceeds a preset number, then determine that the prediction result (or behavior) of the target sample is taking the elevator;

[0309] If there is no target sample with a continuous proportion of overweight and weightlessness greater than a preset value, or the number of target samples continuously exceeds a preset number, then determine that the prediction result (or behavior) of the target sample is leveling.

[0310] When a person is walking, there is basically no small - amplitude overweight and weightlessness exceeding one continuous data frame. Since the periodicity of human walking is relatively obvious, and when a person is in an elevator, in fact, most of the time it is a static state, and the person shows a small - amplitude overweight (weightlessness) - uniform motion - small - amplitude weightlessness (overweight) motion state following the operation of the elevator. This special state is very difficult to appear in other daily behaviors of people, especially maintaining continuous small - amplitude overweight and weightlessness for 2 - 4 s. Therefore, by extracting the proportion of overweight and weightlessness of different amplitudes, it can accurately identify whether a person is on the elevator. That is to say, in the second behavior reference result predicted by this implementation method, the prediction result regarding the elevator is relatively accurate. Therefore, when using the second behavior reference result to correct the fusion result, the accuracy of the result regarding taking the elevator in the fusion result can be improved, thereby enhancing the accuracy of behavior recognition.

[0311] In this embodiment, the proportion of continuous small - amplitude overweight and weightlessness in each data frame is calculated through acceleration. When calculating the small - amplitude proportion, the selection of the small - amplitude overweight and weightlessness threshold is crucial. The upper and lower bounds of the continuous overweight and weightlessness of the elevator refer to the upper and lower bounds of overweight and the upper and lower bounds of weightlessness during the elevator startup process, which need to be separated from behaviors such as standing still and walking. In the experiment, it is found that compared with standing still, the overweight (weightlessness) of the elevator is slightly larger (smaller), but compared with walking, etc., it is slightly smaller (larger), and it is in the middle of the two. Although it is known that it is in the middle currently, how to determine this region to separate it from standing still, walking, etc. is the key. Through a large amount of data statistics and analysis, it is found that when the lower bound of overweight is 10.05, the upper bound of overweight is 11.3, the lower bound of weightlessness is 8.9, and the upper bound of weightlessness is 9.5, it is a relatively reasonable threshold.

[0312] After calculating the overweight and weightlessness of each data frame, judge the overweight and weightlessness in the entire trajectory based on whether there are two consecutive data frames with a proportion of overweight and weightlessness exceeding 2.7, and find the paired overweight and weightlessness. When the overweight and weightlessness in the entire trajectory are not paired or there are two consecutive overweight (weightlessness) occurrences, it is determined that the person is not in the elevator. If no overweight and weightlessness are detected, the trajectory is considered a leveling trajectory. If there is paired overweight and weightlessness, assign the samples between the paired overweight and weightlessness as elevator, and the others as leveling, and then return the result.

[0313] Combined with Figure 17, in a possible implementation manner, the step of correcting the fusion result based on the second behavior as a reference result includes:

[0314] Compare each sample in the second behavior as a reference result and the fusion result. The fusion result is the result obtained after the main model prediction result is fused in both forward and reverse directions.

[0315] If the prediction result of the first sample in the second behavior as a reference result is flat or elevator, modify the second sample corresponding to the first sample in the fusion result according to the second behavior as a reference result;

[0316] If the prediction result of the first sample in the second behavior as a reference result is false (output false), and the prediction result of the second sample corresponding to the first sample in the fusion result is flat, correct the second sample according to the forward trajectory behavior prediction result and / or the reverse trajectory behavior prediction result.

[0317] In this embodiment, a sample refers to a record of a preset length in the prediction result. For example, the prediction result within 2.56 s is one sample. The way of sample comparison is one-to-one comparison. Since the second behavior as a reference result and the fusion result are predicted from the environmental data and communication data corresponding to the same trajectory, the samples of the second behavior as a reference result and the fusion result are compared one-to-one based on time sequence.

[0318] Since the prediction accuracy of the prediction result regarding taking the elevator in the above second behavior as a reference result is high, if the prediction result of the first sample in the second behavior as a reference result is flat or elevator, directly modify the second sample corresponding to the first sample in the fusion result according to the second behavior as a reference result, and modify the prediction result of the second sample to flat or elevator, so as to correct the prediction result regarding taking the elevator in the fusion result.

[0319] In a possible implementation, if the prediction result of the second sample corresponding to the first sample in the fusion result is flat, and the prediction result of the first sample with the second behavior as the reference result is incorrect, the second sample cannot be corrected using the second behavior as the reference result. At this time, to avoid the recognition error caused by deleting the prediction result of the elevator or escalator during the fusion process of the fusion result, the forward trajectory behavior prediction result and / or the reverse trajectory behavior prediction result are used to correct the second sample. For example, if the forward trajectory behavior prediction result is: 00000011100, the reverse trajectory behavior prediction result is 00111000000, and the prediction result after fusing the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result is: 00000000000. Since in the above-mentioned fusion process of the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result, the prediction result of the escalator or elevator may be modified to flat and stationary due to the method of extracting the intersection sample result, in this embodiment, the forward trajectory behavior prediction result and / or the reverse trajectory behavior prediction result are used to correct the fusion result, which can avoid the prediction result of the escalator or elevator being modified to flat and stationary.

[0320] In a possible implementation, since the second behavior as the reference result does not set the samples where the elevator stops and waits on each floor as the elevator, before step S1006, the second behavior as the reference result is corrected. Optionally, the step of correcting the second behavior as the reference result includes: obtaining the elevator samples in the second behavior as the reference result whose prediction result is the elevator, and modifying the other samples between the preset number of elevator samples as elevator samples. That is, the part where the elevator stops and waits between multiple pairs of overweight and weightlessness is set as the elevator, so that the second behavior as the reference result is more accurate.

[0321] Figure 11 The flowchart of the behavior recognition method provided by yet another embodiment of the present application is as Figure 11 shown, and the method includes:

[0322] Step S1101: Obtain the environmental data collected by the sensors of the electronic device, and obtain the communication data of the electronic device. The environmental data includes at least one of acceleration information, geomagnetic information, and angular velocity information;

[0323] Step S1102: Extract data features from the environmental data and the communication data to obtain forward trajectory data features and reverse trajectory data features within a preset time;

[0324] Step S1103: Perform behavior prediction on the forward trajectory data features and the reverse trajectory data features respectively to obtain a forward trajectory behavior prediction result and a reverse trajectory behavior prediction result;

[0325] Step S1104: Perform behavior prediction based on the environmental data and the communication data to obtain the forward trajectory behavior reference result and the reverse trajectory behavior reference result at the preset time;

[0326] Step S1105: Correct the forward trajectory behavior prediction result based on the forward trajectory behavior reference result, and correct the reverse trajectory behavior prediction result based on the reverse trajectory behavior reference result;

[0327] Step S1106: Fuse the corrected forward trajectory behavior prediction result and the corrected reverse behavior prediction result to obtain a fusion result.

[0328] Step S1107: Determine the sample position of the preset landmark point according to the fusion result;

[0329] Step S1108: Adjust the sample position of the preset landmark point, and obtain an adjusted result according to the fusion result adjusted by the sample position of the preset landmark point;

[0330] Step S1109: Predict the second behavior reference result of the trajectory within the preset time according to the acceleration information;

[0331] Step S1110: Correct the adjusted result based on the second behavior reference result to obtain a behavior recognition result.

[0332] The behavior recognition method in this embodiment will be described in combination with modularizing the behavior recognition method. For example, in combination with Figure 13, first, environmental data and communication data (such as WiFi, acceleration information, angular velocity information, geomagnetic information, etc.) are input into a feature extraction module (feature engineering) to perform effective feature extraction. The effective features are those that can identify user behaviors. Then, the forward trajectory features and the reverse trajectory features are respectively input into a preset behavior algorithm model (i.e., the XGboost model), and the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result are respectively obtained based on the preset behavior recognition algorithm model. Then, a preset prediction rule for taking an escalator (i.e., the escalator rule module) is used to predict the environmental data and communication data, and the forward trajectory behavior reference result and the reverse trajectory behavior reference result are obtained. Among them, the prediction rule of the escalator combines a pedometer module to predict the forward trajectory behavior reference result and the reverse trajectory behavior reference result; then, the forward trajectory behavior reference result is used to correct the forward trajectory behavior prediction result, and the reverse trajectory behavior reference result is used to correct the reverse trajectory behavior prediction result, so that the escalator rule module corrects the prediction result output by XGboost. Then, a positive and negative fusion rule module is used to fuse the corrected forward trajectory behavior prediction result and the reverse trajectory behavior prediction result to obtain a fusion result; an escalator landmark adjustment module is used to adjust the landmark points of the fusion result to correct the predicted landmark point positions, and the result after correcting the landmark points is used as the adjusted result. Then, a preset prediction rule for taking an elevator (i.e., the elevator rule module) is used to predict the environmental data and communication data to obtain a second behavior reference result, and the second behavior reference result is used to correct the adjusted result to obtain a behavior recognition result and obtain a trajectory prediction result.

[0333] In this embodiment, after using an algorithm model to predict the user's behavior, the prediction result of the algorithm model is modified multiple times, so that the accuracy of the output behavior recognition result is higher.

[0334] Optionally, in this embodiment, in the prediction of the algorithm model, the correction of the prediction result of the algorithm model, the fusion of the positive and negative prediction results, the elevator correction of the fused prediction result, and the landmark correction, etc., each process improves the recognition accuracy of the user's behavior. Thus, the behavior recognition result obtained through the prediction of the algorithm model, the correction of the prediction result of the algorithm model, the fusion of the positive and negative prediction results, the elevator correction of the fused prediction result, and the landmark correction has a higher accuracy.

[0335] Optionally, in this embodiment, the method of using the forward trajectory behavior reference result to correct the forward trajectory behavior prediction result and the reverse trajectory behavior reference result to correct the reverse trajectory behavior prediction result refers to the above Figures 6 to 8 shown embodiment. The method of fusing the corrected forward trajectory behavior prediction result and the reverse trajectory behavior prediction result refers to the above Figures 3 to 5The embodiments shown. The method of adjusting landmark points for the fusion result refers to the above Figure 9 The embodiments shown. The method of correcting the predicted result after adjusting the sample position of the landmark points by using the second row as the reference result refers to the above Figure 10 The embodiments shown.

[0336] In a possible implementation manner, in the above embodiments, the feature extraction process of the environmental data and communication data can be performed in the following manner to make the recognition result of the preset behavior recognition algorithm model more accurate based on these features.

[0337] In terms of sensor data acquisition:

[0338] During the initial data acquisition, the sampling time window is 2.56 seconds. On the one hand, an overly large time window may affect the recognition accuracy; however, if the recognition window is too small, recognition needs to be performed frequently, increasing the computational load. On the other hand, using 2.56 seconds as the time window is considered because when performing frequency-domain calculations, Fourier transform is required, and using 2 to the Nth power as the time window ensures the integrity of the data participating in the frequency-domain calculation.

[0339] Regarding the sampling time window set by the data collector, the concept of a data frame is proposed. For each type of sensor, a sliding window with 50% data overlap is used to store 2.56 seconds of sensor data, which is called a data frame.

[0340] The sampling frequency of the collector is 100HZ, so each data frame (which can also be called a sample during model training and testing) contains approximately 256 moments of data.

[0341] This embodiment uses different sensor data, either alone or in combination, to extract corresponding features, namely statistical features, time-domain features, and behavioral semantic features.

[0342] The statistical features are respectively applied to calculate the horizontal acceleration, vertical acceleration, acceleration vector, gyroscope, and geomagnetism, obtaining a total of 50-dimensional features. The statistical features reflect the mutual connection between the overall data and consider the data changes during the entire time period from an individual perspective to an overall perspective.

[0343] The time-domain features include first-order integral, second-order integral, and the acceleration Y-Z correlation coefficient. First-order and second-order integrals are calculated for acceleration and gyroscope (angular velocity) data respectively. Semantically, it can be roughly understood as calculating velocity and displacement. According to different human behaviors, such as when a person is stationary, the velocity and displacement are significantly smaller than those when a person is walking. This can help the model make a rough classification of human behaviors. The acceleration Y-Z correlation coefficient refers to the correlation coefficient between the Y-axis and Z-axis of acceleration. When the mobile phone is in a flat-held state, due to the different periodicities presented by a person during walking and going up / down stairs, and different step amplitudes, the correlation coefficient between the Y-axis and Z-axis is also different.

[0344] The time-domain features in total include 5D features: the first-order integral of the sum of acceleration vectors, the second-order integral of the sum of acceleration vectors, the first-order integral of the sum of gyroscope (angular velocity) vectors, the second-order integral of the sum of gyroscope (angular velocity) vectors, and the correlation coefficient between the Y-axis and Z-axis of the sum of acceleration vectors.

[0345] Based on the differences shown by various behaviors in terms of details and data, the following behavioral semantic features are extracted: the proportion of small-amplitude overweight, the proportion of large-amplitude overweight, the proportion of small-amplitude weightlessness, the proportion of large-amplitude weightlessness, the proportion of continuous small-amplitude overweight, the proportion of continuous small-amplitude weightlessness, the maximum turn of a single sample, the maximum turn of a double sample, the average value of geomagnetic fluctuations, the projection of acceleration in the direction of gravity, the WIFI change ratio, etc. The behavioral semantic features in total include 11D features.

[0346] The proportion of small-amplitude overweight, the proportion of large-amplitude overweight, the proportion of small-amplitude weightlessness, the proportion of large-amplitude weightlessness, the proportion of continuous small-amplitude overweight, and the proportion of continuous small-amplitude weightlessness are features captured when analyzing walking and elevator data. When a person is walking, there is basically no overweight or weightlessness exceeding one continuous data frame, and the periodicity of human walking is relatively obvious. When a person is in an elevator, in fact, most of the time it is a stationary state, and the person shows a small-amplitude overweight (weightlessness) - uniform motion - small-amplitude weightlessness (overweight) motion state following the operation of the elevator. This special state is very difficult to appear in other daily human behaviors, especially maintaining continuous small-amplitude overweight / weightlessness for 2 - 4s. Therefore, extracting the proportion of overweight / weightlessness with different amplitudes can accurately identify whether a person is on the elevator. The difference between the proportion of small-amplitude overweight (weightlessness) and the proportion of continuous small-amplitude overweight (weightlessness) is that the proportion of small-amplitude overweight (weightlessness) considers whether each moment within a period of time is within the small-amplitude overweight / weightlessness interval, without considering the intervals at the previous and next moments. After obtaining the number of moments in the small-amplitude overweight / weightlessness interval, divide it by the total number of moments in this period to get the proportion. The proportion of continuous small-amplitude overweight / weightlessness focuses on the number of moments in the longest continuous small-amplitude overweight / weightlessness range within this period, and then divides this number of moments by the total number of moments to get the proportion of continuous small-amplitude overweight / weightlessness.

[0347] The average feature of geomagnetic fluctuations is also the change of the geomagnetic field. For example, in the trajectory of an escalator, the geomagnetic field generally shows a curve that continuously rises or falls. The elevator will generate intense geomagnetic fluctuations when opening or closing the elevator door. Therefore, the fluctuation of the geomagnetic field can reflect the behavior of the human body to a certain extent. The calculation formula of geomagnetic fluctuation is as follows:

[0348] ; ;

[0349] MagSlice represents the data in the time slice from t to (t + windowSize) in the geomagnetic data. ;

[0350] When a person makes a cross - floor movement, in the vertical direction, the acceleration will change, and the position and attitude of the electronic device are uncertain. The projection in the Z - axis direction in the carrier coordinate system of the electronic device cannot be used. Therefore, the projection feature of the acceleration in the gravity direction is calculated. Among them, the environmental data includes the data collected by the nine - axis sensor. Therefore, the gravitational acceleration is obtained by calculating the average value of the acceleration within a sample (2.56 s). The average value within the small window plays a role similar to a simple low - pass filter, which can cancel out the up - and - down swing when a person walks. Especially when a person is stationary, it can achieve a better effect. After obtaining the gravitational acceleration, the projection of the acceleration in the gravity direction can be obtained by using the method of vector projection.

[0351] Regarding the communication data ratio feature, taking WiFi data as an example, the WiFi change ratio feature is also the change of WiFi. The WiFi change ratio actually refers to the ratio of the number of newly appeared WiFi networks at the last moment in the WiFi data to the total number of WiFi networks at the last moment. Among them, the number of newly appeared WiFi networks at the last moment is obtained by comparing with the WiFi networks at the first moment. First, the intersection of the WiFi hardware addresses at the last moment and the first moment is obtained, and then the difference between the WiFi hardware address at the last moment and the intersection is calculated to get the number of newly appeared WiFi networks. Finally, dividing by the total number of WiFi networks at the last moment gives the WiFi change ratio.

[0352] In terms of feature extraction under multi - scale sliding windows:

[0353] Sliding windows of varying sizes are used to extract temporal features for different behaviors and scenarios. When features extracted from a single data frame were fed into the model for training and testing, both accuracy and prediction performance were poor. To improve model accuracy, extensive experimental observations and research revealed that samples extracted from a single data frame do not accurately reflect the semantic features of the preceding and following moments. For example, during elevator travel, there are periods of constant speed. Extracting features from a single data frame reveals that the features at this moment are not significantly different from those of a stationary plane. This is because the data within the window captures too little information, leading to inaccurate model predictions. Based on the principle that the window should encompass the entire cross-layer trajectory features as much as possible while avoiding excessive data from different behaviors, experiments and data observations have led to the use of multi-scale sliding windows of sizes 5, 10, and 15.

[0354] Figure 12 The schematic diagram of the structure of a behavior recognition device provided by the present application is shown. The behavior recognition device 1200 provided by the present application includes: an acquisition module 1201, a feature extraction module 1202, a prediction module 1203 and a fusion module 1204;

[0355] In one possible implementation, the acquisition module 1201 is configured to acquire environmental data collected by a sensor of an electronic device and acquire communication data of the electronic device, where the environmental data includes at least one of acceleration information, geomagnetic information, and angular velocity information;

[0356] The feature extraction module 1202 is used to extract data features from the environmental data and the communication data to obtain forward trajectory data features and reverse trajectory data features within a preset time;

[0357] The prediction module 1203 is configured to perform behavior prediction on the forward trajectory data features and the reverse trajectory data features, respectively, to obtain a forward trajectory behavior prediction result and a reverse trajectory behavior prediction result;

[0358] The fusion module 1204 is configured to fuse the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result to obtain a behavior recognition result.

[0359] It should be understood that the electronic device here is embodied in the form of functional modules. The term "module" here can be implemented in the form of software and / or hardware, and no specific limitation is made thereto. For example, a "module" can be a software program, a hardware circuit, or a combination of the two that implements the above functions. The hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (such as a shared processor, a proprietary processor, or a group of processors, etc.) for executing one or more software or firmware programs, a memory, a combined logic circuit, and / or other suitable components that support the described functions.

[0360] This application also provides an electronic device, including: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the electronic device, cause the electronic device to execute the behavior recognition method described in any one of the above first aspect or possible implementation manners of the first aspect.

[0361] This application also provides a computer-readable storage medium in which a computer program is stored. When the computer program is executed by a processor, the processor is caused to execute the behavior recognition method described in any one of the above first aspect or possible implementation manners of the first aspect.

[0362] This application also provides a chip, the chip includes a processor and a data interface, and the processor reads instructions stored on a memory through the data interface and executes the behavior recognition method described in any one of the first aspect or possible implementation manners of the first aspect.

[0363] Optionally, the chip may further include a memory, instructions are stored in the memory, and the processor is configured to execute the instructions stored on the memory. When the instructions are executed, the processor is configured to execute the behavior recognition method described in any one of the first aspect or possible implementation manners of the first aspect.

[0364] The memory can be a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), or other types of dynamic storage devices that can store information and instructions. It can also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM), or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices. Or it can also be any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, etc.

[0365] In the embodiments of the present application, "and / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent the cases of A existing alone, A and B existing simultaneously, and B existing alone. Where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the preceding and following associated objects. "At least one of the following" and its similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c can represent: a, b, c, a - b, a - c, b - c, or a - b - c, where a, b, and c can be single or multiple.

[0366] Those of ordinary skill in the art can realize that the various units and algorithm steps described in the embodiments disclosed herein can be implemented by a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0367] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0368] In several embodiments provided by the present application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or a part of this technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs.

[0369] As described above, the foregoing is only the specific implementation manner of the present application. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, and all such changes or substitutions should be covered by the protection scope of the present application. The protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A behavior recognition method, characterized in that, The method includes: Obtaining environmental data collected by a sensor of an electronic device, and obtaining communication data of the electronic device, where the environmental data includes at least one of acceleration information, geomagnetic information, and angular velocity information; Performing data feature extraction on the environmental data and the communication data to obtain forward trajectory data features and reverse trajectory data features within a preset time; Performing behavior prediction on the forward trajectory data features and the reverse trajectory data features respectively to obtain a forward trajectory behavior prediction result and a reverse trajectory behavior prediction result; Fusing the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result to obtain a behavior recognition result.

2. The behavior recognition method according to claim 1, characterized in that The step of fusing the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result to obtain a behavior recognition result includes: Obtaining intersection samples in the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result; Determining other samples except the intersection samples; Modifying the other samples according to samples whose prediction results are flat floor still or walking; Obtaining a behavior recognition result based on the modified other samples and the intersection samples.

3. The behavior recognition method according to claim 2, wherein The step of obtaining a behavior recognition result based on the modified other samples and the intersection samples includes: Performing abnormal sample filtering processing on the modified other samples and / or the intersection samples, and obtaining a behavior recognition result with the filtered other samples and the intersection samples.

4. The behavior recognition method according to claim 1, wherein After the step of performing behavior prediction on the forward trajectory data features and the reverse trajectory data features respectively to obtain a forward trajectory behavior prediction result and a reverse trajectory behavior prediction result, it further includes: Performing behavior prediction based on the environmental data and the communication data to obtain a forward trajectory behavior reference result and a reverse trajectory behavior reference result for the preset time; Correcting the forward trajectory behavior prediction result based on the forward trajectory behavior reference result, and correcting the reverse trajectory behavior prediction result based on the reverse trajectory behavior reference result; The step of fusing the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result to obtain a behavior recognition result includes: Fusing the corrected forward trajectory behavior prediction result and the corrected reverse behavior prediction result to obtain a behavior recognition result.

5. The behavior recognition method according to claim 4, wherein The step of correcting the forward trajectory behavior prediction result based on the forward trajectory behavior reference result includes: Sequentially comparing each sample in the forward trajectory behavior reference result and the forward trajectory behavior prediction result; Correcting the sample corresponding to the forward trajectory behavior prediction result according to the sample of the forward trajectory behavior reference result.

6. The behavior recognition method according to claim 5, wherein The step of correcting the sample corresponding to the forward trajectory behavior prediction result according to the sample of the forward trajectory behavior reference result includes: If the prediction result of the first sample of the forward trajectory behavior reference result is walking, correct the second sample corresponding to the first sample in the forward trajectory behavior prediction result and with a confidence level less than a second threshold, where the forward trajectory behavior prediction result is obtained through a preset behavior recognition algorithm model, and the confidence level is the accuracy of the preset behavior recognition algorithm model in recognizing the second sample; If the prediction result of the first sample of the forward trajectory behavior reference result is not walking, determine the second sample corresponding to the first sample in the forward trajectory behavior prediction result; Correct the forward trajectory behavior prediction result according to the matching degree of the prediction results of the first sample and the second sample.

7. The behavior recognition method according to claim 6, wherein The step of correcting the forward trajectory behavior prediction result according to the matching degree of the prediction results of the first sample and the second sample includes at least one of the following: If the prediction result of the first sample of the forward trajectory behavior reference result is taking an escalator, and the prediction result of the second sample of the forward trajectory behavior prediction result is walking or stationary on a flat floor, correct the second sample with reference to the first sample, where the second sample is the sample corresponding to the first sample in the forward trajectory behavior prediction result; If the prediction result of the first sample of the forward trajectory behavior reference result is taking an escalator, and the prediction result of the second sample of the forward trajectory behavior prediction result is taking an elevator, correct the second sample with a confidence level less than a first threshold with reference to the first sample; If the prediction result of the first sample of the forward trajectory behavior reference result is stationary on a flat floor, and the prediction result of the second sample of the forward trajectory behavior prediction result is taking an escalator, correct the second sample with a confidence level less than a second threshold with reference to the first sample.

8. The behavior recognition method according to claim 4, wherein The step of performing behavior prediction based on the environmental data and the communication data to obtain the forward trajectory behavior reference result and the reverse trajectory behavior reference result at the preset time includes: Determine the number of steps corresponding to each sample according to the acceleration information of each sample; If the number of steps is greater than zero, determine that the behavior of the sample is walking and predict that the sample is walking; If the number of steps is equal to zero, determine that the behavior of the sample is non-walking, and determine the prediction result of the sample according to at least one of the number of non-walking samples, the geomagnetic information, and the communication data.

9. The behavior recognition method according to claim 8, characterized in that, The step of determining the prediction result of the sample according to at least one of the number of non-walking samples, the geomagnetic information, and the communication data includes: If the number of non-walking samples is outside a preset range, predict that the sample is stationary on a flat floor; If the number of non-walking samples is within the preset range, predict that the sample is taking an escalator.

10. The behavior recognition method according to claim 9, characterized in that, The step of if the number of non-walking samples is within the preset range, predicting that the sample is taking an escalator includes: If the number of non-walking samples is within the preset range, obtain the geomagnetic variance of the sample corresponding to the number of samples; If the geomagnetic variance is greater than a preset variance, predict that the sample is taking an escalator; If the geomagnetic variance is less than the preset variance, predict that the sample is stationary on a flat floor.

11. The behavior recognition method according to any one of claims 1 to 10, characterized in that, The steps of fusing the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result to obtain a behavior recognition result include: Fuse the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result to obtain a fusion result; Determine the sample position of a preset landmark point according to the fusion result; Adjust the sample position of the preset landmark point, and obtain a behavior recognition result according to the fusion result after the adjustment of the sample position of the preset landmark point.

12. The behavior recognition method according to claim 11, wherein The step of adjusting the sample position of the preset landmark point includes: Determine the number of steps corresponding to each sample according to the acceleration information of each sample; Determine the turning sample position where the number of steps switches between zero and non-zero according to the number of steps of each sample; Align the sample position of the preset landmark point in the fusion result with the turning sample position.

13. The behavior recognition method according to claim 11, wherein, After the step of determining the sample position of the preset landmark point according to the fusion result, it further includes: Determine the number of samples between two adjacent preset landmark points according to the sample position of the preset landmark point; When the number of samples is greater than a preset number, modify the behavior corresponding to the samples between the preset landmark points to flat floor; When the number of samples is less than the preset number, execute the step of adjusting the sample position of the preset landmark point.

14. The behavior recognition method according to claim 1, characterized in that, The steps of fusing the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result to obtain a behavior recognition result include: Fuse the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result to obtain a fusion result; Predict a second behavior reference result of the trajectory within the preset time according to the acceleration information; Correct the fusion result based on the second behavior reference result to obtain a behavior recognition result.

15. The behavior recognition method according to claim 14, characterized in that, The step of correcting the fusion result based on the second behavior reference result includes: Compare each sample in the second behavior reference result and the fusion result; If the prediction result of the first sample in the second behavior reference result is flat floor or elevator, modify the second sample corresponding to the first sample in the fusion result according to the second behavior reference result; If the prediction result of the first sample in the second behavior reference result is incorrect, and the prediction result of the second sample corresponding to the first sample in the fusion result is flat floor, correct the second sample according to the forward trajectory behavior prediction result and / or the reverse trajectory behavior prediction result.

16. The behavior recognition method according to claim 14, characterized in that, The step of predicting the second behavior reference result of the trajectory within the preset time according to the acceleration information includes: Determine the continuous overweight and weightlessness ratio corresponding to each sample according to the acceleration information of each sample; If there are target samples with a continuous overweight and weightlessness ratio greater than a preset value, and the number of target samples continuously exceeds a preset number, determine that the prediction result of the target samples is taking the elevator; If there are no target samples with a continuous overweight and weightlessness ratio greater than a preset value, or the number of target samples continuously exceeds a preset number, determine that the prediction result of the target samples is flat floor.

17. The behavior recognition method according to claim 1, wherein, Both the forward trajectory data features and the reverse trajectory data features include time domain features, statistical features, and behavior semantic features.

18. The behavior recognition method according to claim 6, wherein The preset behavior recognition algorithm model is trained based on the Extreme Gradient Boosting (XGBoost) algorithm.

19. A behavior recognition device, characterized in that, It includes: an acquisition module, a feature extraction module, a prediction module, and a fusion module; The acquisition module is configured to acquire the environmental data collected by the sensors of the electronic device and acquire the communication data of the electronic device, where the environmental data includes at least one of acceleration information, geomagnetic information, and angular velocity information; The feature extraction module is configured to perform data feature extraction on the environmental data and the communication data to obtain forward trajectory data features and reverse trajectory data features within a preset time; The prediction module is configured to perform behavior prediction on the forward trajectory data features and the reverse trajectory data features respectively to obtain a forward trajectory behavior prediction result and a reverse trajectory behavior prediction result; The fusion module is configured to fuse the forward trajectory behavior prediction result and the reverse trajectory behavior prediction result to obtain a behavior recognition result.

20. An electronic device, characterized in that, It includes: one or more processors; a memory; and one or more computer programs, where the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the electronic device, cause the electronic device to execute the behavior recognition method according to any one of claims 1 to 18.

21. A computer-readable storage medium, characterized in that, A computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the processor is caused to execute the behavior recognition method according to any one of claims 1 to 18.

Images