Behavior recognition method, behavior recognition model generation method, and behavior recognition system

Abstract

The application provides a behavior recognition method, a behavior recognition model generation method and a behavior recognition system, relates to the field of electronic devices, and can accurately recognize behavior patterns by using Wi-Fi data and sensor data of a nine-axis sensor. The method is applied to an electronic device and specifically comprises the following steps: acquiring a plurality of groups of behavior data within a preset time length; performing feature extraction on the plurality of groups of behavior data to acquire a plurality of groups of behavior features; obtaining a target behavior recognition result and a target landmark point recognition result based on the plurality of groups of behavior features and a behavior recognition model; determining a correction recognition result according to acceleration data in each group of behavior data in the plurality of groups of behavior data; correcting the target behavior recognition result according to the correction recognition result; and determining a corrected target landmark point recognition result by using the corrected target behavior recognition result.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of electronic devices, and in particular to a behavior recognition method, a behavior recognition model generation method, and a behavior recognition system. BACKGROUND

[0002] With the increasing power of electronic devices (such as mobile phones), electronic devices are increasingly carried and used by people, which has also made electronic devices gradually develop into a tool for monitoring people's behavior habits anytime and anywhere. The number of sensors integrated on electronic devices is constantly increasing with the update and iteration of electronic devices. How to use the sensors and radio frequency signal information equipped by electronic devices to recognize the current motion state or behavior of the user (i.e., behavior recognition) is currently a research hotspot. People spend more than 80% of their time indoors every day, so the recognition of indoor behavior of users is very important in behavior recognition research.

[0003] Currently, the recognition of indoor behavior mainly uses the data of various sensors of electronic devices to put into a machine learning model for training, so as to obtain a complete model that can predict or judge the current user behavior pattern. However, in current research, most of them need to use all sensor data to accurately judge the user's behavior pattern, so once there is a partial sensor failure or nonexistence in the electronic device, the accuracy of the trained model will decrease, and the prediction or judgment result will not be ideal. SUMMARY

[0004] The embodiments of the present application provide a behavior recognition method, a behavior recognition model generation method, and a behavior recognition system, which can accurately recognize the behavior pattern of a user by using Wi-Fi data and sensor data of a nine-axis sensor.

[0005] To achieve the above object, the embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, the present application provides a behavior recognition method applied to an electronic device. The method comprises: the electronic device obtaining a plurality of groups of behavior data within a preset time length; wherein each group of behavior data corresponds to a unit time length, and any group of behavior data in the plurality of groups of behavior data includes Wi-Fi data of the electronic device and sensor data of a nine-axis sensor in the electronic device within a time period corresponding to the any group of behavior data; the plurality of groups of behavior data includes all Wi-Fi data of the electronic device and all sensor data of the nine-axis sensor in the electronic device within the preset time length; the sensor data includes gyroscope sensor data, magnetic sensor data and acceleration sensor data; the acceleration sensor data includes acceleration data; the electronic device performs feature extraction on the plurality of groups of behavior data to obtain a plurality of groups of behavior features; the behavior features include Wi-Fi features and sensor features; the electronic device obtains a target behavior recognition result and a target landmark point recognition result based on the plurality of groups of behavior features and a behavior recognition model; the target behavior recognition result includes a behavior mode corresponding to each group of behavior features in the plurality of groups of behavior features; the target landmark point recognition result includes a time when each behavior mode changes within the preset time length, or a starting time and an ending time of each behavior mode; the behavior recognition model has the ability to determine a behavior mode by using the Wi-Fi features and the sensor features; the behavior mode corresponding to the behavior features includes a floor-holding state or an elevator up-and-down state; the electronic device determines a corrected recognition result according to the acceleration data in each group of behavior data in the plurality of groups of behavior data; the corrected recognition result includes a behavior mode corresponding to each group of behavior data in the plurality of groups of behavior data; the behavior mode corresponding to the behavior data includes a floor-holding state or an elevator up-and-down state; the electronic device corrects the target behavior recognition result according to the corrected recognition result; and the electronic device determines a corrected target landmark point recognition result by using the corrected target behavior recognition result.

[0007] Based on the above technical solution, in this solution, the electronic device can first use a pre-trained behavior recognition model to identify (or predict) the user's behavior pattern within a preset time period. This behavior recognition model is pre-trained and can determine the user's behavior pattern based on sensor features and Wi-Fi features. Since the behavior recognition model relies on relatively little sensor data when judging behavior patterns, to prevent misjudgment, after the behavior recognition model outputs the behavior recognition result, the electronic device can also determine whether it is in an elevator-like state based on acceleration data from the sensor data. Because there is a significant difference in the user's weightlessness data in non-elevator modes (leveling, escalator / stairs / climbing) and elevator / stairs modes, the electronic device can determine the user's behavior pattern within the preset time period based on the weightlessness status of the electronic device, i.e., correct the recognition result. Then, based on this corrected recognition result, the target behavior recognition result and target landmark point recognition result of the behavior recognition model can be corrected, thereby obtaining a more accurate behavior pattern recognition result.

[0008] In one possible design approach of the first aspect, the electronic device acquires multiple sets of behavioral data within a preset time period, including: the electronic device acquiring a sensor dataset and a Wi-Fi dataset within the preset time period; the sensor dataset includes all sensor data from the nine-axis sensor of the electronic device within the preset time period; the Wi-Fi dataset includes all Wi-Fi data from the electronic device within the preset time period; the electronic device performs data preprocessing on the sensor dataset and the Wi-Fi dataset to obtain a behavioral dataset to be determined; the data preprocessing includes data alignment and data completion; the electronic device samples the behavioral dataset to be determined according to a preset sampling method to obtain multiple sets of behavioral data.

[0009] Based on the above technical solution, all sensor data and Wi-Fi data within a preset time period can be acquired, and multiple sets of behavioral data that need to be used as the basis for behavioral pattern judgment can be obtained using appropriate sampling methods. This allows the behavior recognition process to proceed smoothly.

[0010] In one possible design approach of the first aspect, the preset sampling method includes sliding window sampling; the length of the sliding window in sliding window sampling is a unit time duration, and the sliding step size of sliding window sampling is half of the unit time duration.

[0011] In practice, when a user is in a certain behavioral pattern, there may be a moment when they perform a special action that does not belong to that behavioral pattern, causing anomalies in the sensor data at that moment. For example, when a user is in the behavioral pattern of going up or down an elevator, they may swing their arm, causing a significant change in acceleration data. To minimize the possibility of behavioral features derived from behavioral data corresponding to these special actions being input into the behavior recognition model and leading to inaccurate recognition results, the technical solution provided in this application allows for some identical data in two adjacent sets of behavioral data samples. In this way, even if the behavioral pattern represented by the behavioral features obtained after feature extraction from one set of behavioral data is incorrect, the behavioral pattern represented by the behavioral features obtained after feature extraction from subsequent sets of behavioral data is likely to be correct. Furthermore, after inputting behavioral features obtained from multiple sets of behavioral data into the behavior recognition model, the resulting behavior recognition results and landmark point recognition results will have fewer errors. Based on the above description, the above technical solution discloses that the preset sampling method can be sliding window sampling. For example, the length of the sliding window in the feature parameters of sliding window sampling can be a unit time duration, and the sliding step size of the sliding window can be half of the unit time duration.

[0012] In one possible design of the first aspect, the electronic device determines a correction recognition result based on acceleration data in each of multiple sets of behavioral data, including:

[0013] The electronic device determines a first proportion for each set of behavioral data based on the acceleration data in each set of behavioral data. The first proportion is the percentage of the number of first acceleration data in the target behavioral data out of the total number of acceleration data in the target behavioral data. The equivalent gravitational acceleration of the first acceleration data is within a preset range, and for any first acceleration data in the target behavioral data, there is another first acceleration data that is continuous with any first acceleration data. The target behavioral data is the behavioral data to which the first proportion belongs. The preset range includes a preset overweight range and a preset weightlessness range.

[0014] If multiple sets of behavioral data show that the proportion of the first set of first behavioral data is greater than the preset proportion, the multiple sets of first behavioral data are identified as first weightlessness behavioral data. If the equivalent gravitational acceleration of the first acceleration data in the first weightlessness behavioral data is within the preset weightlessness range, the first weightlessness behavioral data is identified as first weightlessness behavioral data.

[0015] If multiple sets of behavioral data show that the proportion of the first set of first behavioral data is greater than the preset proportion, the multiple sets of first behavioral data are identified as first hyperweightlessness behavioral data. If the equivalent gravitational acceleration of the first acceleration data in the first hyperweightlessness behavioral data is within the preset weightlessness range, the first hyperweightlessness behavioral data is identified as first weightlessness behavioral data.

[0016] The electronic device determines the behavior pattern corresponding to each set of behavior data in multiple sets of behavior data based on the first overweight behavior data and the first weightlessness behavior data;

[0017] The electronic device determines the behavioral pattern corresponding to each set of behavioral data from multiple sets of behavioral data as the corrected recognition result.

[0018] Based on the above technical solution, the electronic device can first determine the hypergravity / weightlessness situation within a preset time period based on acceleration data (the time period to which the first hypergravity behavior data belongs is the hypergravity situation, and the time period to which the first weightlessness behavior data belongs is the weightlessness situation), and then determine the behavior pattern corresponding to each set of behavior data within the preset time period. This provides data support for the correction of subsequent target behavior recognition results and target landmark point recognition results.

[0019] In one possible design of the first aspect, after the electronic device determines the first proportion of each group of behavioral data based on the acceleration data in each group of behavioral data, the method further includes: when there is no consecutive first proportion of multiple groups of first behavioral data that is greater than a preset proportion, the electronic device determines the behavioral pattern corresponding to each group of behavioral data in the multiple groups of behavioral data as a flat state; the electronic device determines the behavioral pattern corresponding to each group of behavioral data in the multiple groups of behavioral data as the corrected recognition result.

[0020] The significance of the above technical solution lies in the following: when the first proportion of a certain set of behavioral data is greater than a preset proportion, it can be assumed that within a unit of time corresponding to this set of behavioral data, the user's equivalent gravitational acceleration within a certain continuous time period conforms to the characteristics corresponding to the behavior pattern of going up and down elevators. In other words, this set of behavioral data can be considered to belong to the behavioral data of the behavior pattern of going up and down elevators with a high probability, and the user's behavior pattern within a unit of time corresponding to this set of behavioral data can also be considered to belong to the behavior pattern of going up and down elevators with a high probability. Furthermore, it is understandable that the duration of the acceleration or deceleration process during one elevator operation is 2-4 seconds, and the unit of time corresponding to a set of sample data falls within this range. However, if the user performs actions that affect their equivalent gravitational acceleration (such as raising their arm, walking, etc.) within a unit of time in the leveling state behavior pattern, the electronic device will use the first proportion of the behavioral data within that unit of time to determine that the behavior pattern corresponding to the behavioral data is going up and down elevators. This determination result is subject to bias. Therefore, in order to more accurately determine the sample data corresponding to elevator acceleration or deceleration in the behavior pattern of going up and down elevators, it is only when the first proportion of multiple consecutive sets of first sample data is greater than a preset proportion that the user's behavior pattern within the time period corresponding to the multiple sets of first sample data can be determined with greater probability to be going up and down elevators. In other words, if the electronic device determines that there are no multiple consecutive sets of first behavioral data in which the first proportion is greater than a preset proportion, the user's behavior pattern within the time period (i.e., the preset duration) corresponding to all sets of behavioral data can be determined to be a leveling state.

[0021] In one possible design approach of the first aspect, the electronic device determines a behavioral pattern corresponding to each set of behavioral data in a plurality of sets of behavioral data based on first hypergravity behavior data and first weightlessness behavior data, including:

[0022] If there is a pair of second overweight behavior data and second weightlessness behavior data, the electronic device will determine the behavior pattern corresponding to the behavior data in the second overweight behavior data, the behavior data in the second weightlessness behavior data, and the behavior data between the second overweight behavior data and the second weightlessness behavior data in time sequence as going up and down the elevator.

[0023] The pairing of the second overweight behavior data and the second weightlessness behavior data means that, in multiple sets of behavior data, there is no third overweight behavior data or third weightlessness behavior data between the second overweight behavior data and the second weightlessness behavior data in chronological order; the second overweight behavior data and the third overweight behavior data belong to the first overweight behavior data, and the second weightlessness behavior data and the third weightlessness behavior data belong to the first weightlessness behavior data.

[0024] If there is a fourth overweight behavior data that is not paired with any first weightlessness behavior data, then the electronic system will determine the behavior pattern corresponding to the behavior data in the fourth overweight behavior data as an error; the fourth overweight behavior data belongs to the first overweight behavior data.

[0025] If there is a fourth weightlessness behavior data that is not paired with any first weightlessness behavior data, then the electronic system will determine the behavior pattern corresponding to the behavior data in the fourth weightlessness behavior data as an error; the fourth weightlessness behavior data belongs to the first weightlessness behavior data.

[0026] The electronic device identifies the behavior pattern corresponding to the behavior data other than elevator behavior data and error behavior data from multiple sets of behavior data as the leveling state; the behavior pattern corresponding to the elevator behavior data is going up and down the elevator, and the behavior pattern corresponding to the error behavior data is error.

[0027] Because the changes in equivalent gravitational acceleration during the operation of an elevator reveal a clear pattern: minor instances of weightlessness generally occur in pairs; that is, after a minor instance of weightlessness, a minor instance of weightlessness immediately follows. Therefore, based on the above technical solution, electronic equipment can more accurately determine the user's behavior pattern by identifying whether there is a pair of second weightlessness and second weightlessness behavior data (i.e., determining whether minor instances of weightlessness and weightlessness occur in pairs).

[0028] In one possible design of the first aspect, the electronic device corrects the target behavior recognition result based on the correction recognition result, including: the electronic device corrects the behavior pattern corresponding to the optional behavior data in the correction recognition result to going up and down the elevator, so as to obtain an updated correction recognition result; wherein, among the multiple sets of optional behavior data, the behavior data between two behavior data whose corresponding behavior pattern is going up and down the elevator and which are not consecutive in time order.

[0029] If all behavioral features in the target behavior recognition result correspond to a level state, or if there is a behavioral feature in the target behavior recognition result that corresponds to an elevator up / down behavior, and the behavioral pattern of the corresponding second behavioral data in the updated and corrected recognition result is elevator up / down behavior, and the behavioral pattern of the first behavioral feature in the target behavior recognition result is not elevator up / down behavior, then the electronic device will correct the behavioral pattern of the first behavioral feature in the target behavior recognition result to elevator up / down behavior. Here, the second behavioral data is one set of behavioral data from multiple sets of behavioral data, and the first behavioral feature is one set of behavioral features from multiple sets of behavioral features. The order of the first behavioral feature in the multiple sets of behavioral features after being sorted chronologically is the same as the order of the second behavioral data in the multiple sets of behavioral data after being sorted chronologically.

[0030] If all behavioral features in the target behavior recognition result correspond to a level state, or if there is a behavioral feature in the target behavior recognition result that corresponds to an elevator ride, and if the behavioral pattern of the corresponding third behavioral data in the updated and corrected recognition result is a level state, and the behavioral pattern of the second behavioral feature in the target behavior recognition result is not a level state, then the electronic device will correct the behavioral pattern of the second behavioral feature in the target behavior recognition result to a level state. Here, the third behavioral data is one set of behavioral data from multiple sets of behavioral data, and the second behavioral feature is one set of behavioral features from multiple sets of behavioral features. The order of the second behavioral feature in the multiple sets of behavioral features after being sorted in chronological order is the same as the order of the third behavioral data in the multiple sets of behavioral data after being sorted in chronological order.

[0031] Because the behavior patterns of going up and down an elevator exhibit a clear and regular change in weightlessness compared to the behavior patterns of going on and off a level floor, it can be assumed that the updated and corrected results obtained using angular velocity data are more accurate in determining whether a person is going up or down an elevator. Therefore, the updated and corrected recognition results can be used to correct the target behavior recognition results. In other words, based on the above technical solution, after correcting the target behavior recognition results by updating and correcting the recognition results, a more accurate behavior pattern recognition result can be obtained.

[0032] In one possible design approach for the first aspect,

[0033] The electronic device samples the dataset of the behavior to be determined according to a preset sampling method to obtain multiple sets of behavior data, including: the electronic device uses sliding window sampling to sample the dataset of the behavior to be determined in chronological order to obtain at least one set of positive behavior data; the electronic device uses sliding window sampling to sample the dataset of the behavior to be determined in reverse chronological order to obtain at least one set of negative behavior data.

[0034] Electronic devices extract features from multiple sets of behavioral data to obtain multiple sets of behavioral features, including: electronic devices extract features from at least one set of positive behavioral data to obtain at least one set of positive behavioral features, and extract features from at least one set of negative behavioral data to obtain at least one set of negative behavioral features.

[0035] The electronic device obtains target behavior recognition results and target landmark recognition results based on multiple sets of behavioral features and behavior recognition models. This includes: the electronic device inputting at least one set of positive behavioral features into the behavior recognition model to obtain positive behavior recognition results; the positive behavior recognition results include behavioral patterns corresponding to each set of positive behavioral features in the at least one set of positive behavioral features; the electronic device inputting at least one set of negative behavioral features into the behavior recognition model to obtain negative behavior recognition results; the negative behavior recognition results include behavioral patterns corresponding to each set of negative behavioral features in the at least one set of negative behavioral features; the electronic device fusing the positive behavior recognition results and negative behavior recognition results according to a preset fusion rule to obtain target behavior recognition results, and obtaining target landmark recognition results based on the target behavior recognition.

[0036] In some embodiments, when sampling a specific behavior dataset using a sliding window sampling method, if the preset duration is not an integer multiple of the sliding window length, there may be a set of behavior data with a small amount of data. The recognition result of the behavior features corresponding to this set of behavior data input into the behavior recognition model may have errors. Therefore, to avoid this error and to make the user behavior recognition result and landmark point recognition result obtained by the electronic device within the preset duration more accurate, in the above technical solution, the electronic device can first use a sliding window sampling method to sample the specific behavior dataset in chronological order to obtain at least one set of positive behavior data, and then use a sliding window sampling method to first sample the specific behavior dataset in reverse chronological order to obtain at least one set of negative behavior data. Further, feature extraction can be performed on the at least one set of positive behavior data and at least one set of negative behavior data to obtain at least one set of positive behavior features and at least one set of negative behavior features. Then, the at least one set of positive behavior features is input into the behavior recognition model to obtain the first behavior recognition result of the user within the preset duration; the at least one set of negative behavior features is input into the behavior recognition model to obtain the second behavior recognition result of the user within the preset duration. Then, the first behavior identification result and the second behavior identification result are fused using a specific fusion rule to obtain the target behavior identification result, and the target landmark point identification result is determined based on the target behavior identification result.

[0037] In one possible design of the first aspect, the electronic device corrects the target behavior recognition result based on the correction recognition result, including: if the behavior patterns corresponding to all behavior features included in the target behavior recognition result are all in a flat state, the behavior pattern corresponding to the fifth behavior data in the updated correction recognition result is incorrect, and the behavior patterns corresponding to the behavior data other than the fifth behavior data in the updated correction recognition result are all in a flat state, then the electronic device corrects the target behavior recognition result using the forward behavior recognition result or the reverse behavior recognition result.

[0038] If the final target behavior recognition result shows all behavioral features corresponding to a level state, and if the updated and corrected recognition result shows some behavioral data corresponding to an error (False) and the remaining behavioral data corresponding to a level state (only representing the level state), then it indicates that the behavioral data acquired by the electronic device is quite special, and some behavioral data may actually correspond to the behavior of going up and down an elevator. Therefore, based on the above technical solution, if the target behavior recognition result is obtained by fusing forward and reverse behavior recognition results, and the target behavior recognition result shows all behavioral features corresponding to a level state, and the updated and corrected recognition result shows a fifth behavioral data corresponding to an error, and the updated and corrected recognition result shows behavioral data other than the fifth behavioral data corresponding to a level state, then the target behavior recognition result can be corrected using either the forward or reverse behavior recognition result to obtain a more accurate target behavior recognition result.

[0039] Secondly, this application provides a method for generating a behavior recognition model, applied to a training device. The method includes: the training device acquiring at least one set of sample data and corresponding sample behavior labels; each set of sample data includes sample behavior features; the sample behavior features include sample sensor features and sample Wi-Fi features; the sample behavior labels are used to indicate behavior patterns, including leveling or elevator / ascending / descending status; the training device uses at least one set of sample data as training data and the corresponding sample behavior labels as supervisory information to iteratively train an initial behavior recognition model, obtaining a trained behavior recognition model.

[0040] Based on the above technical solution, supervised learning training can be performed using the acquired sample data and sample behavior labels to obtain a behavior recognition model capable of obtaining behavior patterns using sensor features and Wi-Fi features. This provides the prerequisites for implementing the behavior recognition method provided in the embodiments of this application.

[0041] In one possible design approach of the second aspect, the training device uses at least one set of sample data as training data and the sample behavior labels corresponding to the sample data as supervision information to iteratively train an initial behavior recognition model to obtain a trained behavior recognition model. This includes: the training device initializing the initial behavior recognition model; the training device inputting sample data into the initial behavior recognition model to obtain sample behavior patterns; the training device determining a loss value based on the sample behavior patterns and the sample behavior labels corresponding to the sample data; and the training device iteratively updating the initial behavior recognition model based on the loss value to obtain a trained behavior recognition model.

[0042] Based on the above technical solution, the initial behavior recognition model can be iterated step by step using loss, thereby obtaining a behavior recognition model that meets expectations.

[0043] Thirdly, this application provides an electronic device comprising: a memory and one or more processors; the memory being coupled to the processors; wherein the memory stores computer program code, the computer program code including computer instructions, which, when executed by the processor, cause the electronic device to perform the behavior recognition method provided in the first aspect.

[0044] Fourthly, this application provides a training device, including: a processor and a memory; the memory is used to store executable instructions, and the processor is configured to execute the executable instructions stored in the memory to cause the training device to perform the method for generating a behavior recognition model as provided in the second aspect above.

[0045] Fifthly, this application provides a behavior recognition system, including the electronic device provided in the third aspect and the training device provided in the fourth aspect.

[0046] In a sixth aspect, this application provides a computer-readable storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform the behavior recognition method provided in the first aspect.

[0047] In a seventh aspect, this application provides a computer-readable storage medium including computer instructions that, when executed on a training device, cause the training device to perform the method for generating a behavior recognition model as provided in the second aspect.

[0048] Eighthly, a computer program product containing instructions is provided, which, when run on an electronic device, enables the electronic device to execute the behavior recognition method provided in the first aspect.

[0049] In a ninth aspect, a computer program product containing instructions is provided, which, when run on a training device, enables the training device to execute the method for generating the behavior recognition model provided in the second aspect above.

[0050] The beneficial effects that can be achieved by aspects three through nine can be referred to in terms of the beneficial effects of aspects one, two, or three and any of their possible design methods, and will not be elaborated here. Attached Figure Description

[0051] Figure 1 A schematic diagram illustrating the principle of a behavior recognition method provided in an embodiment of this application;

[0052] Figure 2This is a schematic diagram of the structure of a behavior recognition system provided in an embodiment of this application;

[0053] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0054] Figure 4 A schematic diagram of the software architecture of an electronic device provided in an embodiment of this application;

[0055] Figure 5 A schematic diagram of the algorithm architecture of a behavior recognition method provided in an embodiment of this application;

[0056] Figure 6 This is a schematic diagram of the structure of a training device provided in an embodiment of this application;

[0057] Figure 7 A flowchart illustrating a method for generating a behavior recognition model provided in an embodiment of this application;

[0058] Figure 8 This is a schematic diagram of a sliding window sampling scenario provided in an embodiment of this application;

[0059] Figure 9 A flowchart illustrating a behavior recognition method provided in this application embodiment. Figure 1 ;

[0060] Figure 10 A schematic diagram illustrating a target behavior recognition result provided in an embodiment of this application;

[0061] Figure 11 A schematic diagram illustrating the generation of a target behavior recognition result provided in an embodiment of this application;

[0062] Figure 12 A flowchart illustrating a behavior recognition method provided in this application embodiment. Figure 2 ;

[0063] Figure 13 A schematic diagram illustrating the change in equivalent gravitational acceleration during elevator operation, provided as an embodiment of this application;

[0064] Figure 14 A flowchart illustrating a behavior recognition method provided in this application embodiment. Figure 3 ;

[0065] Figure 15 A schematic diagram illustrating a scenario for determining a behavior model, as provided in an embodiment of this application;

[0066] Figure 16 This application provides an example of generating an updated correction identification result. Figure 1 ;

[0067] Figure 17 This application provides an example of generating an updated correction identification result. Figure 2 ;

[0068] Figure 18 A schematic diagram illustrating the correction of a target behavior recognition result provided in an embodiment of this application;

[0069] Figure 19 A schematic diagram illustrating a process for correcting target behavior recognition results provided in an embodiment of this application;

[0070] Figure 20 This is a schematic diagram of a chip system provided in an embodiment of this application. Detailed Implementation

[0071] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that “ / ” means “or,” for example, A / B can mean A or B; “and / or” in the text is merely a description of the relationship between related objects, indicating that three relationships can exist, for example, A and / or B can mean: A alone, A and B simultaneously, and B alone.

[0072] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0073] The terms "first" and "second" in the following embodiments of this application are for descriptive purposes only and should not be construed as implying relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0074] First, the terms used in the embodiments of this application are explained as follows:

[0075] Elevator: An elevator is a type of elevator. It primarily refers to a vertical lifting machine powered by an electric motor, equipped with a box-shaped cabin. An elevator is a fixed lifting device serving a designated number of floors. An elevator has a car that runs between at least two vertical rigid guide rails. In this application, the elevator mentioned in the embodiments specifically refers to an elevator.

[0076] Nine-axis sensor: The nine-axis sensor includes a three-axis accelerometer, a three-axis gyroscope and a three-axis magnetometer.

[0077] An accelerometer (G-sensor), also known as a gravity sensor, is a device that can sense acceleration in any direction. A triaxial accelerometer works based on the fundamental principle of acceleration. Acceleration is a spatial vector; to accurately understand the motion of an object, its components on its three coordinate axes must be measured. Furthermore, when the direction of the object's motion is unknown beforehand, only a triaxial accelerometer can detect the acceleration signal.

[0078] A gyroscope (GYRO-sensor), also known as a ground sensor, traditionally consists of an internal gyroscope. A three-axis gyroscope can simultaneously measure position, trajectory, and acceleration in six directions. A single-axis gyroscope can only measure quantities in two directions, meaning a system typically requires three gyroscopes. A single three-axis gyroscope can replace three single-axis gyroscopes. The working principle of a three-axis gyroscope is to measure the angle between the vertical axis of the gyroscope rotor and the device in a three-dimensional coordinate system, and calculate the angular velocity. The angle and angular velocity are used to determine the object's motion state in three-dimensional space. A three-axis gyroscope can simultaneously measure six directions: up, down, left, right, forward, and backward (the composite direction can also be decomposed into three-axis coordinates), ultimately determining the device's trajectory and acceleration. In other words, a three-axis gyroscope determines the device's current motion state by measuring its own rotation, such as forward, backward, up, down, left, or right; and whether it is accelerating (angular velocity) or decelerating (angular velocity).

[0079] A magnetometer (M-sensor), also called a geomagnetic sensor or magnetic field meter, is used to test the strength and direction of a magnetic field to locate the position of an object. The principle of a magnetometer is similar to that of a compass, measuring the angle between the current object and the four cardinal directions. A three-axis magnetometer, also known as an electronic compass, is widely used in drones, smartwatches, smartphones, and navigation devices.

[0080] Based on the above introduction, it can be simply understood that: the gyroscope knows "we turned around," the accelerometer knows "we moved forward a few meters," and the magnetometer knows "we are heading west." Therefore, in practical applications, due to the needs of application, error correction, and error compensation, the above sensors (i.e., a nine-axis sensor) are often used in combination to make full use of the strengths of each sensor, resulting in more accurate final calculation results. For example, in Android phones, both the magnetometer and accelerometer are used simultaneously to calculate orientation information.

[0081] As electronic devices (such as mobile phones) become increasingly powerful and ubiquitous, they are evolving into tools for discreetly monitoring human behavior anytime, anywhere. The number of sensors integrated into electronic devices is constantly increasing with each upgrade. Utilizing these sensors and radio frequency signals to identify a user's current movement or behavior (i.e., behavior recognition) is a current research hotspot. Since people spend over 80% of their time indoors, identifying indoor user behavior is crucial in behavior recognition research. Accurately identifying indoor user behavior can significantly advance many research fields. For example, identifying indoor user behavior can determine whether a user has moved between floors, thus assessing their activity range.

[0082] Currently, indoor behavior recognition primarily relies on data from various sensors on electronic devices, which are then used to train machine learning models to obtain a complete model capable of predicting or determining current user behavior patterns. However, most current research requires data from all sensors to accurately identify user behavior patterns. Therefore, if some sensors on the electronic device fail or are missing, the accuracy of the trained model will decrease, resulting in unsatisfactory predictions or judgments.

[0083] To address the aforementioned technical issues, refer to Figure 1As shown, this application provides a behavior recognition method. This method can be applied to scenarios where electronic devices judge the indoor behavioral characteristics or behavioral patterns of their users. In this technical solution, most sensor data is not required; the user's movements up and down elevators and leveling can be accurately identified solely based on the basic nine-axis sensor and Wi-Fi (wireless fidelity) information on the electronic device. Specifically, the electronic device can first acquire sensor data from its nine-axis sensor and Wi-Fi data within a preset time period. Then, multiple behavioral feature groups can be obtained from the sensor data and Wi-Fi data within the preset time period (this process can be called feature engineering). Each behavioral feature group corresponds to a unit of time and includes sensor features and Wi-Fi features. Afterward, multiple behavioral feature groups can be input into a behavior recognition model to obtain the behavior recognition result and landmark point recognition result of the user (i.e., the user holding the electronic device). The behavior recognition result can include the user's behavior patterns within multiple time periods within the preset time period. For example, the user's behavior pattern in time period A is up and down elevators, and the user's behavior pattern in time period B is leveling, etc. Landmark identification results include points of change in the user's behavior pattern within a preset time period, or the start and end points of each behavior pattern. This behavior recognition model is pre-trained and can determine the user's behavior pattern based on sensor and Wi-Fi characteristics. Furthermore, because the behavior recognition model relies on limited sensor data to determine behavior patterns, to prevent misjudgments, after the behavior recognition model outputs its results, the electronic device can also determine whether it is in an elevator / escalator state based on acceleration data from the sensor data. Since there are significant differences in the user's weightlessness / elevation data in non-elevator modes (leveling, escalator / stairs / climbing) and elevator / elevator states, the electronic device can use this weightlessness / elevation data to correct the behavior recognition model's results and landmark identification results, thus obtaining more accurate identification results. (Refer to...) Figure 1 As shown, the specific implementation of correcting the recognition results of the behavior recognition model using acceleration data can be achieved by the elevator rule model in the electronic device.

[0084] Of course, in order to enable the above behavior recognition method to be implemented smoothly, this application also provides a method for generating a behavior recognition model. This method can use at least one set of sample data to train the above behavior recognition model in a supervised learning manner.

[0085] The technical solutions provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0086] The technical solution provided in this application can be applied to, for example... Figure 2 The behavior recognition system shown. (Refer to...) Figure 2 As shown, the behavior recognition system includes a training device 01 and an electronic device 02. The training device 01 primarily acquires samples and trains a behavior recognition model, i.e., it implements the behavior recognition model generation method provided in this application embodiment. The electronic device 02, after acquiring the behavior recognition model from the training device 01, performs corresponding conversions when a user needs to perform behavior recognition on a user belonging to a certain electronic device, i.e., it implements the style conversion method provided in this application embodiment.

[0087] It is understood that the aforementioned electronic device 02 and training device 01 can be two separate devices or the same device. This application does not impose any specific restrictions in this regard.

[0088] For example, the electronic device in this application embodiment can be a mobile phone, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, as well as cellular phone, personal digital assistant (PDA), augmented reality (AR) device, virtual reality (VR) device, artificial intelligence (AI) device, wearable device, in-vehicle device, smart home device and / or smart city device. This application embodiment does not impose any special restrictions on the specific type of electronic device.

[0089] For example, taking a mobile phone as an electronic device, Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown.

[0090] like Figure 3As shown, the electronic device may have multiple cameras 293, such as a front-facing ordinary camera, a front-facing low-power camera, a rear-facing ordinary camera, a rear-facing wide-angle camera, etc. Furthermore, the electronic device may include a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, antenna 1, antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, a headphone jack 270D, a sensor module 280, buttons 290, a motor 291, an indicator 292, a display screen 294, and a subscriber identification module (SIM) card interface 295, etc. The sensor module 280 may include a gyroscope sensor 280A, a magnetic sensor 280B, an accelerometer sensor 280C, a proximity sensor 280D, a fingerprint sensor 280E, a temperature sensor 280F, a touch sensor 280G, and an ambient light sensor 280H. The gyroscope sensor 280A may be a three-axis gyroscope sensor, the magnetic sensor 280B may be a three-axis magnetic sensor, and the accelerometer sensor 280C may be a three-axis accelerometer sensor. The three-axis gyroscope sensor, the three-axis magnetic sensor, and the three-axis accelerometer sensor are collectively referred to as a nine-axis sensor.

[0091] Processor 210 may include one or more processing units, such as application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0092] A controller can be the nerve center and command center of an electronic device. Based on the instruction opcode and timing signals, the controller generates operation control signals to control the fetching and execution of instructions.

[0093] The processor 210 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. This memory can store instructions or data that the processor 210 has just used or that are used repeatedly. If the processor 210 needs to use the instruction or data again, it can directly retrieve it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.

[0094] In some embodiments, the processor 210 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0095] The external memory interface 220 can be used to connect to external non-volatile memory, thereby expanding the storage capacity of the electronic device. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to perform data storage functions. For example, music, video, and other files can be stored in the external non-volatile memory.

[0096] The internal memory 221 may include one or more random access memory (RAM) and one or more non-volatile memory (NVM). The RAM can be directly read and written by the processor 110 and can be used to store executable programs (e.g., machine instructions) of the operating system or other running programs, as well as user and application data. The NVM can also store executable programs and user and application data, and can be pre-loaded into the RAM for direct read and write by the processor 110. In this embodiment, the internal memory 221 may store image files or recorded video files captured by the electronic device in single-lens or multi-lens shooting modes.

[0097] Touch sensor 280G, also known as a "touch device," can be located on display screen 194. The touch sensor 280G and display screen 294 together form a touchscreen, also known as a "touchscreen." Touch sensor 280G detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 294. In other embodiments, touch sensor 280G may also be located on the surface of the electronic device, in a different position than display screen 294.

[0098] In some embodiments, an electronic device may include one or N cameras 293, where N is a positive integer greater than 1. In this embodiment, the type of camera 293 can be distinguished based on hardware configuration and physical location. For example, the multiple cameras included in the camera 293 can be placed on the front and back of the electronic device respectively. The camera placed on the side of the electronic device's display screen 294 can be called a front-facing camera, and the camera placed on the back cover of the electronic device can be called a rear-facing camera. Alternatively, the multiple cameras included in the camera 293 may have different focal lengths and angles of view. A camera with a short focal length and a wide angle of view can be called a wide-angle camera, while a camera with a long focal length and a narrow angle of view can be called a regular camera. The difference in the content of the images captured by different cameras lies in the fact that the front-facing camera is used to capture the scene facing the front of the electronic device, while the rear-facing camera is used to capture the scene facing the back of the electronic device; a wide-angle camera can capture a larger area of ​​scenery within a shorter shooting distance, and the scene captured at the same shooting distance will appear smaller in the image than the scene captured using a regular lens. The length of the focal length and the size of the angle of view are relative concepts without specific parameter limitations. Therefore, wide-angle cameras and ordinary cameras are also relative concepts, and can be distinguished based on physical parameters such as focal length and angle of view.

[0099] The electronic device implements display functions through a GPU, a display screen 294, and an application processor. The GPU is a microprocessor for image editing, connected to the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0100] Electronic devices can achieve shooting functions through ISP, camera 293, video codec, GPU, display 294 and application processor.

[0101] Display screen 294 is used to display images, videos, etc. Display screen 294 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device may include one or N displays 294, where N is a positive integer greater than 1.

[0102] In this embodiment of the application, the display screen 294 can be used to display the interface of an electronic device (e.g., a camera preview interface, a video preview interface, a final preview interface, etc.), and display images captured by any one or more cameras 293 in the interface.

[0103] The charging management module 240 is used to receive charging input from the charger. The charger can be a wireless charger or a wired charger.

[0104] The power management module 241 is used to connect the battery 242, the charging management module 240, and the processor 210. The power management module 241 receives input from the battery 242 and / or the charging management module 240 to power the processor 210, internal memory 521, display 294, camera 293, and wireless communication module 260, etc.

[0105] The wireless communication function of electronic devices can be realized through antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, modem, and baseband processor.

[0106] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in an electronic device can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization.

[0107] The mobile communication module 250 can provide solutions for wireless communication applications, including 2G / 3G / 4G / 5G, in electronic devices.

[0108] The wireless communication module 260 can provide solutions for wireless communication applications in electronic devices, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 260 can be one or more devices integrating at least one communication processing module. The wireless communication module 260 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 210. The wireless communication module 260 can also receive signals to be transmitted from processor 210, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0109] The SIM card interface 295 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 295 to make contact with and detach from the electronic device. The electronic device can support one or more SIM card interfaces. The SIM card interface 295 supports Nano SIM cards, Micro SIM cards, and other SIM cards. Multiple cards can be inserted into the same SIM card interface 295 simultaneously. The SIM card interface 295 is also compatible with external memory cards. The electronic device interacts with the network through the SIM card to achieve functions such as calls and data communication.

[0110] Of course, this is understandable. Figure 3 The illustration shown is merely an example when the electronic device is in the form of a mobile phone. If the electronic device is in the form of a tablet, handheld computer, PC, PDA, wearable device (such as a smartwatch, smart bracelet), or other similar device, the structure of the electronic device may include more advanced features. Figure 3 The fewer structures shown can also include more than Figure 3 The structures shown are not limited here.

[0111] Understandably, the implementation of electronic device functions generally requires not only hardware support but also software cooperation.

[0112] Software support is also required. The software system of electronic devices can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application's embodiment uses a layered architecture... Taking the system as an example, the software structure of the electronic device is illustrated.

[0113] Figure 4 This is a schematic diagram of the layered architecture of the software system of the electronic device provided in this application embodiment. The layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces.

[0114] like Figure 4 As shown, the system used by the electronic device is... Taking the system as an example, in this embodiment of the application, the software of the electronic device is divided into four layers, from top to bottom: application layer, framework layer, system library and Android runtime, HAL layer (hardware abstraction layer) and driver layer (or kernel layer).

[0115] The application layer can include a series of applications. For example... Figure 4 As shown, the application layer can include applications such as camera, gallery, calendar, map, WLAN, Bluetooth, music, video, SMS, and call.

[0116] The application layer may further include behavior recognition applications. In this embodiment, the behavior recognition application can be an application that the electronic device cannot see or use when it is being used normally by the user. It can be enabled after the electronic device receives a specific operation, or it can be started by default.

[0117] This behavior recognition application, once activated, can acquire Wi-Fi data and nine-axis sensor data from the electronic device in real time. After acquiring data for a preset duration, it can identify the behavior patterns of the user (i.e., the person holding the electronic device) within that preset duration based on the acquired data. Alternatively, the behavior recognition application can acquire Wi-Fi data and nine-axis sensor data for a preset duration each time it is activated, and then identify the behavior patterns of the user (i.e., the person holding the electronic device) within that preset duration based on the acquired data.

[0118] For example, the algorithm architecture of this behavior recognition application can be as follows: Figure 5As shown, this behavior recognition application may include an output layer, a core algorithm layer, a data processing layer, and a data layer.

[0119] The data layer is used to acquire Wi-Fi data and sensor data from the nine-axis sensor of the electronic device. This sensor data includes geomagnetic data detected by a three-axis magnetometer, gyroscope data detected by a three-axis gyroscope sensor, and acceleration data detected by a three-axis accelerometer. The data layer can acquire Wi-Fi data from Wi-Fi applications (e.g., WLAN applications) in the application layer of the electronic device, and sensor data detected by the nine-axis sensor from the nine-axis sensor driver in the driver layer.

[0120] The data processing layer includes a data preprocessing module and a feature extraction module. The data preprocessing module specifically performs data alignment, data completion, and data sampling.

[0121] Data alignment refers to matching and aligning Wi-Fi data with sensor data from a nine-axis sensor according to time.

[0122] Data completion specifically refers to supplementing data that is missing at certain times according to preset rules. For example, if Wi-Fi data, gyroscope data, and acceleration data exist at time A, but geomagnetic data is missing, then the geomagnetic data will be supplemented according to the preset rules. The preset rules used for data completion can be any feasible rules, and this application does not impose specific restrictions on them. Of course, in reality, electronic devices may not collect Wi-Fi and sensor data at every moment, but rather at certain intervals (e.g., once every 1ms). Therefore, "certain moments" here can refer to the moments when the electronic device collects Wi-Fi and sensor data.

[0123] Data sampling specifically involves dividing the data acquired by the data layer for a preset duration into multiple groups of behavioral data according to predetermined sampling rules. For example, the sampling rule can be a sliding window sampling method, where the window is continuously slid along a sliding step and a sliding window length to divide the data into multiple groups. For instance, if the sliding window length is 2 seconds, the sliding step is 1 second, and the time duration is 10 seconds, then 9 groups of data will be obtained. The first group of behavioral data corresponds to a time of 0-2 seconds, the second group corresponds to a time of 1-3 seconds, and so on, with the ninth group corresponding to a time of 8-10 seconds.

[0124] Feature extraction involves extracting features from each set of data according to certain feature extraction rules, thereby obtaining Wi-Fi features and sensor features from the nine-axis sensor. These sensor features can include geomagnetic features, gyroscope features, and acceleration features.

[0125] The core algorithm layer includes a behavior recognition module and an elevator correction module. The behavior recognition module includes a pre-trained behavior recognition model. The behavior recognition model is used to input the Wi-Fi features and sensor features of each set of behavior data obtained from the data processing layer into the behavior recognition model to obtain the user's behavior recognition results and landmark point recognition results within a preset time period. In this embodiment, the behavior recognition results can include user behavior patterns within multiple time periods, and these patterns can include at least: leveling and elevator / stair climbing. In practice, leveling can be further divided into leveling stillness and leveling walking. Additionally, behavior patterns can also include escalator / stair climbing and stair climbing, depending on requirements. The user's behavior patterns can differ within different time periods. For example, if the preset time period is 120 seconds, the behavior recognition results can include: user behavior pattern leveling and walking within 0-10 seconds, user behavior pattern leveling and walking within 10-50 seconds, user behavior pattern elevator / stair climbing within 50-80 seconds, and user behavior pattern stair climbing within 80-120 seconds. Landmark identification results include the points where the user's behavior pattern changes within a preset time period, or the start and end points of each behavior pattern.

[0126] The elevator correction module is specifically used to identify elevators based on each set of behavior data obtained from the data layer's acceleration data. This elevator identification result includes whether the elevator is going up or down, or not. The elevator correction module is also used to correct the behavior identification results of the behavior recognition module based on the elevator identification results, thereby correcting the landmark point identification results. Specific implementation details are described in subsequent embodiments and will not be elaborated here. In this embodiment, "elevator" specifically refers to a lift, and the same applies to all subsequent embodiments.

[0127] The output layer is used to output the corrected behavior recognition results and landmark point recognition results.

[0128] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The framework layer includes predefined functions. These may include, for example, an activity manager, a window manager, a view system, a resource manager, a notification manager, an audio service, a camera service, etc., though this embodiment does not impose any limitations on these.

[0129] In this embodiment, if there is no behavior recognition application in the application layer, the framework layer of the electronic device may contain a behavior recognition service with the functionality of the behavior recognition application. The algorithm architecture of the behavior recognition service can be referred to... Figure 5 Show.

[0130] The system library can include multiple functional modules. For example: a surface manager, media libraries, OpenGL ES, and SGL. The surface manager manages the display subsystem and provides 2D and 3D layer blending for multiple applications. The media libraries support playback and recording of various common audio and video formats, as well as still image files. The media libraries support various audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG. OpenGL ES is used for 3D graphics drawing, image rendering, compositing, and layer processing. SGL is a 2D graphics engine.

[0131] The Android runtime consists of the core libraries and the virtual machine. The Android runtime is responsible for scheduling and managing the Android system. The core libraries comprise two parts: one part contains the functionalities that Java calls, and the other part consists of the Android core libraries. The application layer and application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0132] The Hardware Abstraction Layer (HAL) is an interface layer located between the operating system kernel and the hardware circuitry, its purpose being to abstract the hardware. It hides the platform-specific hardware interface details, providing the operating system with a virtual hardware platform, making it hardware-independent and portable across multiple platforms. The HAL provides a standard interface that exposes device hardware functionality to the higher-level Java API framework (i.e., the framework layer). The HAL contains multiple library modules, each implementing an interface for a specific type of hardware component, such as: audio HAL (audio module), Bluetooth HAL (Bluetooth module), camera HAL (camera module), and sensors HAL (sensor module, or Isensor service).

[0133] The kernel layer is the layer between hardware and software. The kernel layer includes at least display drivers, camera drivers, audio drivers, and sensor drivers, but this application does not limit this. Specifically, the sensor drivers can include drivers for each sensor included in the electronic device, such as a nine-axis sensor driver, which can include a three-axis gyroscope sensor driver, a three-axis magnetometer sensor driver, and a three-axis accelerometer sensor driver.

[0134] For example, the training device provided in this application can be a server, which can be a single server, a server cluster consisting of multiple servers, or a cloud computing service center. This application does not impose any specific restrictions on this.

[0135] For example, taking the training device as the server, Figure 6 A schematic diagram of a server structure is shown. (Refer to...) Figure 6 As shown, the server includes one or more processors 601, a communication line 602, and at least one communication interface. Figure 6 (This is merely an example illustration of a communication interface 603 and a processor 601; optionally, a memory 604 may also be included.)

[0136] The processor 601 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present application.

[0137] Communication line 602 may include a communication bus for communication between different components.

[0138] The communication interface 603 can be a transceiver module used to communicate with other devices or communication networks, such as Ethernet, RAN, and wireless local area networks (WLAN). For example, the transceiver module can be a transceiver or similar device. Optionally, the communication interface 603 can also be a transceiver circuit located within the processor 601, used to implement the processor's signal input and signal output.

[0139] The memory 604 can be a device with storage functionality. For example, it can be read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions; random access memory (RAM) or other types of dynamic storage devices capable of storing information and instructions; electrically erasable programmable read-only memory (EEPROM); compact disc read-only memory (CD-ROM) or other optical disc storage; optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.); magnetic disk storage media or other magnetic storage devices; or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory can exist independently and be connected to the processor via communication line 602. The memory can also be integrated with the processor.

[0140] The memory 604 stores computer execution instructions for implementing the scheme of this application, and its execution is controlled by the processor 601. The processor 601 executes the computer execution instructions stored in the memory 604, thereby implementing the behavior recognition model generation method provided in the embodiments of this application.

[0141] Alternatively, in this embodiment, the processor 601 may execute the processing-related functions in the behavior recognition model generation method provided in the following embodiments of this application, and the communication interface 603 may be responsible for communicating with other devices (e.g., electronic devices) or communication networks. This embodiment does not specifically limit this.

[0142] Optionally, the computer execution instructions in the embodiments of this application may also be referred to as application code, and the embodiments of this application do not specifically limit this.

[0143] In a specific implementation, as one embodiment, the processor 601 may include one or more CPUs, for example... Figure 6 CPU0 and CPU1 in the CPU.

[0144] In a specific implementation, as one example, the server may include multiple processors, for example... Figure 6The processors 601 and 607 are described herein. Each of these processors may be a single-core processor or a multi-core processor. The processors herein may include, but are not limited to, at least one of the following: a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a microcontroller unit (MCU), or an artificial intelligence processor, and other computing devices that run software. Each computing device may include one or more cores for executing software instructions to perform calculations or processing.

[0145] In a specific implementation, as one embodiment, the server may further include an output device 605 and an input device 606. The output device 605 communicates with the processor 601 and can display information in various ways. For example, the output device 605 may be a liquid crystal display (LCD), a light-emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector, etc. The input device 606 communicates with the processor 601 and can receive user input in various ways. For example, the input device 606 may be a mouse, keyboard, touchscreen device, or sensing device, etc.

[0146] The server described above can be a general-purpose device or a dedicated device. For example, the server can be a desktop computer, laptop, network server, PDA (personal digital assistant), mobile phone, tablet computer, wireless terminal device, embedded device, the aforementioned terminal device, the aforementioned network device, or something else with... Figure 6 Devices with similar structures. This application does not limit the type of server to specific embodiments.

[0147] The methods described in the following embodiments can all be implemented in electronic devices or training devices with the above-described hardware structure.

[0148] First, to facilitate the successful implementation of the behavior recognition method provided in this application, embodiments of this application provide a method for generating a behavior recognition model. This method can be applied to the training device mentioned in the foregoing embodiments. (Refer to...) Figure 7 As shown, the method may include S701 and S702:

[0149] S701, The training device acquires at least one set of sample data and sample behavior labels corresponding to the sample data.

[0150] Each set of sample data includes sample behavioral characteristics. These characteristics include sample sensor features and sample Wi-Fi features. Sample behavioral tags are used to indicate behavioral patterns. Behavioral patterns can be leveling or using elevators. In practice, leveling can be further divided into stationary leveling and walking leveling. Additionally, behavioral patterns can be used to indicate using escalators or stairs, depending on the requirements.

[0151] In this system, the sample data and its corresponding sample behavior label correspond to the same electronic device. At least one set of sample data is obtained by the training device from multiple electronic devices, or obtained using multiple electronic devices. Each set of sample data corresponds to a unit of time, that is, the sample data is the sample sensor features and Wi-Fi features of its corresponding electronic device within a unit of time.

[0152] In practice, Wi-Fi signals are a type of radio frequency (RF) signal. The attenuation of RF signals is proportional to the propagation distance. This signal attenuation causes users (specifically, their electronic devices) to detect different access point (AP) sets or signals from the same AP at different floors (or locations), resulting in variations in signal strength. This allows for the location of electronic devices within a floor. Specifically, the positioning principle stems from the significant attenuation of RF signals as they propagate through floors. For example, Wi-Fi signals experience approximately 29 dB of attenuation after passing through 50 cm of concrete, which can be used to determine floor height. Based on this, Wi-Fi features can serve as a data foundation for identifying user behavior patterns. However, due to the complexity of modern large buildings, relying solely on Wi-Fi features for user behavior identification carries significant errors. Therefore, to more accurately identify behavior patterns, in addition to obtaining sample Wi-Fi features, it is also necessary to acquire sample sensor features closely related to user actions when obtaining sample behavioral features for training the behavior recognition model. For example, sensor features are extracted from sensor data of a nine-axis sensor (i.e., a three-axis gyroscope sensor, a three-axis magnetometer, and a three-axis accelerometer).

[0153] Sample behavior labels can be behavioral patterns determined on-site when acquiring sample data. For example, if the sample behavior label in a certain sample data is going up and down an elevator, then the sample behavior features in that sample data can be collected in an elevator where the electronic equipment is in operation.

[0154] In this embodiment, to make the behavior recognition model applicable to various scenarios, the sample data acquired in S701 can be collected from electronic devices in scenarios such as schools, office buildings, and shopping malls, with a sample data scale of at least 100,000. This way, the larger the sample data scale, the more continuously the behavior recognition model can be updated and trained, improving the model's generalization ability and enabling the behavior recognition model to accurately identify user behavior patterns based on data from electronic devices acquired in various scenarios.

[0155] It should be noted that the purpose of a behavior recognition model is to at least identify user behavior patterns. The specific number of behavior patterns that can be identified depends on the architecture of the behavior recognition model and the types of sample behavior labels in the sample data. In this embodiment, at least the states of going up and down elevators and leveling up need to be identified. In practice, leveling up can include walking on level and remaining stationary on level. Based on this, all the sample behavior labels mentioned above must include at least two categories: going up and down elevators and leveling up.

[0156] Of course, in reality, user behavior patterns indoors can also include going up and down escalators and stairs. Therefore, to better identify user behavior, the behavior recognition model can also identify behavior patterns that include going up and down stairs and escalators. In this case, the sample behavior labels included in the above sample data can also include going up and down stairs and escalators.

[0157] It should be noted that, in this application, "going up and down the elevator" specifically refers to being in the process of going up or down the elevator, "going up and down the escalator" refers to being in the process of going up or down the escalator, "going up and down the stairs" refers to being in the process of going up or down the stairs, "stationary at the same level" refers to being at the same level and not moving, and "walking at the same level" refers to being at the same level and walking.

[0158] In this embodiment, the sample sensor features and sample Wi-Fi features in the sample behavior characteristics are obtained by acquiring Wi-Fi data and sensor data from the sample electronic device. Based on this, taking the electronic device as an example, S701 specifically includes S7011-S7013:

[0159] S7011, The training device acquires sample Wi-Fi datasets and sample sensor datasets of sample electronic devices within a preset time period.

[0160] The sample Wi-Fi dataset includes all Wi-Fi data from the sample electronic device within a preset time period, and the sample sensor dataset includes all sensor data from the nine-axis sensor of the sample electronic device within a preset time period.

[0161] Wi-Fi data can include all information obtainable by the Wi-Fi module of the sample electronic device, such as all detected access points (APs) and AP signal quality information. The nine-axis sensor includes a three-axis gyroscope, a three-axis magnetometer, and a three-axis accelerometer; therefore, its data can include gyroscope data, magnetometer data, and accelerometer data. Gyroscope data can include motion direction detected by the gyroscope, magnetometer data can include geomagnetic data detected by the magnetometer, and accelerometer data can include acceleration data detected by the accelerometer.

[0162] In some embodiments, the sample Wi-Fi dataset and the sample sensor dataset can be collectively referred to as sample trajectory data. For example, the preset duration can be 120 seconds.

[0163] S7011, the training device preprocesses the sample Wi-Fi dataset and the sample sensor dataset to obtain the original sample dataset.

[0164] Typically, within a preset time period, users may exhibit multiple behavioral patterns, with varying durations and potential similarities between Wi-Fi and sensor data across different patterns. If the behavior recognition model is trained using features extracted from Wi-Fi and sensor data at each moment as sample data, errors in the model's judgment will occur. Therefore, when training the behavior model, a single sample dataset can include Wi-Fi and sensor features within a specific time period (i.e., a unit of time). To facilitate subsequent identification of corresponding Wi-Fi and sensor features within a unit of time (i.e., belonging to the same moment), after obtaining the first sample Wi-Fi and sensor datasets within the preset time period, it is necessary to align the Wi-Fi and sensor data belonging to the same moment and complete any Wi-Fi or sensor data that was not collected at a certain moment for various reasons, resulting in the original sample dataset. The original sample dataset then includes all matched, aligned, and completed Wi-Fi and sensor data within the preset time period.

[0165] Therefore, data preprocessing in this application may include data alignment and data completion. Specifically, data alignment and data completion can be performed as described in the foregoing embodiments. Figure 5 The relevant content in the description will not be repeated here.

[0166] S7012. The training device samples the original sample dataset according to a preset sampling method to obtain at least one set of undetermined sample data.

[0167] In this context, the time length corresponding to each set of undetermined sample data is a unit of time.

[0168] In practice, when a user is in a certain behavioral pattern, there may be moments when they perform actions outside that pattern, causing anomalies in the sensor data at those moments. For example, when a user is in the behavior pattern of going up or down an elevator, swinging their arm causes a significant change in acceleration data. To minimize the impact of these special actions on the accuracy of the behavior recognition model trained in this application, the technical solution provided in this application allows for some shared data between two adjacent sets of undetermined sample data. This way, even if the behavior pattern represented by the sample data obtained after feature extraction from one set of undetermined sample data is incorrect, the behavior pattern represented by the sample data obtained after feature extraction from the subsequent set of undetermined sample data is highly likely to be correct. Consequently, the recognition capability of the ultimately trained behavior recognition model is more accurate.

[0169] Based on the above description, in this embodiment of the application, the preset sampling method used to obtain at least one set of undetermined sample data can be sliding window sampling. Specifically, the length of the sliding window in the characteristic parameters of sliding window sampling can be a unit time duration, and the sliding step size of the sliding window can be half of the unit time duration.

[0170] This unit duration can be approximated by the maximum value among the time lengths of the datasets (composed of multiple Wi-Fi data and multiple sensor data) that indicate the characteristics of each behavior pattern. In the elevator up / down behavior pattern, there is a brief acceleration after the elevator starts and a brief deceleration before it stops, while the other behavior patterns have similar characteristics throughout the entire process. That is, the aforementioned "maximum value among the time lengths" can be the duration of the brief acceleration after the elevator starts or the brief deceleration before it stops. Based on this, the unit duration can be close to the duration of the brief acceleration after the elevator starts or the brief deceleration before it stops. Furthermore, in practice, the duration of the brief acceleration after the elevator starts and the duration of the brief deceleration before it stops can be the same, 2-4 seconds.

[0171] For example, taking a unit duration of 2.56 seconds and a preset duration of 10 seconds as an example, refer to... Figure 8As shown in (a), when using a sliding window to sample in chronological order, the sliding window is initially positioned between 0 and 2.56 seconds within a 10-second timeframe. This range encompasses the data in the original 10-second sample dataset belonging to the 0-2.56-second range, which is used as the first set of undetermined sample data. Subsequently, the sliding window is moved forward by one sliding step, i.e., 1.28 seconds, in chronological order. At this point, the sliding window can encompass the data in the original 10-second sample dataset belonging to the 1.28-3.84-second range, which is used as the second set of undetermined sample data. Then, by moving the sliding window one step in time sequence, we can obtain the following data: 2.56-5.12s is the third group of undetermined sample data, 3.84-6.4s is the fourth group of undetermined sample data, 5.12-7.68s is the fifth group of undetermined sample data, 6.4-8.96s is the sixth group of undetermined sample data, and 7.68-10s is the seventh group of undetermined sample data.

[0172] S7013. The training device extracts features from at least one set of undetermined sample data to obtain at least one set of sample data.

[0173] After the training device acquires the undetermined sample data, in order to facilitate the subsequent training of the model, the specified sample data can be further subjected to feature extraction to obtain sample data including sample behavior features, which can be used as training data to train the behavior recognition model.

[0174] The method for extracting features from the given sample data can be any feasible method, such as extracting features based on a feature extraction model or using a specific feature extraction algorithm.

[0175] Based on the technical solutions of S7011-S7013 mentioned above, at least one set of sample data can be successfully obtained, providing data support for subsequent training of behavior recognition models.

[0176] S702. The training device uses at least one set of sample data as training data and the sample behavior labels corresponding to the sample data as supervision information to iteratively train the initial behavior recognition model and obtain the trained behavior recognition model.

[0177] The initial model framework can adopt an extreme gradient boosting (XGBoost) architecture, a light GBM (light gradient boosting machine) architecture, or any other feasible model architecture.

[0178] S702 may specifically include the following steps:

[0179] 1. Initialize the initial behavior recognition model.

[0180] The initialization of the initial behavior recognition model can specifically involve initializing the weight parameters and bias parameters of the model using any feasible initialization method. Four commonly used initialization methods are Gaussian initialization, Xavier initialization, MSRA initialization, and He initialization. Generally, the bias parameters are initialized to 0, and the weight parameters are randomly initialized. The specific initialization process will not be described in detail in this application.

[0181] Furthermore, when initializing the initial behavior recognition model, the hyperparameters in the initial behavior recognition model are set based on the user experience of the trainers. Hyperparameters are model parameters that do not change during training. For example, taking the initial behavior recognition model using the XGBoost architecture, its initialized hyperparameters and their meanings are shown in Table 1 below.

[0182] Table 1 - Initial Behavior Recognition Model Parameter Table

[0183]

[0184]

[0185] The number of categories determines the range of behavioral patterns that the initial behavior recognition model can identify, and the specific value can be determined according to the requirements.

[0186] 2. The training device inputs the sample data into the initial behavior recognition model to obtain the sample behavior pattern.

[0187] The sample data in step 2 can be any one of at least one set of sample data.

[0188] 3. The training device determines the loss value based on the sample behavior patterns and the corresponding sample behavior labels of the sample data.

[0189] The loss value can be obtained based on any feasible loss function. For example, the mean squared error (MSE) is used, which calculates the Euclidean distance between the predicted value (predictive control decision) and the true value (sample control decision). This application does not impose specific restrictions on this.

[0190] 4. Iteratively update the initial behavior recognition model based on the loss value to obtain the behavior recognition model.

[0191] This step can specifically involve updating the weight and bias parameters in the initial behavior model based on the loss value. After each update, steps 2-4 are repeated until the loss value is less than a preset threshold. The sample data input to the initial behavior recognition model can be different each time this process is repeated. The initial behavior recognition model with a loss value less than the preset threshold is then used as the trained behavior recognition model. The preset threshold can be derived empirically, and it can be assumed that when the loss value is less than the preset threshold, the behavior recognition model's error in recognizing behavior patterns is within an acceptable range.

[0192] In some embodiments, the training device can train the behavior recognition model in Train mode. In Train mode, a five-fold cross-validation algorithm can be used to train and validate the behavior recognition model. Specifically, the five-fold cross-validation algorithm divides the training data into five parts: four parts are used as the training set for training, and one part is used as the validation set to validate the trained behavior recognition model. That is, after the training device obtains at least one set of sample data and the corresponding sample behavior labels, it divides the at least one set of sample data and its corresponding sample behavior labels into five parts. Four parts are used as the training set for training using steps 1-4 as described above, and the other part is used to validate the trained behavior recognition results.

[0193] After one validation, the sample data is shuffled again and divided into five parts. Four parts are selected from these five parts as the training set for training, and the remaining part is used as the validation set for validation. This process is repeated multiple times. If the validation results meet expectations, the trained behavior recognition model is stored using the SaveModel mode. In the five-fold cross-validation algorithm, the shuffling order is determined by a pre-set shuffle order. However, in this application, considering the heterogeneity of different electronic devices and the spatiotemporal differences of the sample data, Shuffle is set to False in the five-fold cross-validation. That is, the dataset (i.e., at least one set of sample data) is not shuffled, but training and validation are performed according to the input order of the dataset. In this case, the four-fold training data and the single-fold validation data will have significant device heterogeneity and spatiotemporal differences, allowing for more accurate observation of the model's generalization ability.

[0194] Based on the technical solutions corresponding to S701 and S702 described above, a behavior recognition model can be trained using supervised learning. This behavior recognition model then has the ability to determine behavior patterns using Wi-Fi features and sensor features. The behavior recognition method provided in subsequent embodiments can then utilize this behavior recognition model to identify user behavior patterns.

[0195] It should be noted that in the above embodiments and subsequent embodiments, the relevant explanation is based on the example of the behavior recognition model having the ability to determine at least two behavior modes, namely, leveling status and going up and down elevators, using Wi-Fi features and sensor features.

[0196] In practice, this behavior recognition model can identify five behavior patterns using Wi-Fi and sensor features. These five behavior patterns include: walking on a level floor, remaining stationary on a level floor, using an elevator, using an escalator, and using stairs. The aforementioned level floor state can include both walking and remaining stationary. If the behavior recognition model needs to recognize these five behavior patterns, during the training process, when acquiring sample data and sample behavior labels, it will acquire sample data and labels corresponding to these five behavior patterns, while adaptively adjusting other relevant content, thereby training a behavior recognition model capable of recognizing these five behavior patterns. Of course, the behavior recognition model used in subsequent embodiments will also have the ability to recognize these five behavior patterns. Furthermore, in this case, if the behavior pattern corresponding to a certain set of behavior data (or behavior feature) in the target behavior recognition result in a subsequent embodiment is a level floor state, then this level floor state can specifically be walking on a level floor or remaining stationary on a level floor. This situation will not be further elaborated upon later.

[0197] The behavior recognition method provided in the embodiments of this application will now be described in conjunction with the accompanying drawings.

[0198] This application provides a behavior recognition method that can be applied to, for example... Figure 1 In the electronic device of the behavior recognition system, refer to Figure 9 As shown, the process of this behavior recognition method may include S901-S919:

[0199] S901, The electronic device acquires sensor data sets and Wi-Fi data sets within a preset time period.

[0200] When the behavior recognition application in the electronic device is activated, the electronic device can begin to implement the behavior recognition method (specifically, it can be implemented by the behavior recognition application). That is, S901 can begin to be executed.

[0201] The sensor dataset includes all sensor data from the electronic device within a preset time period, while the Wi-Fi dataset includes all Wi-Fi data from the electronic device within a preset time period.

[0202] Specifically, the sensor data may include sensor data from a nine-axis sensor in an electronic device. The nine-axis sensor may include a three-axis gyroscope sensor, a three-axis magnetometer sensor, and a three-axis accelerometer sensor. The sensor data may include: gyroscope data detected by the three-axis gyroscope sensor, geomagnetic data detected by the three-axis magnetometer sensor, and acceleration data detected by the three-axis accelerometer sensor.

[0203] In this embodiment of the application, the above-mentioned S901 can be performed by, for example Figure 5 The data layer implementation is shown.

[0204] S902, the electronic device preprocesses the sensor dataset and Wi-Fi dataset to obtain the dataset of undetermined behavior.

[0205] Data preprocessing may include data alignment and data completion. Data alignment and data completion can be performed as described in the preceding embodiments. Figure 5 The relevant content in the description will not be repeated here.

[0206] Typically, within a preset time period, a user may exhibit multiple behavioral patterns, with varying durations for each pattern, and similarities in Wi-Fi and sensor data may exist across different patterns. If features extracted from Wi-Fi and sensor data at each instant are used as behavioral characteristics and input into the behavior recognition model, errors in the model's judgment will occur. Therefore, the behavioral characteristics input into the behavior recognition model should be those extracted from Wi-Fi and sensor data within a single time period. This necessitates the implementation of step S902.

[0207] In addition, referring to the relevant description of the behavior recognition model generation method in the foregoing embodiments, if the behavior recognition model is to be used normally, S902 above also needs to be implemented to ensure that the behavior recognition model can correctly recognize behavior patterns.

[0208] In this embodiment of the application, S902 can be performed by, for example Figure 5 The data processing layer implementation shown

[0209] S903. The electronic device samples the target behavior dataset according to a preset sampling method to obtain multiple sets of behavior data.

[0210] The time duration corresponding to each set of behavioral data is a unit duration. Each set of behavioral data includes Wi-Fi data and sensor data within the corresponding time period. The paired Wi-Fi data and sensor data in each set of behavioral data can exist in chronological order. The size and reason for the unit duration can be found in the relevant descriptions following S7012 in the aforementioned implementation, and will not be repeated here.

[0211] In practice, when a user is in a certain behavioral pattern, there may be moments when they perform actions outside that pattern, causing anomalies in the sensor data at those moments. For example, when a user is in the behavior pattern of going up or down an elevator, swinging their arm causes a significant change in acceleration data. To minimize the risk of behavioral features derived from these special actions being input into the behavior recognition model and leading to inaccurate results, two sets of behavioral data obtained from adjacent sampling may contain some identical data. This way, even if the behavioral pattern represented by the feature extraction from one set of behavioral data is incorrect, the behavioral pattern represented by the feature extraction from subsequent sets of behavioral data is likely to be correct. Consequently, the behavior recognition results and landmark point recognition results obtained by inputting behavioral features derived from at least one set of behavioral data into the behavior recognition model will have fewer errors.

[0212] Based on the above description, the preset sampling method here can be sliding window sampling. For example, the length of the sliding window in the feature parameters of sliding window sampling can be a unit time duration, and the sliding step size of the sliding window can be half of the unit time duration. The implementation of sliding window sampling in chronological order can be as follows: Figure 8 As shown in (a), the specific description can be found after S7012 in the aforementioned embodiment. Figure 8 The expression.

[0213] Referring to the correlation ratio description after S7012 in the aforementioned embodiment, the unit duration can be approximately the duration of a brief acceleration after the elevator starts or a brief deceleration before the elevator stops.

[0214] In this embodiment of the application, S903 can be derived from the foregoing embodiments as follows: Figure 5 The data processing layer implementation is shown.

[0215] S904. Electronic devices extract features from multiple sets of behavioral data to obtain multiple sets of behavioral features.

[0216] In this system, there is a one-to-one correspondence between behavioral features and behavioral data; that is, each set of behavioral data is derived from a set of behavioral features through feature extraction. Each set of behavioral features includes Wi-Fi characteristics and sensor characteristics within the time period to which the behavioral features belong. Feature extraction of behavioral data can be performed in any feasible way, such as extracting features based on a feature extraction model or using a specific feature extraction algorithm.

[0217] In this embodiment of the application, S903 can be derived from the foregoing embodiments as follows: Figure 5 The data processing layer implementation is shown.

[0218] S905, electronic equipment, obtains target behavior recognition results and target landmark point recognition results based on multiple sets of behavioral features and behavior recognition models.

[0219] For example, S905 may involve an electronic device inputting multiple sets of behavioral features into a behavior recognition model to obtain target behavior recognition results and target landmark recognition results. The target behavior recognition results may include the behavior pattern corresponding to each set of behavioral features. Specifically, the behavior pattern corresponding to each set of behavioral features refers to the user's behavior pattern within the time period to which that set of behavioral features belongs. The target landmark recognition results include the time when the behavior pattern changes within a preset duration, or the start point (or start time) and end point (or end time) of each behavior pattern. Here, the behavior recognition model is the behavior recognition model obtained by the behavior recognition model generation method provided in the aforementioned embodiments. This behavior recognition model has the ability to determine behavior patterns using Wi-Fi features and sensor features.

[0220] For example, using multiple sets of behavioral features, including 10 sets of behavioral features, the behavior recognition model can identify behavioral patterns including leveling and escalator / elevator movement, where 0 represents leveling and 2 represents escalator / elevator movement. The recognition results of the behavior recognition model for these 10 sets of behavioral features can be as follows: Figure 10 As shown, the time sequence is 0000222200. Therefore, the target behavior recognition result is 0000222200, and the target landmark recognition result is as follows: the starting point of the first leveling state is the start time of the time period corresponding to the first group of behavioral features; the ending point of the first leveling state is the end time of the time period corresponding to the fourth group of behavioral features; the starting point of going up and down the elevator is the start time of the time period corresponding to the fifth group of behavioral features; the ending point of going up and down the elevator is the end time of the time period corresponding to the eighth group of behavioral features; the starting point of the second leveling state is the start time of the time period corresponding to the ninth group of behavioral features; and the ending point of the second leveling state is the end time of the time period corresponding to the tenth group of behavioral features.

[0221] In this embodiment of the application, S905 can be derived from the foregoing embodiments as follows: Figure 5 The behavior recognition module in the core algorithm layer shown is implemented.

[0222] In some embodiments, when sampling a specific behavior dataset using a sliding window sampling method, if the preset duration is not an integer multiple of the sliding window length, there may be a set of behavior data with a small amount of data. The behavior features corresponding to this set of behavior data may lead to errors in the recognition results input into the behavior recognition model. Therefore, to avoid this error and to make the user behavior recognition results and landmark point recognition results obtained by the electronic device within the preset duration more accurate, the electronic device can use a sliding window sampling method to first sample the specific behavior dataset in chronological order to obtain at least one set of positive behavior data, and then use a sliding window sampling method to first sample the specific behavior dataset in reverse chronological order to obtain at least one set of negative behavior data.

[0223] The implementation of the sliding window sampling method, which samples in chronological order, can be as described in the previous embodiments. Figure 8 The relevant statement in (a) is as follows. For example, taking a unit duration of 2.56 seconds and a preset duration of 10 seconds as an example, refer to... Figure 8 As shown in (b), when using a sliding window to sample in reverse time, the sliding window initially falls within the 7.44-10s range of the 10s dataset. This range encompasses the data from 7.44-10s within the dataset of undetermined behaviors, and is used as the first group of behavior data. Next, the sliding window is moved one step backward (1.28s) in reverse time, at which point it encompasses the data from 6.16-8.72s within the dataset of undetermined behaviors, and is used as the second group of behavior data. Subsequently, the sliding window is moved one step backward in time, resulting in the following groups: 4.88-7.44s as the third group of behavior data, 3.6-6.16s as the fourth group, 2.32-4.88s as the fifth group, 1.04-3.6s as the sixth group, and 0-2.32s as the seventh group.

[0224] Next, feature extraction is performed on at least one set of positive behavior data and at least one set of negative behavior data to obtain at least one set of positive behavior features and at least one set of negative behavior features. Then, at least one set of positive behavior features is input into the behavior recognition model to obtain the first behavior recognition result for the user within a preset time period; at least one set of negative behavior features is input into the behavior recognition model to obtain the second behavior recognition result for the user within a preset time period. Then, the first behavior recognition result and the second behavior recognition result are fused using a specific fusion rule to obtain the target behavior recognition result, and the target landmark point recognition result is determined based on this target behavior recognition result. Specifically, when fusing the first behavior recognition result and the second behavior recognition result, the recognition results of each set of behavior features included in both are sorted in chronological order before fusion. In other words, if the recognition results of each group of behavioral features included in the first behavior recognition result are ABCD after being sorted in chronological order, and the recognition results of each group of behavioral features included in the second behavior recognition result are EFGH after being sorted in chronological order, then the target behavior recognition result in chronological order includes: the fusion result of A and E, the fusion result of B and F, the fusion result of C and G, and the fusion result of D and H.

[0225] For example, assuming at least one set of positive behavioral features and at least one set of negative behavioral features each include 10 sets of behavioral features, the behavior recognition model can identify behavioral patterns including leveling and elevator / ascending / descending states, where 0 represents leveling and 2 represents elevator / ascending / descending states. The recognition results of the behavior recognition model for these 10 sets of positive behavioral features can be as follows: Figure 11 As shown in (a), the sequence in time is 0002222000. Therefore, the positive behavior recognition result is 0002222000. The behavior recognition model's recognition result for these 10 sets of negative behavior features can be seen as follows: Figure 11 As shown in (b), the time sequence is 0000222000. Therefore, the reverse behavior recognition result is 0000222000. If the fusion rule is: if two recognition results with the same order both indicate an elevator going up or down, the fusion result is elevator going up or down; if two recognition results with the same order are different, the fusion result is level ground. Then the target behavior recognition result can be as follows: Figure 11As shown in (c), specifically 0000222000. Therefore, the target landmark identification results are as follows: the starting point of the first leveling state is the start time of the time period corresponding to the first group of behavioral features; the ending point of the first leveling state is the end time of the time period corresponding to the fourth group of behavioral features; the starting point of going up and down the elevator is the start time of the time period corresponding to the fifth group of behavioral features; the ending point of going up and down the elevator is the end time of the time period corresponding to the seventh group of behavioral features; the starting point of the second leveling state is the start time of the time period corresponding to the eighth group of behavioral features; and the ending point of the second leveling state is the end time of the time period corresponding to the tenth group of behavioral features.

[0226] Based on the above statements, combined with Figure 9 , refer to Figure 12 As shown, S903 may include S9031 and S9032, S904 may include S904A, and S905 may include S9051-S9053.

[0227] S9031. Electronic devices use sliding window sampling to sample the dataset of the behavior to be determined in chronological order to obtain at least one set of positive behavior data.

[0228] S9032. Electronic devices use sliding window sampling to sample the dataset of the behavior to be determined in reverse time order to obtain at least one set of reverse behavior data.

[0229] S904A, the electronic device performs feature extraction on at least one set of positive behavior data and at least one set of negative behavior data to obtain at least one set of positive behavior features and at least one set of negative behavior features.

[0230] S9051. The electronic device inputs at least one set of positive behavioral features into the behavior recognition model to obtain positive behavior recognition results.

[0231] Among them, the positive behavior recognition results include the behavior patterns corresponding to each of the positive behavior features in at least one set of positive behavior features.

[0232] S9052. The electronic device inputs at least one set of reverse behavior features into the behavior recognition model to obtain the reverse behavior recognition result.

[0233] The reverse behavior recognition result includes the behavior pattern corresponding to each set of reverse behavior features in at least one set of reverse behavior features.

[0234] S9053. The electronic device fuses the forward behavior recognition result and the reverse behavior recognition result according to the preset fusion rules to obtain the target behavior recognition result, and obtains the target landmark point recognition result based on the target behavior recognition.

[0235] In this case, the behavioral features corresponding to the behavioral patterns included in the target behavior recognition results can be understood as behavioral features obtained by fusing positive and negative behavioral features (but in reality, no fusion is performed).

[0236] Based on such Figure 12 The scheme shown can utilize behavior recognition models to obtain more accurate target behavior recognition results and target landmark point recognition results.

[0237] After obtaining the target behavior recognition results and target landmark point recognition results using the behavior recognition model, the accuracy of the recognition results may be insufficient considering the limited types of sensors the behavior recognition model targets. In reality, when users walk, due to the obvious periodicity of human walking, they generally do not experience small-amplitude weightlessness (i.e., the equivalent gravitational acceleration is within the preset range) for more than one continuous unit of time. However, when the human body is in the behavior pattern of going up and down an elevator, it is actually in a static state most of the time. The movement of the person following the elevator's movement exhibits a motion state of small-amplitude weightlessness – constant speed movement – ​​small-amplitude weightlessness (weightlessness). This special state is very difficult to occur in other daily human behaviors, especially to maintain a continuous small-amplitude weightlessness for 2-4 seconds (i.e., the duration of acceleration after the elevator starts or deceleration before the elevator stops). Therefore, by determining the proportion of the continuous first acceleration data corresponding to small-amplitude weightlessness in each set of behavior data to all acceleration data, it is possible to accurately identify whether a person is on an elevator, and thus correct the target behavior recognition results and target landmark point recognition results determined by the behavior recognition model. Based on this, this application also designs a behavior pattern determination process based on acceleration, namely the subsequent S906-S913.

[0238] S906. The electronic device determines the first proportion of the number of first acceleration data in each group of behavior data to the total number of all acceleration data in that group of behavior data based on the acceleration data in each group of behavior data.

[0239] In each set of behavioral data, for any given first acceleration data point, there exists another first acceleration data point within that set that is continuous with it. In other words, any first acceleration data point in each set of behavioral data is at least temporally continuous with other first acceleration data points. Temporal continuity between two first acceleration data points means that the acquisition times of these two first acceleration data points are consecutive, and there are no other acceleration data acquisition times between their acquisition times.

[0240] Alternatively, the content of S906 above can be described as follows: The electronic device determines the first proportion of each set of behavioral data based on the acceleration data in each set of behavioral data in multiple sets of behavioral data; the first proportion is the first proportion of the number of first acceleration data in the target behavioral data to the total number of all acceleration data in the target behavioral data; the equivalent gravitational acceleration of the first acceleration data is within a preset range, and for any first acceleration data in the target behavioral data, there is another first acceleration data that is continuous with any first acceleration data; the target behavioral data is the set of behavioral data to which the first proportion belongs.

[0241] Specifically, electronic devices can determine the equivalent gravitational acceleration based on the acceleration in the direction of gravity in the acceleration data.

[0242] In this embodiment of the application, the equivalent gravitational acceleration being within a preset range can be referred to as slight weightlessness.

[0243] In practice, when a user does not move within an elevator while going up or down, there are upper and lower thresholds for the user's weightlessness or weight gain. These thresholds refer to the upper and lower limits of weightlessness and weight gain during elevator startup (or during elevator stop). Therefore, this can be used to distinguish the user from other behavioral patterns such as leveling. Experiments have shown that the equivalent gravitational acceleration corresponding to weightlessness or weight gain during elevator startup is slightly larger (smaller) than the equivalent gravitational acceleration at rest on a level floor; however, it is slightly smaller (larger) than the equivalent gravitational acceleration in a leveling state. Through extensive data statistics and analysis, a reasonable threshold is considered to be an equivalent gravitational acceleration of 10.05 for the lower limit of weightlessness, 11.3 for the upper limit, 8.9 for the lower limit, and 9.5 for the upper limit.

[0244] In other words, in this application, for example, the preset range includes a preset overweight range and a preset weightlessness range. The preset overweight range is (10.05, 11.3), and the preset weightlessness range is (8.9, 9.5). When the equivalent gravitational acceleration of a certain acceleration data belongs to (10.05, 11.3) or (8.9, 9.5), then that acceleration data is used as the candidate first acceleration data. If there are multiple candidate first acceleration data in its behavioral data, and they are consecutive in time sequence, then all of the multiple candidate first acceleration data are determined as first acceleration data.

[0245] In this embodiment of the application, S906 can be derived from the foregoing embodiments as follows: Figure 5 The elevator correction module in the core algorithm layer shown is implemented.

[0246] S907. Electronic devices determine whether there are multiple sets of behavioral data where the proportion of the first behavior data in multiple consecutive sets is greater than a preset proportion.

[0247] For example, the preset percentage can be 2.7%. Continuous multiple sets of first behavioral data refer to multiple sets of behavioral data that are consecutive in time sequence. That is, if a sliding window sampling method is used to sample the dataset of behaviors to be performed in time sequence, then the multiple sets of behavioral data obtained from multiple consecutive samplings are considered continuous behavioral data. In S907, the multiple sets of first behavioral data contain two or more sets of values. Multiple sets of first hypergravity behavioral data may exist that are not consecutive; the first hypergravity behavioral data is a combination of multiple consecutive sets of first behavioral data. The multiple sets of first behavioral data contain two or more sets.

[0248] If the proportion of a certain set of behavioral data exceeds a preset proportion, it can be assumed that within a unit of time corresponding to this set of behavioral data, the user's equivalent gravitational acceleration within a certain continuous time period conforms to the characteristics of the behavior pattern of going up and down elevators. In other words, this set of behavioral data can be considered to be behavioral data belonging to the behavior pattern of going up and down elevators with a high probability, and the user's behavior pattern within the unit of time corresponding to this set of behavioral data can also be considered to be belonging to the behavior pattern of going up and down elevators with a high probability.

[0249] Furthermore, it's understandable that the acceleration or deceleration process of an elevator during a single run lasts 2-4 seconds, and the unit duration corresponding to a set of sample data falls within this range. If a user performs actions that affect their equivalent gravitational acceleration (such as raising the arm or walking) within a unit duration of this behavior pattern while the user is at a level, the electronic device will use the first percentage corresponding to that behavior data to determine that the corresponding behavior pattern is going up or down the elevator. However, this determination is biased. Therefore, to more accurately determine the sample data corresponding to elevator acceleration or deceleration within the behavior pattern of going up or down the elevator, only when the first percentage of multiple consecutive sets of first sample data is greater than a preset percentage can it be more probable to conclude that the user's behavior pattern within the time period corresponding to those multiple sets of first sample data is going up or down the elevator.

[0250] In other words, if the electronic device determines that in multiple sets of behavioral data, the proportion of the first behavior data in each consecutive set is greater than a preset proportion, it continues to judge the user's behavior pattern within the time period corresponding to the multiple sets of first behavior data, i.e., executes S909. If the electronic device determines that in multiple sets of behavioral data, there is no situation where the proportion of the first behavior data in each consecutive set is greater than a preset proportion, it determines the user's behavior pattern within the time period (i.e., preset duration) corresponding to all sets of behavioral data as a flat state, i.e., executes S908.

[0251] It should be noted that in practice, the above-mentioned judgment step S907 may not be necessary. The electronic device can execute S909 if it determines whether there are multiple sets of first behavior data in which the first proportion is greater than the preset proportion in consecutive sets of first behavior data; and execute S908 if it determines whether there are multiple sets of behavior data in which the first proportion is greater than the preset proportion in consecutive sets of first behavior data.

[0252] It should be noted that because acceleration data cannot determine whether a state is level or whether one is going up or down escalators or stairs, behavior patterns that are definitely not going up or down stairs are defined as level states.

[0253] Since actual user actions are uncertain, this preset percentage can be derived from a large amount of data, for example, 2.7%. Taking a preset percentage of 2.7% and a set of 100 acceleration data points as an example, if five consecutive acceleration data points in this set of data all correspond to equivalent gravitational accelerations within the preset range, then these five acceleration data points are designated as the first acceleration data. In this case, the first acceleration data accounts for 5% of all acceleration data, which is greater than the preset percentage of 2.7%.

[0254] S908. The electronic device determines that the behavior pattern corresponding to each set of behavior data in multiple sets of behavior data is a flat state.

[0255] S908 is followed by S913.

[0256] For example, refer to Figure 13 As shown, Figure 13 This demonstrates the change in equivalent gravitational acceleration during the operation of an elevator in reality. The vertical axis represents equivalent gravitational acceleration, and the horizontal axis represents time. The two horizontal lines at the top of the waveform represent the upper and lower limits of slight overload, while the two horizontal lines at the bottom represent the upper and lower limits of slight weightlessness. A clear pattern emerges: slight overload and weightlessness generally occur in pairs; that is, after a slight overload, a slight weightlessness immediately follows. Based on this, after S907, if the electronic device determines whether multiple sets of behavioral data have a first proportion greater than a preset proportion in each consecutive set of first behavioral data, and continues to judge the user's behavior pattern within the time period corresponding to those multiple sets of first behavioral data, it can determine whether the overload and weightlessness occur in pairs to identify whether certain behavioral data corresponds to the user's behavior pattern of going up and down the elevator within the time period. This leads to the execution of subsequent steps S909-S913.

[0257] S909. The electronic device determines whether the equivalent gravitational acceleration of the first acceleration data in the first weightlessness behavior data belongs to the preset weightlessness range.

[0258] The first hypergravity behavior data is a collection of multiple consecutive sets of first behavior data. The electronic device determines whether the first acceleration data in the first hypergravity behavior data belongs to a preset hypergravity range, that is, it determines whether the equivalent gravitational acceleration of all consecutive first acceleration data in the first hypergravity behavior data that are within the preset range belongs to a small hypergravity range or a small weightlessness range.

[0259] If the electronic device determines that the equivalent gravitational acceleration of the first acceleration data in the first hypergravity behavior data belongs to the preset hypergravity range, then the electronic device can determine the first hypergravity behavior data as the first hypergravity behavior data, that is, execute S910; if the electronic device determines that the equivalent gravitational acceleration of the first acceleration data in the first hypergravity behavior data does not belong to the preset hypergravity range, then the electronic device can determine the first hypergravity behavior data as the first hypergravity behavior data, that is, execute S911.

[0260] It should be noted that in practice, the above-mentioned judgment step S909 may not be necessary. The electronic device can execute S910 if it determines that the equivalent gravitational acceleration of the first acceleration data in the first hypergravity behavior data is within the preset hypergravity range; and execute S911 if it determines that the equivalent gravitational acceleration of the first acceleration data in the first hypergravity behavior data is not within the preset hypergravity range.

[0261] Furthermore, because the aforementioned sampling of behavioral data already considered the duration of each behavioral data point and the degree of overlap between two adjacent behavioral data points in time sequence, in the first weightlessness behavioral data composed of multiple sets of first behavioral data, all the first acceleration data points are highly likely to belong to the same range, that is, either all belong to the preset weightlessness range or all belong to the preset weightlessness range. If, in the first weightlessness behavioral data, all the first acceleration data points do not belong to the same range, then the range corresponding to the more frequent first angular velocity data points can be determined as the range to which the equivalent gravitational acceleration of the first acceleration data points in the first weightlessness behavioral data belongs. For example, if in a certain first weightlessness behavioral data point, 6 first acceleration data points belong to the preset weightlessness range and 2 first acceleration data points belong to the preset weightlessness range, then the equivalent gravitational acceleration of the first acceleration data points in the first weightlessness behavioral data is considered to belong to the preset weightlessness range.

[0262] S910, The electronic device determines the first hypergravity behavior data from the first hypergravity behavior data.

[0263] S911, The electronic device determines the first weightlessness behavior data from the first hypergravity behavior data.

[0264] S912. The electronic device determines the behavior pattern corresponding to each set of behavior data in multiple sets of behavior data within a preset time period based on the first overweight behavior data and the first weightlessness behavior data.

[0265] Among them, the behavioral patterns corresponding to the behavioral data are the user's behavioral patterns within the time period to which the behavioral data belongs.

[0266] Once the electronic device has determined the first hypergravity behavior data and the first weightlessness behavior data in the first hypergravity behavior data, it can determine the behavior pattern corresponding to each set of behavior data based on whether the first hypergravity behavior data and the first weightlessness behavior data are paired.

[0267] S912 is followed by S913.

[0268] For example, in combination Figure 9 , refer to Figure 14 As shown, S912 may specifically include S9121 and S9122:

[0269] S9121. If there is a pair of second overweight behavior data and second weightlessness behavior data, the electronic device will determine the behavior pattern corresponding to the behavior data in the second overweight behavior data, the behavior data in the second weightlessness behavior data, and the behavior data between the second overweight behavior data and the second weightlessness behavior data in time sequence as going up and down the elevator.

[0270] In terms of time sequence, there is no third overweight or third weightlessness behavior data between the second and second overweight behavior data. The second and third overweight behavior data belong to the first overweight behavior data, and the second and third weightlessness behavior data belong to the first weightlessness behavior data.

[0271] For example, refer to Figure 15 As shown in Figure (a), there are 10 sets of behavioral data within a preset time period, ordered chronologically as follows: A, B, C, D, E, F, G, H, I, J. A and B are both first-order behavioral data with a percentage greater than a preset percentage, D and E are both first-order behavioral data with a percentage greater than a preset percentage, AB combination forms the first overweight behavior data, and DE combination forms the first weightlessness behavior data. Therefore, the user behavior pattern within the time period corresponding to A, B, C, D, and E can be identified as going up and down elevators.

[0272] S9122. If there exists a fourth overweight behavior data that is not paired with any first weightlessness behavior data, then the electronic system will determine the behavior pattern corresponding to the behavior data in the fourth overweight behavior data as an error.

[0273] Among them, the fourth overweight behavior data belongs to the first overweight behavior data. The fourth overweight behavior data is not paired with any first weightlessness behavior data specifically means that, in all behavior data, the next first weightlessness behavior data after the fourth overweight behavior data is specifically the first overweight behavior data in chronological order, or there is no next first weightlessness behavior data after the fourth overweight behavior data.

[0274] If a certain first overweight behavior data point (i.e., the fourth overweight behavior data point) does not pair with any other first weightlessness behavior data point, it indicates that the user experienced a slight overweight state within the time period corresponding to this fourth overweight behavior data point, but without a corresponding slight weightlessness state. In other words, the user may have accelerated in the opposite direction of gravity in the elevator, but without a corresponding deceleration process. Therefore, it cannot be accurately determined whether the user was in an elevator-going-up / down behavior pattern within the time period corresponding to this fourth overweight behavior data point, nor can it be determined whether the user was in a level position. Therefore, it can be considered that the behavior pattern corresponding to this fourth overweight behavior data point is incorrect.

[0275] For example, refer to Figure 15 As shown in (b), there are 10 sets of behavioral data within a preset time period, ordered chronologically as follows: A, B, C, D, E, F, G, H, I, J. A and B are both first-time behavioral data with a percentage greater than a preset percentage, D and E are both first-time behavioral data with a percentage greater than a preset percentage, the combination AB forms the first overweight behavioral data, and the combination DE also forms the first overweight behavioral data. Therefore, the user's behavioral pattern within the time period corresponding to A, B, D, and E can be classified as incorrect.

[0276] S9123. If there is a fourth weightlessness behavior data that is not paired with any first weightlessness behavior data, then the electronic system will determine the behavior pattern corresponding to the behavior data in the fourth weightlessness behavior data as an error.

[0277] The fourth weightlessness behavior data belongs to the first weightlessness behavior data. Similar to S9122, if a certain first weightlessness behavior data (i.e., the fourth weightlessness behavior data) does not pair with any other first weightlessness behavior data, it indicates that the user experienced a slight weightlessness during the time period corresponding to the fourth weightlessness behavior data, but without a corresponding slight weightlessness. In other words, the user may have accelerated in the direction of gravity in the elevator, but without a corresponding deceleration. Therefore, it cannot be accurately determined whether the user was in an elevator-going-up / down behavior pattern during the time period corresponding to the fourth weightlessness behavior data, nor can it be determined whether the user was in a level position. Therefore, it can be considered that the behavior pattern corresponding to the behavior data in the fourth weightlessness behavior data is incorrect.

[0278] S9124. The electronic device determines the behavior pattern corresponding to the behavior data other than elevator behavior data and error behavior data in multiple sets of behavior data as the leveling state.

[0279] Among them, the elevator behavior data corresponds to the behavior pattern of going up and down the elevator, and the error behavior data corresponds to the behavior pattern of error.

[0280] S913. The electronic device determines the behavior pattern corresponding to all behavior data within the set time period as the corrected recognition result.

[0281] In this embodiment, S906-S913 can be derived from the foregoing embodiments as follows: Figure 5 The elevator correction module in the core algorithm layer shown is implemented.

[0282] It should be noted that S906-S913 can be executed simultaneously with S904-S905, or before S904, after S905, or after S904 and before S905, as long as S905 and S913 have been completed before S914. This application does not impose specific restrictions in this regard.

[0283] For example, S906-S913 described above can be derived from, as follows: Figure 5 The elevator correction module in the core algorithm layer shown is implemented.

[0284] Based on the technical solutions corresponding to S906-S913 described above, the electronic device can determine the behavior pattern corresponding to each set of behavior data within a preset time period based on the acceleration data. This provides data support for the correction of subsequent target behavior recognition results and target landmark point recognition results.

[0285] The electronic device uses a behavior recognition model to obtain target behavior recognition results and target landmark point recognition results, and uses acceleration data to identify the behavior patterns corresponding to the behavior data within a preset time period, determining the behavior pattern corresponding to each group of behavior data within the preset time period. Then, the electronic device can use the correction recognition results determined by acceleration to correct the target behavior recognition results, and determine the corrected target landmark point recognition results based on the corrected target behavior recognition results, thus obtaining the corrected target behavior recognition results and the corrected landmark point recognition results. That is, after S913, refer to... Figure 9 As shown, the behavior recognition method also includes S914-S919:

[0286] S914. The electronic device corrects the behavior pattern corresponding to the selectable behavior data in the corrected identification result to going up and down the elevator, so as to obtain an updated corrected identification result.

[0287] Among them, the optional behavioral data are behavioral data between two sets of behavioral data whose corresponding behavioral patterns are going up and down elevators and which are not consecutive in time sequence.

[0288] The significance of S914 lies in the fact that, in reality, elevators may stop at certain floors to wait for passengers. Therefore, if two behavioral data points corresponding to the elevator's up and down behavior patterns are not consecutive in time, the behavioral patterns corresponding to the behavioral data between them should also be considered as elevator up and down. Of course, if all behavioral data in the corrected identification result correspond to the same behavior pattern as the leveling state, or if there is no other behavioral data between the two behavioral data points corresponding to the elevator up and down behavior patterns, then the updated corrected identification result is the same as the corrected identification result.

[0289] In this embodiment of the application, the above-mentioned S914 can be performed by, for example Figure 5 The elevator correction module in the core algorithm layer shown is implemented.

[0290] For example, taking 4 to represent the level state and 2 to represent the elevator going up or down, if the target behavior recognition result includes six behavioral features corresponding to behavioral patterns in chronological order as 422422, then the updated and corrected recognition result obtained after executing the above S914 includes six behavioral data corresponding to behavioral patterns in chronological order as 422222.

[0291] S915. If all behavioral patterns corresponding to behavioral features in the target behavior recognition result are in a level state, or if there is a behavioral pattern corresponding to a behavioral feature in the target behavior recognition result that is going up or down an elevator, and the behavioral pattern corresponding to the second behavioral data in the updated and corrected recognition result is going up or down an elevator, and the behavioral pattern corresponding to the first behavioral feature in the target behavior recognition result is not going up or down an elevator, then the electronic device will correct the behavioral pattern corresponding to the first behavioral feature in the target behavior recognition result to going up or down an elevator.

[0292] The second behavioral data is one set of behavioral data among multiple sets of behavioral data, and the first behavioral feature is one set of behavioral features among multiple sets of behavioral features; the order of the first behavioral feature after sorting the multiple sets of behavioral features in chronological order is the same as the order of the second behavioral data after sorting the multiple sets of behavioral data in chronological order.

[0293] For example, let 0 represent a level floor and 2 represent an elevator going up or down. Figure 16As shown, if the target behavior recognition result includes six behavioral features whose corresponding behavioral patterns are 002200 in chronological order, and the updated and corrected recognition result includes six behavioral data whose corresponding behavioral patterns are 022200 in chronological order, then the electronic device will correct the behavioral pattern corresponding to the second behavioral feature in the target behavior recognition result from 0 to 2. The corrected target behavior recognition result can then be 022200.

[0294] Of course, if the behavior pattern corresponding to the second behavior data in the updated and corrected recognition result is going up and down the elevator, and the behavior pattern corresponding to the first behavior feature in the target behavior recognition result is also going up and down the elevator, then the electronic device will not correct the behavior pattern corresponding to the first behavior feature in the target behavior recognition.

[0295] S916. If all behavioral features in the target behavior recognition result correspond to a level state, or if there is a behavioral feature in the target behavior recognition result that corresponds to an elevator ride, and the behavioral pattern of the third behavioral data in the updated and corrected recognition result is a level state, and the behavioral pattern of the second behavioral feature in the target behavior recognition result is not a level state, then the electronic device will correct the behavioral pattern of the second behavioral feature in the target behavior recognition result to a level state.

[0296] Among them, the third behavioral data is one set of behavioral data in multiple sets of behavioral data, and the second behavioral feature is one set of behavioral features in multiple sets of behavioral features; the order of the second behavioral feature after sorting the multiple sets of behavioral features in chronological order is the same as the order of the third behavioral data after sorting the multiple sets of behavioral data in chronological order.

[0297] For example, let 0 represent a level floor and 2 represent an elevator going up or down. Figure 17 As shown, if the target behavior recognition result includes six behavioral features whose corresponding behavioral patterns are 002220 in chronological order, and the updated and corrected recognition result includes six behavioral data whose corresponding behavioral patterns are 000220 in chronological order, then the electronic device will correct the behavioral pattern corresponding to the third behavioral feature in the target behavior recognition result from 2 to 0. The corrected target behavior recognition result can then be 000220.

[0298] Of course, if the behavior pattern of the corresponding third behavior data in the updated and corrected recognition result is a flat state, and the behavior pattern corresponding to the second behavior feature in the target behavior recognition result is a flat state, then the electronic device will not correct the behavior pattern corresponding to the second behavior feature in the target behavior recognition.

[0299] S917. If all behavioral patterns corresponding to behavioral features in the target behavior recognition result are in a level state, or if there is a behavioral pattern corresponding to a behavioral feature in the target behavior recognition result that is going up or down an elevator, and if the behavioral pattern corresponding to the fourth behavioral data in the updated and corrected recognition result is incorrect, the electronic device will not correct the behavioral pattern corresponding to the third behavioral feature in the target behavior recognition result.

[0300] Among them, the fourth behavior data is one set of behavior data in multiple sets of behavior data, and the third behavior feature is one set of behavior features in multiple sets of behavior features; the order of the third behavior feature in the multiple sets of behavior features after being sorted in chronological order is the same as the order of the fourth behavior data in the multiple sets of behavior data after being sorted in chronological order.

[0301] S918. If the behavior pattern corresponding to all behavioral features in the target behavior recognition result is neither a level state nor an elevator up or down state, the target behavior recognition result shall not be corrected.

[0302] S919, The electronic device uses the corrected target behavior recognition result to determine the corrected target landmark point recognition result.

[0303] Specifically, S919 uses the target behavior recognition result to determine the start and end points of the user's behavior pattern within a preset time period (or the time point of behavior pattern change), thereby obtaining the corrected target landmark recognition result.

[0304] For example, S914-S919 described above can be derived from, as follows: Figure 5 The elevator correction module in the core algorithm layer shown is implemented.

[0305] Based on the technical solution provided in this application, an electronic device can first use a pre-trained behavior recognition model to identify (or predict) the user's behavior pattern within a preset time period. This behavior recognition model is pre-trained and can determine the user's behavior pattern based on sensor features and Wi-Fi features. Since the behavior recognition model relies on relatively little sensor data when judging behavior patterns, to prevent misjudgment, after the behavior recognition model outputs the behavior recognition result, the electronic device can also determine whether it is in an elevator-like state based on acceleration data from the sensor data. Because there are significant differences in the user's weightlessness data in non-elevator modes (leveling, escalator / stairs / climbing) and elevator / stairs modes, the electronic device can determine the user's behavior pattern within the preset time period based on the weightlessness status of the electronic device, i.e., correct the recognition result. Then, based on this corrected recognition result, the target behavior recognition result and target landmark point recognition result of the behavior recognition model can be corrected, thereby obtaining a more accurate behavior pattern recognition result.

[0306] In some embodiments, if the specific implementation of S903-S905 in the behavior recognition method provided in this application is as follows: Figure 12 As shown, then combined Figure 9 , refer to Figure 12 As shown, S906-S913 can specifically be S906A-S913A:

[0307] S906A, the electronic device determines, based on the acceleration data in each group of positive behavior data, the second proportion of the number of second acceleration data in each group of first behavior data where the effective gravitational acceleration is within a preset range and continuous, relative to the total number of all acceleration data.

[0308] Of course, positive behavior data here can also be negative behavior data.

[0309] S907A: The electronic device determines whether at least one set of positive behavior data exists in multiple consecutive sets of first positive behavior data where the proportion of the first behavior data is greater than a preset proportion.

[0310] If the electronic device determines that at least one set of positive behavior data has a first proportion greater than a preset proportion in multiple consecutive sets of first positive behavior data, then execute S909A; if the electronic device determines that at least one set of positive behavior data does not have a first proportion greater than a preset proportion in multiple consecutive sets of first positive behavior data, then execute S908A.

[0311] S908A: The electronic device determines that the user's behavior pattern is in a flat state within a preset time period.

[0312] S909A, the electronic device determines whether the equivalent gravitational acceleration of the first acceleration data in the first weightlessness behavior data belongs to the preset weightlessness range.

[0313] Here, the first hypergravity behavior data is a combination of multiple consecutive sets of first positive behavior data.

[0314] If the electronic device determines that the equivalent gravitational acceleration of the first acceleration data in the first hypergravity behavior data belongs to the preset hypergravity range, then the electronic device can determine the first hypergravity behavior data as the first hypergravity behavior data, that is, execute S910A; if the electronic device determines that the equivalent gravitational acceleration of the first acceleration data in the first hypergravity behavior data does not belong to the preset hypergravity range, then the electronic device can determine the first hypergravity behavior data as the first hypergravity behavior data, that is, execute S911A.

[0315] S910A, the electronic device determines the first hypergravity behavior data from the first hypergravity behavior data.

[0316] S911A, the electronic device determines the first weightlessness behavior data from the first hypergravity behavior data.

[0317] S912A, the electronic device, based on the first overweight behavior data and the first weightlessness behavior data, sets a behavior pattern corresponding to each set of behavior data within a preset time period.

[0318] S913A: The electronic device determines the behavior pattern corresponding to all positive behavior data within a set time period as the corrected recognition result.

[0319] The specific implementations of S906A-S913A can refer to the implementation methods of S906-S913 mentioned above, and will not be repeated here.

[0320] It should be noted that S906A-S913A uses acceleration data from the forward behavior data to determine the correction and identification result. In practice, it can also use acceleration data from both forward and / or reverse behavior data to determine the correction and identification result. If acceleration data from the reverse behavior data is used to determine the correction and identification result, the specific implementation can refer to S906A-S913A above. If acceleration data from both forward and reverse behavior data is used to determine the correction and identification result, the acceleration data from the forward behavior data can be used to determine one correction and identification result, and the acceleration data from the reverse behavior data can be used to determine another correction and identification result, and then the two correction and identification results are fused. The specific implementation principle can refer to the technical solutions corresponding to S9031, S9032, S904A and S9051-S9053 in the aforementioned embodiments, and will not be repeated here.

[0321] In such Figure 12 In the specific example of the behavior recognition method shown, before determining the target behavior recognition result, the behavior recognition model will obtain the positive behavior recognition result obtained by sampling behavior data in chronological order, and the reverse behavior recognition result obtained by sampling behavior data in reverse chronological order.

[0322] If, in the final target behavior recognition result, all behavioral features correspond to a level-floor state in the behavioral pattern, and simultaneously, if the updated and corrected recognition result shows some behavioral data corresponding to a false behavioral pattern, while the remaining behavioral data correspond to a level-floor state (representing only the level-floor state), then it indicates that the behavioral data acquired by the electronic device is somewhat special, and some behavioral data may actually correspond to the behavior pattern of going up or down an elevator. Based on this, in situations such as... Figure 12 Based on the example shown, refer to Figure 12 As shown, the method also includes S918A:

[0323] S918A. If the behavior patterns corresponding to all the behavior features included in the target behavior recognition result are in a flat state, and the behavior pattern corresponding to the fifth behavior data in the update and correction recognition result is incorrect, and the behavior patterns corresponding to the behavior data other than the fifth behavior data in the update and correction recognition result are in a flat state, then the electronic device uses the positive behavior recognition result or the reverse behavior recognition result to correct the target behavior recognition result.

[0324] S918A and later electronic devices can correct the target landmark recognition results based on the corrected target behavior recognition results.

[0325] Among them, using positive behavior recognition results or negative behavior recognition results to correct the target behavior recognition results can be used to determine the positive behavior recognition results or negative behavior recognition results as the corrected target behavior recognition results.

[0326] In one possible implementation, whether the electronic device uses the forward behavior recognition result or the reverse behavior recognition result to correct the target behavior recognition result can be determined based on the first number of elevator rides in the behavior patterns corresponding to all sample features in both the forward and reverse behavior recognition results. If the first number corresponding to the forward behavior recognition result is greater, the electronic device uses the forward behavior recognition result to correct the target behavior recognition result; if the first number corresponding to the reverse behavior recognition result is greater, the electronic device uses the reverse behavior recognition result to correct the target behavior recognition result.

[0327] For example, let 0 represent a level floor, 2 represent going up or down the elevator, and F represent an error. See below. Figure 18As shown, if the target behavior recognition result includes six behavioral patterns corresponding to six behavioral features arranged in chronological order as 000000, the updated and corrected recognition result includes six behavioral patterns corresponding to six behavioral data arranged in chronological order as 00FF00, the forward behavior recognition result includes six behavioral patterns corresponding to six behavioral features arranged in chronological order as 000200, and the reverse behavior recognition result includes six behavioral patterns corresponding to six behavioral features arranged in chronological order as 002200, then the electronic device can use the reverse behavior recognition result to correct the target behavior recognition result. The corrected target behavior recognition result is the reverse behavior recognition result, i.e., 002200. Based on the corrected target behavior recognition results, the corrected landmark point recognition results can include: the starting point of the first leveling state is the starting time of the time period corresponding to the first group of behavioral features, the ending point of the first leveling state is the ending time of the time period corresponding to the second group of behavioral features, the starting point of going up and down the elevator is the starting time of the time period corresponding to the third group of behavioral features, the ending point of going up and down the elevator is the ending time of the time period corresponding to the fourth group of behavioral features, the starting point of the second leveling state is the starting time of the time period corresponding to the fifth group of behavioral features, and the ending point of the second leveling state is the ending time of the time period corresponding to the sixth group of behavioral features.

[0328] To more clearly illustrate the process by which electronic devices correct target behavior recognition results using updated and corrected recognition results, the following section will combine... Figure 12 The process by which the electronic device corrects the target behavior recognition results using the corrected recognition results is explained. (Refer to...) Figure 19 As shown, the process is as follows:

[0329] 1. First, the target behavior recognition results and correction recognition results can be obtained.

[0330] 2. Connect the multiple elevator segments in the corrected identification results to obtain updated corrected identification results.

[0331] In this process, multiple elevator segments in the correction and identification results are connected. Specifically, among all the behavioral patterns corresponding to the behavioral data included in the correction and identification results, the behavioral patterns corresponding to different behavioral data that correspond to the elevator going up and down are determined to be elevator going up and down in chronological order.

[0332] 3. After the electronic device obtains the target behavior recognition results and the updated and corrected recognition results of the behavior recognition model, it will first determine whether all the target behavior recognition results are in a flat state.

[0333] The phrase "all behavior patterns in the target behavior recognition results are in a flat state" specifically means that all behavior patterns in the target behavior recognition results are in a flat state.

[0334] If all target behavior recognition results are in a flat state, it is necessary to determine whether the target behavior recognition results are correct. Specifically, this needs to be determined in conjunction with the updated and corrected recognition results, i.e., proceed to step 4; if not all target behavior recognition results are in a flat state, proceed to step 9.

[0335] 4. Determine whether the elevator was not identified and whether the identification was incorrect in the updated and corrected recognition results.

[0336] The update and correction of the recognition results did not identify the elevator going up and down and the recognition was incorrect. Specifically, this means that among all the behavioral patterns corresponding to the behavioral data in the updated recognition results, there is a second behavioral data whose behavioral pattern is False, and the behavioral patterns corresponding to the other behavioral data are all in the level state.

[0337] If the updated and corrected recognition results do not identify elevators going up and down and are incorrectly identified, it indicates that there may be some behavioral data corresponding to the behavior pattern of going up and down elevators. It is incorrect for the target behavior recognition results to show only level states. In this case, it is necessary to use the forward recognition results or the reverse recognition results to correct the target behavior recognition results, i.e., execute step 5. If the updated and corrected recognition results identify elevators going up and down or are incorrectly identified, it is necessary to combine whether elevators going up and down are identified in the updated and corrected recognition results to decide whether to correct the target behavior recognition results, i.e., execute step 6.

[0338] 5. Correct the target behavior recognition results using positive or negative recognition results.

[0339] If the elevator was not identified during the updated and corrected recognition results, and an error was detected, it can be assumed that the target behavior recognition result obtained by fusing the forward and reverse recognition results is likely erroneous. Furthermore, this error may be caused by the fusion process; therefore, either the forward or reverse recognition results should be used to correct the target recognition.

[0340] Specifically, using the positive or negative recognition results to correct the target recognition means using the positive or negative recognition results as the corrected target behavior recognition result.

[0341] 6. Determine whether the elevator was identified in the updated and corrected recognition results.

[0342] The updated and corrected identification results identified going up and down elevators. Specifically, this means that among all the behavioral data in the updated and corrected identification results, there are some behavioral data that correspond to the behavioral pattern of going up and down elevators.

[0343] If the updated and corrected identification results identify elevators going up and down, it can be considered that there is an error in the target behavior identification results showing all level states. In this case, the updated and corrected identification results need to be used to correct it, i.e., execute 7; if the updated and corrected identification results do not identify elevators going up and down, it indicates that the updated and corrected identification results also show all level states. In other words, the target behavior identification results showing all level states are likely correct. In this case, no correction is needed, i.e., execute 8.

[0344] 7. Use the updated and corrected identification results to correct the target behavior identification results, and obtain the corrected target behavior identification results.

[0345] Specifically, using the updated and corrected identification results to correct the target behavior identification results refers to determining the updated and corrected identification results as the corrected target behavior identification results.

[0346] The implementation of the 7 steps can be as follows: If the behavior pattern corresponding to the second behavior data in the updated and corrected recognition result is going up and down the elevator, and the behavior pattern corresponding to the first behavior feature in the target behavior recognition result is not going up and down the elevator, then the electronic device will correct the behavior pattern corresponding to the first behavior feature in the target behavior recognition result to going up and down the elevator.

[0347] The second behavioral data is one set of behavioral data among multiple sets of behavioral data, and the first behavioral feature is one set of behavioral features among multiple sets of behavioral features; the order of the first behavioral feature after sorting the multiple sets of behavioral features in chronological order is the same as the order of the second behavioral data after sorting the multiple sets of behavioral data in chronological order.

[0348] 8. Do not correct the target behavior recognition results.

[0349] The fact that the updated and corrected recognition results did not identify either the elevator going up or down, nor did they detect any errors, indicates that the updated and corrected recognition results identified all behavioral patterns corresponding to all behavioral data as a leveling state. At this point, the target behavior recognition result also identified all behavioral patterns corresponding to all behavioral features as a leveling state. This means that the updated and corrected recognition results cannot correct the target behavior recognition result, and it is impossible to determine whether there are any recognition errors in the target behavior recognition result. Therefore, no correction is needed for the target behavior recognition result here.

[0350] 9. Determine whether the target behavior recognition results include going up or down an elevator.

[0351] Specifically, the inclusion of "going up and down elevators" in the target behavior recognition results means that among all the behavioral features included in the target behavior recognition results, there are certain behavioral patterns that correspond to the behavior of going up and down elevators.

[0352] If the target behavior recognition result includes going up and down elevators, then it is necessary to determine whether to correct it based on the specific situation in the updated and corrected recognition result, i.e., proceed to step 10; if the target behavior recognition result does not include going up and down elevators, it indicates that the target behavior recognition result contains behavior patterns such as going up and down escalators and stairs, and it is not necessary to use the updated and corrected recognition result for correction, because the updated and corrected recognition result is obtained using acceleration data, and does not make judgments on behavior patterns such as going up and down escalators and stairs. i.e., proceed to step 8.

[0353] 10. Determine whether there is an elevator going up or down in the updated and corrected identification results.

[0354] Specifically, "the presence of elevators in the updated and corrected identification results" means that among all the behavioral data included in the updated and corrected identification results, there are certain behavioral patterns that correspond to the behavior of going up and down elevators.

[0355] If the updated and corrected identification results contain information about elevators going up and down, then the updated and corrected identification results can be used to correct the target behavior identification results, i.e., execute step 11; if the updated and corrected identification results do not contain information about elevators going up and down, then it is necessary to determine whether the updated and corrected identification results are all in a level state, and then decide whether to make corrections, i.e., execute step 12.

[0356] 11. Use the updated and corrected identification results to correct the target behavior identification results.

[0357] Specifically, using updated and corrected recognition results to correct the target behavior recognition results can be done in the following ways:

[0358] If the behavior pattern corresponding to the second behavior data in the updated and corrected recognition result is "going up and down the elevator", and the behavior pattern corresponding to the first behavior feature in the target behavior recognition result is not "going up and down the elevator", then the electronic device will correct the behavior pattern corresponding to the first behavior feature in the target behavior recognition result to "going up and down the elevator". Here, the second behavior data is a set of behavior data among multiple sets of behavior data, and the first behavior feature is a set of behavior features among multiple sets of behavior features. The order of the first behavior feature in the multiple sets of behavior features after sorting in chronological order is the same as the order of the second behavior data in the multiple sets of behavior data after sorting in chronological order.

[0359] If the behavior pattern of the corresponding third behavior data in the updated and corrected recognition result is a flat state, and the behavior pattern corresponding to the second behavior feature in the target behavior recognition result is not a flat state, then the electronic device will correct the behavior pattern corresponding to the second behavior feature in the target behavior recognition result to a flat state; wherein, the third behavior data is a set of behavior data among multiple sets of behavior data, and the second behavior feature is a set of behavior features among multiple sets of behavior features; the order of the second behavior feature in the multiple sets of behavior features after being sorted in chronological order is the same as the order of the third behavior data in the multiple sets of behavior data after being sorted in chronological order.

[0360] If the behavior pattern corresponding to the fourth behavior data in the updated and corrected recognition result is incorrect, the electronic device will not correct the behavior pattern corresponding to the third behavior feature in the target behavior recognition result; wherein, the fourth behavior data is a set of behavior data in multiple sets of behavior data, and the third behavior feature is a set of behavior features in multiple sets of behavior features; the order of the third behavior feature in the multiple sets of behavior features after being sorted in chronological order is the same as the order of the fourth behavior data in the multiple sets of behavior data after being sorted in chronological order.

[0361] 12. Determine whether all the updated and corrected identification results are in a flat state.

[0362] Specifically, "all flat states in the updated and corrected identification results" means that all behavioral patterns corresponding to all behavioral data included in the updated and corrected identification results are in a flat state.

[0363] If the updated and corrected identification results are all level states, and since the target behavior identification results also include elevators going up and down, the result of the updated and corrected identification module is considered correct, and the updated and corrected identification results can be used as the corrected target behavior identification results, i.e., execute 13; if the updated and corrected identification results are not all level states, it indicates that the updated and corrected identification results contain both level states and errors, while the target behavior identification results contain both level states and elevators going up and down, then the target behavior identification results are considered correct, and no correction is needed, i.e., execute 8.

[0364] 13. Use the updated correction results to correct the target behavior recognition results.

[0365] Specifically, step 13 can be used to correct non-layered behavior patterns in the behavior patterns corresponding to all behavior features in the target behavior recognition results to a layered state. That is, if the behavior pattern corresponding to the third behavior data in the recognition results is updated and corrected to a layered state, and the behavior pattern corresponding to the first behavior feature in the target behavior recognition results is not a layered state, then the behavior pattern corresponding to the second behavior feature in the target behavior recognition results is corrected to a layered state. Here, the third behavior data is one set of behavior data from multiple sets of behavior data, and the second behavior feature is one set of behavior features from multiple sets of behavior features; the order of the second behavior feature in the multiple sets of behavior features after being sorted chronologically is the same as the order of the third behavior data in the multiple sets of behavior data after being sorted chronologically.

[0366] In addition, in the actual use of the behavior recognition method provided in the above embodiments, parameters such as accuracy, precision, recall, and harmonic mean fl-score can be used to evaluate the accuracy of the behavior recognition method in recognizing behavior patterns.

[0367] For example, under extensive data verification testing, the technical solutions provided in this application achieve an average recognition accuracy of approximately 95% in behavior recognition using two-class classification (stationary leveling and walking leveling are classified as leveling movement behaviors, while escalator and elevator use are classified as cross-floor behaviors). For three-class classification (stationary leveling and walking leveling are classified as leveling movement behaviors, while escalator and elevator use are separate categories), the average recognition accuracy is approximately 94%. For four-class classification (stationary leveling, walking leveling, escalator and elevator use are separate categories), the recognition accuracy for stationary leveling can reach over 93%, for walking leveling over 97%, for escalator use over 92%, and for elevator use over 97%.

[0368] In terms of landmark recognition, the average delay between the start and end times of all predicted landmarks and the actual landmarks can be controlled within 4 seconds. The recognition accuracy of all landmarks (the percentage of landmarks with recognition errors ≤ 3 sets of behavioral data out of the total number of landmarks) can reach approximately 90%, and the recognition accuracy of all landmarks (the percentage of landmarks with recognition errors ≤ 5 sets of behavioral data out of the total number of landmarks) can reach approximately 95%. Furthermore, for...

[0369] As can be seen, the technical solution provided in this application embodiment can more accurately identify user behavior patterns.

[0370] It is understood that, in order to achieve the aforementioned functions, the electronic device includes corresponding hardware structures and / or software modules for performing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, the embodiments of the present invention can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in a hardware-driven or software-driven manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of the embodiments of this application.

[0371] This application embodiment can divide the above-described electronic device or training device into functional modules according to the above method example. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division; in actual implementation, there may be other division methods.

[0372] This application also provides an electronic device, which includes a memory and one or more processors; the memory is coupled to the processors; wherein the memory stores computer program code, which includes computer instructions, and when the computer instructions are executed by the processor, the electronic device performs the behavior recognition method provided in the foregoing embodiments. The specific structure of this electronic device can be referred to... Figure 3 The structure of the electronic device shown is illustrated.

[0373] This application also provides a training device, which includes a processor and a memory. The memory stores executable instructions, and the processor is configured to execute the executable instructions stored in the memory, so that the training device performs the behavior recognition model generation method provided in the above embodiments. The specific structure of the training device can be found in [reference needed]. Figure 6 The structure of the training device shown is illustrated.

[0374] This application also provides a chip system, such as... Figure 20 As shown, the chip system 1100 includes at least one processor 1101 and at least one interface circuit 1102. The processor 1101 and the interface circuit 1102 are interconnected via lines. For example, the interface circuit 1102 can be used to receive signals from other devices (e.g., the memory of an electronic device). As another example, the interface circuit 1102 can be used to send signals to other devices (e.g., the processor 1101).

[0375] For example, interface circuit 1102 can read instructions stored in memory and send those instructions to processor 1101. When the instructions are executed by processor 1101, the electronic device can perform the various steps in the above embodiments. Of course, the chip system may also include other discrete devices, and this application embodiment does not specifically limit this.

[0376] This application also provides a computer-readable storage medium that includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the behavior recognition method provided in the foregoing embodiments.

[0377] This application also provides a computer program product containing executable instructions that, when run on an electronic device, cause the electronic device to perform the behavior recognition method provided in the foregoing embodiments.

[0378] This application also provides a computer-readable storage medium including computer instructions that, when executed on a training device, cause the training device to perform the behavior recognition model generation method provided in the foregoing embodiments.

[0379] This application also provides a computer program product containing executable instructions that, when run on a training device, cause the training device to execute the behavior recognition model generation method provided in the foregoing embodiments.

[0380] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0381] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0382] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0383] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0384] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0385] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A behavior recognition method applied to electronic devices, characterized in that, The method includes: The electronic device acquires multiple sets of behavioral data within a preset time period; wherein, the time length corresponding to each set of behavioral data is a unit time period, and any set of behavioral data includes the Wi-Fi data of the electronic device and the sensor data of the nine-axis sensor in the electronic device within the time period corresponding to the set of behavioral data; the multiple sets of behavioral data include all Wi-Fi data of the electronic device and all sensor data of the nine-axis sensor in the electronic device within the preset time period; the sensor data includes: gyroscope sensor data, magnetometer sensor data, and accelerometer sensor data; the accelerometer sensor data includes acceleration data; The electronic device extracts features from the multiple sets of behavioral data to obtain multiple sets of behavioral features; the behavioral features include Wi-Fi features and sensor features. The electronic device obtains target behavior recognition results and target landmark recognition results based on the multiple sets of behavioral features and the behavior recognition model. The target behavior recognition results include the behavior pattern corresponding to each set of behavioral features. The target landmark recognition results include the time when each behavior pattern changes within a preset time period, or the start and end times of each behavior pattern. The behavior recognition model has the ability to determine behavior patterns using Wi-Fi features and sensor features. The behavior patterns corresponding to the behavioral features include leveling or going up and down elevators. The electronic device determines the correction and recognition result based on the acceleration data in each of the multiple sets of behavioral data; the correction and recognition result includes the behavioral pattern corresponding to each set of behavioral data; the behavioral pattern corresponding to the behavioral data includes leveling or going up and down the elevator; The electronic device corrects the target behavior recognition result based on the corrected recognition result; The electronic device uses the corrected target behavior recognition result to determine the corrected target landmark point recognition result.

2. The method according to claim 1, characterized in that, The electronic device acquires multiple sets of behavioral data within a preset time period, including: The electronic device acquires sensor datasets and Wi-Fi datasets within a preset time period; the sensor datasets include all sensor data from the nine-axis sensor of the electronic device within the preset time period; the Wi-Fi datasets include all Wi-Fi data from the electronic device within the preset time period. The electronic device performs data preprocessing on the sensor dataset and Wi-Fi dataset to obtain a dataset of behaviors to be determined; the data preprocessing includes data alignment and data completion. The electronic device samples the dataset of behaviors to be determined according to a preset sampling method to obtain the multiple sets of behavior data.

3. The method according to claim 2, characterized in that, The preset sampling method includes sliding window sampling; the length of the sliding window in the sliding window sampling is the unit time duration, and the sliding step size of the sliding window sampling is half of the unit time duration.

4. The method according to any one of claims 1-3, characterized in that, The electronic device determines the correction and recognition result based on the acceleration data in each of the multiple sets of behavioral data, including: The electronic device determines a first proportion of each set of behavioral data based on the acceleration data in each set of behavioral data. The first proportion is the percentage of the number of first acceleration data in the target behavioral data out of the total number of acceleration data in the target behavioral data. The equivalent gravitational acceleration of the first acceleration data is within a preset range, and for any first acceleration data in the target behavioral data, there is another first acceleration data that is continuous with any first acceleration data. The target behavioral data is the behavioral data to which the first proportion belongs. The preset range includes a preset overweight range and a preset weightlessness range. When the proportion of multiple sets of first behavior data is greater than a preset proportion, the multiple sets of first behavior data are determined as first weightlessness behavior data. And when the equivalent gravitational acceleration of the first acceleration data in the first weightlessness behavior data is within the preset weightlessness range, the first weightlessness behavior data is determined as first weightlessness behavior data. When the proportion of multiple sets of first behavior data is greater than a preset proportion, the multiple sets of first behavior data are determined as first hyperweightlessness behavior data. And when the equivalent gravitational acceleration of the first acceleration data in the first hyperweightlessness behavior data is within the preset weightlessness range, the first hyperweightlessness behavior data is determined as first weightlessness behavior data. The electronic device determines the behavior pattern corresponding to each group of behavior data in the plurality of groups of behavior data based on the first overweight behavior data and the first weightlessness behavior data; The electronic device determines the behavioral pattern corresponding to each set of behavioral data in the multiple sets of behavioral data as the correction and recognition result.

5. The method according to claim 4, characterized in that, After the electronic device determines the first proportion of each set of behavioral data based on the acceleration data in each set of behavioral data from the plurality of sets of behavioral data, the method further includes: When the proportion of multiple sets of first behavior data in multiple sets of behavior data is not greater than a preset proportion, the electronic device determines the behavior pattern corresponding to each set of behavior data in the multiple sets of behavior data as a flat state. The electronic device determines the behavioral pattern corresponding to each set of behavioral data in the multiple sets of behavioral data as the correction and recognition result.

6. The method according to claim 5, characterized in that, The electronic device determines the behavioral pattern corresponding to each set of behavioral data in the plurality of sets of behavioral data based on the first overweight behavior data and the first weightlessness behavior data, including: If there is a pair of second overweight behavior data and second weightlessness behavior data, the electronic device determines the behavior pattern corresponding to the behavior data in the second overweight behavior data, the behavior data in the second weightlessness behavior data, and the behavior data between the second overweight behavior data and the second weightlessness behavior data in time sequence as going up and down the elevator. The pairing of the second overweight behavior data and the second weightlessness behavior data means that, in the multiple sets of behavior data, there is no third overweight behavior data or third weightlessness behavior data between the second overweight behavior data and the second weightlessness behavior data in chronological order; the second overweight behavior data and the third overweight behavior data belong to the first overweight behavior data, and the second weightlessness behavior data and the third weightlessness behavior data belong to the first weightlessness behavior data. If there exists a fourth overweight behavior data that is not paired with any of the first weightlessness behavior data, then the electronic system will determine the behavior pattern corresponding to the behavior data in the fourth overweight behavior data as incorrect; the fourth overweight behavior data belongs to the first overweight behavior data. If there exists a fourth weightlessness behavior data that is not paired with any of the first weightlessness behavior data, then the electronic system determines the behavior pattern corresponding to the behavior data in the fourth weightlessness behavior data as incorrect; the fourth weightlessness behavior data belongs to the first weightlessness behavior data. The electronic device determines the behavior pattern corresponding to the behavior data other than elevator behavior data and error behavior data in the multiple sets of behavior data as a leveling state; the behavior pattern corresponding to the elevator behavior data is going up or down the elevator, and the behavior pattern corresponding to the error behavior data is an error.

7. The method according to claim 1, characterized in that, The electronic device corrects the target behavior recognition result based on the corrected recognition result, including: The electronic device corrects the behavior pattern corresponding to the optional behavior data in the corrected recognition result to going up and down the elevator, so as to obtain an updated corrected recognition result; wherein, among the multiple sets of optional behavior data, the behavior pattern corresponding to going up and down the elevator is the behavior data between two behavior data that are not consecutive in time sequence. If all behavioral features in the target behavior recognition result correspond to a level state, or if there is a behavioral feature in the target behavior recognition result that corresponds to an elevator up / down behavior, and the behavioral pattern corresponding to the second behavioral data in the updated and corrected recognition result is an elevator up / down behavior, but the behavioral pattern corresponding to the first behavioral feature in the target behavior recognition result is not an elevator up / down behavior, then the electronic device will correct the behavioral pattern corresponding to the first behavioral feature in the target behavior recognition result to an elevator up / down behavior. Here, the second behavioral data is one set of behavioral data from the multiple sets of behavioral data, and the first behavioral feature is one set of behavioral features from the multiple sets of behavioral features. The order of the first behavioral features after sorting the multiple sets of behavioral features in chronological order is the same as the order of the second behavioral data after sorting the multiple sets of behavioral data in chronological order. If all behavioral features in the target behavior recognition result correspond to a level state, or if there is a behavioral feature in the target behavior recognition result that corresponds to an elevator ride, and if the behavioral pattern of the third behavioral data in the updated and corrected recognition result is a level state, but the behavioral pattern of the second behavioral feature in the target behavior recognition result is not a level state, then the electronic device will correct the behavioral pattern of the second behavioral feature in the target behavior recognition result to a level state; wherein, the third behavioral data is one set of behavioral data among the multiple sets of behavioral data, and the second behavioral feature is one set of behavioral features among the multiple sets of behavioral features; the order of the second behavioral feature after sorting the multiple sets of behavioral features in chronological order is the same as the order of the third behavioral data after sorting the multiple sets of behavioral data in chronological order.

8. The method according to claim 3, characterized in that, The electronic device samples the dataset of behaviors to be determined according to a preset sampling method to obtain the multiple sets of behavioral data, including: The electronic device uses the sliding window sampling to sample the dataset of behaviors to be determined in chronological order to obtain at least one set of positive behavior data; The electronic device uses the sliding window sampling to sample the dataset of undetermined behaviors in reverse chronological order to obtain at least one set of reverse behavior data; The electronic device extracts features from the multiple sets of behavioral data to obtain multiple sets of behavioral features, including: The electronic device performs feature extraction on at least one set of positive behavior data to obtain at least one set of positive behavior features, and performs feature extraction on at least one set of negative behavior data to obtain at least one set of negative behavior features; The electronic device, based on the multiple sets of behavioral features and behavioral recognition models, obtains target behavior recognition results and target landmark point recognition results, including: The electronic device inputs the at least one set of positive behavioral features into the behavior recognition model to obtain a positive behavior recognition result; the positive behavior recognition result includes the behavior pattern corresponding to each set of positive behavioral features in the at least one set of positive behavioral features; The electronic device inputs the at least one set of reverse behavior features into the behavior recognition model to obtain a reverse behavior recognition result; the reverse behavior recognition result includes the behavior pattern corresponding to each set of reverse behavior features in the at least one set of reverse behavior features; The electronic device fuses the forward behavior recognition result and the reverse behavior recognition result according to a preset fusion rule to obtain the target behavior recognition result, and obtains the target landmark point recognition result based on the target behavior recognition.

9. The method according to claim 8, characterized in that, The electronic device corrects the target behavior recognition result based on the corrected recognition result, including: If all behavioral features in the target behavior recognition result correspond to a flat state, and the behavior pattern corresponding to the fifth behavioral data in the updated and corrected recognition result is incorrect, and the behavior patterns corresponding to all behavioral data except the fifth behavioral data in the updated and corrected recognition result are all flat states, then the electronic device uses the positive behavior recognition result or the reverse behavior recognition result to correct the target behavior recognition result.

10. An electronic device, characterized in that, The device includes a memory and one or more processors; the memory is coupled to the processors; wherein the memory stores computer program code, the computer program code including computer instructions, which, when executed by the processor, cause the electronic device to perform the behavior recognition method as described in any one of claims 1-9.

11. A computer-readable storage medium, characterized in that, Includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the behavior recognition method as described in any one of claims 1-9.

12. A behavior recognition system, characterized in that, Including the electronic device as described in claim 10.

Images