Activity recognition method, display method, and electronic device

Abstract

Embodiments of the present application disclose an activity recognition method, a display method and an electronic device. The electronic device can obtain a plurality of poses of the electronic device, and then obtain one or more feature data from the plurality of poses using a plurality of one-dimensional convolution layers, each of the one or more feature data indicating a motion state of a person holding the electronic device, an action of the person or a tool carried by the person, and then the electronic device displays the type of information indicated by the one or more feature data in the form of an icon. According to the embodiments of the present application, the one-dimensional convolution layer extracts feature data representing the activity state of the person, which can extract the feature representing the activity state of the person from multiple dimensions, so that the representation of the activity state of the person is more detailed and accurate. In addition, the activity recognition module in the electronic device of the embodiments of the present application can interact with the map function, so that the electronic device can adaptively and in real time present the activity state of the person on the map interface.

Description

Technical Field

[0001] This application relates to the field of terminals, and more particularly to an activity recognition (AR) method, a display method, and an electronic device. Background Technology

[0002] With the development of positioning technologies (such as those relied upon by the Global Positioning System (GPS)) and navigation technologies (such as those relied upon by the BeiDou Navigation Satellite System), mobile terminals can provide users with location and navigation services based on map functions. Currently, mobile terminals can only provide location-related services such as positioning, routes, and navigation based on map functions, resulting in a poor user experience. Summary of the Invention

[0003] This application provides an activity recognition method, a display method, and an electronic device that can recognize the activity status of a person and display icons representing the activity status of a person on a map interface.

[0004] In a first aspect, embodiments of this application provide an activity recognition method, comprising: obtaining multiple poses of an electronic device within a time period; using multiple one-dimensional convolutional layers to obtain one or more feature data based on the multiple poses, wherein the one or more feature data respectively indicate different types of information of a person holding the electronic device within the time period, and the type of information indicated by each feature data is the person's motion state, the person's action, or the tool carrying the person; and controlling the electronic device to display the type of information indicated by the one or more feature data in the form of icons.

[0005] This method is used to identify the activity state of a person holding an electronic device. The activity state of the person holding the electronic device can be represented by at least one type of information: the person's motion state, the person's actions, and the tool carrying the person. The person's motion state refers to the state of the person relative to a reference frame when performing mechanical movement. The person's motion state includes: stationary, walking, and running. The person's actions include: standing, sitting, squatting, and lying down. The tool carrying the person includes: the vehicle the person is riding (e.g., a car, high-speed rail), an elevator, and an escalator. The pose of the electronic device can reflect the activity state of the person holding the electronic device. Based on this, the electronic device acquires multiple poses of the electronic device within a time period. Then, an activity recognition (AR) module is used to extract one or more feature data from the multiple poses. These one or more feature data respectively indicate different types of information of the person holding the electronic device. Furthermore, the electronic device controls itself to display the types of information indicated by the one or more feature data in the form of icons. In this embodiment, the "time period" can be preset, for example, 5 seconds (s). In addition, the AR module performs feature extraction on multiple poses based on multiple one-dimensional convolutional layers to obtain one or more feature data.

[0006] As can be seen, the embodiments of this application perform feature extraction through multiple one-dimensional convolutional layers. This not only overcomes the limitation of relying solely on the single dimension of a person's motion state during feature extraction model training, resulting in a highly scalable feature extraction model, but also improves the accuracy of the identified activity states because the trained feature extraction model can recognize activity states from multiple dimensions. Furthermore, the electronic device of the embodiments of this application can adaptively and in real-time present the activity states of a person, thereby optimizing the visualization effect of the electronic device and improving the user experience.

[0007] In one possible design, obtaining one or more feature data based on the multiple poses using multiple one-dimensional convolutional layers includes: obtaining a set of feature data based on the multiple poses using the multiple one-dimensional convolutional layers; and obtaining the one or more feature data from the set of feature data. Specifically, the electronic device can use multiple one-dimensional convolutional layers to extract features from multiple poses to obtain a set of feature data representing a person's activity state. Then, the electronic device can extract feature data representing each specific type from the set of feature data to obtain the specific activity of the person. This enables the identification of a person's activity state from multiple dimensions, thereby improving the accuracy of the identified activity state.

[0008] In one possible design, obtaining feature data sets based on multiple poses using the multiple one-dimensional convolutional layers includes: performing feature extraction on the multiple poses using m sequentially combined one-dimensional convolutional layers in a first feature extraction framework to obtain an initial feature data set; and performing feature extraction on the initial feature data set using n sequentially combined backbone models in a second feature extraction framework to obtain the feature data set, where each of the n backbone models contains at least two parallelly combined one-dimensional convolutional layers; where m and n are both positive integers. The AR module includes a first feature extraction framework and a second feature extraction framework, both of which use one-dimensional convolutional layers to perform feature extraction, and different one-dimensional convolutional layers can extract different types of feature data. Therefore, in this embodiment, the AR module performs feature extraction on multiple poses using stacked one-dimensional convolutional layers, which not only extracts multiple types of feature data sets of a person, but also, through feature extraction using multiple one-dimensional convolutional layers, results in higher accuracy of the extracted features for activity recognition.

[0009] In one possible design, performing feature extraction on the multiple poses using m sequentially combined one-dimensional convolutional layers in the first feature extraction framework includes: performing feature extraction on a first initial feature data set using the i-th one-dimensional convolutional layer among the m one-dimensional convolutional layers to obtain a second initial feature data set. The first initial feature data set is the feature data set obtained by the (i-1)-th one-dimensional convolutional layer among the m one-dimensional convolutional layers based on the multiple poses, where i is greater than or equal to 2 and less than or equal to m. In this embodiment, the first feature extraction framework, through sequential feature extraction of multiple poses using multiple one-dimensional convolutional layers, can obtain multiple types of feature data sets from multiple poses of an electronic device. These multiple types of feature data sets are used to characterize the activity state of a person. In actual implementation, in some embodiments, the number of channels in the one-dimensional convolutional layers of the first feature extraction framework can increase sequentially according to the data flow. In other embodiments, the number of channels in each one-dimensional convolutional layer of the first feature extraction framework can be the same, or partially the same. In still other embodiments, the number of channels in the one-dimensional convolutional layers of the first feature extraction framework can decrease sequentially.

[0010] In one possible design, the kernel values ​​of the at least two one-dimensional convolutional layers combined in parallel within each backbone model are different. Each one-dimensional convolutional layer includes a kernel (k), where the value of k indicates the number of consecutive data points from the multiple input data used in each convolution operation of that one-dimensional convolutional layer. Since the multiple input data are sampled based on the time dimension, the value of k is related to the time scale. Based on this, each backbone model can perform feature extraction on the input feature data set from different time scales, ensuring that the extracted feature data set can represent more features to cover multiple features corresponding to the same activity when either the amplitude or speed of the action differs, thereby improving recognition accuracy.

[0011] In one possible design, obtaining the one or more feature data from the feature data set includes: obtaining sub-feature data sets corresponding to each of all or some of the at least one type based on the feature data set; obtaining the feature data with the highest confidence from each of the sub-feature data sets; and using all the feature data with the highest confidence as the one or more feature data sets, where the confidence level indicates the credibility of the behavior corresponding to the feature data. The AR module also includes a classifier that can obtain sub-feature data sets corresponding to each type from the feature data set, and then obtain the feature data with the highest confidence from each sub-feature data set. The feature data with the highest confidence indicates specific information about the person for that type, such as whether the person's specific movement state is static. This allows for the acquisition of features for at least one type of person, thereby enabling the characterization of the person's activity state from multiple dimensions.

[0012] In one possible design, after obtaining the feature data with the highest confidence from each of the sub-feature data groups, the method further includes: calculating the probability that the activity state corresponding to the feature data will transition to each other activity state based on each feature data and the state transition matrix associated with the feature data, obtaining at least one probability value. The state transition matrix includes the probability of transitioning between any two activity states of the type of information indicated by the feature data. When the at least one probability value contains a value greater than a first preset threshold, the activity state data corresponding to the largest probability value among the probability values ​​greater than the first preset threshold is taken as the feature data corresponding to the type. Alternatively, when the at least one probability value does not contain a value greater than the first preset threshold, the feature data with the highest confidence is taken as the feature data corresponding to the type. In actual implementation, the activity state of a person may change at any time. Based on this, in some embodiments, for each type of person information, the electronic device pre-maintains a state transition matrix, which includes the transition probability between any two states in that type. Furthermore, the electronic device calculates the probability that the feature data with the highest confidence will transition to other states based on the feature data with the highest confidence for each type and the corresponding state transition matrix. When any of the obtained probabilities exceeds a first preset threshold, the electronic device uses the transitioned activity state data corresponding to the highest probability as feature data of that type. This feature data indicates the user's possible actual activity state. This implementation method can further improve the accuracy of electronic devices in recognizing a person's activity state.

[0013] In one possible design, controlling the electronic device to display the information indicated by one or more feature data in the form of icons includes: controlling the electronic device to display a map, and controlling at least a portion of the icons to be displayed on the map. Using this implementation, the electronic device can interactively present icons representing the map and the activity status of people, thereby optimizing the visualization of map functions and improving the user experience.

[0014] In one possible design, obtaining multiple poses of an electronic device over a time period includes: acquiring at least one set of sensor data to characterize the poses of the electronic device; detecting whether there are missing data points in each set of sensor data; performing sampling and linear interpolation on the at least one set of sensor data without missing data points to obtain multiple sensor data points; and performing filtering and gravity residual removal operations on the multiple sensor data points to obtain the multiple poses.

[0015] In one possible design, the sensor data includes measurement data from a gyroscope sensor, measurement data from an accelerometer sensor, and measurement data from a magnetometer sensor.

[0016] Secondly, embodiments of this application provide a display method, including: receiving a first operation instruction; and responding to the first operation instruction by displaying an icon on a map interface to represent the activity status of a person, wherein the activity status of the person indicates one or more of the following information: the person's movement status, the person's actions, and the tool carrying the person.

[0017] The icons involved in the embodiments of this application may include at least one of person icons and object icons. In some embodiments, the aforementioned icons are implemented as static images, and the image format includes .jpg, .tif, .img, etc. In other embodiments, the icons are implemented as dynamic images, and the image format includes .gif. In the embodiments of this application, the electronic device displays a map, and the icons are displayed on the map. It can be seen that by adopting the implementation method of the embodiments of this application, the activity status of the person and the map can be interactively presented, thereby enabling the electronic device to adaptively and in real time present the activity status of the person on the map interface, thereby optimizing the visualization effect of the map function and improving the user experience.

[0018] In one possible design, the step of displaying an icon representing a character's activity status on the map interface in response to the first operation command includes: displaying the icon in a floating window on the map interface.

[0019] In one possible design, the step of displaying an icon representing a person's activity status on the map interface in response to the first operation command includes: displaying the icon at the location corresponding to the person associated with the icon on the map interface.

[0020] Thirdly, embodiments of this application provide an electronic device, which includes a processing module, a receiving module, and a display module. The processing module includes functions for executing the first aspect, the second aspect, various possible implementations of the first aspect, and various possible implementations of the second aspect, such as feature data extraction, classification, and display control. The receiving module is used to execute the first aspect, the second aspect, various possible implementations of the first aspect, and various possible implementations of the second aspect, receiving operation instructions and sensor data. The display module is used to execute the first aspect, the second aspect, various possible implementations of the first aspect, and various possible implementations of the second aspect, such as displaying icons.

[0021] Fourthly, embodiments of this application provide an electronic device having the function of implementing one or more methods of the first aspect, the second aspect, various possible implementations of the first aspect, and various possible implementations of the second aspect. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. In one possible design, the structure of the above electronic device includes a processor, a receiver, and a display. The processor is configured to perform functions in the above methods other than information transmission and reception and display. The receiver is used to receive user operation commands. The display is used to display maps and icons. The electronic device may also include a memory coupled to the processor, which stores necessary program instructions and data of the electronic device.

[0022] Fifthly, embodiments of this application provide a computer storage medium storing instructions that, when executed on a computer, cause the computer to perform some or all of the steps of the methods in the first aspect, the second aspect, various possible implementations of the first aspect, and various possible implementations of the second aspect.

[0023] In a sixth aspect, embodiments of this application provide a computer program product that, when run on a computer, causes the computer to perform some or all of the steps of the methods in the first aspect, the second aspect, various possible implementations of the first aspect, and various possible implementations of the second aspect.

[0024] As can be seen, the AR module running on the electronic device in this embodiment extracts feature data representing the activity state of a person from the pose of the electronic device by stacking one-dimensional convolutional layers. This not only offers good scalability but also ensures that the extracted feature data indicates at least one type of person information, resulting in a more detailed and accurate representation of the person's activity state. Furthermore, the interaction between the AR module and the map function allows the electronic device to adaptively and in real-time display the person's activity state on the map interface, thereby optimizing the visualization effect of the map function and improving the user experience. Attached Figure Description

[0025] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1A A schematic diagram of an exemplary hardware structure of the electronic device 100 provided in an embodiment of this application;

[0027] Figure 1B A schematic diagram of an exemplary software architecture of an electronic device 100 provided in an embodiment of this application;

[0028] Figure 2A An exemplary method flowchart of the display method 10 provided in this application embodiment;

[0029] Figure 2B An exemplary method flowchart of the activity identification method 20 provided in this application embodiment;

[0030] Figure 2C An exemplary method flowchart of the activity identification method 200 provided in this application embodiment;

[0031] Figure 2D Provided for the embodiments of this application Figure 2C An exemplary flowchart for obtaining character activity state data based on pose sequence;

[0032] Figure 3A An exemplary schematic diagram of the main interface of a mobile phone provided in an embodiment of this application;

[0033] Figure 3B This is a first exemplary interface diagram of the map display interface provided in the embodiments of this application;

[0034] Figure 3C This is a second exemplary interface diagram of the map display interface provided in the embodiments of this application;

[0035] Figure 3D This is a third exemplary interface diagram of the map display interface provided in the embodiments of this application;

[0036] Figure 3E This is a fourth exemplary interface diagram of the map display interface provided in the embodiments of this application;

[0037] Figure 3F A fifth exemplary interface diagram of the map display interface provided in the embodiments of this application;

[0038] Figure 3G A sixth exemplary interface diagram of the map display interface provided in the embodiments of this application;

[0039] Figure 4 This is a schematic diagram of the data flow of a mobile phone provided in an embodiment of this application;

[0040] Figure 4A This is an exemplary operational flow diagram of the TSE model 40 provided in the embodiments of this application;

[0041] Figure 4BA schematic diagram illustrating the exemplary computational flow of the backbone model 400 provided in this application embodiment;

[0042] Figure 4C A schematic diagram illustrating the exemplary computational flow of the classifier 50 provided in this embodiment of the application;

[0043] Figure 5A A first exemplary interface diagram for user selection of display mode provided in an embodiment of this application;

[0044] Figure 5B This is a second exemplary interface diagram for user selection of display mode provided in an embodiment of this application;

[0045] Figure 5C This is a schematic diagram of a third exemplary interface for user selection of display mode provided in an embodiment of this application;

[0046] Figure 6 An exemplary interface diagram of the user operation interface provided in the embodiments of this application;

[0047] Figure 7A An exemplary schematic diagram of the electronic device 70 provided in this application;

[0048] Figure 7B An exemplary structural diagram of the electronic device 71 provided in this application. Detailed Implementation

[0049] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings.

[0050] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that although the terms first, second, etc., may be used in the following embodiments to describe a class of objects, the objects are not limited to these terms. These terms are only used to distinguish specific objects of that class of objects. For example, the terms first, second, etc., may be used in the following embodiments to describe operation instructions, but operation instructions are not limited to these terms. These terms are only used to distinguish different operation instructions input by the user. Other classes of objects that may be described using the terms first, second, etc. in the following embodiments are similarly not described here.

[0051] This application provides an activity recognition (AR) method and a display method. The electronic device can use an AR module based on one-dimensional convolution principles to obtain the activity status of the user carrying the electronic device, and then display icons related to that activity status on a map interface. This allows the electronic device's map function to present the user's activity status in real-time and dynamically, thereby optimizing the user experience.

[0052] It should be noted that the "person activity state" described in this application embodiment refers to the state of a person as represented by their movement, actions, and environmental elements associated with those actions. This application embodiment can characterize the person activity state from at least one dimension, such as from at least one dimension of the person's movement state, the person's actions, and the tool carrying the person. The person's movement state refers to the state of the person relative to a reference frame when performing mechanical movement. The person's movement state can include at least one of: being stationary, walking, and running. The person's actions refer to physical movements, which can include at least one of: standing, sitting, squatting, and lying down. The tool carrying the person can include: riding a means of transportation (e.g., a car, high-speed rail, etc.), riding an elevator, and riding an escalator. For example, if the person's movement state is "stationary," the person's action is "sitting," and the tool carrying the person is a "high-speed rail," then the person's activity state is, for example, a user riding a high-speed rail. In the process of activity recognition, each of the aforementioned dimensions can be expressed as a type of feature data of a person. For example, the dimension of a person's motion state can be expressed as a first type of feature data, the dimension of a person's action can be expressed as a second type of feature data, and the dimension of the tool carrying the person can be expressed as a third type of feature data, so that electronic devices can recognize the person's activity state based on different types of feature data.

[0053] It should be understood that in actual implementation, the dimensions representing the activity state of a character are not limited to the examples above. Furthermore, the character's motion state, actions, and the tools used to transport the character are not limited to the examples described above. This application's embodiments will not provide further examples of each of these.

[0054] The following describes an electronic device, a user interface (UI) for such an electronic device, and embodiments for using such an electronic device.

[0055] The electronic devices involved in the embodiments of this application can be devices that include posture detection and display functions, such as mobile phones, tablets, wearable devices (e.g., watches, bracelets, helmets, etc.), in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, and other electronic devices. It is understood that the embodiments of this application do not impose any limitations on the specific type of electronic device. The electronic device can be equipped with... Or devices operating systems.

[0056] Figure 1A Taking a smartphone as an example, a schematic diagram of the hardware structure of an electronic device 100 is shown. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, antenna 1, antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0057] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0058] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, memory, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). Different processing units may be independent devices or integrated into one or more processors. In some embodiments, electronic device 100 may also include one or more processors 110.

[0059] The controller can serve as the central nervous system and command center of the electronic device 100. The controller can generate operation control signals based on instruction operation codes and timing signals to complete control tasks such as detection commands.

[0060] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the electronic device 100.

[0061] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0062] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C buses. The processor 110 can couple to the touch sensor 180K, charger, flash, camera 193, etc., through different I2C bus interfaces. For example, the processor 110 can couple to the touch sensor 180K through the I2C interface, enabling the processor 110 and the touch sensor 180K to communicate through the I2C bus interface, thereby realizing the touch function of the electronic device 100.

[0063] The I2S interface can be used for audio communication. In some embodiments, the processor 110 may include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the I2S interface to enable the function of listening to sound through Bluetooth headphones.

[0064] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via the PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface, enabling the function of listening to sound through Bluetooth headphones. Both the I2S interface and the PCM interface can be used for audio communication.

[0065] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface to enable music playback through Bluetooth headphones.

[0066] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 194. The MIPI interface includes a display serial interface (DSI). In some embodiments, the processor 110 and the display screen 194 communicate via the DSI interface to realize the display function of the electronic device 100.

[0067] The GPIO interface can be configured via software. It can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to a camera 193, a display screen 194, a wireless communication module 160, an audio module 170, a sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0068] USB port 130 is a USB standard compliant interface, specifically a Mini USB port, Micro USB port, USB Type-C port, etc. USB port 130 can be used to connect a charger to charge electronic device 100, and can also be used for data transfer between electronic device 100 and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other electronic devices, such as AR devices.

[0069] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0070] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via a USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device 100 via the power management module 141.

[0071] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, display screen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0072] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0073] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0074] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0075] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs an audio signal through an audio device (not limited to speaker 170A, receiver 170B, etc.) or displays the APP's user interface on the display screen 194. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0076] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0077] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0078] In some embodiments, the wireless communication solution provided by the mobile communication module 150 enables the electronic device 100 to communicate with devices in a network (such as a cloud server). The WLAN wireless communication solution provided by the wireless communication module 160 also enables the electronic device 100 to communicate with devices in a network (such as a cloud server).

[0079] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations, as well as graphics rendering. For example, the GPU can render images representing the activity state of a person. Processor 110 may include one or more GPUs, which execute program instructions to generate the display interface of a map application. In this embodiment, the display screen 194 may include a monitor and a touch device. The monitor is used to output display content to the user, such as the application (APP) icon display interface of electronic device 100, the map display interface, and the display interface representing the activity state of a person on the map interface. The touch device is used to receive operation commands input by the user on the display screen 194.

[0080] The display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a micro-LED, a quantum dot light-emitting diode (QLED), etc.

[0081] In some embodiments of this application, when the display panel uses materials such as OLED, AMOLED, and FLED, the display screen 194 can be bent. Here, "the display screen 194 can be bent" means that the display screen can be bent to any angle at any part and can maintain that angle. For example, the display screen 194 can be folded from the middle left to right. It can also be folded from the middle up and down.

[0082] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0083] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0084] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, photos, videos, and other data can be stored on the external memory card.

[0085] The internal memory 121 can be used to store one or more computer programs, which include instructions. The processor 110 can execute the aforementioned instructions stored in the internal memory 121, thereby causing the electronic device 100 to perform the display methods provided in some embodiments of this application, as well as various data processing methods. The internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system; it may also store one or more application programs (such as maps, email, contacts, etc.). The data storage area may store data required during the use of the electronic device 100 (such as algorithms for recognizing human activity states, icon description files, etc.), as well as created data (such as sensor measurement data, etc.). Furthermore, the internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0086] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0087] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0088] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or make hands-free calls through the speaker 170A.

[0089] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a telephone call or voice message, the receiver 170B can be brought close to the ear to listen to the voice.

[0090] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Electronic device 100 may have at least one microphone 170C. In some embodiments, electronic device 100 may have two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 100 may also have three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0091] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0092] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be disposed on display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes. Electronic device 100 determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 194, electronic device 100 detects the intensity of the touch operation based on pressure sensor 180A. Electronic device 100 can also calculate the touch position based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands.

[0093] The gyroscope sensor 180B can detect changes in the pose of the electronic device 100 by detecting the rotation angles of the electronic device 100 around three axes (X, Y, and Z axes), thereby determining the behavior of the user carrying the electronic device 100. For example, if a user is holding the electronic device 100 and walking upstairs, the electronic device 100 continuously undergoes small displacements with the user's movements, and the rotation angles of the electronic device 100 relative to the three axes change continuously with the user's body posture. In this scenario, the electronic device 100 acquires measurement data from the gyroscope sensor 180B, and based on this measurement data, determines changes in the pose of the electronic device 100, thereby determining changes in the user's posture.

[0094] Accelerometer 180E can detect the magnitude of acceleration of electronic device 100 in multiple directions (typically x, y, and z axes), thereby determining the direction of movement of electronic device 100, and further determining the behavior of the user carrying electronic device 100. For example, when a user is walking upstairs while holding electronic device 100, electronic device 100 can acquire the acceleration values ​​detected by accelerometer 180E in the x, y, and z axis directions (i.e., the measurement data of accelerometer 180E). Then, based on the measurement data of accelerometer 180E, the direction of movement of electronic device 100 is determined, and thus, the user's behavior is determined.

[0095] The magnetic sensor 180D may include a Hall sensor. The magnetic sensor 180D can determine the orientation of the display screen 194 by detecting the magnitude and direction of the magnetic field. For example, the electronic device 100 can determine, based on the magnetic sensor 180D, that the display screen 194 is parallel to the ground and facing upwards. Furthermore, the electronic device 100 can correctly label icons, etc., on the interface of a map app based on the orientation of the display screen 194. In some embodiments, when the electronic device 100 is a flexible screen device, the electronic device 100 can detect the folding and unfolding states of the screen based on the magnetic sensor 180D. This will not be described in detail here.

[0096] For ease of description, in the following embodiments of this specification, the data detected by the gyroscope sensor 180B, such as the rotation angle of the gyroscope, will be referred to as "measurement data of the gyroscope sensor 180B"; the data detected by the accelerometer sensor 180E, such as the magnitude and direction of gravity, will be referred to as "measurement data of the accelerometer sensor 180E"; and the data detected by the magnetometer sensor 180D, such as the magnitude and direction of the magnetic field, will be referred to as "measurement data of the magnetometer sensor 180D". The measurement data of the gyroscope sensor 180B, the accelerometer sensor 180E, and the magnetometer sensor 180D can reflect changes in the pose of the electronic device 100, and the measurement data of the accelerometer sensor 180E can reflect the moving speed of the electronic device 100, thereby enabling the determination of the activity status of the user carrying the electronic device 100.

[0097] The barometric pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates altitude using the air pressure value measured by the barometric pressure sensor 180C to assist in positioning and navigation.

[0098] A distance sensor 180F is used to measure distance. Electronic device 100 can measure distance via infrared or laser. In some embodiments, during a shooting scene, electronic device 100 can utilize the distance sensor 180F to measure distance for rapid focusing.

[0099] The proximity sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The LED may be an infrared LED. The electronic device 100 emits infrared light outward through the LED. The electronic device 100 uses the photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The electronic device 100 may use the proximity sensor 180G to detect when a user holds the electronic device 100 close to their ear for a call, so as to automatically turn off the screen to save power. The proximity sensor 180G can also be used in holster mode and pocket mode for automatic unlocking and locking of the screen.

[0100] The ambient light sensor 180L is used to sense the brightness of ambient light. The electronic device 100 can adaptively adjust the brightness of the display screen 194 based on the sensed ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also work with the proximity sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touches.

[0101] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can utilize the characteristics of the collected fingerprints to achieve fingerprint unlocking, accessing application locks, taking photos with fingerprints, answering calls with fingerprints, etc.

[0102] Temperature sensor 180J is used to detect temperature. In some embodiments, electronic device 100 uses the temperature detected by temperature sensor 180J to execute a temperature handling strategy. For example, when the temperature reported by temperature sensor 180J exceeds a threshold, electronic device 100 performs thermal protection by reducing the performance of a processor located near temperature sensor 180J to reduce power consumption. In other embodiments, when the temperature is below another threshold, electronic device 100 heats battery 142 to prevent abnormal shutdown of electronic device 100 due to low temperature. In still other embodiments, when the temperature is below yet another threshold, electronic device 100 boosts the output voltage of battery 142 to prevent abnormal shutdown due to low temperature.

[0103] Touch sensor 180K, also known as a touch panel or touch-sensitive surface, can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also called a "touchscreen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of electronic device 100, in a different position than display screen 194.

[0104] The bone conduction sensor 180M can acquire vibration signals. In some embodiments, the bone conduction sensor 180M can acquire vibration signals from the vibrating bone segments of the human vocal cords. The bone conduction sensor 180M can also contact the human pulse to receive blood pressure signals. In some embodiments, the bone conduction sensor 180M can also be incorporated into headphones to form bone conduction headphones. The audio module 170 can parse the voice signals from the vibrating bone segments of the vocal cords acquired by the bone conduction sensor 180M to realize voice functionality. The application processor can parse heart rate information from the blood pressure signals acquired by the bone conduction sensor 180M to realize heart rate detection functionality.

[0105] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of electronic device 100.

[0106] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can correspond to touch operations performed on different applications (such as taking photos, playing audio, etc.). Motor 191 can also correspond to different vibration feedback effects for touch operations performed on different areas of the display screen 194. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0107] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0108] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to make contact with and separate from the electronic device 100. The electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to realize functions such as calls and data communication. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0109] The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to exemplify the software structure of electronic device 100.

[0110] Figure 1B This is a schematic diagram of the software structure of the electronic device 100 according to an embodiment of this application.

[0111] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system, from top to bottom, consists of the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0112] The application layer can include a series of application packages.

[0113] like Figure 1BAs shown, the application package may include applications such as gallery, calendar, maps, WLAN, music, SMS, calls, navigation, Bluetooth, and video.

[0114] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0115] like Figure 1B As shown, the application framework layer may include a sensor data acquisition module, an AR module, and a map display module. Of course, the application framework layer may also include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, etc., and this embodiment does not impose any limitations on this.

[0116] The sensor data acquisition module is used to acquire sensor data reflecting the attitude of the electronic device 100 and to transmit the acquired sensor data to the AR module. Sensor data includes, for example, measurement data from the gyroscope sensor 180B, the accelerometer sensor 180E, and the magnetometer sensor 180D. In practice, the sensor data acquisition module can acquire measurement data from at least one sensor, thus obtaining at least one set of measurement data. It should be noted that different sensors have different detection periods; therefore, the sensor data acquisition module can acquire the measurement data of each sensor based on its respective detection period. Thus, the frequency at which the sensor data acquisition module acquires measurement data from each sensor can be different. For example, the sensor data acquisition module acquires the measurement data from the magnetometer sensor 180D at a frequency of 500 Hz, and the sensor data acquisition module acquires the measurement data from the accelerometer sensor 180E at a frequency of 200 Hz.

[0117] It is understood that the measurement data of the gyroscope sensor 180B, the accelerometer sensor 180E, and the magnetometer sensor 180D described above are merely illustrative and do not constitute a limitation on the sensor data involved in the embodiments of this application. In some other embodiments, the sensor data may also include more or fewer sensor measurement data that can characterize the pose of the electronic device 100.

[0118] The AR module can be used to obtain human activity status data based on the aforementioned sensor data, and to transmit the human activity status data to the map display module. The AR module may include a data preprocessing module, a feature extraction module, and a classification module. Figure 1B(Not shown in the original text). The data preprocessing module can be used to preprocess the sensor data, and the feature extraction module can be used to extract features from the preprocessed data to obtain a set of feature data representing the activity state of a person. The classification module can be used to extract at least one type of feature data from the feature data set, where each feature data is a one-dimensional data point representing the activity state of a person. In this embodiment, both the feature extraction module and the classification module are based on the algorithm principle of one-dimensional convolution, as detailed below. In some embodiments, the model implementing the feature extraction module is also called a time series embedding (TSE) model, and the classification module can also be called a classifier; this embodiment does not limit this.

[0119] The map display module can be used to manage the display strategy of the map function in the electronic device 100, including determining the display mode of icons and the display position of icons. The map display module can also be used to render icons to be displayed. Afterwards, the map display module can control the map APP to display icons representing human activity status on the display screen 194 of the electronic device 100 according to the display strategy, presenting the user with a visual representation of behavioral status (such as...). Figures 3B to 3E (The display effect shown).

[0120] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0121] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0122] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface that includes a camera icon.

[0123] The phone manager is used to provide communication functions for electronic device 100. For example, it manages call status (including connection and disconnection).

[0124] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0125] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of download completion or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting alert sounds, vibrating electronic devices, and flashing indicator lights.

[0126] The Android Runtime consists of core libraries and a virtual machine. The Android Runtime is responsible for scheduling and managing the Android system.

[0127] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0128] See also Figure 1B The system libraries and kernel layer below the application framework layer can be called the underlying system. The underlying system includes the underlying display system that provides display services. For example, the underlying display system includes the display driver in the kernel layer and the surface manager in the system library.

[0129] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0130] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0131] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0132] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0133] A 2D graphics engine is a graphics engine for 2D drawing.

[0134] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0135] The UI (User Interface) involved in this application, used to implement display functions, is a medium interface for interaction and information exchange between an application or operating system and the user. It converts the internal form of information into a form acceptable to the user. The user interface of an application is written in source code using specific computer languages ​​such as Java or Extensible Markup Language (XML). This source code is parsed and rendered on an electronic device, ultimately presenting user-recognizable content such as videos, images, text, and buttons. Controls, also known as widgets, are the basic elements of the user interface. Typical controls include toolbars, menu bars, text boxes, buttons, scroll bars, images, and text. The attributes and content of controls in the interface are defined using tags or nodes, such as XML tags. <textview> 、 <imgview> 、 <videoview>Nodes define the controls contained in the interface. A node corresponds to a control or property in the interface, and after parsing and rendering, the node is presented as the content visible to the user. In addition, many applications, such as hybrid applications, often contain web pages within their interfaces. A web page, also known as a page, can be understood as a special control embedded in the application interface. Web pages are source code written in a specific computer language, such as Hypertext Markup Language (HTML), Cascading Style Sheets (CSS), JavaScript (JS), etc. Web page source code can be loaded and displayed as user-readable content by a browser or a web page display component with browser-like functionality. The specific content contained in a web page is also defined through tags or nodes in the web page source code; for example, HTML uses tags or nodes to define the content. 、 、 <video> 、 <canvas>to define the elements and attributes of a web page.

[0136] A common form of user interface is a graphic user interface (GUI), which refers to a user interface that displays in a graphical manner. It can be an icon, window, control, etc. interface element displayed in the display screen of an electronic device, wherein the control can include an icon, button, menu, tab, text box, dialog box, status bar, navigation bar, Widget, etc. visual interface element.

[0137] In some existing implementations, the electronic device does not have the function of detecting the activity state of the person, or the electronic device has the function of detecting the activity state of the person, but there is no demand, so the function of detecting the activity state of the person is independent of the map display function, and further, the electronic device cannot provide a service of displaying the activity state of the person through the map, and can only provide a service directly related to the position of the user carrying the electronic device, resulting in a relatively single service item associated with the map function, and a low user experience.

[0138] Based on this, as Figure 2A shown, the embodiment of the present application provides a display method 10, which comprises: an electronic device 100 receives a first operation instruction, and then the electronic device 100 displays an icon representing the activity state of the person on a map interface in response to the first operation instruction.

[0139] In some embodiments, the icon includes a person icon, for example, a person vector diagram (such as Figure 3C "Icon in walking implementation scenario" in FIG. 1). In other embodiments, the icon includes an object icon, for example, an elevator vector diagram, an escalator vector diagram, a bicycle vector diagram, etc. In other embodiments, the icon includes a person icon and an object icon, wherein the person icon and the object icon are displayed in combination, for example, the person vector diagram is adjacent to the stairs (such as Figure 3C "Icon in stair climbing implementation scenario" in FIG. 1).

[0140] In some embodiments, the aforementioned icon is implemented as a static image (such as Figure 3C "Icon in static implementation scenario" in FIG. 1), and the image format includes.jpg format,.tif format,.img format, etc. In other embodiments, the icon is implemented as a dynamic image (such as Figure 3F and Figure 3G the person icon in FIG. 1), and the image format includes.gif format. In other embodiments, when the icon includes a person icon and an object icon, at least one of the person icon and the object icon is implemented as a dynamic image, such as Figure 3D the person icon in FIG. 1 is a dynamic image, and the stair icon is a static image. Herein, no limitation is made.

[0141] In other embodiments, the electronic device 100 may also preset at least one display mode, each display mode indicating how the icon is displayed on the map app interface. For example, the first display mode indicates that the icon is displayed floating on the map app interface (e.g., Figure 3B As shown), the second display mode indicator icon is embedded in the map app's interface, and the map app's interface is presented as a 2D map (e.g., ...). Figure 3D As shown), the third display mode indicator icon is embedded in the interface of the map app, and the interface of the map app is presented as a 3D image. Based on this, the display method 10 may further include: the electronic device 100 receiving a second operation command input by the user, and then the electronic device 100 displaying an icon representing the activity status of a person in the display mode indicated by the second operation command.

[0142] As can be seen, by adopting the implementation method of this application embodiment, the character's activity status and the map can be presented interactively, thereby enabling the electronic device 100 to adaptively and in real time present the character's activity status on the map interface, thereby optimizing the visualization effect of the map function and improving the user experience.

[0143] It should be noted that in actual implementation, the map function of electronic device 100 can be implemented through the map APP installed on electronic device 100, or through the map plugin or map mini-program of other APPs in electronic device 100, such as the map plugin in instant messaging APP.

[0144] See Figure 2B , Figure 2B An activity recognition method 20 is illustrated. When an electronic device 100 executes the activity recognition method 20, it can obtain icons representing a person's activity state, as per the display method 10. The activity recognition method 20 may include: the electronic device 100 obtaining multiple poses of itself within a time period; then, using multiple one-dimensional convolutional layers to obtain one or more feature data based on the multiple poses, whereby the one or more feature data respectively indicate different types of information about the person holding the electronic device 100 within that time period. The type of information indicated by each feature data may be the person's motion state, the person's action, or the tool carrying the person. Furthermore, the electronic device 100 is controlled to display the type of information indicated by the one or more feature data in the form of icons.

[0145] For example, electronic device 100 can control electronic device 100 to display a map, and after obtaining the icon corresponding to each feature data, control the corresponding icon to be displayed on the map, so as to realize the display function of display method 10.

[0146] In this embodiment, the "time period" can be flexibly set according to requirements and is not limited here. In one optional example, the time period is, for example, 5 seconds.

[0147] In this embodiment, the electronic device 100 obtains one or more feature data based on multiple poses. Specifically, this may include using multiple one-dimensional convolutional layers to obtain a feature data set based on multiple poses, wherein the feature data set includes at least one type of information about the person's activity state. Furthermore, the electronic device 100 obtains feature data of each of the aforementioned types from the feature data set to obtain one or more feature data sets.

[0148] See Figure 2C , Figure 2C An activity recognition method 200 is illustrated, which is a possible implementation of activity recognition method 20. Activity recognition method 200 includes: an electronic device 100 acquiring at least one set of measurement data, wherein each set of measurement data corresponds to a sensor. Then, the electronic device 100 performs data preprocessing on the at least one set of measurement data to obtain multiple poses of the electronic device 100. Next, the electronic device 100 performs feature extraction on the multiple poses using a time series embedding (TSE) model to obtain a feature data set. Then, the electronic device 100 uses a classifier to classify the feature data set to obtain feature data of each type.

[0149] For example, the preprocessing of at least one set of measurement data by the electronic device 100 may include: the electronic device 100 verifying each set of measurement data to determine whether there are any anomalies (e.g., missing data) in the acquired measurement data; then, sampling the verified measurement data (i.e., measurement data without anomalies) to standardize the time intervals between the sets of measurement data. Afterward, the electronic device 100 performs linear interpolation on each sampled set of data to obtain a set of data. Further, the electronic device 100 performs noise reduction, gravity residual removal, and other operations on the obtained set of data to obtain multiple poses of the electronic device 100. Then, the electronic device 100 obtains human activity state data based on these multiple poses.

[0150] Figure 2D An exemplary computational flow for obtaining human activity state data based on multiple poses is illustrated. This computational flow is implemented based on the TSE model and a classifier.

[0151] The TSE model described in this application embodiment may include a first feature extraction framework and a second feature extraction framework. The first feature extraction framework includes m sequentially combined one-dimensional convolutional layers, and the second feature extraction framework includes n sequentially combined backbone models. Each backbone model includes at least two parallelly combined one-dimensional convolutional layers. The number of one-dimensional convolutional layers in different backbone models may be the same or different. m and n are both positive integers. After receiving multiple poses, the first feature extraction framework performs feature extraction on the multiple poses to obtain an initial feature data set. Then, the second feature extraction framework performs feature extraction on the initial feature data set based on different time scales to obtain a time-related feature data set. Further, the classifier extracts at least one type of feature data from the feature data set to obtain the person's activity state data.

[0152] For example, in the first feature extraction framework, the first one-dimensional convolutional layer receives multiple poses, and then performs feature extraction on these poses to obtain a first initial feature data set. Next, the second one-dimensional convolutional layer in the first feature extraction framework performs feature extraction on the first initial feature data set to obtain a second initial feature data set. And so on, the m-th one-dimensional convolutional layer in the first feature extraction framework performs feature extraction on the data set extracted by the (m-1)-th one-dimensional convolutional layer to obtain the initial feature data set.

[0153] The operating principle of the one-dimensional convolutional model is to extract features from the input one-dimensional features by channel number, with each channel extracting different features than the other channels. Therefore, the larger the number of channels in a convolutional layer, the higher the accuracy of the features extracted by that layer for activity recognition. Based on this, in some embodiments, the number of channels in the one-dimensional convolutional layer of the first feature extraction framework can increase sequentially according to the data flow. For example, the first one-dimensional convolutional layer in the first feature extraction framework has 32 channels, and the second one-dimensional convolutional layer has 64 channels. This allows the first feature extraction framework to progressively extract deeper features from the multiple poses, thereby ensuring that the initial feature data set represents the character's activity state according to preset conditions.

[0154] Of course, the sequential increase in the number of channels in the one-dimensional convolutional layers of the first feature extraction framework is merely an illustrative implementation and does not constitute a limitation on the embodiments of this application. In other embodiments, the number of channels in each one-dimensional convolutional layer of the first feature extraction framework may be the same, or partially the same. In other embodiments, the number of channels in the one-dimensional convolutional layers of the first feature extraction framework may also decrease sequentially. The embodiments of this application do not limit this.

[0155] It should be noted that the first feature extraction framework only needs to perform preliminary feature extraction on these multiple poses. Therefore, the kernel (k) values ​​of the m one-dimensional convolutional layers in the first feature extraction framework are the same. Here, the value of k indicates the number of consecutive pose data points from the multiple poses used in each convolution operation. Furthermore, these multiple poses are obtained based on temporal sampling; therefore, the value of k is related to the time scale, indicating that the initial feature data set is only related to one time scale.

[0156] In practice, different individuals performing the same activity may have different amplitudes and speeds of movement, resulting in different features per unit time. Therefore, to ensure that the extracted feature data set can represent more features, after obtaining the initial feature data set, the second feature extraction framework further performs feature extraction on the initial feature data set from a time scale to obtain the final feature data set.

[0157] The first backbone model in the second feature extraction framework receives an initial feature data set. Then, this first backbone model performs feature extraction on the initial feature data set from a time scale to obtain a first feature data set. Subsequently, the second backbone model in the second feature extraction framework performs feature extraction on the first feature data set from a time scale to obtain a second feature data set. This process continues, with the nth backbone model in the second feature extraction framework performing feature extraction on the data set obtained by the (n-1)th backbone model to obtain the aforementioned feature data set. The implementation process of the n backbone models in the second feature extraction framework is similar; the following explanation uses the first backbone model as an example, where the first backbone model is any one of the n backbone models.

[0158] For example, the first backbone model can input the input feature data sets into at least two one-dimensional convolutional layers. Each of the at least two one-dimensional convolutional layers performs a convolution operation on the input feature data sets and outputs the resulting data sets. Then, the first backbone model combines the data sets output by the at least two one-dimensional convolutional layers into a single data set and outputs it. The k-values ​​of the at least two one-dimensional convolutional layers in the first backbone model are different. For example, the k-value of the first one-dimensional convolutional layer is 5, and the k-value of the second one-dimensional convolutional layer is 9. Therefore, the time scales corresponding to these two one-dimensional convolutional layers are different. Accordingly, these two one-dimensional convolutional layers extract features from the same feature data set from two different time scales, thereby obtaining feature data sets at multiple time scales.

[0159] It is understandable that the implementation process of each of the aforementioned n backbone models can refer to the implementation process of the first backbone model, and will not be repeated here.

[0160] It should be noted that the number of channels in at least two one-dimensional convolutional layers belonging to the same backbone model can be the same or different, and the number of channels in one-dimensional convolutional layers of different backbone models can also be the same or different; this application does not limit this. Furthermore, according to the data flow, the number of channels in the one-dimensional convolutional layers of the second feature extraction framework backbone model can gradually increase. For example, according to the data flow, the number of channels in the one-dimensional convolutional layer of the first backbone model of the second feature extraction framework is 32, and the number of channels in the one-dimensional convolutional layer of the nth backbone model is 64.

[0161] The feature data set obtained by the TSE model contains feature data for each dimension of a person's activity state. Furthermore, there may be multiple feature data for each dimension. Based on this, the embodiments of this application can extract feature data for at least one dimension from the feature data set to accurately obtain the specific activities of a person in the corresponding dimension.

[0162] See you again Figure 2D The classifier described in this application embodiment may include y classification models, where y is a positive integer. After the feature data set is input into the classifier, each of the y classification models extracts one type of feature data from the feature data set to obtain y types of feature data. These y types of feature data represent the activity state of a person from y dimensions.

[0163] In practice, any classification model can obtain a sub-feature data set of the corresponding type of the classification model in the feature data set based on the operation of a one-dimensional convolutional layer, and then obtain the feature data of that type based on the obtained sub-feature data set.

[0164] For example, taking y = 3, the classifier includes three types of classification models. The first type represents the person's activity state from the dimension of motion state; the second type represents the person's activity state from the dimension of action; and the third type represents the person's activity state from the dimension of the tool carrying the person. After the feature data set is input into the classifier, the first type of classification model obtains sub-feature data sets representing the person's motion state from the feature data set, and then obtains the first type of feature data from the obtained sub-feature data sets, indicating the person's specific motion state. The second type of classification model obtains sub-feature data sets representing the person's action from the feature data set, and then obtains the second type of feature data from the obtained sub-feature data sets, indicating the person's specific action. Similarly, the third type of classification model obtains sub-feature data sets representing the tool carrying the person from the feature data set, and then obtains the third type of feature data from the obtained sub-feature data sets, indicating the specific tool carrying the person. The first type, second type, and third type of feature data constitute the person's activity state data. In other embodiments, y can also be 1, meaning the classifier includes a classification model of one type. In this case, the classifier can obtain feature data of one type. This type can, for example, characterize the person's activity state from any dimension, including the person's motion state, the person's actions, and the tool carrying the person. Accordingly, in this embodiment, the person's activity state data contains only one feature data. Details are omitted here.

[0165] It should be noted that in some embodiments, the electronic device 100 can directly use the feature data output by the aforementioned classifier as the user's activity state data. In other embodiments, the electronic device 100 can smooth the feature data output by the aforementioned classifier to obtain the user's possible actual activity state, and then use the smoothed data as the user's activity state data.

[0166] Understandable Figure 2D This description is merely illustrative and does not constitute a limitation on the technical solutions of the embodiments of this application. In actual implementation, the number of one-dimensional convolutional layers in the above model can be flexibly set according to requirements. In some other embodiments, the number of channels and the value of k in each convolutional layer, as well as the number of feature types corresponding to the classifier, can be other, which will not be elaborated here. In addition, this specification does not show all the implementation scenarios to which this application applies. In other implementation scenarios, other implementation means based on the technical ideas of this application are also within the protection scope of this application.

[0167] As can be seen, the present application embodiment performs feature extraction by stacking one-dimensional convolutional layers. This not only breaks through the limitation of relying on only one dimension of the character's motion state when training the feature extraction model (i.e., the TSE model and the classifier), making the obtained feature extraction model highly scalable, but also improves the accuracy of the identified character's activity state because the trained feature extraction model can identify the character's activity state from multiple dimensions.

[0168] It should be noted that in some embodiments, the activity state of the person presented by the electronic device 100 is the activity state of the user carrying the electronic device 100. In this embodiment, the electronic device 100 executes the aforementioned activity recognition method to obtain the activity state data of the user carrying the electronic device 100, and then displays an icon representing the user's activity state on the map interface. In other embodiments, the activity state of the person presented by the electronic device 100 comes from another electronic device, and is the activity state of the user carrying that other electronic device. In this embodiment, that other electronic device executes the aforementioned activity recognition method to obtain the activity state data of the user carrying that other electronic device, and then sends the user's activity state data to the electronic device 100, so that the electronic device 100 displays an icon representing the corresponding user's activity state on the map interface.

[0169] For ease of understanding, the following embodiments of this application will be described using the following methods: Figure 1A and Figure 1B Taking a mobile phone with the structure shown as an example, the display method and activity recognition method provided in the embodiments of this application will be specifically described in conjunction with the accompanying drawings.

[0170] For example, the mobile phone may have a map app installed. The following describes an embodiment of this application by taking the example of the mobile phone recognizing the activity status of the user carrying the mobile phone (hereinafter referred to as the user) and displaying the activity status of the user carrying the mobile phone on the interface of the map app.

[0171] Example 1:

[0172] Before implementing the display method described in the embodiments of this application, the mobile phone, for example, receives a user's click input command to run a map app. Figure 3A As shown, Figure 3A The illustrated GUI 31 is an exemplary display interface of a mobile phone desktop. GUI 31 includes a navigation bar, status bar, and interface elements for various apps. For example, the phone includes apps such as Map 310, Email, Wallet, and Smart Life. After receiving a user's instruction to click on Map 310, the phone responds by running Map 310. Figure 3A The illustrated GUI31 has been updated to Figure 3B The GUI32 diagram is shown.

[0173] like Figure 3B As shown, Figure 3B The illustrated GUI 32 is an exemplary display interface of the main interface of the map 310. GUI 32 includes a map interface 321, an icon interface 322, a personal center interface 323, and a shortcut function trigger control 324. The map interface 321 can be a map interface centered on the user's location. The icon interface 322 includes an icon 3221 representing a person, an icon 3222 representing stairs, and a marker 3223. The icon 3221 can be, for example, a dynamic upward-climbing form (not shown in the figure). The marker 3223 indicates the activity state represented by the icon interface 322; in this embodiment, the marker 3223 reads "climbing stairs," indicating that the icon interface 322 indicates the person's activity state is climbing stairs. The map interface 321 and the icon interface 322 can be displayed as two separate layers. The map interface 321 is displayed on the main screen of the phone, while the icon interface 322 is displayed as a small window floating in the upper right corner of the phone. The personal center interface 323 provides the user with the main interface for accessing the map app's function settings, thus presenting the user with entry points to various function settings. For example, users can access the personal center interface through the personal center interface 323 and see the settings entry point. Then, users can enter the main settings interface from the settings entry point and select the display mode of the icon interface 322. See details. Figures 5A to 5C The description of the corresponding embodiments is not detailed here. The shortcut function trigger control 324 includes entry points for triggering shortcut functions, such as layer 3241, reporting, ride-hailing, and voice packs. Users can access the settings interface of the corresponding function through the respective entry point. For example, users can access the settings interface for whether the icon interface 322 is displayed through layer 3241, see details below. Figure 6 The description of the corresponding embodiments is not detailed here.

[0174] It should be pointed out that, Figure 3B The illustrated GUI 32 is for illustrative purposes only. In actual implementation, the phone can receive user input commands to drag the small window corresponding to the icon interface 322. Furthermore, the phone can respond to these drag commands and move the position of the small window corresponding to the icon interface 322. For example, the phone can respond to these drag commands and move the small window corresponding to the icon interface 322 to the upper left corner of the phone screen.

[0175] Understandable. Figure 3B The icon interface 322 shown in the illustration is merely illustrative and does not constitute a limitation on the embodiments of this application. In the embodiments of this application, the icon interface displayed on the mobile phone is associated with the user's actual activities. For example, as shown... Figure 3C As shown, Figure 3C The diagram illustrates eight icon interfaces and the activity status indicators for each interface. The activity status of the character represented by these eight icon interfaces, as indicated by the corresponding activity status indicators, will not be elaborated here.

[0176] In other embodiments, after receiving a user's instruction to click on map 310, the character's activity status icon can be associated with the user's location. An exemplary display effect of the corresponding embodiment is shown below. Figure 3D The GUI33 diagram is shown in the illustration. Further details are omitted here.

[0177] Figure 3B and Figure 3C The icons shown represent the user's activity status as identified by the phone's AR module. This is followed by interaction between the AR module and the phone's map display module, resulting in the displayed effects. As can be seen, this implementation allows the phone's AR module and map display function to interact, enabling the phone to recognize the user's activity status in real time and adaptively display icons representing that activity on the map interface. This enriches the map interface's display and improves the user experience.

[0178] After receiving the user's operation instruction to click on the map 310, the mobile phone identifies the user's activity status and displays an icon interface 322 that represents the user's activity status.

[0179] like Figure 4 The schematic data flow diagram shows that after receiving the user's operation command to click on map 310, the phone's sensor data acquisition module acquires measurement data from, for example, the gyroscope sensor, the accelerometer sensor, and the magnetometer sensor, and transmits these measurement data to the phone's AR module. The phone can sample the aforementioned measurement data based on the frequency at which each sensor detects the measurement data. For example, the phone's sensor data acquisition module samples at a frequency of 500Hz to obtain the gyroscope sensor's measurement data, at 400Hz to obtain the accelerometer sensor's measurement data, and at 200Hz to obtain the magnetometer sensor's measurement data.

[0180] In some embodiments, the measurement data from the aforementioned sensors may be referred to as an inertial measurement unit (IMU) sequence. Specifically, the measurement data from each sensor may be referred to as a set of IMU sequences. The term "IMU sequence" is used hereinafter in this specification. In this embodiment, "sequence" refers to multiple data points arranged in chronological order.

[0181] The AR module of a mobile phone can include three functional modules: a data preprocessing module, a TSE model, and a classifier. The following describes the activity recognition process of the AR module in conjunction with these three functional modules.

[0182] See you again Figure 4 The data preprocessing module in the AR module receives each of the above IMU sequences, and then sequentially performs loss check, sampling interpolation, Gaussian filtering and gravity residual removal operations on each IMU sequence.

[0183] The data loss check is implemented as follows: For each IMU sequence, the data preprocessing module checks whether the time difference between any two adjacent data points in the IMU sequence is greater than 50 milliseconds (ms). If the time difference between any two adjacent data points is not greater than 50 ms, it indicates that there is no data loss in the IMU sequence, and the data preprocessing module can perform subsequent operations on the IMU sequence. If the time difference between any two adjacent data points is greater than 50 ms, the data preprocessing module determines that there is data loss in the IMU sequence and will not continue to perform operations on the IMU sequence.

[0184] The sampling interpolation is implemented as follows: Within a preset time window, the data preprocessing module upsamples each group of IMU sequences without data loss at a frequency of 1000Hz, and then downsamples the upsampled sequences at a frequency of 100Hz to align the times of the asynchronous IMU sequences. Afterwards, the data from each downsampled sequence is linearly interpolated to obtain a time series. The preset time window can be, for example, 1 second (s) to 2 seconds.

[0185] The Gaussian filter is specifically implemented as follows: the data preprocessing module uses a Gaussian filter to smooth the time series to filter out noise. For example, the time smoothing factor of the Gaussian filter can be set to 10, that is, the bandwidth parameter of the time kernel function is set to 10.

[0186] The specific implementation of removing gravity residuals is as follows: the data preprocessing module obtains the gravitational acceleration amplitudes in the x-axis, y-axis and z-axis directions of the filtered time series respectively. Then, based on the gravitational acceleration amplitudes in each direction, the gravity in the corresponding direction is removed to obtain the pose sequence of the mobile phone.

[0187] Furthermore, the data preprocessing module transmits the pose sequence to the TSE model of the AR module. The TSE model obtains the feature sequence based on the pose sequence using multiple one-dimensional convolutional layers.

[0188] Figure 4A This illustrates the computational flow of a TSE model 40. For example... Figure 4A As shown, the feature extraction framework 41 of the TSE model 40 includes a one-dimensional convolutional layer 411, a one-dimensional convolutional layer 412, a one-dimensional convolutional layer 413, and a data filtering module 414. The feature extraction framework 42 includes a backbone model 421, a backbone model 422, a backbone model 423, a backbone model 424, a one-dimensional convolutional layer 425, a one-dimensional convolutional layer 426, a linear computation module 427, a linear computation module 428, and a global average pooling (GAP) layer 429.

[0189] In this model, the k-values ​​of one-dimensional convolutional layers 411, 412, and 413 are all set to 3, and the stride (step, s) is set to 1. One-dimensional convolutional layer 411 has 32 channels, one-dimensional convolutional layer 412 has 64 channels, and one-dimensional convolutional layer 413 has 128 channels. The pooling window size (ps) of the data filtering module 414 is set to 3, and s is set to 2. The k-values ​​of one-dimensional convolutional layers 425 and 426 are both set to 1, and s is set to 1. Furthermore, the total number of channels in the parallel-combined one-dimensional convolutional layers in backbone model 421 is the same as the total number of channels in the parallel-combined one-dimensional convolutional layers in backbone model 422, for example, both being 128. The total number of channels in the parallel-combined one-dimensional convolutional layers in backbone model 423 is the same as the total number of channels in the parallel-combined one-dimensional convolutional layers in backbone model 424, for example, both being 256.

[0190] It's important to note that the value of `s` indicates the number of consecutive data points to slide backward when retrieving data from the sequence during the operation. For example, in the sequence "001101100101010", if `k` is 3, it indicates that three consecutive data points are retrieved each time. If `s` is 1 and the first set of data retrieved is "001", then one data point is slide backward from the first set before retrieving the second set. In this example, the second set of data is "011". If `s` is 2, then two data points are slide backward from the first set before retrieving the second set, which is "110".

[0191] After receiving the pose sequence transmitted by the data preprocessing module, a one-dimensional convolutional layer 411 extracts features from the pose sequence from 32 angles (32 channels) to obtain feature sequence 01. Then, a one-dimensional convolutional layer 412 extracts features from feature sequence 01 from 64 angles (64 channels) to obtain feature sequence 02. A one-dimensional convolutional layer 413 extracts features from feature sequence 02 from 128 angles (128 channels) to obtain feature sequence 03. Feature sequence 03 is extracted progressively from the pose sequence. Typically, some feature data in feature sequence 03 are repetitive. Therefore, the data filtering module 414 performs pooling operations on feature sequence 03 to remove repetitive feature data and noise, obtaining feature sequence 04. This feature sequence 04 is the aforementioned initial feature sequence. Then, feature sequence 01 is input into the feature extraction framework 42 to continue feature extraction from feature sequence 04 on a time scale.

[0192] It should be noted that ps indicates the number of consecutive data points in feature sequence 03 obtained by the data filtering module 414 each time it performs a pooling operation. In this embodiment, the value of ps is 3, which instructs the data filtering module 414 to obtain 3 consecutive feature data points from feature sequence 03 each time to perform a pooling operation.

[0193] Furthermore, after expanding feature sequence 04, backbone model 421 extracts features through at least two one-dimensional convolutional layers to obtain feature sequence 05. The k values ​​of the at least two one-dimensional convolutional layers in backbone model 421 are different, thus extracting features from feature sequence 04 on a temporal scale. Subsequently, backbone models 422, 423, and 424 are sequentially combined to progressively extract features from the feature sequences obtained by the previous backbone model on a temporal scale, resulting in feature sequence 12. The implementation process of each backbone model from backbone models 421 to 424 can be found in [reference needed]. Figure 4B The illustrative embodiments are not described in detail here.

[0194] The backbone model focuses more on extracting features from the time scale. In order to ensure the comprehensiveness of the features in the obtained feature sequence, the feature extraction framework 42 can also extract the original features of the feature sequence through one-dimensional convolutional layer 425 and one-dimensional convolutional layer 426 respectively, and add the extracted original features to the feature sequence obtained by the backbone model, so that the features represented by the obtained feature sequence are more comprehensive.

[0195] For example, during the process of progressively extracting features from feature sequence 04 using backbone models 421 and 422 to obtain feature sequence 06, a one-dimensional convolutional layer 425 extracts features from the original features of feature sequence 04 to obtain feature sequence 07. Then, the linear computation module 427 superimposes feature sequence 06 and feature sequence 07 to obtain feature sequence 08, which contains both time-scale features and the original features of feature sequence 04. Subsequently, backbone models 423 and 424 progressively extract features from feature sequence 08, and backbone model 424 outputs feature sequence 09. A one-dimensional convolutional layer 426 extracts features from the original features of feature sequence 08 to obtain feature sequence 10. Then, the linear computation module 428 superimposes feature sequence 09 and feature sequence 10 to obtain feature sequence 11, which contains both time-scale features and the original features of feature sequence 08. Finally, feature sequence 11 is input into the GAP layer 429.

[0196] Based on the computational principle of one-dimensional convolutional layers, the backbone model 424 and the one-dimensional convolutional layer 426 expand each one-dimensional feature to 256 dimensions, making feature sequence 11 a divergent feature sequence. Therefore, feature sequence 11 can be converged before classification. For example, the GAP layer 429 obtains one-dimensional features by weighted averaging of each group of 256 features output by the linear computation module 428, achieving convergence of feature sequence 11 and outputting the converged feature sequence 12. Feature sequence 12 is the feature sequence obtained by the TSE model 40 based on the pose sequence. Then, the TSE model 40 inputs feature sequence 12 into the classifier.

[0197] Figure 4B This illustrates the computational flow of a backbone model 400. For example... Figure 4B As shown, the backbone model 400 includes one-dimensional convolutional layers 401, 402, 403, 404, and 405, a data filtering module 406, a connection module 407, a normalization module 408, and a linear computation module 409. In this embodiment, the number of channels in one-dimensional convolutional layers 401 to 405 is 32.

[0198] Combination Figure 4B Feature sequence 4001 is input to one-dimensional convolutional layer 401 and data filtering module 406. One-dimensional convolutional layer 401 expands feature sequence 4001 based on 32 channels to obtain feature sequence 4002. Feature sequence 4002 is then input to one-dimensional convolutional layers 402, 403, and 404 respectively. Data filtering module 406 filters feature sequence 4001 to obtain feature sequence 4003, which is then input to one-dimensional convolutional layer 405. Further, one-dimensional convolutional layer 402 outputs feature sequence 4001-1 to connection module 407, one-dimensional convolutional layer 403 outputs feature sequence 4001-2 to connection module 407, one-dimensional convolutional layer 404 outputs feature sequence 4001-3 to connection module 407, and one-dimensional convolutional layer 405 outputs feature sequence 4001-4 to connection module 407.

[0199] In this context, the k-values ​​of one-dimensional convolutional layers 402, 403, 404, and 405 are all different, indicating that the time scales at which the one-dimensional convolutional layers 402 to 405 perform convolution operations are different. That is, the one-dimensional convolutional layers 402 to 405 extract features from the feature sequence 4001 based on different time scales. Correspondingly, feature sequences 4001-1, 4001-2, 4001-3, and 4001-4 are feature sequences of feature sequence 4001 at different time scales. For example, the k-value of one-dimensional convolutional layer 402 is 7, the k-value of one-dimensional convolutional layer 403 is 11, the k-value of one-dimensional convolutional layer 404 is 15, and the k-value of one-dimensional convolutional layer 405 is 1.

[0200] Furthermore, the connection module 407 connects feature sequences 4001-1, 4001-2, 4001-3, and 4001-4 to obtain feature sequence 4004, and inputs feature sequence 4004 into the normalization module 408. Feature sequence 4004 corresponds to 128 dimensions. Then, the normalization module 408 performs a normalization operation on feature sequence 4004 and outputs feature sequence 4005. Subsequently, the linear computation module 409 performs linear computation on feature sequence 4005 to output feature sequence 4006. Feature sequence 4006 is the feature sequence obtained by the backbone model 400 based on feature sequence 4001.

[0201] In this embodiment, the connection module 407 connects feature sequences 4001-1, 4001-2, 4001-3, and 4001-4 in a certain order to form a sequence. In some embodiments, the connection module 407, for example, uses feature sequence 4001-1 as the 0th to 31st feature data, feature sequence 4001-2 as the 32nd to 63rd feature data, feature sequence 4001-3 as the 64th to 95th feature data, and feature sequence 4001-4 as the 96th to 127th feature data, combining them to obtain feature sequence 4004. It should be understood that the combination order of the above connection operations is only illustrative and does not constitute a limitation on the embodiments of this application. In other embodiments, the linking module 407 may also combine feature sequences 4001-1, 4001-2, 4001-3 and 4001-4 in other orders to obtain a feature sequence.

[0202] Understandable Figure 4B The illustrated backbone model 400 is merely illustrative and does not constitute a limitation on the embodiments of this application. In other embodiments, the number of parallel one-dimensional convolutional layers in the backbone model can be other numbers, such as 8. The number of channels in the one-dimensional convolutional layer and the k value of each parallel one-dimensional convolutional layer can also be other values. For example, the number of channels in the one-dimensional convolutional layer is 64. Furthermore, in a scenario where a feature extraction framework includes multiple backbone models, the number of channels in the one-dimensional convolutional layers of the different backbone models can be different. For example, in feature extraction framework 42, the number of channels in the one-dimensional convolutional layers of backbone models 421 and 422 is 32, and the number of channels in the one-dimensional convolutional layers of backbone models 423 and 424 is 64.

[0203] Figure 4C This illustrates the computational flow of a classifier 50. For example... Figure 4C As shown, classifier 50 includes classification model 51, classification model 52, and classification model 53. In this embodiment, classification model 51 obtains a first type of target feature data based on feature sequence 12, which is used, for example, to characterize user behavior. Classification model 52 obtains a second type of target feature data based on feature sequence 12, which is used, for example, to characterize user posture. Classification model 53 obtains a third type of target feature data based on feature sequence 12, which is used, for example, to characterize user activity scenarios.

[0204] For example, TSE model 40 inputs feature sequence 12 into classifier 50, that is, TSE model 40 inputs feature sequence 12 into classification models 51, 52, and 53 respectively. Then, classification models 51, 52, and 53 perform classification calculations on feature sequence 12 respectively to obtain target feature data of the corresponding type. The implementation process of classification calculations by classification models 51, 52, and 53 is similar; the following description uses classification model 51 as an example.

[0205] The classification model 51 includes a fully connected layer 511, a dropout layer 512, a fully connected layer 513, a dropout layer 514, and a softmax layer 515. The fully connected layer 511, for example, obtains a sequence representing user behavior from the feature sequence 12, resulting in feature sequence 121, and inputs feature sequence 121 into the dropout layer 512. The dropout layer 512 fits the feature data in feature sequence 121 to discard some non-critical features (e.g., walking features in a running feature sequence), resulting in feature sequence 122. Furthermore, to enhance the accuracy of the obtained feature sequence, the fully connected layer 513 and the dropout layer 514 sequentially perform calculations on feature sequence 122. The dropout layer 514 outputs feature sequence 123 and inputs feature sequence 123 into the softmax layer 515. The calculation process of the fully connected layer 513 and the dropout layer 514 can be found in the calculation process of the fully connected layer 511 and the dropout layer 512. For example, the softmax layer 515 obtains the largest feature data from the feature sequence 123 as the target feature data representing the user's specific behavior.

[0206] The modules contained in classification model 52 and classification model 53 are similar to those contained in classification model 51. The implementation process of classification model 52 obtaining target feature data representing user posture based on feature sequence 12 and the implementation process of classification model 53 obtaining target feature data representing user activity scene based on feature sequence 12 are similar to the implementation process of classification model 51 obtaining target feature data representing user behavior based on feature sequence 12, and will not be described in detail here.

[0207] In some embodiments, "target feature data" can specifically be implemented as the feature data with the highest confidence level, or the specific activity state identifier indicated by the feature data with the highest confidence level. Here, confidence level is used to characterize the degree of credibility that a user's activity state is a specific state; the higher the confidence level, the higher the credibility that the user's activity state is the corresponding state. For example, in the dimension of user behavior, if the confidence level for a user's specific behavior being walking is 0.6, and the confidence level for a user's specific behavior being running is 0.9, then the credibility of the user's specific activity being running is relatively high.

[0208] It should be noted that the fully connected layer in classifier 50 can be implemented as a one-dimensional convolutional layer.

[0209] It is understood that classifier 50 is merely illustrative and does not constitute a limitation on the embodiments of this application. In other embodiments, the classifier may also include more or fewer classification models. Furthermore, each classification model may also include more or fewer fully connected layers and dropout layers. The embodiments of this application are not limited in this regard.

[0210] Furthermore, in some embodiments, the three target feature data obtained by classifier 50 are transmitted as user activity state data to the map display module of the mobile phone, so that the map display module can obtain and control the display of the corresponding icons.

[0211] In other embodiments, the three target feature data obtained by classifier 50 are input into AR smoothing module to predict the user's possible actual activity state. Then, the data obtained by AR smoothing module to characterize the user's possible actual activity state is used as the user's activity state data and input into the mobile phone's map display module so that the map display module can obtain and control the display of the corresponding icon.

[0212] The AR smoothing module maintains a state transition matrix for user activity states, which includes the transition probabilities between any two states. Taking the three dimensions mentioned above as examples, in this embodiment, the AR smoothing module maintains three state transition matrices respectively. These three matrices maintain the transition probabilities between any two behaviors in the user behavior dimension, between any two postures in the user pose dimension, and between any two activity scenes in the user activity scene dimension. After receiving the above three target feature data, the AR smoothing module obtains the possible specific activities of the user in each of the three dimensions based on these three target feature data and the three state transition probabilities.

[0213] For example, taking user behavior as an example, the AR smoothing module, based on a preset smoothing algorithm, calculates the probability of the behavior corresponding to the user behavior transitioning to other behaviors based on the target feature data and state transition matrix corresponding to the user behavior. When any of the obtained probabilities is greater than a first preset threshold, the AR smoothing module determines the transitioned behavior corresponding to the highest probability as the user's possible actual behavior, and then uses this transitioned behavior as the behavior representing the user's activity state. When all the obtained probabilities are less than the first preset threshold, the AR smoothing module uses the target feature data corresponding to the user behavior as the behavior representing the user's activity state. The first preset threshold can be set according to empirical values; in one possible implementation, the first preset threshold is, for example, 98%.

[0214] For example, if the target feature data corresponding to a user's behavior indicates walking, the AR module calculates the probability of the user's behavior changing from walking to standing still, and the probability of the user's behavior changing from walking to running, based on the state transition matrix corresponding to the user's behavior. If both probabilities are less than 98%, the AR smoothing module determines that the user's specific activity is walking. If the probability of the user switching from walking to running is greater than or equal to 98%, the AR smoothing module determines that the user's actual behavior is likely running, and thus, uses running as the specific behavior representing the user's activity state.

[0215] The AR smoothing module performs a smoothing process on the user's posture based on the target feature data and state transition matrix corresponding to the user's posture, and performs a smoothing process on the user's activity scene based on the target feature data and state transition matrix corresponding to the user's activity scene. The process is similar to the above description and will not be detailed here.

[0216] It should be noted that the aforementioned state transition matrix can be obtained by training a hidden Markov model (HMM).

[0217] Furthermore, the phone can maintain a correspondence between each specific activity and its icon description information. The icon indicated by the icon description information is an image representing the corresponding activity. Based on this, after receiving user activity status data, the map display module can obtain the icon description information corresponding to the user activity status data, and then render the icon according to the icon description information and display the corresponding icon on the map app's interface.

[0218] For example, when the user activity state data includes the aforementioned three target feature data, each target feature data corresponds to an icon description. The mobile phone can then perform rendering according to the icon description information corresponding to each of the three target feature data, for example, obtaining the content contained in the icon interface 322, and subsequently, the mobile phone displays GUI 32.

[0219] Example 2:

[0220] Furthermore, users can trigger the display of a character's activity status in relation to their location, as needed. For example, in... Figure 3B Based on the GUI32 shown, in response to the user's operation command to zoom in on the icon interface 322 with two fingers, the phone associates the character's activity status with the user's location and displays accordingly. Figure 3B The illustrated GUI32 is updated to Figure 3D The GUI33 shown in the diagram.

[0221] like Figure 3D As shown, Figure 3D The illustrated GUI 33 is another exemplary map display interface. GUI 33 includes a map interface 331, a user center interface, and shortcut function trigger controls. The map interface 331 is a map interface centered on the user's location, and the area of ​​the map interface 331 indicating the user's location displays icons 3221 and 3222. Icons 3221 and 3222 are shown below. Figure 3B The corresponding embodiments are described in detail here. In this embodiment, the display position of icon 3221 on the map interface 331 can change according to the user's actual location, and icon 3222 can be displayed or hidden adaptively according to the user's actual behavior. This allows the map app to display the user's activity status in real time, accurately, and vividly, improving the user's viewing experience. The personal center interface and quick function trigger controls are as follows... Figure 3B The corresponding embodiments are described, and will not be detailed here.

[0222] It should be noted that, for the convenience of users, the color and / or brightness of icons 3221 and 3222 in GUI33 can be significantly different from the color and / or brightness of other parts of the map interface 331. For example, if the color of other parts of the map interface 331 is light blue, the color of icons 3221 and 3222 can be black.

[0223] also, Figure 3D The illustrated map interface 331 is presented in a 2D map display mode. In other embodiments, the map interface 331 may also be presented in other display modes, such as a 3D map mode. This application embodiment does not limit this.

[0224] Understandable. Figure 3D The icons shown are for illustrative purposes only and do not constitute a limitation on the embodiments of this application. In the embodiments of this application, the icons displayed on the mobile phone are associated with the user's actual activities. For example, as shown... Figure 3E As shown, Figure 3E Eight illustrative map interfaces are shown, with each icon representing a different character activity state. The activity states represented by these icons are as follows: Figure 3E As indicated by the symbol, it will not be elaborated upon here.

[0225] As described in the foregoing embodiments, after receiving a user's click on map 310 in GUI 31, GUI 31 can directly switch to the display interface of GUI 33. In this embodiment, the user can also disconnect the association between the character's activity status and the user's location as needed. For example, in Figure 3D Based on the GUI33 shown, in response to the user's pinch operation command, the phone disconnects the character's activity state from the user's location, and accordingly, Figure 3D The illustrated GUI33 has been updated to Figure 3B The GUI32 diagram is shown.

[0226] Example 3:

[0227] exist Figure 3D In corresponding implementation scenarios, in some embodiments, the mobile phone can also control the map to display recommendation information based on the user's activity status and location. This recommendation information indicates areas recommended to the user. An exemplary display interface for the mobile phone to display recommendation information is as follows: Figure 3F and Figure 3G As shown.

[0228] For example, a mobile phone can determine areas where people appear relatively frequently and their specific activity states when they appear in those areas based on the acquired batch data. Areas where people appear relatively frequently are usually popular areas favored by users; in this embodiment, these areas are designated as areas to be recommended. The mobile phone can then store the corresponding areas and the user's specific activity states. When a person's activity state is detected to be a specific activity state, and the nearest distance between the user's location and the corresponding area is less than or equal to a preset distance, the mobile phone's map display module triggers the display of recommended information, which indicates the corresponding area to be recommended.

[0229] The mobile phone can identify areas where the number of people visiting within a certain period exceeds a second preset threshold as areas with relatively high frequency of occurrence. This period could be, for example, three months. The second preset threshold can be flexibly set based on different areas. For example, when the area is a pedestrian street in the city center, the second preset threshold could be 100,000. Or, when the area is a walking path in a residential area, the second preset threshold could be 2,000. Furthermore, the preset distance can be flexibly set based on the actual scenario. For example, when the implementation scenario is inside a shopping mall, the preset distance could be 50 meters. Or, when the implementation scenario is a park, the preset distance could be 150 meters.

[0230] like Figure 3F As shown, Figure 3F The illustrated GUI 34 is an exemplary display interface for a map app to display recommended information. GUI 34 includes a map interface 341, and the colors of the icons 3411 and the edge lines of the region 3412 are different from the colors of other areas of the map interface 341. In this embodiment, icon 3411 represents the user's activity state as running. The edge line of region 3412, for example, represents a hiking trail, and region 3412 may contain an identifier 3414, the content of which is the name of region 3412, "Hiking Trail". After the mobile phone detects that the user's activity state is running, it can determine the region corresponding to the running state according to a pre-stored correspondence, such as region 3412. Then, the mobile phone detects the user's location and obtains the location 3413 in GUI 34. If the mobile phone determines that the nearest distance between location 3413 and area 3412 is less than or equal to 150 meters, the mobile phone's map display module controls the edge line of area 3412 to be displayed in a different color than other parts of the map interface 341, thus obtaining GUI 34, and recommending the hiking trail to the user in a highlighted manner.

[0231] In other embodiments, the edge line of region 3412 may be displayed in a blinking manner.

[0232] like Figure 3G As shown, Figure 3G The illustrated GUI 35 is another exemplary display interface for a map app to display recommended information. GUI 35 includes a map interface 351, on which icons 3511 and dialog boxes 3512 are displayed. The content of the dialog box 3512 is, for example, "Here are some items you might like!" The color of the icon 3511 on the map interface 351 is different from the color of other areas of the map interface 351. In this embodiment, the user's environment is, for example, a shopping mall. Icon 3511 represents the user's activity state as walking. The mobile phone can determine areas in the mall with high user frequency based on pre-stored data. Then, if the mobile phone determines that the user's location is less than or equal to 50 meters from the nearest area, the mobile phone's map display module displays the dialog box 3512 in the corresponding area, resulting in GUI 35, thereby recommending popular stores to the user in the form of prompts.

[0233] In other embodiments, the phone may display a sound while simultaneously displaying dialog box 3512 to alert the user.

[0234] It should be understood that the above-described embodiments of mobile phones displaying recommendation information are merely illustrative and do not constitute a limitation on the embodiments of this application. In other embodiments, the mobile phone may also display recommendation information in other ways, or a combination of other ways.

[0235] As can be seen, by adopting this implementation method, the mobile phone can adaptively recommend popular areas to the user based on the user's activity status and location, thereby providing more visual services to the user based on the map and optimizing the user experience.

[0236] Example 4:

[0237] As described in the foregoing embodiments, the mobile phone provides multiple display modes for the map app. Therefore, users can choose the display mode for the map app themselves. Figures 5A to 5C This illustrates an exemplary display interface during the process of a mobile phone responding to a user's selection operation.

[0238] Combination Figure 3B The illustrated embodiment describes a map app whose user center provides interfaces for various function settings. For example... Figure 5A As shown, Figure 5A The illustrated GUI51 is an example display of the main interface of the map app's user center. GUI51 includes settings interfaces for various commonly used functions and user personalization functions. Commonly used functions include offline maps, ride-hailing, and frequently used addresses. User personalization functions include my contributions, I am a merchant, and Settings 511. After receiving the user's command to click Settings 511, the phone responds to the command and enters the main settings interface. Figure 5A The illustrated GUI51 has been updated to Figure 5B The GUI52 shown.

[0239] like Figure 5B As shown, Figure 5B The illustrated GUI52 is an example display interface of the main settings screen of a map app. GUI52 includes settings items such as general settings, navigation settings, voice settings, and display settings 521. After receiving a user's command to click on display settings 521, the phone responds to the command and enters the display settings interface. Figure 5B The illustrated GUI52 has been updated to Figure 5C The GUI53 shown in the illustration.

[0240] like Figure 5C As shown, Figure 5C The illustrated GUI53 is an example display interface of a map app's display settings interface. GUI53 includes display mode options such as floating display 531 and 2D embedded display 532. For example, after receiving the user's operation command to select floating display 531, the phone displays the icon interface as a small window floating above the map interface, such as... Figure 3B and Figure 3C As shown. After receiving the user's operation command to select the 2D embedded display 532, the phone embeds the icon into the map interface for display, as shown. Figure 3D and Figure 3E As shown.

[0241] Understandable. Figure 5C This is merely an illustrative description and does not constitute a limitation on the display modes involved in the embodiments of this application. In the embodiments of this application, the mobile phone may also include more or fewer display mode options. For example, in some other embodiments, the mobile phone may also include options for icon display color and icon style, etc.

[0242] Example 5:

[0243] The scenarios illustrated in Examples 1 to 4 are all scenarios where the map has the function of displaying the activity status of people enabled. In actual operation, users can trigger the function of displaying the activity status of people to be turned off as needed. Figure 6 This illustrates an exemplary display interface of a mobile phone responding to a user's action to close the app.

[0244] Combination Figure 3B The illustrated embodiment describes a map app whose main interface includes an entry point for shortcut functions of layer 3241. Users can disable the display of activity status using the functions provided by layer 3241. Figure 6 As shown, Figure 6 The illustrated GUI 60 is an exemplary display interface of the layer settings interface. GUI 60 includes options for map display items such as classic map, public transport, and AR map 601. For example, classic map and AR map 601 are checked, meaning they are selected. In this embodiment, AR map 601 instructs the map app to display the aforementioned icon. After receiving a user's click on AR map 601, the phone responds to this instruction to de-display the aforementioned icon and switch AR map 601 to an unchecked state. Consequently, the aforementioned icon will no longer be displayed on the map interface. Of course, when the user clicks on AR map 601 again, the phone responds to this instruction to display the aforementioned icon again and switch AR map 601 to a checked state. Consequently, the aforementioned icon will reappear on the map interface.

[0245] As can be seen, by adopting this implementation method, users can flexibly set whether to display the character's activity status on the map interface, thereby optimizing the user experience.

[0246] Understandable Figures 3A to 6 This is merely an illustrative description and does not constitute a limitation on the embodiments of this application. In other embodiments, the map display interface may differ depending on the version of the map app. The color and style of the icons in the above embodiments may also be other, and the display mode of the recommendation information may also be other; this application does not limit these aspects.

[0247] Furthermore, the above embodiments are merely illustrative examples using mobile phones and do not constitute a limitation on the embodiments of this application. In other embodiments, the electronic devices involved in the embodiments of this application can also be foldable screen devices, tablet computers, etc. This is not a limitation.

[0248] In summary, the AR module running on the electronic device involved in this application extracts the activity state features of a person from the pose sequence of the electronic device by stacking one-dimensional convolutional layers. This not only offers good scalability but also allows for the extraction of features representing the person's activity state from multiple dimensions, resulting in a more detailed and accurate representation of the person's activity state. Furthermore, the interaction between the AR module and the map function enables the electronic device to adaptively and in real-time present the person's activity state on the map interface, thereby optimizing the visualization effect of the map function and improving the user experience.

[0249] The above embodiments describe the various solutions of the activity recognition and display methods provided in this application from the perspectives of the hardware structure, software architecture, and actions performed by the hardware and software components of the electronic device. Those skilled in the art should readily recognize that, in conjunction with the processing steps described in the embodiments disclosed herein, such as obtaining poses and feature data, this application can be implemented not only in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0250] For example, the aforementioned electronic device 100 can implement the corresponding functions in the form of functional modules. Figure 7A As shown, the electronic device 70 may include a processing module 701, a receiving module 702, and a display module 703. The processing module 701 includes an AR module and can be used to perform the above-described... Figures 2B to 5C The illustrated embodiments involve functions such as feature data extraction, classification, and display control. The receiving module 702 can be used to perform the above-described functions. Figures 2A to 6 The illustrated embodiments involve operation commands and the reception of sensor data. The display module 703 can be used to execute the above. Figures 2A to 6 The illustrated embodiments involve functions such as interface display.

[0251] It is understandable that the above division of modules is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. For example, the function of processing module 701 can be integrated into a processor, the function of receiving module 702 can be integrated into a transceiver, and the function of display module 703 can be integrated into a display. Figure 7B As shown, the electronic device 71 may include a processor 711, a transceiver 712, a display 713, and a memory 714. The memory 714 is coupled to the processor 711 and stores the necessary computer program 715 for the electronic device 71. The processor 711 is configured to invoke the computer program 715 to execute. Figures 2A to 6 The transceiver 712 is configured to receive various user input commands and sensor data from the electronic device. The display 713 is configured to display a map of the electronic device 71 and icons indicating the user's activity status. Further details are omitted here.

[0252] In a specific implementation, corresponding to an electronic device, this application also provides a computer storage medium, wherein the computer storage medium, located in any device, can store a program, which, when executed, can perform actions including... Figures 2A to 6 Some or all of the steps in the various embodiments of the provided method. The storage medium in any device may be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0253] One or more of the above modules or units can be implemented by software, hardware, or a combination of both. When any of the above modules or units is implemented by software, the software exists as computer program instructions and is stored in memory. The processor can be used to execute the program instructions and implement the above method flow. The processor can include, but is not limited to, at least one of the following: a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a microcontroller unit (MCU), or an artificial intelligence processor, etc., and various computing devices that run software. Each computing device may include one or more cores for executing software instructions to perform calculations or processing. The processor can be built into a SoC (System-on-a-Chip) or an application-specific integrated circuit (ASIC), or it can be a separate semiconductor chip. In addition to the cores for executing software instructions to perform calculations or processing, the processor may further include necessary hardware accelerators, such as field-programmable gate arrays (FPGAs), PLDs (programmable logic devices), or logic circuits that implement dedicated logic operations.

[0254] When the above modules or units are implemented in hardware, the hardware can be any one or any combination of CPU, microprocessor, DSP, MCU, artificial intelligence processor, ASIC, SoC, FPGA, PLD, special purpose digital circuit, hardware accelerator or non-integrated discrete device, which can run the necessary software or perform the above method flow independently of software.

[0255] When the above modules or units are implemented using software, they can be implemented in whole or in part as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0256] It should be understood that in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments.

[0257] The various parts of this specification are described in a progressive manner. Similar or identical parts between the different embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, the device and system embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant details can be found in the description of the method embodiments.

[0258] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.< / canvas> < / video> < / videoview> < / imgview> < / textview>

Claims

1. An activity recognition method, characterized in that, include: Obtain multiple poses of an electronic device within a time period; Multiple one-dimensional convolutional layers are used to obtain one or more feature data based on the multiple poses. The one or more feature data respectively indicate different types of information about the person holding the electronic device during the time period. The type of information indicated by each feature data is the person's motion state, the person's action, or the tool carrying the person. The electronic device is controlled to display the type of information indicated by one or more feature data in the form of icons; The step of obtaining one or more feature data based on the multiple poses using multiple one-dimensional convolutional layers includes: Feature extraction is performed on the multiple poses using m one-dimensional convolutional layers that are sequentially combined in the first feature extraction framework to obtain an initial feature data set; Using n backbone models sequentially combined in the second feature extraction framework, feature extraction is performed on the initial feature data set to obtain a feature data set. One or more feature data are then selected from the feature data set. Each of the n backbone models contains at least two one-dimensional convolutional layers combined in parallel, and m and n are both positive integers. The kernel k values ​​of the at least two one-dimensional convolutional layers combined in parallel in each backbone model are different. The first backbone model forms a data set from the outputs of at least two one-dimensional convolutional layers, and the value of k represents the number of consecutive poses among the multiple poses used in each convolution operation of the one-dimensional convolutional layer. The step of filtering one or more feature data from the feature data group includes: Based on the feature data set, obtain a sub-feature data set corresponding to each type in all or some of the at least one type; The feature data with the highest confidence level is obtained from each of the sub-feature data groups, and all the feature data with the highest confidence level are used as the one or more feature data, wherein the confidence level is used to indicate the degree of credibility of the behavior corresponding to the feature data; After obtaining the feature data with the highest confidence from each of the sub-feature data groups, the process further includes: Based on each feature data and the state transition matrix associated with the feature data, calculate the probability that the activity state corresponding to the feature data will transition to each other activity state, and obtain at least one probability value. The state transition matrix includes the probability that any two activity states in the type of information indicated by the feature data will transition to each other. When at least one probability value contains a value greater than a first preset threshold, the transitioned activity state data corresponding to the largest probability value among the probability values ​​greater than the first preset threshold is used as the feature data corresponding to the type; or, When none of the at least one probability value contains a value greater than a first preset threshold, the feature data with the highest confidence level is taken as the feature data corresponding to the type.

2. The method as described in claim 1, characterized in that, The step of performing feature extraction on the multiple poses using m sequentially combined one-dimensional convolutional layers in the first feature extraction framework includes: The first initial feature data group is extracted by the i-th one-dimensional convolutional layer among the m one-dimensional convolutional layers to obtain the second initial feature data group. The first initial feature data group is the feature data group obtained by the (i-1)-th one-dimensional convolutional layer among the m one-dimensional convolutional layers according to the multiple poses, where i is greater than or equal to 2 and less than or equal to m.

3. The method as described in claim 1 or 2, characterized in that, The screen of the electronic device also displays a map, with at least some of the icons displayed on the map.

4. The method as described in claim 1, characterized in that, Obtain multiple poses of an electronic device over a time period, including: Acquire at least one set of sensor data to characterize the pose of the electronic device; Detect whether there are any missing data points in each group of sensor data in the at least one group of sensor data; Sampling and linear interpolation are performed on at least one set of sensor data that has no missing data to obtain multiple sensor data; Filtering and gravity residual removal operations are performed on the multiple sensor data to obtain the multiple poses.

5. The method as described in claim 4, characterized in that, The sensor data includes measurement data from the gyroscope sensor, the accelerometer sensor, and the magnetometer sensor.

6. An electronic device, characterized in that, include: A processor, a memory, and a display, wherein the processor is coupled to the memory, the memory is used to store programs or instructions, and the processor executes the programs or instructions to obtain multiple poses of an electronic device over a time period; The processor is further configured to use multiple one-dimensional convolutional layers to obtain one or more feature data based on the multiple poses, wherein the one or more feature data respectively indicate different types of information of the person holding the electronic device during the time period, and the type of information indicated by each feature data is the person's motion state, the person's action, or the tool carrying the person; The display is configured to display, in response to control by the processor, the type of information indicated by the one or more feature data in the form of icons; Specifically, the processor is used for: Feature extraction is performed on the multiple poses using m one-dimensional convolutional layers that are sequentially combined in the first feature extraction framework to obtain an initial feature data set; Using n backbone models sequentially combined in the second feature extraction framework, feature extraction is performed on the initial feature data set to obtain a feature data set. One or more feature data are then selected from the feature data set. Each of the n backbone models contains at least two one-dimensional convolutional layers combined in parallel, and m and n are both positive integers. The kernel k values ​​of the at least two one-dimensional convolutional layers combined in parallel in each backbone model are different. The first backbone model forms a data set from the outputs of at least two one-dimensional convolutional layers, and the value of k represents the number of consecutive poses among the multiple poses used in each convolution operation of the one-dimensional convolutional layer. The processor is specifically used for: Based on the feature data set, obtain a sub-feature data set corresponding to each type in all or some of the at least one type; The feature data with the highest confidence level is obtained from each of the sub-feature data groups, and all the feature data with the highest confidence level are used as the one or more feature data, wherein the confidence level is used to indicate the degree of credibility of the behavior corresponding to the feature data; The processor is also used for: After obtaining the feature data with the highest confidence from each of the sub-feature data groups, the probability of the activity state corresponding to the feature data transitioning to each other activity state is calculated based on each feature data and the state transition matrix associated with the feature data, to obtain at least one probability value. The state transition matrix includes the probability of any two activity states transitioning to each other in the type of information indicated by the feature data. When at least one probability value contains a value greater than a first preset threshold, the transitioned activity state data corresponding to the largest probability value among the probability values ​​greater than the first preset threshold is used as the feature data corresponding to the type; or, When none of the at least one probability value contains a value greater than a first preset threshold, the feature data with the highest confidence level is taken as the feature data corresponding to the type.

7. The electronic device as claimed in claim 6, characterized in that, The processor performs feature extraction on the multiple poses using m one-dimensional convolutional layers sequentially combined in the first feature extraction framework, including: The processor is configured to perform feature extraction on the first initial feature data group using the i-th one-dimensional convolutional layer among the m one-dimensional convolutional layers to obtain a second initial feature data group. The first initial feature data group is a feature data group obtained by the (i-1)-th one-dimensional convolutional layer among the m one-dimensional convolutional layers based on the multiple poses, where i is greater than or equal to 2 and less than or equal to m.

8. The electronic device as claimed in claim 6 or 7, characterized in that, The display is also used to display a map and to display at least a portion of the icons on the map.

9. A computer-readable storage medium, characterized in that, Includes a computer program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1 to 5.

10. A computer program product, characterized in that, Includes instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 5.

Images