State recognition and model training method, equipment, system and medium

Abstract

The invention discloses a state recognition and model training method, device and system and a medium, and relates to the technical field of object recognition. The state recognition method is applied to the wearable device, and comprises the following steps: acquiring sensing data about the state of a target object; extracting features of the sensing data based on a carried first sub-model, so that the first sub-model outputs an intermediate execution result with the features; and transmitting the intermediate execution result to terminal equipment carrying a second sub-model, so that the terminal equipment processes the intermediate execution result to obtain a first state recognition result. Wherein the first sub-model and the second sub-model are obtained by slicing a pre-established target state recognition model. According to the method, the features of the sensing data are extracted through the sub-models obtained through model slicing processing, so that the intermediate execution result with the features is transmitted between the wearable device and the terminal device, and compared with transmission of all original data, the data transmission amount is reduced.

Description

Technical Field

[0001] The present application relates to the field of object recognition technology, and in particular to a method, device, system and medium for state recognition and model training thereof. Background Art

[0002] State recognition of target objects, such as posture recognition, motion state recognition, and physical health status recognition, has a wide range of applications in many fields. For example, in the medical and health industry, real-time recognition of human motion posture can help medical staff understand the patient's recovery status. Alternatively, real-time monitoring of the elderly's motion posture can quickly trigger an alarm when their posture is abnormal. In the sports industry, recording athletes' motion posture can help evaluate their performance, analyze the technical key points of running, jumping, walking, and other movements, and provide data support for sports training. In daily life, based on the user's historical data, a personalized motion spatiotemporal matrix can be constructed for the user, and product recommendations can be made at the appropriate time, or the user's daily data can be recorded for reference.

[0003] Currently, the most common way to achieve status recognition is to transmit all the raw sensor data detected by the wearable device to the terminal device through a communication module such as a Bluetooth module for status recognition. However, this brings a large amount of data transmission, occupies a large amount of communication module bandwidth, and leads to increased data packet loss and energy consumption. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a method, device, system and medium for state recognition and model training thereof, so as to at least partially solve the above-mentioned technical problems.

[0005] In order to achieve the above-mentioned purpose, the first aspect of the present application provides a state recognition method based on model slicing, which is applied to a wearable device and includes: obtaining sensor data about the state of a target object; extracting features of the sensor data based on a first sub-model, so that the first sub-model outputs an intermediate execution result with the features; and transmitting the intermediate execution result to a terminal device equipped with a second sub-model, so that the terminal device processes the intermediate execution result based on the second sub-model to obtain a first state recognition result. Wherein, the first sub-model and the second sub-model are sub-models obtained by slicing a pre-established target state recognition model.

[0006] In an embodiment of the present application, the wearable device is further equipped with the second sub-model, and the state recognition method further includes: processing the intermediate execution result based on the second sub-model to obtain a second state recognition result.

[0007] The second aspect of the present application provides a state recognition method based on model slicing, which is applied to a terminal device and includes: obtaining an intermediate execution result transmitted by a wearable device, wherein the intermediate execution result has the characteristics of sensor data about the state of a target object, and the characteristics are obtained by the wearable device extracting the sensor data based on a first sub-model; and processing the intermediate execution result based on a second sub-model to obtain a first state recognition result. The first sub-model and the second sub-model are sub-models obtained by slicing a pre-established target state recognition model.

[0008] In an embodiment of the present application, the state identification method also includes: transmitting the intermediate execution result to a server equipped with the second sub-model, so that the server processes the intermediate execution result based on the second sub-model to obtain a third state identification result; and / or transmitting the first state identification result to the server.

[0009] The third aspect of the present application provides a wearable device, comprising: an acquisition module for acquiring sensor data about the state of a target object; a first processing module for extracting features of the sensor data based on a first sub-model, so that the first sub-model outputs an intermediate execution result with the features; and a first transmission module for transmitting the intermediate execution result to a terminal device equipped with a second sub-model, so that the terminal device processes the intermediate execution result based on the second sub-model to obtain a first state recognition result. The first sub-model and the second sub-model are sub-models obtained by slicing a pre-established target state recognition model.

[0010] In an embodiment of the present application, the wearable device is further equipped with the second sub-model, and the first processing module is further used to process the intermediate execution result based on the second sub-model to obtain a second state recognition result.

[0011] In a fourth aspect, the present application provides a terminal device, comprising: a second transmission module for acquiring an intermediate execution result transmitted by a wearable device, wherein the intermediate execution result has characteristics of sensor data about the state of a target object, and the characteristics are obtained by the wearable device extracting the sensor data based on a first sub-model; and a second processing module for processing the intermediate execution result based on a second sub-model to obtain a first state recognition result. The first sub-model and the second sub-model are sub-models obtained by slicing a pre-established target state recognition model.

[0012] In an embodiment of the present application, the second transmission module is also used to: transmit the intermediate execution result to a server equipped with the second sub-model, so that the server processes the intermediate execution result based on the second sub-model to obtain a third state recognition result; and / or transmit the first state recognition result to the server.

[0013] The fifth aspect of the present application provides a state recognition system, including a wearable device and a terminal device. The wearable device is configured to: obtain sensor data about the state of the target object; extract the features of the sensor data based on the first sub-model carried by it to obtain an intermediate execution result with the features; and transmit the intermediate execution result to the terminal device. The terminal device is configured to: obtain the intermediate execution result from the wearable device; and process the intermediate execution result based on the second sub-model carried by it to obtain a first state recognition result. Wherein, the first sub-model and the second sub-model are sub-models obtained by slicing a pre-established target state recognition model.

[0014] In an embodiment of the present application, the target state recognition model is a neural network model, the first sub-model is a convolutional sub-model including the input layer and convolutional layer of the neural network model, and the second sub-model is a fully connected sub-model including the fully connected layer and output layer of the neural network model.

[0015] In an embodiment of the present application, the wearable device is further configured to process the intermediate execution result based on the carried second sub-model to obtain a second state recognition result.

[0016] In the embodiment of the present application, the second sub-model carried by the wearable device is also subjected to model quantization processing.

[0017] The sixth aspect of the present application provides a state recognition model training method, including: training a pre-built initial state recognition model based on state sample data to obtain a target state recognition model; performing model slicing processing on the target state recognition model to obtain a first sub-model and a second sub-model, wherein the first sub-model is used to extract the features of the sensor data about the state of the target object to obtain an intermediate execution result with the features, and the second sub-model is used to process the intermediate execution result to obtain a state recognition result; and importing the first sub-model into a wearable device and importing the second sub-model into a terminal device.

[0018] In an embodiment of the present application, the state recognition model training method further includes: importing the second sub-model into the wearable device.

[0019] In an embodiment of the present application, the state recognition model training method further includes: performing model quantization processing on the second sub-model before importing the second sub-model into the wearable device.

[0020] In an embodiment of the present application, the target state recognition model includes an input layer, a convolutional layer, a fully connected layer and an output layer, and the model slicing processing of the target state recognition model includes: splitting between the convolutional layer and the fully connected layer, so that the first sub-model is a convolutional sub-model including the input layer and the convolutional layer, and the second sub-model is a fully connected sub-model including the fully connected layer and the output layer.

[0021] In the seventh aspect of the present application, an electronic device is provided, comprising: a memory configured to store instructions; and a processor configured to call the instructions from the memory and to implement any of the above-mentioned state recognition methods, or any of the above-mentioned state recognition model training methods when executing the instructions.

[0022] In an eighth aspect, the present application provides a machine-readable storage medium having stored thereon instructions for enabling a machine to execute any of the above-described state recognition methods, or to execute any of the above-described state recognition model training methods.

[0023] Through the above technical solution, the embodiment of the present application extracts the features of the sensor data through the sub-model obtained by model slicing processing, so that the intermediate execution results with the features are transmitted between the wearable device and the terminal device. Compared with transmitting all the original sensor data, the data transmission volume is greatly reduced, thereby reducing the bandwidth occupancy of the communication module, avoiding increased data packet loss and increased energy consumption due to occupying a large amount of communication module bandwidth.

[0024] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the following detailed description, they are used to explain the embodiments of the present application but do not constitute a limitation on the embodiments of the present application. In the accompanying drawings:

[0026] Figure 1 The following schematically shows a flow chart of a state recognition method based on model slicing according to the first embodiment of the present application;

[0027] Figure 2 The following schematically shows a flow chart of a state recognition method based on model slicing according to the second embodiment of the present application;

[0028] Figure 3The following schematically shows a functional structure diagram of a wearable device according to the third embodiment of the present application;

[0029] Figure 4 The following schematically shows a functional structure diagram of a terminal device according to the fourth embodiment of the present application;

[0030] Figure 5 The following schematically shows a functional structure diagram of a state recognition system according to the fifth embodiment of the present application;

[0031] Figure 6 The following schematically shows a structural diagram of an exemplary recognition model according to an embodiment of the present application;

[0032] Figure 7 The following schematically shows a flow chart of a state recognition model training method according to the sixth embodiment of the present application;

[0033] Figure 8 The figure schematically shows a flow chart of the cooperation between the wearable device and the terminal device for performing motion posture prediction according to the seventh embodiment of the present application;

[0034] Figure 9 A schematic diagram schematically illustrates the minimum operating device of the deployment posture prediction method or system according to the seventh embodiment of the present application; and

[0035] Figure 10 The structural diagram of an electronic device according to the eighth embodiment of the present application is schematically shown.

[0036] Description of Reference Numerals

[0037] 110, acquisition module; 120, first processing module; 130, first transmission module.

[0038] 210. Second transmission module; 220. Second processing module.

[0039] 100. Wearable devices; 200. Terminal devices. DETAILED DESCRIPTION

[0040] To make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0041] It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of this application are in compliance with the relevant provisions of national laws and regulations. In the embodiments of this application, certain software, components, models, and other existing solutions in the industry may be mentioned. These should be considered as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of this application, but it does not mean that the applicant has or will necessarily use such solutions.

[0042] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement status, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indications will also change accordingly.

[0043] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0044] Example 1

[0045] Figure 1 The following schematically shows a flow chart of a state recognition method based on model slicing according to the first embodiment of the present application. Figure 1 As shown, the first embodiment of the present application provides a state recognition method based on model slicing, which is applied to a wearable device, and the state recognition method may include the following steps S110-S130.

[0046] Step S110: Acquire sensor data about the state of the target object.

[0047] The target object state includes, but is not limited to, posture, motion state, and physical health status. For step S110, for example, an accelerometer or an inertial measurement unit (IMU) can be used to collect multi-dimensional state data of the target object in different states. In a further example, the accelerometer is a sensor circuit system that can collect the three-dimensional linear acceleration, three-dimensional angular rate, and three-dimensional magnetic field strength of the target object, and its core is a nine-axis inertial sensor chip.

[0048] Step S120: extracting features of the sensor data based on the installed first sub-model, so that the first sub-model outputs an intermediate execution result having the features.

[0049] Step S130: transmitting the intermediate execution result to a terminal device equipped with a second sub-model, so that the terminal device processes the intermediate execution result based on the second sub-model to obtain a first state recognition result.

[0050] In steps S120 and S130, the first and second sub-models are obtained by slicing a pre-established target state recognition model. That is, the target state recognition model is divided into a first sub-model for feature extraction and a second sub-model for state recognition, and the two sub-models are respectively imported into the wearable device and the terminal device for application. Model slicing is also referred to as model fragmentation, which means slicing the model into sub-models for different layers according to its hierarchical structure.

[0051] While step S130 above indicates that the terminal device performs state recognition to obtain a first state recognition result, in other examples, the wearable device may also perform state recognition directly. In this way, the wearable device can be equipped with the second sub-model, so that the state recognition method further includes: processing the intermediate execution result based on the second sub-model on the wearable device to obtain a second state recognition result.

[0052] It should be noted that the target state recognition model can be obtained by training a neural network with sample data, and the specific process will be described in detail in other embodiments below, so it will not be repeated here.

[0053] The state recognition method of the first embodiment of the present application is applied to a wearable device. It extracts the features of the sensor data through the sub-model obtained by model slicing processing, so that the intermediate execution results with the features are transmitted between the wearable device and the terminal device. Compared with transmitting all the original sensor data, the data transmission volume is greatly reduced, thereby reducing the bandwidth occupied by the communication module, avoiding the increase in data packet loss and energy consumption caused by occupying a large amount of communication module bandwidth. In addition, the first embodiment of the present application can obtain a second state recognition result by the wearable device equipped with a second sub-model, so that the wearable device does not need to rely on a terminal device with stronger computing power, storage capacity and endurance to realize the state recognition of the target object.

[0054] Example 2

[0055] Figure 2 The following schematically shows a flow chart of a state recognition method based on model slicing according to the second embodiment of the present application. Figure 2As shown, the second embodiment of the present application provides a state recognition method based on model slicing, which is applied to a terminal device, and the state recognition method may include the following steps S210-S220.

[0056] Step S210: Obtain the intermediate execution result transmitted by the wearable device.

[0057] The intermediate execution result has features of sensor data about the state of the target object, and the features are obtained by extracting the sensor data by the wearable device based on the first sub-model carried by the wearable device.

[0058] Step S220: Process the intermediate execution result based on the second sub-model to obtain a first state recognition result.

[0059] The first sub-model and the second sub-model are sub-models obtained by slicing a pre-established target state recognition model.

[0060] In the above-mentioned step S220, the terminal device performs state identification, thereby obtaining a first state identification result. However, in other examples, the terminal device may also only transfer the intermediate execution result of step S210, and the state identification is completed by the background server, etc. In this way, following step S210, the state identification method of this embodiment may also include: transmitting the intermediate execution result to a server equipped with the second sub-model, so that the server processes the intermediate execution result based on the second sub-model to obtain a third state identification result. That is, taking the terminal device as a smart phone as an example, the mobile phone may be equipped with the second sub-model, so that after receiving the intermediate execution result, the mobile phone directly identifies the state; the mobile phone may also not be equipped with the second sub-model, and it only acts as a data transfer device to transmit the intermediate execution result to the server, and the server is equipped with the second sub-model to perform state identification.

[0061] In addition, similar to transmitting the intermediate execution result to the server, the first state identification result may also be transmitted to the server.

[0062] It should be noted that for more implementation details and effects of this embodiment, please refer to the above-mentioned embodiment 1, which will not be described in detail here.

[0063] Example 3

[0064] Based on the same inventive concept as that of the first embodiment, the third embodiment of the present application provides a wearable device. Figure 3As shown, the wearable device includes: an acquisition module 110, used to obtain sensor data about the state of the target object; a first processing module 120, used to extract features of the sensor data based on the first sub-model carried, so that the first sub-model outputs an intermediate execution result with the features; and a first transmission module 130, used to transmit the intermediate execution result to a terminal device equipped with a second sub-model, so that the terminal device processes the intermediate execution result based on the second sub-model to obtain a first state recognition result.

[0065] The first sub-model and the second sub-model are sub-models obtained by slicing a pre-established target state recognition model.

[0066] In a preferred embodiment, the wearable device may also be equipped with the second sub-model, and the first processing module 120 is further configured to process the intermediate execution result based on the second sub-model to obtain a second state recognition result.

[0067] In this example, the acquisition module 110, the first processing module 120, and the first transmission module 130 can be implemented by a minimum functional system that runs a target state recognition model on a wearable device. Suitable for wearable devices that are lightweight, easy to wear, have certain data processing and transmission capabilities, have limited battery life, and can display calculation results to the user, this minimum functional system generally has the following hardware modules:

[0068] 1) Sensor module: used to collect real-time motion status data and transmit the data to the computing module, etc.

[0069] 2) Computing module: such as a microprocessor, which has smaller computing functions and a memory of less than 256 kilobytes (256KB). It is used to load algorithm models, receive data from sensor modules and calculate results, and can also call other module functions.

[0070] 3) Storage module: such as a memory, used to store the model method, intermediate results of the algorithm model and other functional programs.

[0071] 4) Bluetooth module: used to transmit the intermediate results of the algorithm model to the terminal device, etc.

[0072] 5) Display module: used to display the results.

[0073] 6) Power module: used to power the equipment.

[0074] Based on this, the acquisition module 110 is implemented, for example, using a sensor module herein, such as the aforementioned accelerometer or inertial measurement unit (IMU); the first processing module 120 is implemented, for example, using a computing module herein; and the first transmission module 130 is implemented, for example, using a Bluetooth module herein. Furthermore, the storage module herein can also be used to store intermediate execution results and second state recognition results.

[0075] For more implementation details and effects of Example 3 of the present application, please refer to the above-mentioned Example 1 for understanding, and will not be repeated here.

[0076] Example 4

[0077] Based on the same inventive concept as that of the second embodiment, the fourth embodiment of the present application provides a terminal device. Figure 4 As shown, the terminal device includes: a second transmission module 210 for acquiring an intermediate execution result transmitted by a wearable device, wherein the intermediate execution result has characteristics of sensor data about the state of the target object, and the characteristics are obtained by the wearable device extracting the sensor data based on the first sub-model; and a second processing module 220 for processing the intermediate execution result based on the second sub-model to obtain a first state recognition result. The first sub-model and the second sub-model are sub-models obtained by slicing a pre-established target state recognition model.

[0078] In a preferred embodiment, the second transmission module 210 is also used to: transmit the intermediate execution result to a server equipped with the second sub-model, so that the server processes the intermediate execution result based on the second sub-model to obtain a third state recognition result; and / or transmit the first state recognition result to the server.

[0079] Generally, terminal devices have stronger computing power, storage capacity, and battery life than wearable devices, and can also be equipped with wireless network modules (WIFI modules, 4G modules, or 5G modules, etc.), suitable for more advanced applications. Taking smartphones as an example, they generally include the following hardware modules:

[0080] 1) Wireless module: such as WIFI module, used to upload data to the cloud.

[0081] 2) Computing module: For example, a processor, which has powerful computing capabilities and a memory greater than 4 gigabytes (4GB); used to load algorithm models, read data from the memory module and calculate results, and can also call other module functions.

[0082] 3) Storage module: such as memory (or storage), used to store the intermediate results of the model method, algorithm model and other functional programs.

[0083] 4) Bluetooth module: used to receive the intermediate results of the algorithm model.

[0084] 5) Display module: used to display the results.

[0085] 6) Power module: used to power the equipment.

[0086] Based on this, the second processing module 220 can be implemented by the computing module built into the smartphone, and the second transmission module 210 can be implemented by using its Bluetooth module and wireless module. In addition, the storage module can also be used to store the first state recognition result and intermediate execution results.

[0087] For more implementation details and effects of the terminal device of this embodiment, please refer to Example 2 and will not be repeated here.

[0088] Example 5

[0089] like Figure 5 As shown, the fifth embodiment of the present application provides a state recognition system, including a wearable device 100 and a terminal device 200. The wearable device 100 is configured to: obtain sensor data about the state of the target object; extract features of the sensor data based on the first sub-model carried by it to obtain an intermediate execution result having the features; and transmit the intermediate execution result to the terminal device. In addition, the terminal device 200 is configured to: obtain the intermediate execution result from the wearable device; and process the intermediate execution result based on the second sub-model carried by it to obtain a first state recognition result.

[0090] The first sub-model and the second sub-model are sub-models obtained by slicing a pre-established target state recognition model.

[0091] In the example, the wearable device 100 can be a smart bracelet, a smart watch, a smart glove, a smart wristband, a smart dress, a smart pants, a smart shoe, etc.; the terminal device 200 can be a smart phone, a laptop, a tablet computer, a smart router, a smart conversation wizard, etc.

[0092] In a preferred embodiment, the target state recognition model is a neural network model, the first sub-model is a convolutional sub-model including the input layer and convolutional layer of the neural network model, and the second sub-model is a fully connected sub-model including the fully connected layer and output layer of the neural network model. Figure 6 As shown, it can be designed as one convolutional layer and three fully connected layers.

[0093] In a preferred embodiment, the wearable device 100 is further configured to process the intermediate execution result based on the carried second sub-model to obtain a second state recognition result.

[0094] In a more preferred embodiment, the second sub-model carried by the wearable device is further subjected to model quantization processing. Model quantization processing refers to mapping model parameters into numerical forms (such as integers) that occupy less memory according to certain rules. This can greatly reduce the model's use of system memory resources without significantly reducing model accuracy.

[0095] by Figure 6 For example, the first sub-model is a convolutional sub-model with a small number of parameters, including a convolutional layer, which can be used to extract vector data features, and the second sub-model is a fully connected sub-model including a fully connected layer, a large number of parameters, and can be used to classify data features. On the basis of model segmentation, the fully connected sub-model carried by the wearable device 100 is quantized to form a lightweight deep neural network model, which helps to reduce the computing resource usage of the deep neural network model in the wearable device 100. In addition, the obtained lightweight deep neural network model greatly reduces the computing resources it occupies, so that the computing resources of the wearable device 100 can be directly used to realize state recognition. Without considering other more advanced application scenarios, there is no need to request state recognition from the terminal device through a communication module such as a Bluetooth module, thereby achieving the purpose of reducing the amount of transmitted data.

[0096] Note that on the wearable device 100, not both the convolutional sub-model and the fully connected sub-model are quantized, meaning they are not fully quantized. This is because full quantization results in a loss of data precision. Therefore, retaining the unquantized state of the convolutional sub-model helps achieve a more accurate intermediate execution result. This intermediate execution result is then transmitted by the wearable device 100 to the terminal device 200, which has greater processing power, to help ensure recognition accuracy and enable more advanced applications on the terminal device 200. For example, the second state recognition result obtained by the wearable device 100 may only include the name of the current state, while the first state recognition result obtained by the terminal device 200 may also include a state prediction result. This means that the terminal device can perform more advanced state predictions in addition to state recognition. In other words, the wearable device 100 transmits the unquantized intermediate execution result to the terminal device 200 for prediction calculation, thereby avoiding the reduction in recognition accuracy caused by the quantization of the fully connected sub-model. However, it should be noted that if high model prediction accuracy is not required, the terminal device 200 can also use the final execution result after the model quantization process for state prediction.

[0097] Therefore, the first sub-model is imported into the wearable device, and the second sub-model is imported into the terminal device compatible with the wearable device. The former is a quantized lightweight neural network model, which helps reduce the wearable device's energy consumption, while the second sub-model uses the former's intermediate execution results rather than the entire original state data, which is larger in volume. This helps reduce the Bluetooth transmission bandwidth between the wearable device and the terminal device. Furthermore, quantization processing always causes a certain loss in recognition accuracy. However, the terminal device performs state recognition and higher-level processing based on the unquantized intermediate execution results, avoiding the impact of recognition accuracy loss on processing accuracy.

[0098] In summary, the state recognition system of the embodiment of the present application includes a second sub-model that has undergone model quantization processing. This makes it easy to deploy on wearable devices and reduces the model capacity of the target state recognition model, thereby reducing resource consumption during model runtime. Consequently, the system can improve the efficiency of target object state recognition, for example, improving the efficiency of human motion posture recognition.

[0099] For more implementation details and effects of the state recognition system of this embodiment, please refer to Example 3 and Example 4, which will not be described in detail here.

[0100] Example 6

[0101] Figure 7 This is a flow chart of a state recognition model training method according to the sixth embodiment of the present application. Figure 7 As shown, the following steps S710-S730 are included:

[0102] Step S710 , training a pre-built initial state recognition model based on state sample data to obtain a target state recognition model.

[0103] Step S720: performing model slicing processing on the target state recognition model to obtain a first sub-model and a second sub-model.

[0104] The first sub-model is used to extract features of the sensor data about the state of the target object to obtain an intermediate execution result having the features, and the second sub-model is used to process the intermediate execution result to obtain a state recognition result.

[0105] Step S730: import the first sub-model into the wearable device, and import the second sub-model into the terminal device.

[0106] The implementation of steps S710 - S730 is described in detail below through examples.

[0107] 1. Constructing a target model for step S710.

[0108] First, the state data of the target object is obtained through the acceleration sensor, such as three-dimensional acceleration, linear velocity, magnetic field strength data, etc. A set of state data at time t is called a frame, and a frame of data is defined as:

[0109] x t ={a1,a2,a3,g1,g2,g3,m1,m2,m3} (1)

[0110] Among them, a1, a2, and a3 are acceleration values, g1, g2, and g3 are gyroscope values representing linear velocity, and m1, m2, and m3 are magnetic field strength values.

[0111] For example, in the example of motion posture monitoring, people can be arranged to wear wearable devices and perform various types of motion postures, so that the acceleration sensor can collect multi-dimensional state data and store the sensor data in text form, such as in a computer or memory, so as to form a corresponding data set later. For example, the data set can be defined as follows:

[0112] X′={x1,x2,x3,......,x T} (2)

[0113] Combined with the 9 parameters of formula (1), the dataset X′ is a T*9 matrix, where T is the total number of frames.

[0114] The collected data is then preprocessed to obtain state sample data. For example, preprocessing includes but is not limited to denoising and labeling data categories. For denoising, data can be cleaned using established denoising algorithms, such as sliding average filtering, Fourier transform combined with filtering, and other denoising algorithms. Denoising removes or modifies data that deviates significantly from the standard, reducing the proportion of abnormal data and making the dataset more standardized. Furthermore, the cleaned dataset is defined as X for the next step of data classification.

[0115] Further data classification is performed, for example, in the cleaned dataset X, consecutive k frames of data are labeled as a certain state type. Specifically, taking the state as posture as an example, the action posture types performed by the researchers are C types of postures such as standing, sitting, lying, falling, washing hair, changing clothes, sweeping the floor, mopping the floor and turning over, which are defined as P1, P2, ..., P C , the labeled dataset can be defined as:

[0116] X t i ={x t ,x t+1 ,x t+2 ,......,x t+k}∈P i,t∈[1,Tk],i∈[1,C] (3)

[0117] Then, according to the needs of the usage scenario, the state sample data is divided into a training set and a test set. For example, a certain amount of data is randomly selected from the data set corresponding to the state sample data to form a test set, and the rest is formed into a training set, which provides a basis for subsequent model training and testing. Then, according to the recognition target, for example, according to the action characteristics and recognition requirements of the target object, a neural network is used to prepare an initial state recognition model, such as Figure 6 For more advanced state prediction, mature solutions such as LSTM and Transformer can be used to train the corresponding initial prediction model.

[0118] The initial state recognition model is further trained and validated using the training and test sets to obtain the target state recognition model. This model training and performance validation ensure that both model accuracy and real-time performance meet requirements. This is a mature solution in the existing technology and will not be further elaborated.

[0119] 2. Model slicing in step S720.

[0120] In the example, the target state recognition model includes an input layer, a convolutional layer, a fully connected layer and an output layer, and the model slicing processing of the target state recognition model includes: splitting between the convolutional layer and the fully connected layer, so that the first sub-model is a convolutional sub-model including the input layer and the convolutional layer, and the second sub-model is a fully connected sub-model including the fully connected layer and the output layer.

[0121] That is, the target state recognition model is sliced to obtain the corresponding Figure 6 The convolution sub-model and the fully connected sub-model are used, the former is used as the first sub-model to extract the features of the sensor data about the state of the target object to obtain an intermediate execution result with the features, and the latter is used as the second sub-model to process the intermediate execution result to obtain a state recognition result.

[0122] like Figure 6 The recognition model shown is a standard neural network model, which contains a variety of basic neural network layers: input layer, convolutional neural network layer (referred to as convolutional layer), fully connected network layer (referred to as fully connected layer), output layer. Among them, the convolutional neural network layer is used to extract the continuous sensor data X of formula (3) t i The fully connected network layer is used to classify the features. Figure 6 The recognition model is sliced as follows:

[0123] 1) The input layer and convolution layer are divided into convolution sub-models (denoted as Mconv ), M conv The output result is X fla , will be used as input to the fully connected sub-model when the model is running; and the output result X fla It will be copied and sent to the terminal device through the communication module for other use;

[0124] 2) The fully connected layer and the output layer are divided into fully connected sub-models (denoted as M fc ), M fc Accept X fla As input, the output result is the result of gesture recognition P i ,The results will be displayed in the display module of the wearable device.

[0125] 3. Regarding the model import in step S730.

[0126] That is, the trained neural network model is deployed to the target hardware. The deployment process usually includes model format conversion (such as to ONNX, TensorRT, etc.), model optimization (such as pruning, quantization), device environment configuration (installation of drivers and inference frameworks), and final inference operation.

[0127] In a preferred embodiment, in addition to importing the first sub-model into the wearable device and importing the second sub-model into the terminal device, the second sub-model can also be imported into the wearable device, and before being imported into the wearable device, the second sub-model can be subjected to model quantization processing.

[0128] Back to the example, Figure 6 Specifically, the model quantization technology M is used to quantize the fully connected sub-model. fc For example, weight quantization technology can be used to process the quantization process, so that the quantization process is defined as:

[0129]

[0130] Where W is the high-precision parameter value, which is the fully connected layer weight parameter and bias parameter in step S506; s is the scaling factor, which is used to adjust the quantization range and precision; Round means rounding the result to an integer. The scaling factor is calculated based on the dynamic range of the parameter, for example, using the following formula:

[0131]

[0132] Where max(|W|) is the maximum absolute value of the parameter; b is the target bit width.

[0133] On the wearable device side, M fcSaving to disk and loading to memory both require a lot of computing resources of wearable devices, and for M fc Although the quantization technology may reduce the recognition accuracy to a certain extent, it can greatly reduce the consumption of computing resources. In addition, the memory usage after quantization is also reduced. For example, in this example, the memory usage can be reduced by 75%. conv The number of parameters is very small, and M conv The intermediate results need to be sent to the terminal device for more advanced applications, such as state prediction. Therefore, in order to avoid the loss of prediction result accuracy caused by quantization, the M conv Perform quantization operations.

[0134] Based on this, through this example, wearable devices can achieve state recognition without relying on terminals, and terminal devices can achieve state recognition and more advanced state prediction. They can also provide users with online browsing history data, dangerous posture warnings, personalized recommendations and other functions based on the prediction results.

[0135] In summary, through this example, the embodiments of the present application have at least the following advantages over the prior art:

[0136] First, it uses a neural network model with higher precision than traditional recognition algorithms, which can recognize more types of states with high recognition accuracy.

[0137] Secondly, the neural network model used for state recognition is sliced and quantized, which reduces the computing resource usage of the neural network model, making it suitable for deployment in wearable devices, enabling wearable devices to perform state recognition independently without relying on terminal devices.

[0138] Thirdly, in addition to state recognition, it can also take into account state prediction, and can further provide applications such as online browsing history data, dangerous posture warnings, and personalized recommendations, making up for the defect of existing technologies that do not consider multiple application scenarios such as state recognition and state prediction at the same time.

[0139] Fourthly, the intermediate execution results of the input data of the neural network model obtained by model slicing are transmitted to the terminal device to start more advanced applications such as state prediction. The intermediate execution results do not lose accuracy due to the model quantization processing, thereby greatly reducing the bandwidth occupancy and transmission data volume of Bluetooth communication between the wearable device and the terminal device, and ensuring the accuracy of executing more advanced applications.

[0140] For other implementation details and effects of this embodiment, please refer to the other embodiments mentioned above and will not be described again here.

[0141] Example 7

[0142] This embodiment seven is an application example corresponding to each of the above embodiments. This application example follows the examples of wearable devices and terminal devices in the above embodiments. Taking motion posture as an example, it specifically introduces how to use the embodiment of this application to perform higher-end posture prediction. Figure 8 This is a flow chart of the cooperation between the wearable device and the terminal device for motion posture prediction in the seventh embodiment of the present application, which shows a complete motion posture recognition and prediction process; Figure 9 It is the minimum operating system for deploying the posture prediction method or system of the embodiment of the present application, and it can be understood that those skilled in the art can Figure 9 Add other hardware modules or new functions based on the existing hardware.

[0143] like Figure 8 As shown, the posture prediction process performed by the embodiment of the present application includes the following key steps:

[0144] Step S801: The acceleration sensor collects posture data.

[0145] For example, an acceleration sensor continuously collects posture data at a specific frame rate to obtain real-time posture data, and obtains historical posture data to form sample data.

[0146] Step S802: The real-time posture data is preliminarily processed in the wearable device processor to form an input matrix, namely the input matrix X in the above formula (3): t i .

[0147] Step S803: In the wearable device processor, Input to M conv , get the feature intermediate result X conv , and X conv Transmitted to the terminal device through the Bluetooth module in the wearable device.

[0148] The reason for selecting the intermediate feature results (corresponding to the intermediate execution results described above) for transmission is that convolution processes the raw data to obtain the data's feature values, which reduces the transmission size of the transmission module. This reduces bandwidth usage, Bluetooth module power consumption, and packet loss.

[0149] It is worth noting that the convolutional layer can be trained on the pattern of the original input data to learn specific features. For example, in image processing, the lower convolutional layers may detect simple features such as edges and corners, while the higher convolutional layers will gradually capture complex patterns (such as object shapes or categories). The output of the convolutional layer is X conv It can reflect the local features of the original data, the spatial distribution of features, etc., which is very important for modeling the contextual relationships of inputs such as images and speech.

[0150] Step S804: In the wearable device processor, the intermediate result X conv Continue to enter M fc The final recognition result is obtained, and the recognition result can be displayed to the user through the display module.

[0151] Step S805: This step can be performed simultaneously with step S804. The terminal device receives the intermediate result X. conv and stored in the memory.

[0152] Step S806: The terminal device uses the prediction model to predict the intermediate result X. conv Processing can be performed to predict future user behavior. At the same time, the wireless module can be used to upload data and results, and the display model can be used to present the data and structure to the user.

[0153] Among them, the prediction model is, for example, an LSTM model, which can train a posture prediction model based on posture data to obtain the user's possible future behaviors, so as to predict and monitor the user's posture movements.

[0154] In this example, the Bluetooth module transmits the intermediate results of the neural network model instead of directly transmitting the original data, which reduces the amount of transmitted data and the bandwidth occupied by the Bluetooth module, resulting in less power consumption and lower packet loss rate for the wearable device. Quantitative analysis shows that bandwidth occupancy is reduced by 80%.

[0155] In a further example, the terminal device can also cooperate with the cloud (such as a cloud server), for example, uploading posture recognition results, posture prediction results, etc. to the cloud through the WIFI module in the terminal device, and the cloud comprehensively utilizes various data to provide users with personalized recommendations, monitoring, alarms and other advanced applications.

[0156] In summary, Example 7 of the present application provides a low-power, low-packet-loss-rate gesture prediction solution that can identify more motion gesture categories using fewer computing resources. Intermediate gesture recognition results can also be reused in terminal devices, thus satisfying a variety of application scenarios such as gesture recognition and gesture prediction. Specifically, this embodiment of the present application enables real-time recognition and comprehensive monitoring of motion gestures while reducing wearable device energy consumption and Bluetooth transmission bandwidth.

[0157] Example 8

[0158] Embodiment 8 of the present application provides an electronic device, such as Figure 10 As shown, it includes: a memory configured to store instructions; and a processor configured to call the instructions from the memory and to implement the state recognition method described in the above embodiment one when executing the instructions, or to implement the state recognition model training method described in embodiment six.

[0159] Among them, when executing the state recognition method, the electronic device is, for example, a minimum control unit composed of a memory and a processor integrated in a wearable device; when executing the state recognition model training method, the electronic device is, for example, an independent controller, which obtains corresponding sub-models based on the coordination of the built-in processor and memory, so that the two sub-models can be subsequently imported into the wearable device and the terminal device for application respectively.

[0160] For more implementation details and effects of the eighth embodiment, please refer to the aforementioned embodiments and will not be described in detail here.

[0161] An embodiment of the present application also provides a machine-readable storage medium, which stores instructions for enabling a machine to execute the state recognition method described in the above embodiment one, or the state recognition model training method described in the above embodiment six.

[0162] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0163] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0164] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0165] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0166] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0167] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0168] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0169] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0170] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A state recognition method based on model slicing, characterized in that: The state recognition method is applied to a wearable device and includes: Acquiring sensory data about the state of the target object; Extracting features of the sensor data based on the installed first sub-model, so that the first sub-model outputs an intermediate execution result having the features; and Transmitting the intermediate execution result to a terminal device equipped with a second sub-model, so that the terminal device processes the intermediate execution result based on the second sub-model to obtain a first state recognition result; The first sub-model and the second sub-model are sub-models obtained by slicing a pre-established target state recognition model.

2. The state recognition method according to claim 1, characterized in that: The wearable device is further equipped with the second sub-model, and the state recognition method further includes: The intermediate execution result is processed based on the second sub-model to obtain a second state recognition result.

3. A state recognition method based on model slicing, characterized in that: The state recognition method is applied to a terminal device and includes: Obtaining an intermediate execution result transmitted by the wearable device, wherein the intermediate execution result has a feature of sensor data about a state of a target object, and the feature is obtained by extracting the sensor data by the wearable device based on a first sub-model carried therein; and Processing the intermediate execution result based on the second sub-model to obtain a first state recognition result; The first sub-model and the second sub-model are sub-models obtained by slicing a pre-established target state recognition model.

4. The state recognition method according to claim 3, characterized in that: The state recognition method further includes: transmitting the intermediate execution result to a server equipped with the second sub-model, so that the server processes the intermediate execution result based on the second sub-model to obtain a third state recognition result; and / or The first state recognition result is transmitted to the server.

5. A wearable device, characterized in that: include: an acquisition module for acquiring sensor data about the state of the target object; A first processing module, configured to extract features of the sensor data based on a first sub-model, so that the first sub-model outputs an intermediate execution result having the features; as well as A first transmission module is configured to transmit the intermediate execution result to a terminal device equipped with a second sub-model, so that the terminal device processes the intermediate execution result based on the second sub-model to obtain a first state recognition result; The first sub-model and the second sub-model are sub-models obtained by slicing a pre-established target state recognition model.

6. A terminal device, characterized in that: include: a second transmission module, configured to obtain an intermediate execution result transmitted by the wearable device, wherein the intermediate execution result has a feature of the sensor data regarding the state of the target object, and the feature is obtained by extracting the sensor data by the wearable device based on the first sub-model; as well as A second processing module, configured to process the intermediate execution result based on the second sub-model carried therein to obtain a first state recognition result; The first sub-model and the second sub-model are sub-models obtained by slicing a pre-established target state recognition model.

7. A state recognition system, characterized in that: Including wearable devices and terminal devices, The wearable device is configured to: acquire sensor data about a state of a target object; extract features of the sensor data based on a first sub-model to obtain an intermediate execution result having the features; and transmit the intermediate execution result to the terminal device; The terminal device is configured to: obtain the intermediate execution result from the wearable device; and process the intermediate execution result based on the second sub-model carried therein to obtain a first state recognition result; The first sub-model and the second sub-model are sub-models obtained by slicing a pre-established target state recognition model.

8. The state recognition system according to claim 7, characterized in that: The target state recognition model is a neural network model, the first sub-model is a convolutional sub-model including an input layer and a convolutional layer of the neural network model, and the second sub-model is a fully connected sub-model including a fully connected layer and an output layer of the neural network model.

9. The state recognition system according to claim 7, characterized in that: The wearable device is further configured to process the intermediate execution result based on the carried second sub-model to obtain a second state recognition result.

10. The state recognition system according to claim 9, characterized in that: The second sub-model carried by the wearable device is also subjected to model quantization processing.

11. A state recognition model training method, characterized in that: include: Based on the state sample data, the pre-built initial state recognition model is trained to obtain the target state recognition model; Performing model slicing processing on the target state recognition model to obtain a first sub-model and a second sub-model, wherein the first sub-model is used to extract features of the sensor data about the target object state to obtain an intermediate execution result having the features, and the second sub-model is used to process the intermediate execution result to obtain a state recognition result; as well as Import the first sub-model into the wearable device, and import the second sub-model into the terminal device.

12. The state recognition model training method according to claim 11, characterized in that: The state recognition model training method further includes: importing the second sub-model into the wearable device.

13. The state recognition model training method according to claim 12, characterized in that: The state recognition model training method further includes: performing model quantization processing on the second sub-model before importing the second sub-model into the wearable device.

14. The state recognition model training method according to claim 11, characterized in that: The target state recognition model includes an input layer, a convolutional layer, a fully connected layer, and an output layer, and performing model slicing processing on the target state recognition model includes: A split is performed between the convolutional layer and the fully connected layer, so that the first sub-model is a convolutional sub-model including the input layer and the convolutional layer, and the second sub-model is a fully connected sub-model including the fully connected layer and the output layer.

15. A machine-readable storage medium, characterized in that The machine-readable storage medium stores instructions for enabling a machine to execute the state recognition method according to any one of claims 1-4, or to execute the state recognition model training method according to any one of claims 11-14.

Images