Behavior recognition model establishment and recognition method and device based on ear-mounted device

Abstract

The invention relates to the technical field of artificial intelligence, and provides a behavior recognition model establishment and recognition method and device based on ear-mounted equipment. The method comprises the following steps: acquiring first behavior data acquired by a mobile intelligent terminal and second behavior data acquired by an ear-wearing device to construct a training sample; utilizing a feature extractor to obtain a feature extraction result corresponding to the training sample; inputting the feature extraction result into a classifier and a label predictor to obtain a behavior identification prediction result and a classification result; and calculating a behavior identification prediction result and a target loss corresponding to a classification result, performing back propagation according to the target loss, updating network parameters in the feature extractor, the tag predictor and the classifier, and obtaining a behavior identification model of the ear-wearing equipment composed of the trained feature extractor, tag predictor and classifier. According to the embodiment of the invention, the data acquisition cost and training cost of the behavior recognition model based on the ear-mounted device can be reduced.

Description

TECHNICAL FIELD

[0001] The present specification relates to the technical field of artificial intelligence, and in particular to a behavior recognition model establishment method and device based on an ear-wearable device. BACKGROUND

[0002] An ear-wearable device is a complex device with multi-modal sensing and environmental perception capabilities, which can be used for human activity recognition. However, due to the dynamic and unpredictable nature of human head movement, the motion signals captured by the ear-wearable device generally exhibit more significant changes and are less stable compared to the stable motion signals captured by the inertial measurement unit sensors in traditional mobile devices (such as smartphones). Therefore, the classification task of human behavior activities based on the ear-wearable device faces the problem of high implementation difficulty. In the prior art, in order to ensure the classification accuracy of human behavior activities, a large amount of human behavior activity data needs to be collected by the ear-wearable device, resulting in a significant increase in data collection and training costs. Therefore, there is an urgent need for a behavior recognition model establishment method based on an ear-wearable device. The behavior recognition model established based on the ear-wearable device is not only trained by collecting part of the human behavior activity data by the ear-wearable device, but also assisted by the stable motion signals captured by the mobile device (such as a smartphone), thereby reducing the data collection and training costs of the behavior recognition model based on the ear-wearable device. SUMMARY

[0003] In view of the fact that in the current technology, in order to ensure the classification accuracy of human behavior activities, a large amount of human behavior activity data needs to be collected by the ear-wearable device, resulting in a significant increase in data collection and training costs, the present solution is proposed to overcome the above problems or at least partially solve the above problems.

[0004] In one aspect, the purpose of some embodiments of the present specification is to provide a behavior recognition model establishment method based on an ear-wearable device, the method comprising:

[0005] obtaining first behavior data collected by a mobile intelligent terminal and second behavior data collected by an ear-wearable device;

[0006] constructing training samples according to the first behavior data and the second behavior data;

[0007] performing feature extraction on the training samples using a feature extractor to obtain a feature extraction result;

[0008] inputting the feature extraction result into a label predictor to obtain a behavior recognition prediction result;

[0009] inputting the feature extraction result into a classifier to determine whether the training samples come from the mobile intelligent terminal or the ear-wearable device, and obtaining a corresponding classification result;

[0010] Based on a preset loss function, the target loss corresponding to the behavior recognition prediction result and classification result is calculated, and backpropagation is performed according to the target loss to update the network parameters in the feature extractor, label predictor and classifier, so as to extract the domain-invariant features corresponding to the first behavior data and the second behavior data, and optimize the behavior recognition prediction result corresponding to the training sample according to the domain-invariant features.

[0011] Repeat the above steps of updating the network parameters in the feature extractor, label predictor, and classifier using training samples until the preset convergence condition is met, to obtain the behavior recognition model of the ear-worn device composed of the trained feature extractor, label predictor, and classifier.

[0012] Furthermore, after acquiring the first behavioral data collected via a mobile smart terminal and the second behavioral data collected via an ear-worn device, the process further includes:

[0013] The first and second rows of data are preprocessed.

[0014] Further, the first and second line data are preprocessed, including:

[0015] The first behavioral data is denoised using a first filter, and the second behavioral data is denoised using a second filter.

[0016] And / or,

[0017] Data augmentation processing is performed based on the first and second behavioral data;

[0018] And / or,

[0019] The first and second rows of data are normalized.

[0020] Furthermore, training samples are constructed based on the first row of data and the second row of data, including:

[0021] Training samples are selected from the first row of data and the second row of data, such that the ratio between the training samples from the first row of data and the training samples from the second row of data exceeds a preset threshold.

[0022] Furthermore, the feature extractor is constructed using a bidirectional long short-term memory neural network to capture temporal features and dependencies in the training samples.

[0023] Further, based on a preset loss function, a target loss corresponding to the behavior recognition prediction result and the classification result is calculated, and network parameters in the feature extractor, the label predictor and the classifier are updated according to the target loss, so as to extract domain-invariant features corresponding to the first behavior data and the second behavior data, and optimize the behavior recognition prediction result corresponding to the training sample according to the domain-invariant features.

[0024] The behavior recognition prediction loss of the behavior recognition prediction result and the classification loss of the classification result are calculated respectively by using a preset loss function, and based on a preset gradient reversal layer, the target loss is calculated according to the behavior recognition prediction loss and the classification loss.

[0025] According to the target loss, the network parameters in the feature extractor, the label predictor and the classifier are updated by using a back propagation algorithm, so as to extract domain-invariant features corresponding to the first behavior data and the second behavior data, and optimize the behavior recognition prediction result corresponding to the training sample according to the domain-invariant features.

[0026] Further, in the forward propagation process, the gradient reversal layer is an identity function; in the back propagation process, the gradient reversal layer reverses the gradient sign before passing from the next layer to the previous layer, so as to perform a negative operation on the classification loss.

[0027] On the other hand, some embodiments of the present specification also provide an ear-mounted device-based behavior recognition model establishment device, which comprises:

[0028] An acquisition module is configured to acquire first behavior data collected by a mobile intelligent terminal and second behavior data collected by an ear-mounted device;

[0029] A construction module is configured to construct a training sample according to the first behavior data and the second behavior data;

[0030] An extraction module is configured to perform feature extraction on the training sample by using a feature extractor, to obtain a feature extraction result;

[0031] A recognition module is configured to input the feature extraction result into a label predictor, to obtain a behavior recognition prediction result;

[0032] A classification module is configured to input the feature extraction result into a classifier, to determine whether the training sample is from a mobile intelligent terminal or from an ear-mounted device, to obtain a corresponding classification result;

[0033] an updating module configured to calculate a target loss corresponding to the behavior recognition prediction result and the classification result based on a preset loss function, and update network parameters in the feature extractor, the label predictor and the classifier according to the target loss, so as to extract domain-invariant features corresponding to the first behavior data and the second behavior data, and optimize the behavior recognition prediction result corresponding to the training sample according to the domain-invariant features;

[0034] a repeating module configured to repeat the step of updating the network parameters in the feature extractor, the label predictor and the classifier using the training sample until a preset convergence condition is reached, so as to obtain a behavior recognition model of the ear-wearable device composed of the feature extractor, the label predictor and the classifier after training.

[0035] Based on the same inventive concept, some embodiments of the present specification aim to provide an ear-wearable device-based behavior recognition method, which comprises:

[0036] receiving behavior data to be processed collected by an ear-wearable device;

[0037] inputting the behavior data to be processed into an ear-wearable device-based behavior recognition model trained using the method according to any one of the preceding embodiments, so as to obtain a behavior recognition result of the behavior data to be processed.

[0038] Based on the same inventive concept, in another aspect, some embodiments of the present specification further provide an ear-wearable device-based behavior recognition device, which comprises:

[0039] a receiving module configured to receive behavior data to be processed collected by an ear-wearable device;

[0040] a recognition module configured to input the behavior data to be processed into an ear-wearable device-based behavior recognition model trained using the method according to any one of the preceding embodiments, so as to obtain a behavior recognition result of the behavior data to be processed.

[0041] In another aspect, some embodiments of the present specification further provide a computer device, which comprises a memory, a processor, and a computer program stored in the memory, and when the computer program is run by the processor, the instructions of the above method are executed.

[0042] In another aspect, some embodiments of the present specification further provide a computer storage medium having a computer program stored thereon, and when the computer program is run by a processor of a computer device, the instructions of the above method are executed.

[0043] In another aspect, some embodiments of the specification also provide a computer program product comprising a computer program, which, when executed by a processor of a computer device, performs the instructions of the above method.

[0044] Some embodiments of the specification provide one or more technical solutions, which at least have the following technical effects:

[0045] Embodiments of the specification first acquire first behavior data collected by a mobile intelligent terminal and second behavior data collected by an ear-mounted device, and construct a training sample according to the first behavior data and the second behavior data, then extract a feature extraction result corresponding to the training sample, and input the feature extraction result into a label predictor and a classifier respectively to obtain a behavior recognition prediction result and a classification result, calculate a target loss according to the two prediction results of the behavior recognition prediction result and the classification result, perform back propagation, and thus improve the ability of a feature extractor to extract domain-invariant features corresponding to the first behavior data and the second behavior data, so as to utilize the internal relationship between the first behavior data and the second behavior data for behavior recognition, realize training of a behavior recognition model based on the ear-mounted device by using the first behavior data to assist the second behavior data, and reduce data collection cost and training cost.

[0046] The above description is only a summary of some technical solutions of the specification, in order to more clearly understand the technical means of some embodiments of the specification, which can be implemented according to the content of the specification, and in order to make the above and other purposes, features and advantages of some embodiments of the specification more obvious and easy to understand, the following will specifically describe the embodiments of the specification. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of some embodiments of the specification or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or the prior art description. Obviously, the drawings in the following description are only some embodiments of the specification, and those skilled in the art can obtain other drawings according to these drawings without creating any creative labor. In the drawings:

[0048] Figure 1 An implementation system schematic diagram of a behavior recognition model based on an ear-mounted device is shown in some embodiments of the specification;

[0049] Figure 2 A flowchart of a behavior recognition model based on an ear-mounted device is shown in some embodiments of the specification;

[0050] Figure 3Fig. 1 shows an angular velocity data diagram collected by an ear-wearable device in some embodiments of the present specification;

[0051] Figure 4 Fig. 4 shows a network parameter updating step in some embodiments of the present specification;

[0052] Figure 5 Fig. 5 shows a training process of an ear-wearable device-based behavior recognition model in some embodiments of the present specification;

[0053] Figure 6 Fig. 6 shows a flowchart of an ear-wearable device-based behavior recognition method in some embodiments of the present specification;

[0054] Figure 7 Fig. 7 shows a structure of an ear-wearable device-based behavior recognition model establishment device in some embodiments of the present specification;

[0055] Figure 8 Fig. 8 shows a structure of an ear-wearable device-based behavior recognition device in some embodiments of the present specification;

[0056] Figure 9 Fig. 9 shows a structure of a computer device provided in some embodiments of the present specification.

[0057]

Explanation of Reference Signs

[0058] 101, terminal;

[0059] 102, server;

[0060] 701, acquisition module;

[0061] 702, construction module;

[0062] 703, extraction module;

[0063] 704, recognition module;

[0064] 705, classification module;

[0065] 706, updating module;

[0066] 707, repetition module;

[0067] 801, reception module;

[0068] 802, recognition module;

[0069] 902, computer device;

[0070] 904, processor;

[0071] 906, memory;

[0072] 908, drive mechanism;

[0073] 910, input / output interface;

[0074] 912, input device;

[0075] 914, output device;

[0076] 916, presentation device;

[0077] 918, graphical user interface;

[0078] 920, network interface;

[0079] 922, communication link;

[0080] 924, communication bus. DETAILED DESCRIPTION

[0081] In order to make the person skilled in the art better understand the technical solutions in the specification, the technical solutions in the specification will be described clearly and completely in the following by combining the drawings in some embodiments of the specification. Obviously, the described embodiments are only some of the embodiments of the specification, not all. Based on some embodiments in the specification, all other embodiments obtained by the person skilled in the art without creative labor should belong to the protection scope of the specification.

[0082] It should be noted that the terms "first", "second", and the like in the specification and claims of the present document and the above-described drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data used in this way can be exchanged under appropriate circumstances, so that the embodiments of the present document described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, device, product or equipment that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0083] It should be noted that the acquisition, storage, use, processing, etc. of data in the technical solutions of the present application comply with the relevant provisions of relevant laws and regulations.

[0084] It should be noted that in the embodiments of the present application, some software, components, models and the like in the industry may be mentioned, which should be considered as exemplary, and the purpose is only to illustrate the feasibility of the technical scheme of the present application, but does not mean that the applicant has or will necessarily use the scheme.

[0085] As shown in Figure 1 An embodiment of the application is shown in the figure, which is a system schematic diagram of a behavior recognition model establishment method based on an ear-wearable device, which can include a terminal 101 and a server 102. The terminal 101 and the server 102 communicate through a network, which can include a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof, and is connected to a website, a user device (such as a computing device), and a backend system. The staff can send a behavior recognition model establishment request based on an ear-wearable device to the server 102 through the terminal 101. After receiving the behavior recognition model establishment request based on the ear-wearable device, the server 102 calls the data in the database for calculation and processing to obtain the modeling result, and sends the modeling result to the terminal 101, so that the staff can process the business according to the modeling result.

[0086] In the embodiments of the present application, the server 102 can be a standalone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and basic cloud computing services such as big data and artificial intelligence platforms.

[0087] In an optional embodiment, the terminal 101 can include, but is not limited to, self-service terminal devices, desktop computers, tablet computers, notebook computers, smart wearable devices, and the like. Optionally, the operating system running on the electronic device can include, but is not limited to, the Android system, the IOS system, Linux, Windows, and the like. Of course, the terminal 101 is not limited to the above-mentioned electronic devices with certain entities, and it can also be software running in the above-mentioned electronic devices.

[0088] In addition, it should be noted that Figure 1 The figure shown is only one application environment provided by the present disclosure, and in actual application, a plurality of terminals 101 can also be included, which is not limited in the present application.

[0089] Figure 2is a flowchart of a behavior recognition model establishment method based on an ear-wearable device provided by an embodiment of the present application. The specification provides method operation steps as described in the embodiments or flowcharts, but more or fewer operation steps can be included based on conventional or non-inventive labor. The order of steps listed in the embodiments is only one of the many execution orders, and does not represent the only execution order. In actual system or device product execution, the method order shown in the embodiments or the drawings can be executed in sequence or in parallel. Specifically, as shown in Figure 2 The method can include the following steps:

[0090] S201: Obtain first behavior data collected by a mobile intelligent terminal and second behavior data collected by an ear-wearable device.

[0091] In some embodiments, the mobile intelligent terminal refers to a computer intelligent device that can be used on the move, and most of the time refers to a mobile phone or a smart phone with multiple application functions and a tablet computer. The ear-wearable device includes an in-ear earphone, a semi-in-ear earphone, etc. The user can perform autonomous activities while wearing the ear-wearable device. The first behavior data and the second behavior data respectively refer to motion signals captured by an inertial measurement unit (IMU) sensor embedded in the mobile intelligent terminal or the ear-wearable device when the user carries the mobile intelligent terminal or wears the ear-wearable device. Compared with the first behavior data, due to the dynamic nature and unpredictability of the user's head movement, the second behavior data often presents a more significant fluctuation trend, and therefore, in terms of user behavior recognition, using the second behavior data for user behavior recognition often faces higher challenges in implementation difficulty.

[0092] In some embodiments, in order to improve the data effectiveness of the first behavior data and the second behavior data, after obtaining the first behavior data collected by the mobile intelligent terminal and the second behavior data collected by the ear-wearable device, further comprising:

[0093] Preprocessing the first behavior data and the second behavior data.

[0094] Specifically, the preprocessing process can include:

[0095] Using a first filter to perform noise reduction processing on the first behavior data, and using a second filter to perform noise reduction processing on the second behavior data.

[0096] And / or,

[0097] Performing data enhancement processing according to the first behavior data and the second behavior data.

[0098] And / or,

[0099] The first and second rows of data are normalized.

[0100] It can be understood that in some embodiments, both the first row of data and the second row of data contain noise. Therefore, a filter can be used to denoise the first row of data and the second row of data. Of course, in some embodiments, the first row of data can also be the denoised data, in which case there is no need to denoise the first row of data.

[0101] For the second-row data, head movements significantly interfere with the acquisition process, resulting in substantial noise. Therefore, noise reduction processing can effectively improve the data quality of the second-row data. However, eliminating head movement interference from raw IMU data is not easy because the frequencies of body or head movements are distributed over a wide range and overlap within certain limits. On one hand, human activity frequencies are typically between 0 and 20 Hz, with 98% of human activity frequencies below 10 Hz. On the other hand, due to the self-filtering function of body structure, the frequency of motion signals recorded from the head is usually low, often below 5.8 Hz. Therefore, low-pass filtering can be used to mitigate head movement interference and reduce noise in the second-row data.

[0102] See attached document Figure 3 The diagram shows the spectrum of angular velocity data collected by an ear-worn device while the user is running. Figure 2 As can be seen, the main frequency components of the angular velocity data are concentrated in the low-frequency range (e.g., below 5Hz). However, when running is accompanied by more vigorous head movements, significant lobes will appear in the high-frequency range. Therefore, considering the signal frequency during human movement, the second filter is a low-pass filter. In a typical embodiment, the second filter can be a low-pass filter with a cutoff frequency of 5Hz, which can effectively filter out unwanted interference in the high-frequency range, including interference introduced by unstable head movements, while retaining information signals related to body movement.

[0103] In some embodiments, data augmentation processing based on the first and second behavior data can be used to subsequently enhance the robustness of the ear-worn device-based behavior recognition model to ear-worn device orientation and data heterogeneity. Specifically, the acceleration and gyroscope amplitude in the first and second behavior data can be augmented using the following formula:

[0104]

[0105] Among them, a mag Indicates the resultant acceleration, ax represents the acceleration in the x-axis direction, a y represents the acceleration in the y-axis direction, a z represents the acceleration in the z-axis direction, g mag represents the synthesized gyroscope amplitude, g x represents the gyroscope amplitude in the x-axis direction, g y represents the gyroscope amplitude in the y-axis direction, g z represents the gyroscope amplitude in the z-axis direction.

[0106] Through the preprocessing step of the above formula, the influence of the device direction can be reduced, and a signal representation independent of the device orientation is provided for enhancing the generalization ability of the behavior recognition model based on the ear-wearable device between different devices and domains.

[0107] Further, in some embodiments, since different types of data in the first behavior data and the second behavior data can be distributed in different data ranges, the data comparability is affected, which may affect the subsequent model training efficiency. For example, since the gyroscope is more sensitive to motion, the accelerometer data is always more stable than the gyroscope parameter data during the same activity of the user, resulting in that the data measured by the accelerometer and the gyroscope are always in different ranges. In order to alleviate this influence, the range difference between the acceleration reading and the gyroscope reading can be reduced by normalizing the data measured by the accelerometer and the gyroscope (such as acceleration data).

[0108] S202: constructing training samples according to the first behavior data and the second behavior data;

[0109] In some embodiments, when constructing the training samples, the training samples are selected from the first behavior data and the second behavior data respectively, so that the ratio between the training samples from the first behavior data and the training samples from the second behavior data exceeds a preset threshold.

[0110] It can be understood that the first behavior data is taken as a source domain, and the second behavior data is taken as a target domain, and the training samples are selected from the source domain and the target domain respectively. The behavior data between different domains implies the same internal features. That is, the domain-invariant features, which can be used for efficient and accurate behavior recognition.

[0111] It should be noted that, since the first behavior data has low cost of data collection, and existing open source data sets can provide ready-made first behavior data, compared with behavior recognition only using a large amount of second behavior data, by extracting domain invariant features between the first behavior data and the second data with the help of the first behavior data, behavior recognition is performed through the domain invariant features, which can greatly reduce the data collection cost of the second behavior data, and since the interference of head movement leads to high training cost of behavior recognition using the second behavior data, behavior recognition through the domain invariant features can reduce the training cost.

[0112] On this basis, in order to ensure that the data collection cost and the training cost are effectively reduced when constructing the training sample, a large number of training samples from the first behavior data and a small number of training samples from the second behavior data are often used, which also means that the preset threshold is a considerable number, and in typical embodiments, the preset threshold often exceeds 100, that is, the number of training samples from the first behavior data is at least 100 times the number of training samples from the second behavior data.

[0113] S203: performing feature extraction on the training sample by using a feature extractor to obtain a feature extraction result;

[0114] In some embodiments, the feature extractor is constructed by using a bidirectional long short-term memory neural network to capture the time sequence features and the dependency relationship in the training sample. In typical embodiments, the size of the hidden layer of the feature extractor is 16, the number of layers is 2, the training sample input by the feature extractor is two-dimensional data with a sequence length of 100, which is from the first behavior data and the second behavior data respectively, and the full connection layer uses a ReLU activation function for label predictor and classifier.

[0115] S204: inputting the feature extraction result into a label predictor to obtain a behavior recognition prediction result;

[0116] In some embodiments, it is obvious that the behavior recognition model obtained only by using the training sample from the first behavior data cannot accurately obtain the user behavior based on the ear-wearable device, therefore, in addition to inputting the feature extraction result into the label predictor, the feature extraction result also needs to be input into a classifier for subsequent extraction of domain invariant features.

[0117] S205: inputting the feature extraction result into a classifier to determine whether the training sample is from a mobile intelligent terminal or from an ear-wearable device to obtain a corresponding classification result;

[0118] In some embodiments, the classifier can be regarded as a binary classifier, which determines whether the current training sample is from the first behavior data or the second behavior data according to the feature extraction result, which is equivalent to determining whether it is from the mobile intelligent terminal or the ear-wearable device.

[0119] In some embodiments, the target loss is composed of a behavior recognition prediction loss of the behavior recognition prediction result and a classification loss of the classification result, wherein the classification loss is constructed based on the gradient reversal layer, and is used to encourage the ear-wearable device-based behavior recognition model to recognize the domain-invariant features corresponding to the first behavior data and the second behavior data, so as to gradually realize the minimization of the domain difference in the training process, so that the ear-wearable device-based behavior recognition model has the ability to extract domain-invariant features.

[0120] In some embodiments, the target loss is composed of a behavior recognition prediction loss of the behavior recognition prediction result and a classification loss of the classification result, wherein the classification loss is constructed based on the gradient reversal layer, and is used to encourage the ear-wearable device-based behavior recognition model to recognize the domain-invariant features corresponding to the first behavior data and the second behavior data, so as to gradually realize the minimization of the domain difference in the training process, so that the ear-wearable device-based behavior recognition model has the ability to extract domain-invariant features.

[0121] Specifically, referring to the accompanying drawings, Figure 4 In some embodiments, based on the preset loss function, the target loss corresponding to the behavior recognition prediction result and the classification result is calculated, and the network parameters in the feature extractor, the label predictor and the classifier are updated according to the target loss by back propagation, so as to extract the domain-invariant features corresponding to the first behavior data and the second behavior data, and optimize the behavior recognition prediction result corresponding to the training sample according to the domain-invariant features.

[0122] S401: Using a preset loss function, respectively calculating a behavior recognition prediction loss of the behavior recognition prediction result and a classification loss of the classification result, and based on a preset gradient reversal layer, calculating a target loss according to the behavior recognition prediction loss and the classification loss;

[0123] S402: According to the target loss, the network parameters in the feature extractor, the label predictor and the classifier are updated by using the back propagation algorithm, so as to extract the domain-invariant features corresponding to the first behavior data and the second behavior data, and optimize the behavior recognition prediction result corresponding to the training sample according to the domain-invariant features.

[0124] It can be understood that, in some embodiments, referring to the accompanying drawings, Figure 5An illustrated training process of an ear-wearable device based behavior recognition model, the output of the feature extractor enters two different paths, one path is used for label prediction, and the other path is used for binary classification, the task of the label predictor is to predict the correct activity label of the training sample, and the task of the classifier is to determine whether the training sample is from a mobile smart terminal or from an ear-wearable device, so that the loss function uses gradient reversal, and the target loss is calculated according to the output of the label predictor and the classifier, and then the back propagation is realized.

[0125] Specifically, the preset loss function is constructed based on the gradient reversal layer, in the forward propagation process, the gradient reversal layer is an identity function, allowing data to pass unchanged, in the backward propagation process, the gradient reversal layer reverses the gradient sign before passing from the next layer to the previous layer, to take the negative operation on the classification loss, the purpose is to minimize the label prediction error, and at the same time encourage the feature extractor to learn domain-invariant features.

[0126] Further, the loss function can be expressed as:

[0127]

[0128] Wherein, E(θ f ,θ y ,θ d ) represents the loss function, G f ,G y ,G d represent the feature extractor, the label predictor and the classifier respectively, θ f ,θ y ,θ d represent the parameters of the feature extractor, the label predictor and the classifier respectively, λ represents the trade-off parameter, L y represents the label prediction loss, L d represents the classification loss, x i represents the input, and y i represents the label.

[0129] This also means that during the training phase, the target loss is determined by the difference between the label prediction loss and the classification loss, used to minimize the behavior recognition error and maximize the binary classification error. Specifically, the smaller the label prediction loss, the smaller the behavior recognition error; the larger the classification loss, the larger the binary classification error. When the label prediction loss is smaller and the classification loss is larger, the behavior recognition model based on the ear-worn device can not only accurately identify user behavior but also confuse the training samples from mobile smart terminals or ear-worn devices. The feature extractor and classifier compete against each other during training, ultimately achieving a balance between the label prediction loss and the classification loss. During backpropagation, the existence of the gradient reversal layer in the loss function is reflected by the negative sign before the classification loss term. Gradient reversal is achieved by negating the classification loss.

[0130] S207: Repeat the above steps of updating the network parameters in the feature extractor, label predictor and classifier using training samples until the preset convergence condition is met, and obtain the behavior recognition model of the ear-worn device composed of the trained feature extractor, label predictor and classifier.

[0131] To ensure the ability of the behavior recognition model based on ear-worn devices to extract domain-invariant features and the recognition accuracy, the target loss when terminating iterative training should satisfy either the sum of the label prediction loss and the classification loss being less than a preset loss value, or the number of iterations exceeding a preset number, thereby obtaining the trained feature extractor, label predictor, and classifier.

[0132] In some embodiments, it can be understood that, firstly, first behavioral data collected by a mobile smart terminal and second behavioral data collected by an ear-worn device are acquired. Training samples are then constructed based on the first and second behavioral data. Subsequently, feature extraction results corresponding to the training samples are extracted and input into a label predictor and a classifier, respectively, to obtain behavior recognition prediction results and classification results. The target loss is calculated based on these two prediction results and backpropagation is performed to improve the ability of the feature extractor to extract domain-invariant features corresponding to the first and second behavioral data. This allows behavior recognition to be performed using the inherent relationship between the first and second behavioral data, enabling the use of the first behavioral data to assist the training of a behavior recognition model based on an ear-worn device, thereby reducing data acquisition and training costs.

[0133] Based on the same inventive concept, some embodiments of this specification also provide a behavior recognition method based on an ear-worn device, see reference. Figure 6 As shown, in some embodiments, the method may include:

[0134] S601: Receives behavioral data to be processed collected by the ear-worn device;

[0135] S602: input the to-be-processed behavior data into the ear-wearable device-based behavior recognition model trained according to the method of any one of the preceding embodiments, to obtain a behavior recognition result of the to-be-processed behavior data.

[0136] It should be noted that although the operations of the method of the present application are described in a specific order in the above embodiments and drawings, this does not require or imply that the operations must be performed in this specific order, or that all of the shown operations must be performed to achieve the desired result. Additionally or alternatively, certain steps can be omitted, multiple steps can be combined into one step, and / or one step can be divided into multiple steps.

[0137] Corresponding to the ear-wearable device-based behavior recognition model establishment method described above, some embodiments of the present specification also provide an ear-wearable device-based behavior recognition model establishment device, which is described with reference to Figure 7 In some embodiments, the device can include:

[0138] The acquisition module 701 is configured to acquire first behavior data collected by a mobile intelligent terminal and second behavior data collected by an ear-wearable device.

[0139] The construction module 702 is configured to construct training samples according to the first behavior data and the second behavior data.

[0140] The extraction module 703 is configured to perform feature extraction on the training samples by using a feature extractor, to obtain a feature extraction result.

[0141] The recognition module 704 is configured to input the feature extraction result into a label predictor, to obtain a behavior recognition prediction result.

[0142] The classification module 705 is configured to input the feature extraction result into a classifier, to determine whether the training samples come from a mobile intelligent terminal or an ear-wearable device, and to obtain a corresponding classification result.

[0143] The update module 706 is configured to calculate a target loss corresponding to the behavior recognition prediction result and the classification result based on a preset loss function, and to update network parameters in the feature extractor, the label predictor, and the classifier according to the target loss, to extract domain-invariant features corresponding to the first behavior data and the second behavior data, and to optimize the behavior recognition prediction result corresponding to the training samples according to the domain-invariant features.

[0144] The repeating module 707 is configured to repeat the steps of updating the network parameters in the feature extractor, the label predictor and the classifier using the training samples until a preset convergence condition is reached, to obtain a behavior recognition model of the ear-mounted device composed of the trained feature extractor, the label predictor and the classifier.

[0145] Corresponding to the ear-mounted device-based behavior recognition method described above, some embodiments of the present specification also provide an ear-mounted device-based behavior recognition apparatus. Referring to FIG. 8, in some embodiments, the apparatus can include: Figure 8

[0146] The receiving module 801 is configured to receive behavior data to be processed collected by an ear-mounted device.

[0147] The identifying module 802 is configured to input the behavior data to be processed into an ear-mounted device-based behavior recognition model trained by the method according to any one of the preceding embodiments, to obtain a behavior recognition result of the behavior data to be processed.

[0148] For ease of description, the above apparatus is described in various units in terms of functions and is described respectively. Of course, the functions of the units can be implemented in one or more software and / or hardware in the implementation of the present specification.

[0149] It should be noted that in the embodiments of the present specification, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved are information and data authorized by the user and authorized by all parties.

[0150] It should be noted that the computer program product of the present specification is a software product for implementing the method of the present specification mainly through a computer program.

[0151] The embodiments of the present specification also provide a computer device. As shown in FIG. 9, the computer device can include a receiving module 901, an identifying module 902 and a processing module 903. Figure 9 ​As shown, in some embodiments of the specification, the computer device 902 can include one or more processors 904, such as one or more central processing units (CPUs) or graphics processing units (GPUs), each of which can implement one or more hardware threads. The computer device 902 can also include any memory 906 for storing any kind of information, such as code, settings, data, etc., and in particular embodiments, a computer program stored on the memory 906 and executable on the processor 904, which when executed by the processor 904, can perform the instructions of the method of any of the embodiments described above. Without limitation, for example, the memory 906 can include any one or combination of the following: any type of RAM, any type of ROM, a flash memory device, a hard disk, an optical disk, etc. More generally, any memory can use any technology for storing information. Further, any memory can provide volatile or non-volatile retention of information. Further, any memory can represent a fixed or removable component of the computer device 902. In one case, the computer device 902 can perform any operation of the associated instructions when executed by the processor 904, which are stored in any memory or combination of memories. The computer device 902 also includes one or more drive mechanisms 908 for interacting with any memory, such as a hard disk drive mechanism, an optical disk drive mechanism, etc.

[0152] The computer device 902 can also include an input / output interface 910 (I / O) for receiving various input (via input devices 912) and for providing various output (via output devices 914). One particular output mechanism can include a presentation device 916 and an associated graphical user interface 918 (GUI). In other embodiments, the input / output interface 910 (I / O), input devices 912, and output devices 914 can also not be included, just as a computer device in a network. The computer device 902 can also include one or more network interfaces 920 for exchanging data with other devices via one or more communication links 922. One or more communication buses 924 couple the above-described components together.

[0153] The communication links 922 can be implemented in any manner, such as through a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication links 922 can include any combination of hardwired links, wireless links, routers, gateway functionality, name servers, etc., governed by any protocol or combination of protocols.

[0154] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or a combination thereof. Figure 1 an apparatus with a processor-based platform adapted to perform the functions specified in the flowchart block or blocks.

[0155] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or a combination thereof. Figure 1 an apparatus with a processor-based platform adapted to perform the functions specified in the flowchart block or blocks.

[0156] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or a combination thereof. Figure 1 an apparatus with a processor-based platform adapted to perform the functions specified in the flowchart block or blocks.

[0157] In one typical configuration, a computer device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0158] The memory can include non-persistent memory and / or volatile memory, such as a random access memory (RAM) including a cache area for the temporary storage of data. The memory can also include non-volatile memory, such as a read only memory (ROM) device, a floppy disk, a flexible disk, hard disk, or a tape, cloud storage, flash memory card, or another suitable data storage device. The memory can store computer readable instructions or one or more software programs, including one or more of an operating system, an on-body device application program, and related components such as one or more software modules.

[0159] Computer-readable media includes permanent and non-permanent, movable and non-movable media, which can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computer device. According to the definition in this specification, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0160] Those skilled in the art will appreciate that embodiments of the present specification can be provided as methods, systems or computer program products. Therefore, embodiments of the present specification can take the form of entirely hardware embodiments, entirely software embodiments or embodiments combining software and hardware aspects. Moreover, embodiments of the present specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0161] Embodiments of the present specification can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Embodiments of the present specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are connected through a communication network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including storage devices.

[0162] It should also be understood that, in embodiments of the present specification, the term "and / or" is merely a description of an association relationship between associated objects, which means that there can be three relationships. For example, A and / or B can mean that there are three cases of A alone, A and B together, and B alone. In addition, the character " / " herein generally represents that the associated objects before and after are in an "or" relationship.

[0163] Each of the embodiments in the present specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, for the system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments.

[0164] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the embodiments of the present specification. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any suitable manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0165] The above only describes the embodiments of the present application and is not intended to limit the present application. The present application can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. A method for establishing a behavior recognition model based on an ear-wearable device, the method comprising: The method comprises: acquiring first behavior data collected by a mobile intelligent terminal and second behavior data collected by an ear-mounted device; constructing a training sample according to the first behavior data and the second behavior data; extracting features from the training sample by using a feature extractor to obtain a feature extraction result; inputting the feature extraction result into a label predictor to obtain a behavior recognition prediction result; inputting the feature extraction result into a classifier to determine whether the training sample is from the mobile intelligent terminal or from the ear-mounted device, and obtaining a corresponding classification result; based on a preset loss function, calculating a target loss corresponding to the behavior recognition prediction result and the classification result, and updating network parameters in the feature extractor, the label predictor and the classifier according to the target loss to extract domain-invariant features corresponding to the first behavior data and the second behavior data, and to optimize the behavior recognition prediction result corresponding to the training sample according to the domain-invariant features; repeating the step of updating the network parameters in the feature extractor, the label predictor and the classifier by using the training sample until a preset convergence condition is reached, to obtain a behavior recognition model of the ear-mounted device composed of the trained feature extractor, the label predictor and the classifier.

2. The method of claim 1, wherein, After acquiring the first behavior data collected by the mobile intelligent terminal and the second behavior data collected by the ear-mounted device, further comprising: preprocessing the first behavior data and the second behavior data.

3. The method of claim 2, wherein, The preprocessing of the first behavior data and the second behavior data comprises: using a first filter to perform noise reduction processing on the first behavior data, and using a second filter to perform noise reduction processing on the second behavior data; and / or, performing data enhancement processing on the first behavior data and the second behavior data; and / or, performing normalization processing on the first behavior data and the second behavior data.

4. The method of claim 1, wherein, The construction of the training sample according to the first behavior data and the second behavior data comprises: selecting the training sample from the first behavior data and the second behavior data so that the ratio between the training sample from the first behavior data and the training sample from the second behavior data exceeds a preset threshold.

5. The method of claim 1, wherein, The feature extractor is constructed by using a bidirectional long short-term memory neural network to capture the time sequence features and the dependency relationship in the training sample.

6. The method of claim 1, wherein, Based on a preset loss function, the target loss corresponding to the behavior recognition prediction result and the classification result is calculated, and the network parameters in the feature extractor, the label predictor and the classifier are updated according to the target loss to extract domain-invariant features corresponding to the first behavior data and the second behavior data, and to optimize the behavior recognition prediction result corresponding to the training sample according to the domain-invariant features, comprising: using a preset loss function to calculate a behavior recognition prediction loss of the behavior recognition prediction result and a classification loss of the classification result, respectively, and based on a preset gradient reversal layer, the target loss is calculated according to the behavior recognition prediction loss and the classification loss; According to the target loss, network parameters in the feature extractor, the label predictor and the classifier are updated by using a back propagation algorithm to extract domain-invariant features corresponding to the first behavior data and the second behavior data, and to optimize the behavior recognition prediction result corresponding to the training sample according to the domain-invariant features.

7. The method of claim 6, wherein, In the forward propagation process, the gradient inversion layer is an identity function; in the back propagation process, the gradient inversion layer reverses the gradient sign before passing from a later layer to an earlier layer to perform a negative operation on the classification loss.

8. A method for ear-wearable device based behavior recognition, the method comprising: The method comprises: receiving behavior data to be processed collected by an ear-wearable device; inputting the behavior data to be processed into an ear-wearable device-based behavior recognition model trained by any one of the methods of claims 1-7 to obtain a behavior recognition result of the behavior data to be processed. 9.A device for establishing a behavior recognition model based on an ear-wearable device, the device comprising: The device comprises: an acquisition module configured to acquire first behavior data collected by a mobile intelligent terminal and second behavior data collected by an ear-wearable device; a construction module configured to construct a training sample according to the first behavior data and the second behavior data; an extraction module configured to perform feature extraction on the training sample by using a feature extractor to obtain a feature extraction result; an identification module configured to input the feature extraction result into a label predictor to obtain a behavior recognition prediction result; a classification module configured to input the feature extraction result into a classifier to determine whether the training sample is from a mobile intelligent terminal or from an ear-wearable device to obtain a corresponding classification result; an update module configured to calculate a target loss corresponding to the behavior recognition prediction result and the classification result based on a preset loss function, and to update network parameters in the feature extractor, the label predictor and the classifier according to the target loss to extract domain-invariant features corresponding to the first behavior data and the second behavior data, and to optimize the behavior recognition prediction result corresponding to the training sample according to the domain-invariant features; a repetition module configured to repeat the steps of updating the network parameters in the feature extractor, the label predictor and the classifier by using the training sample until a preset convergence condition is reached to obtain an ear-wearable device-based behavior recognition model composed of the feature extractor, the label predictor and the classifier after training.

10. An ear-worn device based behavior recognition apparatus, comprising: The device comprises: a receiving module configured to receive behavior data to be processed collected by an ear-wearable device; an identification module configured to input the behavior data to be processed into an ear-wearable device-based behavior recognition model trained by any one of the methods of claims 1-7 to obtain a behavior recognition result of the behavior data to be processed.

11. A computer device comprising a memory, a processor, and a computer program stored on the memory, wherein, The computer program, when executed by the processor, performs the instructions of the method according to any one of claims 1-8.

12. A computer storage medium having stored thereon a computer program, characterized in that The computer program, when executed by the processor of the computer device, performs the instructions of the method according to any one of claims 1-8.

13. A computer program product, characterised in that, The computer program product comprises a computer program, which, when executed by the processor, performs the instructions of the method according to any one of claims 1-8.

Images