An online adaptation method based on pattern matching and mobile perception scenarios

Abstract

The present invention discloses an online adaptation method based on pattern matching and mobile perception scenarios. The method comprises: performing classification and identification by comparing the similarity between a target sample and a template in a template library to obtain a classification result; monitoring feedback from an application or user in real time for the obtained classification result; combining the feedback result with the active learning theory to determine the representativeness of the current target sample, which reflects the importance of the target sample for classification and identification; and updating the template library with a set strategy based on the corresponding representativeness to achieve online adaptation. Utilizing the present invention, developers can implement an application system that supports online continuous adaptation on mobile devices while ensuring storage and computing efficiency.

Description

Technical Field

[0001] The present invention relates to the field of mobile perception technology, and more specifically, to an online adaptation method based on pattern matching and mobile perception scenarios. Background Art

[0002] With the rapid development of micro-electromechanical systems, smart devices such as mobile terminals have become an integral part of our daily lives. Technological advancements are enabling the integration of an increasing number of sensors into mobile devices. Using mobile devices to perceive and provide services to users has become a hot research topic. Human behavior recognition is one of the most important branches of human-centric mobile perception, with common applications encompassing gesture recognition and inferring daily behaviors. However, most current recognition systems still face significant difficulties when deployed in real-world scenarios. Affected by the various unknown and changing conditions in real-world scenarios, system recognition accuracy is significantly reduced. In user-centric scenarios, in particular, individual differences among users, as well as dynamic uncertainties such as usage habits and shifting interests, pose significant challenges to the performance of recognition systems.

[0003] Existing recognition technologies mainly start from the training of perception models, minimizing the impact on user experience by reducing training overhead. However, they ignore the changes in conditions during actual use, resulting in poor recognition results. Summary of the Invention

[0004] The purpose of the present invention is to overcome the above-mentioned shortcomings of the prior art and provide an online adaptation method based on pattern matching and motion perception scene. The method comprises the following steps:

[0005] By comparing the similarity between the target sample and the template in the template library, classification and recognition are performed to obtain the classification result;

[0006] Monitor feedback from applications or users in real time based on the classification results.

[0007] Combined with the feedback results, the representativeness of the current target sample is judged according to the active learning theory. The representativeness reflects the importance of the target sample for classification and recognition.

[0008] Based on the corresponding representativeness, the template library is updated with the set strategy to achieve online adaptation.

[0009] Compared with the existing technology, the advantages of the present invention are that it provides a classification and recognition algorithm based on pattern matching and active learning theory, and modifies the template library through a sample replacement strategy, thereby ensuring computing and storage efficiency. For mobile sensing scenarios where the present invention is applied, there is no need for cloud participation, and it can quickly adapt to changes in user and environmental conditions in real time under complex feedback conditions. In addition, while ensuring storage and computing efficiency, the present invention designs a method for mobile devices that can continuously adapt online to meet the needs of adapting to dynamic conditions.

[0010] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0012] Figure 1 is a flow chart of an online adaptation method based on pattern matching and motion perception scenarios according to an embodiment of the present invention;

[0013] Figure 2 is a schematic diagram of an interactive application scenario based on identifying vibration knocking signals based on a motion sensor according to an embodiment of the present invention;

[0014] Figure 3 is a schematic diagram of updating a core template library according to one embodiment of the present invention;

[0015] Figure 4 This is a diagram of experimental results in a motion monitoring scenario according to an embodiment of the present invention;

[0016] Figure 5 FIG. 4 is a diagram showing experimental results in a gesture recognition scenario according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention.

[0018] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0019] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0020] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0021] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0022] The present invention provides a classification and recognition method (or algorithm) based on pattern matching and mobile perception scenarios, which is suitable for the continuous online adaptation environment and user conditions of mobile smart devices. Theoretically, it can be applied to all mobile perception scenarios that implement classification based on similarity and can obtain user feedback.

[0023] See also Figure 1 As shown, the provided method includes the following steps.

[0024] S1: For the received test samples, they are combined with the templates in the template library and classified according to the similarity.

[0025] In this step, the template library is initialized, and classification and recognition are performed by comparing the similarity between the test sample and the templates in the template library.

[0026] S2, receiving feedback

[0027] During the algorithm's operation, feedback is monitored in real time to determine the label category. For example, the methods of real-time feedback monitoring include the following:

[0028] S21, no feedback from the user / application, the classification result is used as the label of the test sample;

[0029] S22, the user / application has feedback, and the user / application feedback is used as the label of the test sample.

[0030] S3, based on the feedback, generates a set of candidate template library versions and calculates the sample representativeness based on active learning theory.

[0031] Specifically, in step S3, combining the immediate feedback and judging the representativeness of the current test sample (single) based on active learning theory includes:

[0032] S31, according to the test sample x * Feedback label y * , generate a candidate template library version set H.

[0033] S32, based on active learning theory and the given loss function l(h(x),y) that meets bounded conditions and is based on similarity, calculate the importance weighted loss values ​​of the candidate template library h in the template library version set H in sequence:

[0034] L(h;S)=∑ (x,y,w)∈S w·l(h(x),y) (1)

[0035] Here, S is the sample set from the original template library. This sample set should be consistent with the entities in the sample set used for classification in step S1 to ensure computational and storage efficiency. l(h(x), y) is a similarity-based loss function. The weight w is calculated using a weighting function based on key information corresponding to the original template (e.g., representativeness, update time, etc.). The weight reflects the importance of the sample to the model at the current moment.

[0036] S33: Update the template library version set H according to the minimum weighted loss value of the template library, and remove the template libraries that do not meet the conditions in advance.

[0037] S34, based on active learning theory, according to the formula: p = max f,g∈H；y∈Y l(f(x * ),y)-l(g(x * ),y), calculate the test sample x * The inverse of representativeness (the smaller the p-value, the more representative the sample).

[0038] In one embodiment, in step S31, the label y is fed back based on the test sample. * The generated template library version set H has the following characteristics:

[0039] S311 , each template library element h in the template library version set H contains the same number of templates (samples).

[0040] S312, the difference between each template library version set template library element h and the original template library is that a label is y * The sample is replaced by the test sample, so the size of the template library version set |H| is equal to the size of the template library with label y * The number of samples.

[0041] In step S32, the loss function combines a similarity measurement method based on a positive definite kernel function definition, which has a mathematically guaranteed bounded condition. A practical loss function definition is expressed as:

[0042] l(h(x),y)=max(0,E(D [h(S)＝y] (x))-min D [h(s)≠y] (x)) (2)

[0043] Where E(·) represents the average value calculation excluding the closest distance sample.

[0044]

[0045] Formula (3) is a vector consisting of the cosine distances (defined as in the equation on the right) between the input sample x and the sample(s) that meet the subscript conditions. The cosine distance is defined as:

[0046]

[0047] Formula (4) is a regularized kernel function (which mathematically guarantees the boundedness of the distance), where u and v are any two samples, and κ can be any selected positive definite kernel function related to the application scenario, which is used to measure the similarity between samples. Since the classification model based on similarity comparison involves the calculation of key distance information in the loss function during the recognition process, this ensures the computational efficiency of this type of loss function. The choice of positive definite kernel function includes but is not limited to linear kernel function, radial basis kernel function, etc. Different kernel functions have different characteristics (mainly affecting the calculation of the bound), and can be appropriately adjusted according to the actual platform or algorithm requirements (such as sample characteristics, computational efficiency, and storage).

[0048] In one embodiment, the weight function in step S32 satisfies any scenario-related strategy, including template time, etc. For example, a practical weight function is defined as:

[0049]

[0050] The weight function of formula (5) requires additional recording of the sample time. For ease of understanding, the superscript is added as the template sample input time t, and p is used t Indicates the representativeness calculated by step S34 at the time of template entry. In the function definition of formula (5), T is the current time, and It is the moment when a certain feedback event is triggered during use, for example, the moment when the feedback label matches the template library and the output category is different. Through the weight function, the algorithm has memory (monotonically decreasing exponential function) and forgetting (by control) capability, thus avoiding the impact of erroneous (referring to samples with incorrect feedback labels caused by users or applications) or outdated (referring to situations where the user's definition of the current category has changed) templates on the update.

[0051] In one embodiment, step S33 updates the template library version set H according to the weighted loss value based on the standard sample complexity of active learning theory, and has the probability of eliminating unqualified template libraries in advance.

[0052] S4, calculates the loss value of the candidate template library according to the active learning theory, and selects the candidate template library with the minimum loss value.

[0053] In this step, the original templates and corresponding key information (eg, representativeness, update time, etc.) are further combined to update the template library according to specific strategies to meet the needs of real-time adaptation.

[0054] In one embodiment, updating the template library in combination with the original template in step S4 includes:

[0055] S41, using the test sample x * , feedback label y * , and the representative p initialization element s=(x * ,y * ,w * =1 / p), update the sample set S=S∪{s}.

[0056] S42, using the sample set S updated in step S41, recalculate the loss value of each candidate template library h in the template library version set H (updated in step S33) (the formula is the same as step S32), and select the template library with the smallest loss value for update

[0057] h t =arg min h∈H L(h;S) (6)

[0058] S43, after the update, in order to maintain the stability of storage and operation, further update the sample set S = S / {x t}, where x t is not included in h t The tested sample x * The replaced template instance. At this point, a round of template library updates is completed.

[0059] It should be understood that while the above examples illustrate the concept of the present invention using test samples, in actual applications, online adaptation can be achieved by executing the above process on received target samples. Furthermore, the method or system implemented based on the present invention can be applied to classification and recognition in a variety of scenarios, such as behavior recognition and gesture recognition.

[0060] Figure 2 It is an interactive application scenario that realizes the engineering design of system initialization recognition model and interactive application, including: system detection and segmentation and preprocessing of tapping signals; initialization of template library; calculation of similarity with template and output of classification results; receiving real-time feedback from input method application or user; completing the update of template library according to scenario-related loss function and weight function. Figure 1The corresponding algorithm steps enable the model to have the ability of online real-time adaptation. The practical application of the algorithm mainly involves the selection of the similarity calculation method. For this scenario, a linear kernel function (i.e., the inner product between samples) can be selected in the calculation of the loss function and the similarity. After calculation, the upper bound of the corresponding loss function is 2, which means that in this embodiment, when calculating the loss function value, the loss function value needs to be normalized according to the value range [0,2] to ensure that the final loss function value is within the range [0,1].

[0061] Figure 3 Schematic diagram of updating the template library. During the working process of the system based on the present invention, the template library h used for classification and recognition and the template sample set S involved in the calculation of test sample representativeness and template library loss value share the same sample and label storage space. In each round of update, the number of template samples in the template library will remain consistent (e.g. Figure 3 In this way, the computing and storage efficiency of the system implemented based on the present invention is guaranteed.

[0062] In order to further verify the effect of the present invention, an experimental verification was carried out, and the experimental results are shown in Figure 4 and Figure 5 As shown, Figure 4 This is a motion monitoring scenario. The simulated application uses the mode of the classification results in a sliding window with a length of the vertical axis as real-time feedback. Figure 5 It is a gesture recognition scenario (TAPRINT), which involves tests including error rate, user feedback frequency, and usage content.

[0063] In the experimental verification, compared with the traditional domain adaptation (labeled as [DA], which refers to the scheme without online adaptation after initialization of samples) algorithm (nearest neighbor 5NN and the baseline MetaSense based on meta-learning training method), in the motion monitoring based on motion sensors (such as Figure 4 PAMAP2, HHAR, WHAR) and gesture recognition (as shown Figure 5 In two scenarios (TAPRINT in [1]), the two implementations of the continuous update method proposed in the present invention (including the random update algorithm RND and the algorithm CODA involved in the present invention) can achieve significant performance improvements (macro F1 scores), demonstrating the feasibility of continuous adaptation; among them, the algorithm of the present invention (CODA) can achieve more stable performance scores (macro F1 scores) in multiple groups of experiments simulating actual applications, further demonstrating the adaptability of the algorithm to motion perception scenarios. Among them, the application experiment of the motion monitoring scenario mainly simulates a window-based inference strategy; the experiment of the actual application of gesture recognition includes the differences in feedback error rate Fallibility, feedback frequency Fatigue and the use of context.

[0064] In summary, the present invention provides a classification and recognition algorithm based on pattern matching and mobile perception scenarios, which is suitable for continuous online adaptation of mobile devices or wearable devices to the environment and user conditions. It adopts a design with storage and computational efficiency, which is conducive to the continuous online adaptation of mobile devices and can meet the needs of dynamic adaptation conditions. The present invention applies active learning to guide the update of the classification and recognition model based on similarity. By defining a loss function based on similarity, the representativeness / importance of the current test sample for the model can be obtained. In addition, the loss function is combined with the design of the weight function to further ensure the effectiveness of the update.

[0065] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0066] Computer-readable storage medium can be a tangible device that can keep and store the instructions used by the instruction execution device.Computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device or any suitable combination thereof.More specific examples (non-exhaustive list) of computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove having instructions stored thereon, and any suitable combination thereof.Computer-readable storage medium used herein is not interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (for example, light pulses by fiber optic cables), or electrical signals transmitted by wires.

[0067] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0068] The computer program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, Python, and conventional procedural programming languages ​​such as "C" language or similar programming languages. The computer readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), is personalized by utilizing the state information of the computer readable program instructions, and the electronic circuit can execute the computer readable program instructions, thereby realizing various aspects of the present invention.

[0069] Various aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0070] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0071] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0072] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of an instruction, and the module, program segment or part of the instruction contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are all equivalent.

[0073] While various embodiments of the present invention have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. An online adaptation method based on pattern matching and mobile perception scenarios, comprising the following steps: By comparing the similarity between the target sample and the template in the template library, classification and recognition are performed to obtain the classification result; Monitor feedback from applications or users in real time based on the classification results. Combined with the feedback results, the representativeness of the current target sample is judged according to the active learning theory. The representativeness reflects the importance of the target sample for classification and recognition. Based on the corresponding representativeness, the template library is updated with the set strategy to achieve online adaptation; The step of combining the feedback results and judging the representativeness of the current target sample based on active learning theory includes: According to the target sample x * Feedback label y * , generate a candidate template library version set H; The weighted loss values ​​of the candidate template libraries h in the template library version set H are calculated in sequence, which can be expressed as: Where x represents the sample, y represents the label, l(h(x), y) is the loss function based on similarity, S is the sample set of the original template library, and w is the weight function; The template library corresponding to the minimum weighted loss value is used to update the template library version set H. The update is based on the standard sample complexity of active learning theory, and there is a probability that template libraries that do not meet the conditions will be eliminated in advance; According to the formula: p = max f,g∈H；y∈Y l(f(x * ),y)-l(g(x * ),y), calculate the target sample x * Representativeness, Y is the set of labels; Among them, for the target sample x * Feedback label y * , generate a candidate template library version set H with the following characteristics: Each template library element h in the template library version set H contains the same number of templates; For each template library version set, the difference between the template library element h and the original template library is that a tag y * The sample is replaced by the target sample, so that the size of the template library version set |H| is equal to the size of the template library with label y * The number of samples.

2. The method according to claim 1, characterized in that The real-time monitoring of feedback from applications or users includes: In the case where there is no feedback from the user or application, the classification result is used as the label of the target sample; In the case where there is feedback from users or applications, the label of the target sample is determined based on the feedback results.

3. The method according to claim 1, characterized in that The loss function is expressed as: l(h(x),y)=max(0,E(D [h(S)＝y] (x))-min D [h(S)≠y] (x)) Where E(·) represents the average value calculation excluding the closest distance sample, is a vector consisting of the cosine distances between the input sample x and the samples that meet the subscript conditions, u and v are any two samples, and κ is a positive definite kernel function used to measure the similarity between samples.

4. The method according to claim 1, wherein The weight function includes the template time and is expressed as: Among them, t is the sample entry time, p t It represents the representativeness calculated at the time of sample entry, T is the current time, It is the moment when the feedback event is triggered during use.

5. The method according to claim 1, wherein The updating of the template library based on the corresponding representativeness and the set strategy includes: With the target sample x * , feedback label y * , and the representative p initialization element s=(x * ,y * ,w * =1 / p), and update the sample set S=S∪{s}; Use the updated sample set S to recalculate the loss value of each candidate template library h in the template library version set H, and select the template library with the smallest loss value for update; Further update the sample set S=S / {x t }, where x t is not included in h t and is the target sample x * The template to replace.

6. The method according to claim 3, characterized in that The positive definite kernel function is selected according to the application scenario, including a linear kernel function and a radial basis kernel function.

7. A computer-readable storage medium having a computer program stored thereon, wherein: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

8. A computer device comprising a memory and a processor, wherein a computer program capable of being run on the processor is stored in the memory, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 6 are implemented.

Images