Finger pressing state detection, training method, electronic device, and medium

Abstract

The application relates to the technical field of signal detection, and discloses a finger pressing state detection method, a training method, an electronic device and a medium, which are used for improving the generalization ability of finger pressing state detection. The method part comprises the following steps: pre-processing an ultrasonic echo signal to obtain model input data, wherein the ultrasonic echo signal is an ultrasonic echo signal corresponding to ultrasonic wave signals emitted by an ultrasonic array to a pressing area; and inputting the model input data into a trained neural network model, so that the neural network model outputs a finger pressing state.

Description

Technical Field

[0001] This application relates to the field of signal detection technology, and in particular to a finger pressing state detection, training method, electronic device and medium. Background Technology

[0002] When using electronic devices such as mobile phones and computers with touch functionality, users need to touch the screen or other touch areas with their fingers to perform operations or recognition. For example, touch to unlock, single click to confirm, double click to take a screenshot, etc. Generally, the electronic device first captures the action and then triggers the corresponding touch function.

[0003] An ultrasonic system emits ultrasonic waves, receives reflected sound waves, and obtains digital signals after a series of signal conversions. Because sound wave propagation is affected by acoustic impedance, different reflecting surfaces have different acoustic impedances, resulting in differences in ultrasonic echo intensity. Therefore, there are differences in ultrasonic echo intensity between the finger and the touch contact surface, and between the air and the touch contact surface. These differences can be used to detect user finger movements and determine the finger's pressing state.

[0004] In traditional methods, under the condition of keeping the ultrasonic emission frequency and temperature the same, the finger pressing state (lifting or pressing, etc.) can be detected by setting a benchmark threshold. However, the ultrasonic signal is affected by factors such as emission frequency, temperature, ultrasonic flight time and ultrasonic module differences. The signal strength varies greatly under different conditions. Therefore, judging the finger pressing state by the threshold requires a large amount of data covering different conditions and setting corresponding benchmark thresholds. Covering all different situations is difficult, resulting in poor generalization ability of traditional detection methods. Summary of the Invention

[0005] This application provides a finger pressing state detection, training method, electronic device, and medium to solve the technical problem of poor generalization ability in traditional finger pressing state detection schemes.

[0006] Based on the above technical issues, the following solutions are provided:

[0007] Firstly, a method for detecting finger pressing status is provided, the method comprising:

[0008] The ultrasonic echo signal is preprocessed to obtain the model input data. The ultrasonic echo signal is the ultrasonic echo signal corresponding to the ultrasonic array emitting ultrasonic signals to the pressing area.

[0009] The input data of the model is input into the trained neural network model so that the neural network model outputs the finger pressing state.

[0010] In one implementation, the neural network model includes a neural network model based on a recurrent neural network; or a neural network model based on a convolutional neural network.

[0011] In one implementation:

[0012] The neural network model includes a first classification decision layer and a multi-layer recurrent neural network, and the output of the multi-layer recurrent neural network is connected to the first classification decision layer.

[0013] or;

[0014] The neural network model includes a second classification decision layer and a single-layer recurrent neural network, the output of which is connected to the second classification decision layer.

[0015] In one implementation, the preprocessing of the ultrasonic echo signal to obtain model input data includes:

[0016] The ultrasonic echo signal of a single frame is sampled at multiple points in the spatial domain to obtain the spatial sampling data corresponding to the ultrasonic echo signal of a single frame.

[0017] The spatial sampling data is converted into a first data vector;

[0018] Based on the first data vector, the model input data of the neural network model is obtained.

[0019] In one implementation, the step of spatially sampling a single frame of the ultrasonic echo signal to obtain spatial sampling data corresponding to the single frame of the ultrasonic echo signal includes:

[0020] According to the preset row and column intervals, the ultrasonic echo signal of a single frame is spatially sampled to obtain the spatial sampling data corresponding to the ultrasonic echo signal of a single frame.

[0021] In one implementation, obtaining the model input data of the neural network model based on the first data vector includes:

[0022] The first data vector is normalized, and the normalized first data vector is the model input data of the neural network model.

[0023] In one implementation, the preprocessing of the ultrasonic echo signal to obtain model input data includes:

[0024] The echo intensity feature value of the single frame of the ultrasonic echo signal acquired at the current time step, and the echo intensity feature values ​​corresponding to other time steps, are converted into a second data vector, wherein the other time steps are the time steps before the current time step.

[0025] Based on the second data vector, the model input data of the neural network model is obtained.

[0026] In one implementation, the other time steps are the N-1 consecutive time steps preceding the current time step N.

[0027] In one implementation, obtaining the model input data of the neural network model based on the second data vector includes:

[0028] The second data vector is normalized, and the normalized second data vector is the model input data of the neural network model.

[0029] In one implementation, the finger pressing state output by the neural network model includes: a classified state or a classified state to be determined, wherein the classified state is any one of at least two categories of pressing states.

[0030] In one implementation, the at least two types of pressing states include a finger-touching state and a finger-non-touching state; or, the at least two types of pressing states include a finger-touching state, a finger-non-touching state, a critical state from touch to non-touch, and a critical state from non-touch to touch.

[0031] In one implementation, after inputting the model input data into the trained neural network model so that the neural network model outputs the finger pressing state, the method further includes:

[0032] When the output of the finger pressing state is the classification determination state, the classification determination state is taken as the final finger pressing state, and the output of the classification determination state is updated to the state memory sequence according to the time sequence.

[0033] When the output finger pressing state is the classification pending state, the last recorded pressing state in the state memory sequence is taken as the final finger pressing state, or the pressing state with the highest state probability in the state memory sequence is taken as the final output finger pressing state, and the state memory sequence is updated accordingly.

[0034] Secondly, a method for training a neural network model is provided, the method comprising:

[0035] Acquire training sample data, which includes finger pressing state sample data and its corresponding pressing state classification label. The finger pressing state sample data is constructed based on ultrasonic echo sample signals, which are ultrasonic reflection signals after ultrasonic wave sample signals are emitted into the pressing area.

[0036] Based on the training sample data, the neural network model is trained until a neural network model that meets the preset model conditions is obtained.

[0037] The trained neural network model is used to output the finger pressing state.

[0038] In one implementation, the neural network model includes a neural network model based on a recurrent neural network; or a neural network model based on a convolutional neural network.

[0039] In one implementation:

[0040] The neural network model includes a first classification decision layer and a multi-layer recurrent neural network, and the output of the multi-layer recurrent neural network is connected to the first classification decision layer.

[0041] or;

[0042] The neural network model includes a second classification decision layer and a single-layer recurrent neural network, the output of which is connected to the second classification decision layer.

[0043] In one implementation, the finger pressing state sample data includes a first sample data vector;

[0044] The first sample data vector is obtained by transforming the spatial domain sample data, which is obtained by performing multi-point spatial sampling based on a single frame of ultrasonic echo sample signal.

[0045] In one implementation, the spatial sampling sample data is obtained by spatially sampling a single frame of the ultrasonic echo sample signal at a preset interval.

[0046] In one implementation, the first sample data vector is a normalized sample data vector.

[0047] In one implementation, the finger pressing state sample data includes a second sample data vector;

[0048] The second sample data vector is obtained by converting the echo intensity feature values ​​corresponding to multiple single-frame ultrasound echo sample signals, wherein the multiple single-frame ultrasound echo sample signals are single-frame ultrasound echo sample signals corresponding to multiple different time steps.

[0049] In one implementation, the multiple different time steps are consecutive time steps.

[0050] In one implementation, the second sample data vector is a normalized sample data vector.

[0051] In one implementation, the finger pressing state output by the neural network model includes: a classified state or a classified state, wherein the classified state is any one of at least two categories of pressing states.

[0052] In one implementation, the at least two types of pressing states include a finger-touching state and a finger-non-touching state; or, the at least two types of pressing states include a finger-touching state, a finger-non-touching state, a critical state from touch to non-touch, and a critical state from non-touch to touch.

[0053] Thirdly, a finger pressing state detection device is provided, the finger pressing state detection device comprising:

[0054] The processing module is used to preprocess the ultrasonic echo signal to obtain the model input data. The ultrasonic echo signal is the ultrasonic echo signal corresponding to the ultrasonic array emitting ultrasonic signals to the pressing area.

[0055] The input module is used to input the model input data into the trained neural network model so that the neural network model outputs the finger pressing state.

[0056] Fourthly, a neural network model training device is provided, the neural network model training device comprising:

[0057] The acquisition module is used to acquire training sample data, which includes finger pressing state sample data and its corresponding pressing state classification label. The finger pressing state sample data is constructed based on ultrasonic echo sample signals, which are ultrasonic reflection signals after ultrasonic wave sample signals are emitted into the pressing area.

[0058] The training module is used to train the neural network model based on the training sample data until a neural network model that meets the preset model conditions is obtained. The trained neural network model is used to output the finger pressing state.

[0059] Fifthly, an electronic device, wherein the finger pressing state detection device includes an ultrasonic piezoelectric sensor, an analog-to-digital converter, a data processing center, and a central controller, wherein the ultrasonic piezoelectric sensor array, the analog-to-digital converter, and the data processing center are all connected to the central controller;

[0060] The ultrasonic piezoelectric sensor is used to respond to the central controller emitting ultrasonic signals to the pressing area and receiving ultrasonic echo signals.

[0061] The analog-to-digital converter is used to perform digital-to-analog conversion on the ultrasonic echo signal in response to the central controller;

[0062] The data processing center, in response to the central controller, implements the steps of the finger press state detection method as described in any of the preceding claims.

[0063] A sixth aspect is an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the neural network model training method as described in any of the preceding claims.

[0064] In a seventh aspect, a readable storage medium is provided, the readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method as described in any of the preceding claims.

[0065] Eighthly, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps of the method described in any of the preceding claims.

[0066] In one of the aforementioned solutions, the ultrasonic echo signal is preprocessed to obtain model input data. This model input data is then input into a trained neural network model, causing the neural network model to output the corresponding finger pressing state. Different finger pressing states correspond to different ultrasonic echo signal intensity characteristics. The preprocessed ultrasonic echo signal, i.e., the model input data, characterizes the intensity characteristic changes. The trained neural network model outputs the finger pressing state based on the model input data. This eliminates the need for comparison by setting a benchmark threshold, thus eliminating the differences between different ultrasonic modules (e.g., different ultrasonic piezoelectric sensors used in different electronic devices). The same network model parameters have better generalization ability across different ultrasonic modules. Furthermore, it can effectively eliminate differences in various scenarios (temperature, frequency, etc.) and different finger states, covering a variety of special scenarios such as high temperature, low temperature, and wet fingers, exhibiting stronger generalization ability and accuracy. Attached Figure Description

[0067] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 This is a schematic diagram of a system framework for a processing system according to one embodiment of this application;

[0069] Figure 2 This is a schematic diagram illustrating the working principle of the signal transmission model and signal reception model of the ultrasonic array in one embodiment of this application;

[0070] Figure 3 This is a schematic diagram comparing the intensity of ultrasonic echo signals in a finger-touching state and a finger-non-touching state in one embodiment of this application;

[0071] Figure 4 This is a flowchart illustrating a finger pressing state detection method according to one embodiment of this application;

[0072] Figure 5 This is a schematic diagram of a model architecture for a neural network model based on a multilayer recurrent neural network according to an embodiment of this application;

[0073] Figure 6 This is a schematic diagram of a model architecture for a neural network model based on a single-layer recurrent neural network according to an embodiment of this application;

[0074] Figure 7 yes Figure 5 or Figure 6 A schematic diagram of the node relationships of a neuron after the hidden layer is expanded;

[0075] Figure 8 This is a schematic diagram of a data processing flow in a finger pressing state detection method of this application when a spatial multi-point detection scheme is adopted;

[0076] Figure 9 This is a schematic diagram of a data processing flow in a finger pressing state detection method of this application when a timing detection scheme is adopted;

[0077] Figure 10 This is a comparative schematic diagram of the falling edge of the down state and the rising edge of the up state in this application;

[0078] Figure 11 This is a flowchart illustrating a process of using state memory sequences to assist decision-making in one embodiment of this application;

[0079] Figure 12 This is another flowchart illustrating the use of state memory sequences to assist decision-making in one embodiment of this application;

[0080] Figure 13 This is a schematic diagram of a finger pressing state detection device according to one embodiment of this application;

[0081] Figure 14 This is a schematic diagram of a neural network model training device according to one embodiment of this application. Detailed Implementation

[0082] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0083] It should be understood that this application can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. Rather, providing these embodiments will make the disclosure thorough and complete, and will fully convey the scope of this application to those skilled in the art. In the drawings, for clarity, the dimensions of layers and regions, as well as their relative dimensions, may be exaggerated. The same reference numerals denote the same elements throughout.

[0084] It should be understood that when an element or layer is referred to as "on," "adjacent to," "connected to," or "coupled to" other elements or layers, it may be directly on, adjacent to, connected to, or coupled to other elements or layers, or there may be intervening elements or layers. Conversely, when an element is referred to as "directly on," "directly adjacent to," "directly connected to," or "directly coupled to" other elements or layers, there are no intervening elements or layers. It should be understood that although the terms first, second, third, etc., may be used to describe various elements, components, areas, layers, and / or portions, these elements, components, areas, layers, and / or portions should not be limited by these terms. These terms are only used to distinguish one element, component, area, layer, or portion from another element, component, area, layer, or portion. Therefore, without departing from the teachings of this application, the first element, component, area, layer, or portion discussed below may be referred to as a second element, component, area, layer, or portion.

[0085] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. When used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “comprising” and / or “including,” when used in this specification, identify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups. When used herein, the term “and / or” includes any and all combinations of the associated listed items.

[0086] To fully understand this application, detailed structures and steps will be presented in the following description to illustrate the technical solutions proposed in this application. Preferred embodiments of this application are described in detail below; however, in addition to these detailed descriptions, this application may have other implementation methods.

[0087] This application provides a finger pressing state detection scheme for detecting finger pressing states, including but not limited to finger touch states and finger non-touch states. Based on the detected finger pressing states, this scheme can be applied to fingerprint recognition scenarios or other applications involving touch functionality. For example, when a user is using an electronic device such as a mobile phone or computer, they need to touch the pressing area (such as the screen or other touchable area, such as a physical contact surface) to perform operations or recognition. Exemplary touch functions include, but are not limited to, touch unlock, single-click confirmation, double-click screenshot, etc., and are not specifically limited to any particular function. Generally, the electronic device needs to first capture the finger pressing state before triggering the corresponding touch function. Furthermore, the aforementioned electronic devices include products with touch functionality requirements such as mobile phones, computers, and tablets, and are not specifically limited to any particular device.

[0088] To facilitate understanding of the embodiments of this application, a processing system framework used in the embodiments of this application will be introduced first, such as... Figure 1 As shown, the processing system framework provided in this application includes a finger press state detection device and a touch function device, wherein, in Figure 1 In the examples provided, the touch-screen device includes, but is not limited to, a fingerprint unlocking device; it can also be other touch-screen systems, and this application does not limit the scope of the embodiments. The processing system is described below using fingerprint unlocking as an example.

[0089] As a framework example, the finger pressing state detection device includes an ultrasonic piezoelectric sensor, an analog-to-digital converter (ADC), a storage unit, a data processing center, and a central controller. The ultrasonic piezoelectric sensor, ADC, storage unit, and data processing center are all connected to the central controller. Exemplarily, the ultrasonic piezoelectric sensor includes an ultrasonic array and a coupling layer. The ultrasonic array can be a PVDF (polyvinylidene fluoride) array, and its specific configuration is not limited. The ADC can be a high-precision ADC, and its specific configuration is not limited. The central controller controls the ultrasonic piezoelectric sensor array to complete the excitation and retrieval of the ultrasonic array. The ADC performs digital signal conversion on the analog ultrasonic echo signal. The converted ultrasonic echo signal is sent to the data processing center via the storage unit. Based on the ultrasonic echo signal, the data processing center uses the finger pressing state detection method provided in this embodiment to detect the finger pressing state.

[0090] The fingerprint unlocking device includes a fingerprint processing algorithm module. After detecting a finger pressing state, the fingerprint processing algorithm module performs preliminary processing of the fingerprint data, and then performs further processing through a fingerprint recognition algorithm to complete related fingerprint registration, fingerprint recognition unlocking and other functions.

[0091] It should be noted that, Figure 1 This is merely an illustration and does not limit the actual processing framework used.

[0092] Continue reading Figure 2 As shown, Figure 2 Taking the pressed area as the screen as an example, a schematic diagram of the ultrasonic signal transmission model and ultrasonic echo signal reception model of the ultrasonic piezoelectric sensor is shown. The ultrasonic array of the ultrasonic piezoelectric sensor emits ultrasonic signals to the screen at a set fixed frequency, penetrating the screen to reach the ridge and valley of the finger's epidermal layer. Because the screen is in contact with the ridge (fingerprint protrusion), the finger valley (fingerprint indentation) is not in contact with the screen. The ultrasonic echo signal without finger coverage is used as the Base, and the ultrasonic echo signal with finger coverage is used as the Touch. The difference between the Touch and the Base is the texture image of the ridge, i.e., the fingerprint texture.

[0093] Due to the significant difference in acoustic impedance between human skin and air, the reflectivity of ultrasonic signals varies when they reach different contact surfaces. Therefore, the intensity of the ultrasonic echo signal under finger-touching conditions differs from that under no-touch conditions (no covering on the screen, i.e., non-touch). This difference can be used to determine the finger pressing state. The reflectivity under touching and non-touching conditions can be calculated as follows:

[0094]

[0095] Among them, z air ,z finger ,z s These represent the impedance of air, the impedance of human skin, and the impedance of the screen, respectively.

[0096] Generally, the impedance of a screen is much greater than that of air, which is approximately 430 Rayleighs. For example, the impedance of currently used screens is approximately 13 M Rayleighs. When a finger is not touching the screen, the impedance of the air is negligible when ultrasonic waves travel from the screen to the screen-to-air interface, resulting in almost total reflection of the ultrasonic signal. However, when a finger is touching the screen, both reflection and transmission occur simultaneously. Some ultrasonic waves pass through the finger's skin, while others are reflected. Therefore, the intensity of the ultrasonic echo signal is greater when the finger is not touching the screen, while it decreases significantly when the finger is touching the screen. From the above analysis, it can be seen that the intensity of the ultrasonic echo signal varies under different finger pressure states. Based on these characteristics of ultrasonic echo signal intensity variation, the finger pressure state can be determined.

[0097] As the background description states, traditional solutions require extensive data coverage under diverse conditions and the establishment of corresponding baseline thresholds. Covering all different situations presents challenges. For instance, signal strength varies significantly even under the same conditions. The ultrasonic echo signal changes even more dramatically depending on the finger's state; for example, the ultrasonic echo signal strength is lower when the finger is dry, and higher when the finger is wet. Furthermore, signal strength differs at different temperatures, with higher intensity at high temperatures and lower intensity at low temperatures. This results in poor generalization ability and low accuracy for the proposed solutions.

[0098] Therefore, in this application embodiment, considering the large differences in ultrasonic echo signal intensity under different conditions, which leads to poor generalization of traditional solutions, a finger pressing state detection scheme based on a neural network model is proposed. The following embodiments will be described in detail.

[0099] like Figure 4 As shown, with Figure 1 Taking the finger pressing state detection system shown as an example, this application embodiment provides a finger pressing state detection method based on a neural network model, including the following steps:

[0100] S10: Preprocess the ultrasonic echo signal to obtain model input data. The ultrasonic echo signal is the ultrasonic echo signal corresponding to the ultrasonic array emitting ultrasonic signals to the pressing area.

[0101] The electronic device includes a finger-pressing detection device and a pressing area, which can be a touchscreen or other touch-sensitive location, such as a touch panel, etc., with no specific limitation. The electronic device uses an ultrasonic piezoelectric sensor array to emit ultrasonic signals to the pressing area at a fixed frequency, and uses the ultrasonic piezoelectric sensor array to receive the reflected signals formed by the emitted ultrasonic signals, i.e., ultrasonic echo signals. These are converted into digital ultrasonic echo signals by an analog-to-digital converter and then sent to a data processing center via a storage unit. The data processing center preprocesses the ultrasonic echo signals in the storage unit to obtain model input data.

[0102] S20: Input the model input data into the trained neural network model so that the neural network model outputs the finger pressing state.

[0103] In this embodiment, a neural network model based on the output of ultrasonic echo signals to represent finger pressing states is pre-trained. This neural network model is trained using a large amount of training sample data. This model includes, but is not limited to, neural network models built based on Recurrent Neural Networks (RNNs) or Convolutional Neural Networks (CNNs). The network parameters of this neural network model are obtained by training on pre-constructed finger pressing data covering multiple scenarios, temperatures, and other conditions. Each scenario's finger pressing data includes a pressing classification label. During model training, the model can be trained using a pre-defined training method until a suitable neural network model is obtained, i.e., the trained neural network model. For details on the model training process, please refer to the description in subsequent embodiments.

[0104] In this step, the data processing center preprocesses the ultrasonic echo signal to obtain model input data. This model input data is then fed into the trained neural network model, causing the model to output the corresponding finger pressing state. For example, the trained neural network model can be built into an electronic device or deployed on a server for use as a model call; no specific limitation is made.

[0105] As can be seen, this embodiment provides a finger pressing state detection method based on a neural network model. By preprocessing the ultrasonic echo signal to obtain model input data, the model input data is input into the trained neural network model so that the neural network model outputs the corresponding finger pressing state. Different finger pressing states correspond to different ultrasonic echo signal intensity characteristics. The preprocessed ultrasonic echo signal, i.e., the model input data, represents the intensity characteristic change. The trained neural network model outputs the finger pressing state based on the model input data. It eliminates the need for setting a benchmark threshold for comparison, thus eliminating the differences between different ultrasonic modules (e.g., different ultrasonic piezoelectric sensors used in different electronic devices). The same network model parameters have better generalization ability across different ultrasonic modules. Moreover, it can effectively eliminate the differences under various different scenarios (temperature, frequency, etc.) and different finger states, and can cover a variety of special scenarios, such as high temperature, low temperature, and wet fingers, with stronger generalization ability and accuracy.

[0106] It should be noted that in the embodiments of this application, the neural network model can be constructed based on a recurrent neural network (RNN) or a convolutional neural network (CNN), and there is no specific limitation. Taking the construction of the neural network model based on a recurrent neural network as an example, in one embodiment, the neural network model includes a first classification decision layer and a multi-layer recurrent neural network, the output of which is connected to the corresponding first classification decision layer; or, the neural network model includes a second classification decision layer and a single-layer recurrent neural network, the output of which is connected to the second classification decision layer. That is to say, the recurrent neural network used can be a single-layer recurrent neural network or a multi-layer recurrent neural network. It should be understood that the first classification decision layer and the second classification decision layer are only used to distinguish between single-layer and multi-layer networks, and both can be the same classification decision layer.

[0107] The single-layer or multi-layer recurrent neural network used can be determined according to actual needs and is not specifically limited. Since finger pressing state detection is a pressing state detection process over a relatively long period of time, and in a recurrent neural network, each current state is related to a previous or subsequent state, this embodiment of the application, through a neural network model based on a recurrent neural network, will help to detect the finger pressing state more accurately by associating it with previous or subsequent states.

[0108] To facilitate understanding of the subsequent detection scheme, the neural network model based on a recurrent neural network provided in the embodiments of this application will be introduced below.

[0109] like Figure 5 As shown in the example, the neural network model includes a first classification decision layer and a multi-layer recurrent neural network. The output of the multi-layer recurrent neural network is connected to the first classification decision layer. Since the multi-layer recurrent neural network is related to the temporal state, it is expanded according to the temporal state. Therefore, the multi-layer recurrent neural network includes hidden layer networks corresponding to different temporal states. Each hidden layer network represents a network architecture corresponding to a different temporal state. It can be seen that the hidden layer network includes n hidden layers, where n≥2. The n hidden layers in the hidden layer network are connected sequentially, and the output of the hidden layer in the previous temporal state is passed to the same hidden layer in the next temporal state. The output of the last hidden layer of the hidden layer network, which is the output of the multi-layer recurrent neural network, is connected to the first classification decision layer. Figure 5 The classification decision layer in the text is the first classification decision layer.

[0110] For details, please continue reading Figure 5 As shown, in this example, x is expanded according to the time sequence. t0 ,x t1 ,...,x tnThese represent the input data of the hidden layer network in a multi-layer recurrent neural network at different time sequences t0-tn, respectively. t0 ,y t1 ,...,y tn This represents the finger pressing state output by the classification decision layer at the corresponding time step. The hidden layer network includes hidden layer 1, hidden layer 2, ..., hidden layer n. The output of hidden layer n is connected to the input of the classification decision layer. The value of each hidden layer in the hidden layer network of the previous time step is passed to the hidden layer of the same layer in the next time step. For example, the output of hidden layer 1 at time step t0 is passed to the hidden layer 1 at time step t1, and the output of hidden layer 1 at time step t1 is passed to the hidden layer 1 at time step t2. The passed content includes the state vector of the hidden layer output. For example, hidden layer 1 at time step t0 passes state vector 1 to hidden layer 1 at time step t1. The output of the next hidden layer will be passed to the previous hidden layer, and so on for hidden layers of the same layer or other layers. Figure 5 As shown, these details will not be elaborated upon here. It is also worth noting that multi-layer recurrent neural networks also have weight matrix parameters, which will be discussed in more detail when introducing single-layer recurrent neural networks. The specific number of hidden layers in a multi-layer recurrent neural network can be adjusted, as can the size of the model input data; however, no specific limitations are imposed on the embodiments described in this application.

[0111] like Figure 6 As shown in the example, the neural network model includes a second classification decision layer and a single-layer recurrent neural network. The output of the single-layer recurrent neural network is connected to the second classification decision layer. Since the single-layer recurrent neural network is related to the temporal state, it is expanded according to the temporal state. Therefore, the single-layer recurrent neural network includes hidden layer networks corresponding to different temporal states. Each hidden layer network represents a network architecture corresponding to a different temporal state. The hidden layer network of this single-layer recurrent neural network contains only one hidden layer, and the output of the hidden layer in the previous temporal state is passed to the hidden layer in the next temporal state. The output of the hidden layer is connected to the second classification decision layer. Unlike multi-layer recurrent neural networks, single-layer recurrent neural networks contain only a single hidden layer.

[0112] Specifically, such as Figure 6 As shown, in Figure 6 In the example, we will use three timing sequences as examples for further explanation. t-1 ,x t1 ,x t+1 y represents the input data of the hidden layer network corresponding to the three time series. t-1 ,y t1 ,,y t+1These represent the finger pressing states output by the classification decision layer at the corresponding time sequence. Each hidden layer network includes one hidden layer. The output of hidden layer 1 at time sequence t-1 is passed to the hidden layer at time sequence t1, and the output of the hidden layer at time sequence t1 is passed to the hidden layer at time sequence t+1.

[0113] In the single-layer or multi-layer recurrent neural networks mentioned above, the weight matrices U, W, and V are linear relationship parameters in the recurrent neural network. Let's continue with... Figure 6 Taking the single-layer recurrent neural network shown as an example, the working principle of this neural network model based on the single-layer recurrent neural network will be further explained. The network framework of the neural network model based on the single-layer recurrent neural network has the following expression:

[0114] y t =φ(O t )

[0115] O t =g(V·S) t )

[0116] S t =f(U·x t +W·S t-1 )

[0117] Where f(*) and g(*) represent the activation functions of the corresponding layers, and φ(*) represents the classification decision algorithm used in the classification decision layer. For example, the classification decision algorithm can be a binary classification or multi-class classification algorithm, without specific limitations. t S represents the value of the hidden layer corresponding to time t, which is the aforementioned state vector. t-1 S represents the value of the hidden layer corresponding to time series t-1. t+1 This represents the value of the hidden layer corresponding to time series t+1, O t This represents the output of the hidden layer corresponding to time t, O t-1 This represents the output of the hidden layer corresponding to time series t-1, O t+1 y represents the output of the classification decision layer corresponding to time series t+1. t y represents the output of the classification decision layer corresponding to time series t. t-1 y represents the output of the classification decision layer corresponding to time series t-1. t+1 Let represent the output of the classification decision layer at time t+1. The weight matrix U represents the weight matrix from the model input to the hidden layer, the weight matrix V represents the weight matrix from the hidden layer to the classification decision layer, and the weight matrix W represents the weight matrix from the hidden layer of the previous time series to the hidden layer of the corresponding next time series. Therefore, the input vector... After the classification decision layer outputs and makes decisions, the final output is the finger pressing state of that time sequence. It can be seen that the output state of each time sequence is not only affected by the current input, but also related to the information of the previous time sequence.

[0118] Furthermore, in one embodiment, the number of neurons in the hidden layer of the single-layer or multi-layer recurrent neural network can also be adjusted, depending on the specific training parameters. This application does not impose any limitations on this aspect. As an example, taking a single-layer recurrent neural network as an example, its network with nodes can be unfolded as follows... Figure 7 As shown. x t h represents the input vector of the current time series t. 11 h 21 ,...h m1 h represents the value of a hidden layer neuron node at time t. 12 h 21 ,...h m2 Let f1, f2, ... h be the values ​​of the neurons in the hidden layer at the previous time step t-1. c Let y represent the activation function of the hidden layer at the current time t. t This represents the finger pressing state output by the classification decision layer at time t.

[0119] It should be noted that the above embodiments are based on a neural network model constructed based on a recurrent neural network. Based on this, the embodiments of this application also provide at least two detection schemes. The two detection schemes differ in the processing of model input data and each has its own advantages, which will be described in detail below.

[0120] The first approach is a multi-point airspace detection scheme.

[0121] like Figure 8 As shown, in one embodiment, based on the neural network model constructed using the recurrent neural network described above, step S10, namely, preprocessing the ultrasonic echo signal to obtain model input data, includes:

[0122] S101A: Perform multi-point spatial sampling on a single frame of the ultrasonic echo signal to obtain spatial sampling data corresponding to the single frame of the ultrasonic echo signal.

[0123] S102A: Convert the spatial sampling data into a first data vector;

[0124] S103A: Based on the first data vector, obtain the model input data of the neural network model.

[0125] This embodiment is one of the above-mentioned multi-point spatial detection schemes. When the data processing center receives a single frame of the ultrasonic echo signal, it performs multi-point spatial sampling on the single frame of the ultrasonic echo signal to obtain the spatial sampling data corresponding to the single frame of the ultrasonic echo signal.

[0126] like Figure 2 As shown, Figure 2 Taking the pressed area as the screen as an example, a schematic diagram of the ultrasonic signal transmission model and ultrasonic echo signal reception model of the ultrasonic piezoelectric sensor is shown. The ultrasonic array of the ultrasonic piezoelectric sensor emits ultrasonic signals to the screen at a set fixed frequency, penetrating the screen to reach the ridge and valley of the finger's epidermal layer. Because the screen is in contact with the ridge (fingerprint protrusion), the finger valley (fingerprint indentation) is not in contact with the screen. The ultrasonic echo signal without finger coverage is used as the Base, and the ultrasonic echo signal with finger coverage is used as the Touch. The difference between the Touch and the Base is the texture image of the ridge, i.e., the fingerprint texture.

[0127] Generally, the impedance of a screen is much greater than that of air. Air impedance is approximately 430 Rayleighs. For example, assuming the current screen impedance is approximately 13 M Rayleighs, when an ultrasonic wave travels from the screen to the air surface where the finger is not touching it, the air impedance is negligible, and the ultrasonic signal is almost completely reflected. However, when a finger is touching it, both reflection and transmission occur simultaneously. Some ultrasonic signals pass through the finger's skin, while others are reflected. Therefore, the ultrasonic echo signal intensity is greater when the finger is not touching it, while it decreases significantly when the finger is touching it. Figure 3 As shown, assuming a sampling frequency of 50Hz, repeating the finger-cover-lift-cover-lift action for 20 seconds to collect 1000 frames of data, the following results are obtained: Figure 3 The graph shown represents the relationship between ultrasonic echo signals and finger pressing states. "up" indicates a non-touching state, "down" indicates a touching state, the signal value is the average intensity across the entire screen, and the horizontal axis represents the frame number. Figure 3 It can also be seen that the ultrasonic echo signal intensity value corresponding to the ridge position is low when touched by a finger, which differs from the ultrasonic echo signal intensity value at the trough position. Therefore, based on the above characteristics, the ultrasonic echo intensity variation feature can be extracted using a multi-point sampling method in the spatial domain.

[0128] For example, assuming the spatial size of a single-frame ultrasonic echo signal after analog-to-digital conversion is m*n, each point in the spatial domain of the single-frame ultrasonic echo signal represents the ultrasonic echo signal intensity corresponding to that point. The spatial multi-point detection scheme involves spatial multi-point sampling of the single-frame ultrasonic echo signal. Based on the obtained spatial sampling data, a first data vector matching the input size of the recurrent neural network is obtained. Then, based on the first data vector, the model input data of the neural network model is obtained and used as the network input. The model input data is input into the recurrent neural network in the neural network model. After the weight calculation of the recurrent neural network, the output of the recurrent neural network is given to the classification decision layer in the neural network model so that the classification decision layer outputs the current finger pressing state.

[0129] As can be seen, this embodiment provides a finger pressing state detection method based on a neural network model. Combining the aforementioned neural network model constructed based on a recurrent neural network, the network input of the recurrent neural network is formed by spatially sampling multiple points of a single frame of ultrasonic echo signal to identify the finger pressing state. In this embodiment, besides improving applicability and generalization by not using a sampling benchmark threshold comparison method, the use of multi-point spatial sampling also reduces the amount of data in the model input data, thus improving processing efficiency.

[0130] In conjunction with the above embodiments, in one embodiment, step S101A, namely, performing spatial sampling on the acquired single frame of the ultrasonic echo signal to obtain spatial sampling data corresponding to the single frame of the ultrasonic echo signal, includes: performing spatial sampling on the single frame of the ultrasonic echo signal according to a preset row and column interval to obtain spatial sampling data corresponding to the single frame of the ultrasonic echo signal.

[0131] For example, assuming the spatial size of a single frame of ultrasound echo signal data is m*n, by sampling the Base and Touch frames of size m*n with a row interval of r1 and a column interval of c1, an m1*n1 matrix is ​​obtained. Finally, this sampling matrix can form a 1*(m1*n1) matrix vector, which is the first data vector, and serves as the input to the aforementioned recurrent neural network, i.e., the input to the neural network model. It should be noted that the row interval r1 and column interval c1 can be set according to requirements and are not specifically limited.

[0132] In this embodiment, when performing multi-point spatial sampling of the ultrasonic echo signal, a preset row and column interval can be used for sampling. In addition to reducing the amount of data and improving the generalization ability, the interval sampling method can effectively reduce the loss of ultrasonic echo signal variation characteristics, so as to ensure the accuracy of subsequent detection.

[0133] Of course, in other embodiments, the specific sampling interval of the preset row and column interval can be determined according to the actual situation, and this application embodiment does not limit it; in addition, a non-row and column interval spatial multi-point sampling method can also be adopted, and this application embodiment does not limit it either.

[0134] In one embodiment, step S103A, namely obtaining the model input data of the neural network model based on the first data vector, includes: normalizing the first data vector, wherein the normalized first data vector is the model input data of the neural network model.

[0135] In this embodiment, normalizing the transformed first data vector refers to normalizing all vector elements in the first data vector so that the element values ​​are within a uniform range. Therefore, the normalized first data vector serves as the model input data for the neural network model, effectively reducing the computational load of the subsequent neural network model and thus improving the overall efficiency of finger pressure state detection. It should be noted that in other embodiments, the transformed first data vector can also be used as model input data; no specific limitation is imposed.

[0136] The foregoing embodiments described the spatial multi-point detection scheme provided by the embodiments of this application. The second scheme provided by the embodiments of this application is described below.

[0137] The second approach is a timing detection approach.

[0138] like Figure 9 As shown, in one embodiment, based on the neural network model constructed using the recurrent neural network described above, step S10, namely, preprocessing the ultrasonic echo signal to obtain model input data, includes:

[0139] S101B: The echo intensity feature value of the single frame of the ultrasonic echo signal acquired at the current time step, and the echo intensity feature values ​​corresponding to other time steps, are converted into a second data vector, wherein the other time steps are the time steps before the current time step.

[0140] S102B: Obtain the model input data of the neural network model based on the second data vector.

[0141] This embodiment is one of the above-mentioned time-series detection schemes. After receiving the ultrasonic echo signals from each time step, the data processing center calculates the ultrasonic echo intensity characteristic value of the single-frame ultrasonic echo signal at each time step, which is used as the intensity value corresponding to each time step. Taking a data time step sequence N as an example, assuming the current time step is T N Current time step TN The corresponding echo intensity characteristic value is Therefore, this embodiment will set the current time step T. N The echo intensity feature values ​​of the acquired single-frame ultrasonic echo signal, along with the echo intensity feature values ​​corresponding to other time steps, are transformed into a second data vector that matches the input size of the recurrent neural network. Other time steps are the current time step T. N Based on the previous time step, the model input data of the neural network model is obtained according to the second data vector.

[0142] like Figure 10 The diagram shows a comparison of ultrasound echo intensity changes between the up and down states. The horizontal axis represents time, and the vertical axis represents ultrasound echo intensity. Figure 10 It can be seen that, in continuous time, the state change from up to down has a clear falling edge (e.g. Figure 10 (Left image) The state change from down to up has a clear rising edge (e.g.) Figure 10 (See the right image in the middle). Therefore, the current finger pressing state can be further determined by the continuous temporal changes in different frames.

[0143] As can be seen, this embodiment provides a finger pressing state detection method based on a neural network model. Combining the aforementioned neural network model built on a recurrent neural network, the echo intensity feature value of the single frame of the ultrasonic echo signal acquired at the current time step, and the echo intensity feature values ​​corresponding to other time steps, are transformed into the network input of the recurrent neural network to identify the finger pressing state. In this embodiment, besides improving applicability and generalization by not using a sampling benchmark threshold comparison method, the spatial domain changes during the up-down motion of the finger. By using a multi-time-step spatial sequence as model input, the finger pressing state can be detected through temporal spatial state changes. This allows for the capture of continuous motion and is applicable to more scenarios, such as fully covering changes in ultrasonic echo intensity under high or low temperature conditions, thereby achieving more accurate or diverse finger pressing state detection.

[0144] In conjunction with the above embodiments, in one embodiment, the other time steps are the N-1 consecutive time steps preceding the current time step N. That is, in this embodiment, the echo intensity feature value of the single frame of the ultrasonic echo signal acquired at the current time step N, and the echo intensity feature values ​​corresponding to the N-1 consecutive time steps preceding the current time step N, are converted into a second data vector, i.e.:

[0145]

[0146] for example, Represents the second data vector. These represent the echo intensity characteristic values ​​corresponding to the current time step N and the previous N-1 consecutive time steps, respectively.

[0147] In this embodiment, the finger pressing state can be detected by changes in the spatial domain state over consecutive time steps, enabling the capture of more precise continuous movements and thus achieving more accurate or diverse finger pressing state detection. Of course, in other embodiments, the other time steps can also be the echo intensity feature values ​​corresponding to previously consecutive time steps; specific embodiments in this application are not limited in this regard.

[0148] In one embodiment, step S102B, namely obtaining the model input data of the neural network model based on the second data vector, includes: normalizing the second data vector, wherein the normalized second data vector is the model input data of the neural network model.

[0149] In this embodiment, normalizing the transformed second data vector refers to normalizing all vector elements in the second data vector so that the element values ​​are within a uniform range. Therefore, the normalized second data vector serves as the model input data for the neural network model, effectively reducing the computational load of the subsequent neural network model and thus improving the overall efficiency of finger pressure state detection. It should be noted that in other embodiments, the transformed second data vector can also be used as model input data; there is no specific limitation.

[0150] In one embodiment, the echo intensity feature value includes the average echo intensity of a single frame of the ultrasonic echo signal, or the echo intensity feature value includes the minimum echo intensity value in a single frame of the ultrasonic echo signal, without any specific limitation. Using the average echo intensity value allows for a more comprehensive capture of the single-frame signal characteristics; while using the minimum echo intensity value simplifies the calculation.

[0151] For example, in the above embodiments, the average echo intensity can be an arithmetic mean or other forms of mean. The specific embodiments of this application are not limited, and can be set according to requirements to ensure the flexibility of the solution.

[0152] It should be understood that the above detection scheme embodiments are illustrated using a neural network model based on a recurrent neural network. In other embodiments, when using other neural network models, such as convolutional neural networks, the same detection scheme can be used to process the model input data, and no specific limitation is made.

[0153] It should be noted that, depending on the needs, the classification decision layer in the above neural network model can adopt a binary classification algorithm, a three-class classification algorithm, or other multi-class classification methods; no specific limitation is made.

[0154] In one embodiment, the finger pressing state output by the neural network model includes: a classified determined state or a classified undetermined state. The classified determined state is any pressing state among at least two categories. The classified determined state refers to a pressing state that is already clear and can be used for subsequent touch function applications, such as a finger touch state, a finger non-touch state, or other pressing states. The classified undetermined state refers to a pressing state that still needs further confirmation.

[0155] In this embodiment, the classification decision layer in the above neural network model can adopt a binary classification algorithm, a tri-class classification algorithm, or other multi-class classification methods to improve the adaptability of the scheme and application scenarios, and thus have higher application value.

[0156] In one embodiment, the at least two categories of pressing states include a finger-touching state and a finger-non-touching state; or, the at least two categories of pressing states include a finger-touching state, a finger-non-touching state, a critical state from touch to non-touch, and a critical state from non-touch to touch.

[0157] In other words, in one case, the finger pressing state output by the neural network model is a finger touching state, a finger not touching state, or an undetermined state; in another case, the finger pressing state output by the neural network model is a finger touching state, a finger not touching state, a critical state between touching and not touching, a critical state between not touching and touching, or an undetermined state.

[0158] Of course, in other embodiments, the finger pressing state output by the neural network model may only have a classification to determine the state. For example, when using binary classification, in one case, the finger pressing state output by the neural network model is either a finger touching state or a finger not touching state; in another case, the finger pressing state output by the neural network model is either a finger touching state, a finger not touching state, a critical state between touching and not touching, or a critical state between not touching and touching.

[0159] In conjunction with the above embodiments, in one embodiment, when the finger pressing state output by the neural network model includes a classified determined state or a classified undetermined state, after step S20, that is, after inputting the model input data into the trained neural network model so that the neural network model outputs the finger pressing state, the method further includes:

[0160] S101: When the output of the finger pressing state is the classification determination state, the classification determination state is taken as the final finger pressing state, and the output of the classification determination state is updated to the state memory sequence according to the time sequence.

[0161] S102: When the output finger pressing state is the classification pending state, take the pressing state recorded last in the state memory sequence as the final finger pressing state, and update it to the state memory sequence in chronological order.

[0162] For example, assuming the classification determines the state as including a first type of state and a second type of state, this embodiment assists in the decision-making of the finger pressing state by setting a continuously updated state memory sequence. A pending state refers to a situation where the current neural network model cannot output whether it is a first type of state or a second type of state, but is in a state where the final finger pressing state needs to be further determined. In this case:

[0163] When the output finger pressing state is the first type of state, the first type of state is taken as the final finger pressing state, and the output first type of state is updated to the state memory sequence according to the time sequence.

[0164] When the output finger pressing state is the second type of state, the second type of state is taken as the final finger pressing state, and the output second type of state is updated to the state memory sequence according to the time sequence.

[0165] When the output finger pressing state is the classification pending state, the last recorded pressing state is taken from the latest state memory sequence as the final finger pressing state and updated to the state memory sequence in chronological order.

[0166] The first type of state and the second type of state are different finger pressing states. The specific state is not limited. For example, in one embodiment, the first type of state is the finger touch state and the second type of state is the finger non-touch state.

[0167] Taking the first type of state as the finger touch state and the second type of state as the finger non-touch state as an example, it should be understood that the finger may be in the up or down state for a long time. In this case, the data, especially in the time-series detection scheme, may not have rising or falling edges. To address this, a state memory sequence is introduced. A fixed-length state memory sequence MemSeq is set. The classification decision layer includes multi-class states, including 0-up, 1-down, and 2-maintain. That is, the output of the classification decision layer is 0, which represents the finger non-touch state; output is 1, which represents the finger touch state; and output is 2, which represents the undetermined state. The final finger pressing state is determined by combining the output of the neural network model with the state memory sequence MemSeq.

[0168] Specifically, please refer to Figure 11 As shown, when the finger pressing state output by the neural network model is the finger touch state, that is, the O output by the classification decision layer... t When y = 1, the final finger pressing state is y. t =1, and output the finger touch state y according to the timing sequence. t =1 corresponds to updating the state memory sequence MemSeq; when the finger pressing state output by the neural network model is the non-touch state of the finger, that is, the O output by the classification decision layer. t =0, final finger pressing state y t =0, and output the non-touch state y of the finger according to the timing sequence. t =0 corresponds to updating the state memory sequence MemSeq; when the finger pressing state output by the neural network model is the undetermined state, that is, the O output by the classification decision layer. t =2, then from the latest state memory sequence MemSeq, obtain the finger pressing state that was last recorded as the final output, i.e., y. t =MemSeq[-1], and update the state to the memory sequence MemSeq. In summary, taking the above three-class classification case as an example, that is:

[0169] When the classification decision layer outputs an up or down state, the final output of the finger pressing state is the corresponding classification output state (up or down state), and the state memory sequence MemSeq is updated accordingly.

[0170] When the classification decision layer outputs the "maintain" state, the preceding states recorded in the state memory sequence MemSeq are used to determine whether the final output finger pressing state is up or down, and the state memory sequence MemSeq is updated accordingly.

[0171] In this embodiment, the state memory sequence MemSeq is used to assist decision-making, which can improve the richness of the state, make the detection scheme more accurate, and provide a corresponding processing mechanism when the neural network model cannot accurately output the up or down state, thus ensuring the feasibility of the scheme. Moreover, it can achieve relatively high accuracy when facing different signal sampling frequencies.

[0172] In conjunction with the above embodiments, in one embodiment, when the finger pressing state output by the neural network model includes a classified determined state or a classified undetermined state, after step S20, that is, after inputting the model input data into the trained neural network model so that the neural network model outputs the finger pressing state, the method further includes:

[0173] S201: When the output of the finger pressing state is the classification determination state, the classification determination state is taken as the final finger pressing state, and the output of the classification determination state is updated to the state memory sequence according to the time sequence.

[0174] S202: When the output finger pressing state is the classified pending state, the pressing state with the highest state probability is obtained from the latest state memory sequence as the final output finger pressing state, and the state memory sequence is updated accordingly.

[0175] For example, when using a three-class classification, the finger pressing state of the neural network model includes a first-class state, a second-class state, or a state of undetermined classification. The undetermined state refers to a situation where the current neural network model cannot output whether it is a first-class state or a second-class state, but is in a state where the final finger pressing state needs to be further determined. In this case:

[0176] When the output finger pressing state is the first type of state, the first type of state is taken as the final finger pressing state, and the output first type of state is updated to the state memory sequence according to the time sequence.

[0177] When the output finger pressing state is the second type of state, the second type of state is taken as the final finger pressing state, and the output second type of state is updated to the state memory sequence according to the time sequence.

[0178] When the output finger pressing state is the classified pending state, the pressing state with the highest state probability is obtained from the latest state memory sequence as the final finger pressing state, and the state memory sequence is updated accordingly.

[0179] The first type of state and the second type of state are different finger pressing states. The specific state is not limited. For example, in one embodiment, the first type of state is the finger touch state and the second type of state is the finger non-touch state.

[0180] In this embodiment, a fixed-length state memory sequence MemSeq is also set. The classification decision layer includes multi-class states, including 0-up, 1-down, and 2-maintain. The final finger pressing state is determined by combining the output of the neural network model with the records of the state memory sequence MemSeq. The difference from the above embodiment is that in this embodiment, when the output of the classification decision layer is maintain, the final output finger pressing state is determined by combining the highest probability of the state recorded in the state memory sequence MemSeq.

[0181] For example, using the above three-category classification, please refer to [link / reference]. Figure 12 As shown, when the finger pressing state output by the neural network model is the finger touch state, that is, the O output by the classification decision layer... t =1, then the final finger pressing state y t =1, and output the finger touch state y according to the timing sequence. t =1 corresponds to updating the state memory sequence MemSeq; when the finger pressing state output by the neural network model is the non-touch state of the finger, that is, the O output by the classification decision layer. t =0, then the final finger pressing state y t =0, and output the non-touch state y of the finger according to the timing. t =0 corresponds to updating the state memory sequence MemSeq; when the finger pressing state output by the neural network model is the classification pending state, that is, the O output by the classification decision layer. t =2, then from the state memory sequence MemSeq, the pressing state with the highest probability is taken as the final finger pressing state. Specifically, as an example, first determine whether the number of times 0 appears in the state memory sequence MemSeq is greater than the number of times 1 appears. If so, then y t =0; if not, then y t =1, and update the final state to the memory sequence MemSeq. In summary, that is:

[0182] When the classification decision layer outputs an up or down state, the final finger pressing state is the corresponding classification output (up or down state), and the state memory sequence MemSeq is updated accordingly.

[0183] When the classification decision layer outputs the maintain state, the state memory sequence MemSeq is taken as the final finger pressing state, with the highest probability of 0 or 1 among the states recorded. The state memory sequence MemSeq is then updated accordingly.

[0184] This embodiment presents another scheme that uses state memory sequence MemSeq to assist decision-making. This can improve the richness of states, resulting in higher accuracy of the detection scheme. It also provides a corresponding processing mechanism when the neural network model cannot accurately output up or down states, ensuring feasibility and achieving relatively high accuracy even with different signal sampling frequencies. In addition, the use of prior state probability processing can make the scheme more fault-tolerant in practical applications.

[0185] It should be understood that in the above embodiments, the length of the state memory sequence MemSeq is adjustable and not specifically limited. For example, it can be an infinitely long sequence with a length of 1 to infinity; or it can be a fixed sequence, set according to requirements. Moreover, the above state memory sequence can also be used to assist decision-making in other multi-class cases. The above three-class classification is only an example and does not limit the embodiments of this application.

[0186] In one embodiment, the method can further determine the user's finger action by using the state memory sequence MemSeq recorded. For example, a state memory sequence MemSeq of [1,1,1,1,1,1] can be regarded as a long press; a state memory sequence MemSeq of [0,0,1,1,0,0] can be regarded as a single click; and a state memory sequence MemSeq of [1,1,0,0,1,1] can be regarded as a double click, thereby enabling the detection of more finger actions.

[0187] In this embodiment, by further utilizing the state memory sequence MemSeq, finger movements can be quickly detected based on previous finger pressing state detection results.

[0188] In one embodiment, when implementing the above embodiments, the data acquisition frequency can be adjusted according to the requirements. It is not limited to 50Hz, but can also be other acquisition frequencies. The higher the data acquisition frequency, the more intensity change points there are in the up-down process, which is beneficial to improving the detection accuracy and efficiency.

[0189] It should be noted that the above embodiments take the final output result as up or donw as an example. In other embodiments, other final classification states may also be included. The neural network model can be set for targeted training depending on the needs, and there is no specific limitation.

[0190] In summary, the embodiments of this application provide a finger pressing state detection method based on a neural network model, which brings the following technical effects: it eliminates the differences between different ultrasound modules without the need to set a benchmark threshold for comparison; the same network model parameters have better generalization ability among different ultrasound modules; and it can effectively eliminate the differences under various different scenarios (temperature, frequency, etc.) and different finger states, and can cover a variety of special scenarios, such as high temperature, low temperature, and wet fingers, with stronger generalization ability and accuracy.

[0191] The above embodiments describe a finger pressing state detection method based on a neural network model provided by the embodiments of this application. In implementing the above method, it is necessary to train a corresponding neural network model. The neural network model training method provided by the embodiments of this application is described below.

[0192] In one embodiment, a neural network model training method is provided, comprising:

[0193] S01: Obtain training sample data, which includes finger pressing state sample data and its corresponding pressing state classification label. The finger pressing state sample data is constructed based on ultrasonic echo sample signals, which are ultrasonic reflection signals after ultrasonic wave sample signals are emitted into the pressing area.

[0194] S02: Based on the training sample data, train the neural network model until a neural network model that meets the preset model conditions is obtained;

[0195] In this embodiment, the model architecture of the neural network model needs to be constructed first and training sample data needs to be collected. The training sample data includes finger pressing state sample data and its corresponding pressing state classification label. The finger pressing state sample data is constructed based on ultrasonic echo sample signals. The ultrasonic echo sample signals are ultrasonic reflection signals after ultrasonic wave sample signals are emitted into the pressing area. Then, based on the training sample data, the neural network model is trained until a neural network model that meets the preset model conditions is obtained. The trained neural network model is used to output the finger pressing state.

[0196] It should be understood that, as an example, the parameters of this neural network model are obtained through supervised learning on a large amount of training sample data. The model parameters are trained on a previously constructed dataset covering finger pressing states across multiple scenarios and temperatures, and the information obtained in each scenario has a pressing classification label. In specific training, the parameter values ​​that minimize the loss function can be solved using stochastic gradient descent, and the parameters are updated through backpropagation using the chain rule until the optimal model parameters are obtained. The model parameters include model weight coefficients, and the preset model conditions include at least the aforementioned optimal conditions for model parameters.

[0197] It should be noted that, depending on the application requirements, the required neural network model can be specifically trained to match the input data during the application. In one embodiment, the aforementioned neural network model includes, but is not limited to, a neural network model based on a recurrent neural network or a neural network model based on a convolutional neural network.

[0198] Furthermore, the neural network model includes a first classification decision layer and a multi-layer recurrent neural network, wherein the output of the multi-layer recurrent neural network is connected to the first classification decision layer;

[0199] or;

[0200] The neural network model includes a second classification decision layer and a single-layer recurrent neural network, the output of which is connected to the second classification decision layer.

[0201] For details on the model architecture, please refer to the aforementioned description; it will not be repeated here.

[0202] It should be noted that, based on the aforementioned spatial multi-point detection scheme and temporal detection scheme, corresponding training data needs to be constructed specifically for targeted application during model training, which will be described below.

[0203] In one embodiment, the finger pressing state sample data includes a first sample data vector; the first sample data vector is obtained by converting spatial domain sample data, which is obtained by performing multi-point spatial sampling based on a single frame ultrasonic echo sample signal.

[0204] In one embodiment, the spatial sampling sample data is obtained by spatially sampling a single frame of the ultrasonic echo sample signal at a preset interval.

[0205] In one embodiment, the first sample data vector is a normalized sample data vector.

[0206] In this embodiment, for the airspace multi-point detection scheme, the neural network model required for the airspace multi-point detection scheme can be trained in a targeted manner. For more information, please refer to the relevant description of the foregoing embodiment. The difference is that the model input data at this time is training sample data.

[0207] In one embodiment, the finger pressing state sample data includes a second sample data vector;

[0208] The second sample data vector is obtained by converting the echo intensity feature values ​​corresponding to multiple single-frame ultrasound echo sample signals, wherein the multiple single-frame ultrasound echo sample signals are single-frame ultrasound echo sample signals corresponding to multiple different time steps.

[0209] In one embodiment, the echo intensity characteristic value includes the average echo intensity of a single frame of the ultrasonic echo signal, or the echo intensity characteristic value includes the minimum echo intensity value in a single frame of the ultrasonic echo signal.

[0210] In one embodiment, the plurality of different time steps are consecutive time steps.

[0211] In one embodiment, the second sample data vector is a normalized sample data vector.

[0212] In this embodiment, for the time-series detection scheme, the neural network model required for the time-series detection scheme can be trained specifically. For more information, please refer to the relevant description of the foregoing embodiment. The difference is that the model input data at this time is training sample data.

[0213] It should be noted that, depending on the detection requirements, the classification decision layer algorithm in the neural network model can be set, such as binary, tri-class, or other multi-class classification, as well as the specific classification type for each category. When constructing sample data, corresponding classification labels are set according to the classification decision layer algorithm, thereby causing the trained neural network model to output the corresponding classification state.

[0214] In one embodiment, the finger pressing state output by the neural network model includes: a classified state or a classified state, wherein the classified state is any one of at least two categories of pressing states.

[0215] In one embodiment, the at least two categories of pressing states include a finger-touching state and a finger-non-touching state; or, the at least two categories of pressing states include a finger-touching state, a finger-non-touching state, a critical state from touch to non-touch, and a critical state from non-touch to touch.

[0216] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0217] The foregoing mainly describes the method provided in the embodiments of this application. The following describes the embodiments of related devices, media and equipment provided in the embodiments of this application.

[0218] In one embodiment, a finger pressing state detection device is provided, which corresponds one-to-one with the finger pressing state detection method described in the above embodiments. For example... Figure 13 As shown, the finger press state detection device includes a processing module 101 and an input module 102. Detailed descriptions of each functional module are as follows:

[0219] Processing module 101 is used to preprocess the ultrasonic echo signal to obtain model input data, wherein the ultrasonic echo signal is the ultrasonic echo signal corresponding to the ultrasonic array emitting ultrasonic signal to the pressing area;

[0220] The input module 102 is used to input the model input data into the trained neural network model so that the neural network model outputs the finger pressing state.

[0221] In conjunction with the previous embodiment, in one embodiment, the neural network model includes a neural network model built based on a recurrent neural network; or a neural network model built based on a convolutional neural network.

[0222] In one embodiment, in conjunction with the previous embodiment:

[0223] The neural network model includes a first classification decision layer and a multi-layer recurrent neural network, and the output of the multi-layer recurrent neural network is connected to the first classification decision layer.

[0224] or;

[0225] The neural network model includes a second classification decision layer and a single-layer recurrent neural network, the output of which is connected to the second classification decision layer.

[0226] In one embodiment, the processing module 101 is configured to:

[0227] The ultrasonic echo signal of a single frame is sampled at multiple points in the spatial domain to obtain the spatial sampling data corresponding to the ultrasonic echo signal of a single frame.

[0228] The spatial sampling data is converted into a first data vector;

[0229] Based on the first data vector, the model input data of the neural network model is obtained.

[0230] In one embodiment, the processing module 101 is configured to:

[0231] According to the preset row and column intervals, the ultrasonic echo signal of a single frame is spatially sampled to obtain the spatial sampling data corresponding to the ultrasonic echo signal of a single frame.

[0232] In one embodiment, the processing module 101 is configured to:

[0233] The first data vector is normalized, and the normalized first data vector is the model input data of the neural network model.

[0234] In one embodiment, the processing module 101 is configured to:

[0235] The echo intensity feature value of the single frame of the ultrasonic echo signal acquired at the current time step, and the echo intensity feature values ​​corresponding to other time steps, are converted into a second data vector, wherein the other time steps are the time steps before the current time step.

[0236] Based on the second data vector, the model input data of the neural network model is obtained.

[0237] In one embodiment, the echo intensity characteristic value includes the average echo intensity of a single frame of the ultrasonic echo signal, or the echo intensity characteristic value includes the minimum echo intensity value in a single frame of the ultrasonic echo signal.

[0238] In one embodiment, the other time steps are the N-1 consecutive time steps preceding the current time step N.

[0239] In one embodiment, the processing module 101 is configured to:

[0240] The second data vector is normalized, and the normalized second data vector is the model input data of the neural network model.

[0241] In one embodiment, the finger pressing state output by the neural network model includes: a classified state or a classified state, wherein the classified state is any one of at least two categories of pressing states.

[0242] In one embodiment: the at least two categories of pressing states include a finger touching state and a finger non-touching state; or; the at least two categories of pressing states include a finger touching state, a finger non-touching state, a critical state from touching to non-touching, and a critical state from non-touching to touching.

[0243] In one embodiment, the processing module 101 is further configured to:

[0244] When the output of the finger pressing state is the classification determination state, the classification determination state is taken as the final finger pressing state, and the output of the classification determination state is updated to the state memory sequence according to the time sequence.

[0245] When the output finger pressing state is the classification pending state, the last recorded pressing state in the state memory sequence is taken as the final finger pressing state, or the pressing state with the highest state probability in the state memory sequence is taken as the final output finger pressing state, and the state memory sequence is updated accordingly.

[0246] In summary, the embodiments of this application provide a finger pressing state detection device based on a neural network model, which brings the following technical effects: it eliminates the differences between different ultrasonic modules without the need to set a benchmark threshold for comparison; the same network model parameters have better generalization ability across different ultrasonic modules; and it can effectively eliminate differences under various scenarios (temperature, frequency, etc.) and under different finger states, covering a variety of special scenarios, such as high temperature, low temperature, and wet fingers, with stronger generalization ability and accuracy; more technical effects can be found in the corresponding descriptions of the foregoing method embodiments.

[0247] In one embodiment, a neural network model training device is provided, which corresponds one-to-one with the neural network model training method described in the above embodiments. For example... Figure 14 As shown, the neural network model training device includes an acquisition module 201 and a training module 202. Detailed descriptions of each functional module are as follows:

[0248] The acquisition module 201 is used to acquire training sample data, which includes finger pressing state sample data and its corresponding pressing state classification label. The finger pressing state sample data is constructed based on ultrasonic echo sample signals, which are ultrasonic reflection signals after ultrasonic wave sample signals are emitted into the pressing area.

[0249] The training module 202 is used to train the neural network model based on the training sample data until a neural network model that meets the preset model conditions is obtained. The trained neural network model is used to output the finger pressing state.

[0250] In conjunction with the previous embodiment, in one embodiment, the neural network model includes a neural network model built based on a recurrent neural network; or a neural network model built based on a convolutional neural network.

[0251] In conjunction with the previous embodiment, in one embodiment, the neural network model includes a first classification decision layer and a multi-layer recurrent neural network, wherein the output of the multi-layer recurrent neural network is connected to the first classification decision layer;

[0252] or;

[0253] The neural network model includes a second classification decision layer and a single-layer recurrent neural network, the output of which is connected to the second classification decision layer.

[0254] In one embodiment, the finger pressing state sample data includes a first sample data vector;

[0255] The first sample data vector is obtained by transforming the spatial domain sample data, which is obtained by performing multi-point spatial sampling based on a single frame of ultrasonic echo sample signal.

[0256] In one embodiment, the spatial sampling sample data is obtained by spatially sampling a single frame of the ultrasonic echo sample signal at a preset interval.

[0257] In one embodiment, the first sample data vector is a normalized sample data vector.

[0258] In one embodiment, the finger pressing state sample data includes a second sample data vector;

[0259] The second sample data vector is obtained by converting the echo intensity feature values ​​corresponding to multiple single-frame ultrasound echo sample signals, wherein the multiple single-frame ultrasound echo sample signals are single-frame ultrasound echo sample signals corresponding to multiple different time steps.

[0260] In one embodiment, the plurality of different time steps are consecutive time steps.

[0261] In one embodiment, the second sample data vector is a normalized sample data vector.

[0262] In one embodiment, the echo intensity characteristic value includes the average echo intensity of a single frame of the ultrasonic echo signal, or the echo intensity characteristic value includes the minimum echo intensity value in a single frame of the ultrasonic echo signal.

[0263] In one embodiment, the finger pressing state output by the neural network model includes: a classified state or a classified state, wherein the classified state is any one of at least two categories of pressing states.

[0264] In one embodiment, the at least two categories of pressing states include a finger-touching state and a finger-non-touching state; or, the at least two categories of pressing states include a finger-touching state, a finger-non-touching state, a critical state from touch to non-touch, and a critical state from non-touch to touch.

[0265] For specific limitations regarding the aforementioned device, please refer to the relevant limitations of the method above, which will not be repeated here. Each module in the aforementioned device can be implemented entirely or partially through software, hardware, or a combination thereof. Each module can be embedded in hardware within or independently of the processor in an electronic or computer device, or it can be stored in software within the memory of a computer device, so that the processor can call and execute the operations corresponding to each module.

[0266] In one embodiment, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of a neural network model training method as described in any of the preceding embodiments.

[0267] In one embodiment, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the finger press state detection method as described in any of the preceding embodiments.

[0268] Specifically, in one embodiment, such as Figure 1 As shown, an electronic device is provided, including an ultrasonic piezoelectric sensor, an analog-to-digital converter, a data processing center, and a central controller, wherein the ultrasonic piezoelectric sensor array, the analog-to-digital converter, and the data processing center are all connected to the central controller;

[0269] The ultrasonic piezoelectric sensor is used to respond to the central controller emitting ultrasonic signals to the pressing area and receiving ultrasonic echo signals.

[0270] The analog-to-digital converter is used to perform digital-to-analog conversion on the ultrasonic echo signal in response to the central controller;

[0271] The data processing center responds to the central controller by implementing the steps of the finger press state detection method as described above.

[0272] In one embodiment, a readable storage medium is provided, the readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the aforementioned neural network model training method or finger press state detection method.

[0273] In one embodiment, a computer program product is provided, including a computer program, characterized in that, when executed by a processor, the computer program implements the steps of the aforementioned neural network model training method or finger press state detection method.

[0274] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Furthermore, any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory.

[0275] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0276] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for detecting finger pressing status, characterized in that, The method includes: A single frame of ultrasonic echo signal is sampled at multiple points in the spatial domain to obtain spatial sampling data corresponding to the single frame of ultrasonic echo signal. The spatial sampling data is converted into a first data vector to obtain model input data based on the first data vector. The ultrasonic echo signal is the ultrasonic echo signal corresponding to the ultrasonic array emitting ultrasonic signals to the pressing area. The input data of the model is input into the trained neural network model so that the neural network model outputs the finger pressing state based on the intensity change characteristics of the ultrasonic echo signal in the spatial domain. The finger pressing state includes the finger touching state and the finger not touching state.

2. The finger pressing state detection method according to claim 1, characterized in that: The neural network model includes a first classification decision layer and a multi-layer recurrent neural network, and the output of the multi-layer recurrent neural network is connected to the first classification decision layer. or; The neural network model includes a second classification decision layer and a single-layer recurrent neural network, the output of which is connected to the second classification decision layer.

3. The finger pressing state detection method according to claim 1, characterized in that, The step of spatially sampling a single frame of the ultrasonic echo signal to obtain spatial sampling data corresponding to that single frame of the ultrasonic echo signal includes: According to the preset row and column intervals, the ultrasonic echo signal of a single frame is spatially sampled to obtain the spatial sampling data corresponding to the ultrasonic echo signal of a single frame.

4. The finger pressing state detection method according to any one of claims 1-3, characterized in that, The finger pressing state output by the neural network model includes: a classified state or a classified state, wherein the classified state is any pressing state among at least two categories.

5. The finger pressing state detection method according to claim 4, characterized in that: The at least two categories of pressing states include the finger touching state and the finger not touching state; or; The at least two categories of pressing states include the finger touching state, the finger not touching state, the critical state between touching and not touching, and the critical state between not touching and touching.

6. The finger pressing state detection method according to claim 4, characterized in that, After inputting the model input data into the trained neural network model so that the neural network model outputs the finger pressing state, the method further includes: When the output of the finger pressing state is the classification determination state, the classification determination state is taken as the final finger pressing state, and the output of the classification determination state is updated to the state memory sequence according to the time sequence. When the output finger pressing state is the classification pending state, the last recorded pressing state in the state memory sequence is taken as the final finger pressing state, or the pressing state with the highest state probability in the state memory sequence is taken as the final output finger pressing state, and the state memory sequence is updated accordingly.

7. A method for training a neural network model, characterized in that, The method includes: Training sample data is acquired, including finger pressing state sample data and its corresponding pressing state classification labels. The finger pressing state sample data is constructed based on ultrasonic echo sample signals, which are the ultrasonic echo signals corresponding to the ultrasonic array emitting ultrasonic sample signals to the pressing area. The finger pressing state sample data includes a first sample data vector, which is obtained by transforming spatial domain sampling sample data. The spatial domain sampling sample data is obtained by performing multi-point spatial sampling based on a single frame of ultrasonic echo sample signals. Based on the training sample data, the neural network model is trained until a neural network model that meets the preset model conditions is obtained. The trained neural network model is used to output the finger pressing state based on the intensity variation characteristics of the ultrasonic echo signal in the spatial domain. The finger pressing state includes the finger touching state and the finger not touching state.

8. The neural network model training method according to claim 7, characterized in that: The neural network model includes a first classification decision layer and a multi-layer recurrent neural network, and the output of the multi-layer recurrent neural network is connected to the first classification decision layer. or; The neural network model includes a second classification decision layer and a single-layer recurrent neural network, the output of which is connected to the second classification decision layer.

9. The neural network model training method according to claim 7, characterized in that, The spatial sampling data is obtained by spatially sampling a single frame of the ultrasonic echo sample signal at a preset interval.

10. The neural network model training method according to any one of claims 7-9, characterized in that, The finger pressing state output by the neural network model includes: a classified state or a classified state, wherein the classified state is any pressing state among at least two categories.

11. The neural network model training method according to claim 10, characterized in that: The at least two categories of pressing states include a finger-touching state and a finger-non-touching state; or; The at least two categories of pressing states include a finger-touching state, a finger-non-touching state, a critical state between touching and non-touching, and a critical state between non-touching and touching.

12. An electronic device, characterized in that, The electronic device includes an ultrasonic piezoelectric sensor, an analog-to-digital converter, a data processing center, and a central controller, wherein the ultrasonic piezoelectric sensor array, the analog-to-digital converter, and the data processing center are all connected to the central controller; The ultrasonic piezoelectric sensor is used to respond to the central controller emitting ultrasonic signals to the pressing area and receiving ultrasonic echo signals. The analog-to-digital converter is used to perform digital-to-analog conversion on the ultrasonic echo signal in response to the central controller; The data processing center responds to the central controller by implementing the steps of the finger pressing state detection method as described in any one of claims 1-6.

13. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the steps of the neural network model training method as claimed in any one of claims 7 to 11.

14. A readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 11.

Images