Finger pressing state detection method, finger pressing state training method, electronic equipment and medium

Abstract

The invention relates to the technical field of signal detection, and discloses a finger pressing state detection method, a training method, electronic equipment and a medium, which are used for improving the generalization ability of finger pressing state detection. The method comprises the steps that ultrasonic echo signals are preprocessed, model input data are obtained, and the ultrasonic echo signals are corresponding ultrasonic echo signals after an ultrasonic array transmits ultrasonic signals to a pressing area; and inputting the model input data into a trained neural network model to enable the neural network model to output a finger pressing state.

Description

Technical Field

[0001] The present application relates to the field of signal detection technology, and in particular to a finger pressing state detection, training method, electronic device and medium. Background Art

[0002] When using electronic devices such as mobile phones and computers with touch functions, users need to touch the screen or other touch positions with their fingers to perform operations or recognition, such as touch unlock, single click to confirm, double click to take a screenshot, etc. Generally, the electronic device captures the action first and then triggers the corresponding touch function.

[0003] The ultrasonic system can transmit ultrasonic waves, receive reflected sound waves, and obtain digital signals after a series of signal conversions. Since the propagation of sound waves is affected by acoustic impedance, the acoustic impedance of different reflection surfaces is different, and the ultrasonic echo intensity is also different. Therefore, there are differences in the ultrasonic echo intensity between the finger and the touch contact surface, and between the air and the touch contact surface. The above differences can be used to detect the user's finger movements and determine the finger pressing status.

[0004] In traditional solutions, while keeping the ultrasonic emission frequency, temperature and other conditions the same, the finger pressing state (lifted or pressed, etc.) can be detected by a set benchmark threshold. However, ultrasonic signals are affected by factors such as emission frequency, temperature, ultrasonic flight time and differences in ultrasonic modules. There are large differences in signal strength under different conditions. Therefore, judging the finger pressing state by threshold requires a large amount of data covering different conditions and setting corresponding benchmark thresholds. It is difficult to cover all different situations, resulting in poor generalization ability of traditional detection solutions. Summary of the invention

[0005] The present application provides a finger pressing state detection, training method, electronic device and medium to solve the technical problem of poor generalization ability in traditional finger pressing state detection solutions.

[0006] Based on the above technical problems, the following solutions are provided:

[0007] In a first aspect, a method for detecting a finger pressing state is provided, the method comprising:

[0008] Preprocessing the ultrasonic echo signal to obtain model input data, wherein the ultrasonic echo signal is an ultrasonic echo signal corresponding to the ultrasonic array transmitting an ultrasonic signal to the pressing area;

[0009] The model input data is input into the trained neural network model so that the neural network model outputs the finger pressing state.

[0010] In one implementation, the neural network model includes a neural network model constructed based on a recurrent neural network; or a neural network model constructed based on a convolutional neural network.

[0011] In one implementation:

[0012] The neural network model includes a first classification decision layer and a multi-layer recurrent neural network, and the output of the multi-layer recurrent neural network is connected to the first classification decision layer;

[0013] or;

[0014] The neural network model includes a second classification decision layer and a single-layer recurrent neural network, and the output of the single-layer recurrent neural network is connected to the second classification decision layer.

[0015] In one implementation, the preprocessing of the ultrasonic echo signal to obtain model input data includes:

[0016] Performing spatial multi-point sampling on the single-frame ultrasonic echo signal to obtain spatial sampling data corresponding to the single-frame ultrasonic echo signal;

[0017] Converting the spatial domain sampling data into a first data vector;

[0018] According to the first data vector, model input data of the neural network model is obtained.

[0019] In one implementation, performing spatial domain sampling on the single frame of the ultrasonic echo signal to obtain spatial domain sampling data corresponding to the single frame of the ultrasonic echo signal includes:

[0020] According to the preset row and column intervals, spatial domain sampling is performed on the single frame of the ultrasonic echo signal to obtain spatial domain sampling data corresponding to the single frame of the ultrasonic echo signal.

[0021] In one implementation, obtaining model input data of the neural network model according to the first data vector includes:

[0022] The first data vector is normalized, and the first data vector after normalization is model input data of the neural network model.

[0023] In one implementation, the preprocessing of the ultrasonic echo signal to obtain model input data includes:

[0024] Converting the echo intensity characteristic value of the single frame of the ultrasonic echo signal acquired at the current time step and the echo intensity characteristic values ​​corresponding to other time steps into a second data vector, wherein the other time steps are the time steps before the current time step;

[0025] According to the second data vector, model input data of the neural network model is obtained.

[0026] In one implementation, the other time steps are the N-1 consecutive time steps before the current time step N.

[0027] In one implementation, obtaining model input data of the neural network model according to the second data vector includes:

[0028] The second data vector is normalized, and the second data vector after normalization is the model input data of the neural network model.

[0029] In one implementation, the finger pressing state output by the neural network model includes: a classification determined state or a classification pending state, and the classification determined state is any one of at least two categories of pressing states.

[0030] In one implementation, the at least two categories of press states include a finger touch state and a finger non-touch state; or; the at least two categories of press states include a finger touch state, a finger non-touch state, a critical state from touch to non-touch, and a critical state from non-touch to touch.

[0031] In one implementation, after inputting the model input data into the trained neural network model so that the neural network model outputs the finger pressing state, the method further includes:

[0032] When the output finger pressing state is the classification determination state, the classification determination state is used as the final finger pressing state, and the output classification determination state is updated to the state memory sequence in a time sequence;

[0033] When the output finger pressing state is the classification pending state, the last pressing state recorded in the state memory sequence is taken as the final finger pressing state, or the pressing state with the highest state probability in the state memory sequence is taken as the final output finger pressing state, and it is updated accordingly to the state memory sequence.

[0034] In a second aspect, a neural network model training method is provided, the method comprising:

[0035] Acquire training sample data, wherein the training sample data includes finger pressing state sample data and its corresponding pressing state classification label, wherein the finger pressing state sample data is constructed based on an ultrasonic echo sample signal, and the ultrasonic echo sample signal is an ultrasonic reflection signal after an ultrasonic sample signal is emitted to a pressing area;

[0036] Based on the training sample data, the neural network model is trained until a neural network model that meets the preset model conditions is obtained;

[0037] The trained neural network model is used to output the finger pressing state.

[0038] In one implementation, the neural network model includes a neural network model constructed based on a recurrent neural network; or a neural network model constructed based on a convolutional neural network.

[0039] In one implementation:

[0040] The neural network model includes a first classification decision layer and a multi-layer recurrent neural network, and the output of the multi-layer recurrent neural network is connected to the first classification decision layer;

[0041] or;

[0042] The neural network model includes a second classification decision layer and a single-layer recurrent neural network, and the output of the single-layer recurrent neural network is connected to the second classification decision layer.

[0043] In one implementation, the finger pressing state sample data includes a first sample data vector;

[0044] The first sample data vector is obtained by transforming the spatial domain sampling sample data, and the spatial domain sampling sample data is obtained by performing spatial domain multi-point sampling based on a single frame of ultrasonic echo sample signal.

[0045] In one implementation, the spatial domain sampling sample data is obtained by performing spatial domain sampling on a single frame of the ultrasonic echo sample signal at a preset interval.

[0046] In one implementation, the first sample data vector is a normalized sample data vector.

[0047] In one implementation, the finger pressing state sample data includes a second sample data vector;

[0048] The second sample data vector is obtained by converting echo intensity feature values ​​corresponding to a plurality of single-frame ultrasonic echo sample signals, respectively, and the plurality of single-frame ultrasonic echo sample signals are single-frame ultrasonic echo sample signals corresponding to a plurality of different time steps.

[0049] In one implementation, the plurality of different time steps are continuous time steps.

[0050] In one implementation, the second sample data vector is a normalized sample data vector.

[0051] In one implementation, the finger pressing state output by the neural network model includes: a classification determined state or a classification pending state, and the classification determined state is any one of at least two categories of pressing states.

[0052] In one implementation, the at least two categories of press states include a finger touch state and a finger non-touch state; or; the at least two categories of press states include a finger touch state, a finger non-touch state, a critical state from touch to non-touch, and a critical state from non-touch to touch.

[0053] In a third aspect, a finger pressing state detection device is provided, the finger pressing state detection device comprising:

[0054] A processing module, used for preprocessing the ultrasonic echo signal to obtain model input data, wherein the ultrasonic echo signal is an ultrasonic echo signal corresponding to the ultrasonic array transmitting the ultrasonic signal to the pressing area;

[0055] An input module is used to input the model input data into the trained neural network model so that the neural network model outputs the finger pressing state.

[0056] In a fourth aspect, a neural network model training device is provided, the neural network model training device comprising:

[0057] An acquisition module, used to acquire training sample data, wherein the training sample data includes finger pressing state sample data and its corresponding pressing state classification label, wherein the finger pressing state sample data is constructed based on an ultrasonic echo sample signal, and the ultrasonic echo sample signal is an ultrasonic reflection signal after an ultrasonic sample signal is emitted to a pressing area;

[0058] The training module is used to train the neural network model based on the training sample data until a neural network model that meets the preset model conditions is obtained. The trained neural network model is used to output the finger pressing state.

[0059] In a fifth aspect, an electronic device, the finger pressing state detection device comprises an ultrasonic piezoelectric sensor, an analog-to-digital converter, a data processing center and a central controller, wherein the ultrasonic piezoelectric sensor array, the analog-to-digital converter and the data processing center are all connected to the central controller;

[0060] The ultrasonic piezoelectric sensor is used to respond to the central controller to transmit an ultrasonic signal to the pressing area and receive an ultrasonic echo signal;

[0061] The analog-to-digital converter is used to perform digital-to-analog conversion on the ultrasonic echo signal in response to the central controller;

[0062] The data processing center responds to the central controller to implement the steps of the finger pressing state detection method as described in any one of the above.

[0063] In a sixth aspect, an electronic device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the neural network model training method as described in any one of the foregoing items are implemented.

[0064] In a seventh aspect, a readable storage medium is provided, wherein the readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the methods described above are implemented.

[0065] In an eighth aspect, a computer program product is provided, comprising a computer program, which implements the steps of any of the aforementioned methods when executed by a processor.

[0066] In one of the above-mentioned schemes, the ultrasonic echo signal is preprocessed to obtain model input data, and the model input data is input into the trained neural network model so that the neural network model outputs the corresponding finger pressing state. The ultrasonic echo signal intensity characteristics corresponding to different finger pressing states are different. The preprocessed ultrasonic echo signal, that is, the model input data, characterizes the intensity characteristic change. The trained neural network model outputs the finger pressing state based on the model input data. There is no need to set a benchmark threshold comparison method, and the differences between different ultrasonic modules (for example, different electronic devices use different ultrasonic piezoelectric sensors). The same network model parameters have better generalization ability between different ultrasonic modules, and can effectively eliminate the differences in various scenarios (temperature, frequency, etc.) and different finger states. It can cover a variety of special scenarios, such as high temperature, low temperature, fingers wet with water, etc., and has stronger generalization ability and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0068] Figure 1 It is a system framework schematic diagram of a processing system in one embodiment of the present application;

[0069] Figure 2 It is a schematic diagram of the working principle of the signal transmission model and the signal receiving model of the ultrasound array in one embodiment of the present application;

[0070] Figure 3 It is a schematic diagram comparing the ultrasonic echo signal strength in a finger touching state and a finger non-touching state in an embodiment of the present application;

[0071] Figure 4 It is a flow chart of a method for detecting finger pressing state in one embodiment of the present application;

[0072] Figure 5 It is a schematic diagram of a model architecture of a neural network model based on a multi-layer recurrent neural network in one embodiment of the present application;

[0073] Figure 6 It is a schematic diagram of a model architecture of a neural network model based on a single-layer recurrent neural network in one embodiment of the present application;

[0074] Figure 7 yes Figure 5 or Figure 6 A schematic diagram of the relationship between neuron nodes after the hidden layer is expanded;

[0075] Figure 8 This is a schematic diagram of a data processing flow when a spatial multi-point detection scheme is adopted in a finger pressing state detection method of the present application;

[0076] Fig. 9 It is a schematic diagram of a data processing flow when a timing detection scheme is adopted in a finger pressing state detection method of the present application;

[0077] Fig.10 It is a comparison diagram of the falling edge of the down state and the rising edge of the up state in this application;

[0078] Fig.11 This is a schematic diagram of a process of using a state memory sequence to assist decision making in one embodiment of the present application;

[0079] Fig.12 This is another flow chart of using state memory sequence to assist decision making in one embodiment of the present application;

[0080] Fig.13 is a structural schematic diagram of a finger pressing state detection device in one embodiment of the present application;

[0081] Fig.14 It is a structural schematic diagram of a neural network model training device in one embodiment of the present application. DETAILED DESCRIPTION

[0082] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0083] It should be understood that the present application can be implemented in different forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, providing these embodiments will make the disclosure thorough and complete and fully convey the scope of the present application to those skilled in the art. In the accompanying drawings, the sizes and relative sizes of layers and regions may be exaggerated for clarity. The same reference numerals throughout represent the same elements.

[0084] It should be understood that when an element or layer is referred to as "on, adjacent to, connected to or coupled to other elements or layers, it may be directly on, adjacent to, connected to or coupled to other elements or layers, or there may be intervening elements or layers. On the contrary, when an element is referred to as "directly on, directly adjacent to, directly connected to or directly coupled to other elements or layers, there are no intervening elements or layers. It should be understood that although the terms first, second, third, etc. may be used to describe various elements, components, regions, layers and / or parts, these elements, components, regions, layers and / or parts should not be limited by these terms. These terms are only used to distinguish one element, component, region, layer or part from another element, component, region, layer or part. Therefore, without departing from the teachings of the present application, the first element, component, region, layer or part discussed below may be represented as a second element, component, region, layer or part.

[0085] The purpose of the terms used herein is only to describe specific embodiments and is not intended to be limiting of the present application. When used herein, the singular forms "one", "an" and "said / the" are also intended to include plural forms, unless the context clearly indicates otherwise. It should also be understood that the terms "consisting of" and / or "comprising", when used in this specification, determine the presence of the features, integers, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or groups. When used herein, the term "and / or" includes any and all combinations of the relevant listed items.

[0086] In order to thoroughly understand the present application, detailed structures and steps will be presented in the following description to illustrate the technical solution proposed by the present application. The preferred embodiments of the present application are described in detail below, but in addition to these detailed descriptions, the present application may also have other implementation methods.

[0087] The embodiment of the present application provides a finger pressing state detection scheme for detecting the finger pressing state, for example, including but not limited to the finger touch state and the finger non-touch state, etc. Based on the above detected finger pressing state, the scheme can be applied to fingerprint recognition scenarios or other applications including touch function scenarios. For example, when a user is using an electronic device such as a mobile phone or a computer, the user needs to touch the pressing area (such as the screen or other touchable area, such as a physical contact panel) with his finger to operate or identify. Exemplarily, the above touch functions include but are not limited to: touch to unlock, single-click to confirm, double-click to screenshot, etc., which are not specifically limited. In general, the electronic device needs to capture the finger pressing state before triggering the corresponding touch function. In addition, the above electronic devices include mobile phones, computers, tablets and other product devices with touch function requirements, which are not specifically limited.

[0088] To facilitate understanding of the embodiments of the present application, a processing system framework used in the embodiments of the present application is first introduced. Figure 1 As shown, the processing system framework provided by the present application includes a finger pressing state detection device and a touch function device, wherein Figure 1 In the example, the touch function device includes but is not limited to the fingerprint unlocking function device, and may also be other touch function systems, which are not limited in the embodiment of the present application. The following describes the processing system by taking fingerprint unlocking as an example.

[0089] As a framework example, the finger pressing state detection device includes an ultrasonic piezoelectric sensor, an analog-to-digital converter, a storage unit, a data processing center and a central controller, wherein the ultrasonic piezoelectric sensor, the analog-to-digital converter, the storage unit and the data processing center are respectively connected to the central controller. Exemplarily, the ultrasonic piezoelectric sensor includes structures such as an ultrasonic array and a coupling layer. The ultrasonic array can be a PVDF (polyvinylidene fluoride) array, which is not specifically limited. The analog-to-digital converter can be a high-precision analog-to-digital converter, which is not specifically limited. The central controller is used to control the ultrasonic piezoelectric sensor array to complete the excitation and recovery of the ultrasonic array. The analog-to-digital converter is used to complete the digital signal conversion of the simulated ultrasonic echo signal. The ultrasonic echo signal after analog-to-digital conversion is sent to the data processing center via the storage unit. The data processing center uses the finger pressing state detection method provided in the embodiment of the present application based on the ultrasonic echo signal to detect the finger pressing state.

[0090] The fingerprint unlocking function device includes a fingerprint processing algorithm module. After detecting the finger pressing state, the fingerprint processing algorithm module performs preliminary processing of the fingerprint data and further processes it through the fingerprint recognition algorithm to complete related fingerprint registration, fingerprint recognition unlocking and other functions.

[0091] It should be noted that Figure 1 This is only an illustration and does not limit the actual processing framework used.

[0092] Continue reading Figure 2 As shown, Figure 2 Taking the screen as the pressing area, the schematic diagram of the ultrasonic signal emission model and ultrasonic echo signal receiving model of the ultrasonic piezoelectric sensor is shown. The ultrasonic array of the ultrasonic piezoelectric sensor transmits ultrasonic signals to the screen at a set fixed frequency, penetrating the screen to reach the ridge and valley position of the finger epidermis. Since the screen fits the ridge position (fingerprint convexity), the finger valley position (fingerprint concave) does not contact the screen. The ultrasonic echo signal in the state of no finger coverage is used as the Base, and the ultrasonic echo signal in the state of finger coverage is used as the Touch. The difference between Touch and Base is the texture image of the ridge, that is, the fingerprint texture.

[0093] Due to the large difference in acoustic impedance between the human skin and the air, the reflectivity of the ultrasonic signal when it propagates to different contact surfaces is different. Therefore, there is a difference between the ultrasonic echo signal intensity in the finger touch state and the ultrasonic echo signal intensity in the empty state (there is no cover on the screen, that is, the finger is not touching the screen). This difference can be used to determine the finger pressing state. The reflectivity calculation method under touch and non-touch can be as follows:

[0094]

[0095] Among them, z air ,z finger ,z s They represent the impedance of air, human skin and screen respectively.

[0096] In general, the impedance of the screen is much greater than the impedance of the air. The impedance of the air is about 430Rayl. For example, the impedance of the screen currently used is about 13MRayl. When the finger is not touching, when the ultrasonic wave is transmitted from the screen to the screen contact surface of the air, the impedance of the air can be ignored, and the ultrasonic signal is almost completely reflected; while when the finger is touching, reflection and transmission will occur at the same time, part of the ultrasonic signal will pass through the finger epidermis, and part of the ultrasonic signal will be reflected. Therefore, when the finger is not touching, the ultrasonic echo signal intensity is greater, and when the finger is touching, the ultrasonic echo signal intensity shows a significant decrease. From the above analysis, it can be seen that under different finger pressing states, the ultrasonic echo signal intensity is in a changing state. According to the above ultrasonic echo signal intensity change characteristics, it can be used to judge the finger pressing state.

[0097] As introduced in the background, traditional solutions require a large amount of data covering different scene conditions and setting corresponding benchmark thresholds. It is difficult to cover all different situations. For example, the signal strength under the same conditions also varies greatly. When the finger is in different states, the ultrasonic echo signal changes vary even more. For example, when the finger is dry, the ultrasonic echo signal strength is lower, while when the finger is wet or wet, the ultrasonic echo signal strength is higher. In addition, the signal strength will also be different at different temperatures. Under high temperature conditions, the ultrasonic echo signal strength is higher, while under low temperature conditions, the ultrasonic echo signal strength is lower. This results in poor generalization ability of the solution and low accuracy.

[0098] Therefore, in the embodiments of the present application, based on the large differences in ultrasonic echo signal strength under different conditions, which leads to the problem of poor generalization of traditional solutions, a finger press state detection scheme based on a neural network model is proposed, which is described in detail through each embodiment below.

[0099] like Figure 4 As shown, Figure 1 Taking the finger pressing state detection system shown in the figure as an example, the embodiment of the present application provides a finger pressing state detection method based on a neural network model, comprising the following steps:

[0100] S10: Preprocessing the ultrasonic echo signal to obtain model input data, wherein the ultrasonic echo signal is an ultrasonic echo signal corresponding to the ultrasonic array emitting an ultrasonic signal to the pressing area.

[0101] The electronic device includes a finger pressing state detection device and a pressing area, and the pressing area can be a touch screen or other touch positions, such as a touch panel, etc., which is not specifically limited. The electronic device uses the ultrasonic array of the ultrasonic piezoelectric sensor to transmit an ultrasonic signal to the pressing area at a fixed frequency, and uses the ultrasonic array of the ultrasonic piezoelectric sensor to receive the reflected signal formed by the transmitted ultrasonic signal, that is, the ultrasonic echo signal, which is converted into a digital form by an analog-to-digital converter and reaches the data processing center through a storage unit. The data processing center preprocesses the ultrasonic echo signal in the storage unit to obtain model input data.

[0102] S20: Inputting the model input data into the trained neural network model so that the neural network model outputs the finger pressing status.

[0103] In the embodiment of the present application, a neural network model that outputs the finger pressing state based on the ultrasonic echo signal will be pre-trained. The neural network model is trained based on a large amount of training sample data. The neural network model includes but is not limited to a neural network model constructed based on a recurrent neural network (RNN) or a convolutional neural network (CNN). The network parameters of the neural network model are obtained by training the finger pressing data covering multiple scenes, multiple temperatures and other conditions constructed in the early stage, and the finger pressing data obtained in each scene has classification label information of the pressing. In the model training, the model training method can be set to perform training until a neural network model that meets the requirements is obtained, that is, a trained neural network model is obtained. For the content of model training, please refer to the description of the subsequent embodiments.

[0104] In this step, the data processing center pre-processes the ultrasonic echo signal, obtains the model input data, and then inputs the model input data into the trained neural network model so that the neural network model outputs the corresponding finger pressing state. Exemplarily, the trained neural network model can be built into an electronic device or deployed on a server side and used in the form of model call, which is not specifically limited.

[0105] It can be seen that in this embodiment, a finger pressing state detection method based on a neural network model is provided, by preprocessing the ultrasonic echo signal to obtain model input data, and the model input data is input into the trained neural network model so that the neural network model outputs the corresponding finger pressing state, and the ultrasonic echo signal intensity characteristics corresponding to different finger pressing states are different. The preprocessed ultrasonic echo signal, that is, the model input data, characterizes the intensity characteristic change, and the trained neural network model outputs the finger pressing state based on the model input data. There is no need to set a benchmark threshold comparison method, and the differences between different ultrasonic modules (for example, different electronic devices use different ultrasonic piezoelectric sensors) can be eliminated. The same network model parameters have better generalization ability between different ultrasonic modules, and can effectively eliminate the differences in various scenarios (temperature, frequency, etc.) and under different finger states, and can cover a variety of special scenarios, such as high temperature, low temperature, fingers wet with water, etc., with stronger generalization ability and accuracy.

[0106] It should be noted that in the embodiments of the present application, the neural network model can be constructed based on a recurrent neural network RNN ​​or a convolutional neural network CNN, without specific limitation. Taking the construction of the neural network model based on a recurrent neural network as an example, in one embodiment, the neural network model includes a first classification decision layer and a multi-layer recurrent neural network, and the output of the multi-layer recurrent neural network is connected to the corresponding first classification decision layer; or, the neural network model includes a second classification decision layer and a single-layer recurrent neural network, and the output of the single-layer recurrent neural network is connected to the second classification decision layer. In other words, the recurrent neural network used can be a single-layer recurrent neural network or a multi-layer recurrent neural network. It should be understood that the first classification decision layer and the second classification decision layer are only expressions used to distinguish between single-layer and multi-layer networks, and both can be the same classification decision layer.

[0107] The single-layer recurrent neural network or multi-layer recurrent neural network used can be determined according to actual needs and is not specifically limited. Since the finger pressing state detection is a pressing state detection process over a long period of time, and in the recurrent neural network, each current state is related to the previous state or the next state, therefore, the embodiment of the present application uses a neural network model based on a recurrent neural network, through the association of the previous or previous state, it will be helpful to more accurately detect the finger pressing state.

[0108] To facilitate understanding of the subsequent detection scheme, the neural network model constructed based on the recurrent neural network provided in the embodiment of the present application is first introduced below.

[0109] like Figure 5 As shown, as an example, the neural network model includes a first classification decision layer and a multi-layer recurrent neural network, the output of the multi-layer recurrent neural network is connected to the first classification decision layer, because the multi-layer recurrent neural network is related to the time series state, it is unfolded according to the time series state, then the multi-layer recurrent neural network includes hidden layer networks corresponding to different time series, and the hidden layer networks respectively represent the network architectures corresponding to different time series. It can be seen that the hidden layer network includes n hidden layers, n≥2, and the n hidden layers in the hidden layer network are connected in sequence, and the output of the hidden layer of the hidden layer network of the previous time series is transmitted to the same hidden layer of the next time series. The output of the last hidden layer of the hidden layer network, that is, the output of the multi-layer recurrent neural network, is connected to the first classification decision layer. Figure 5 The classification decision layer in is the first classification decision layer.

[0110] For more details, please refer to Figure 5 As shown, in this example, according to the time series expansion, x t0 ,x t1 ,...,x tnThey represent the input data of the hidden layer network corresponding to different time series t0-tn in the multi-layer recurrent neural network, y t0 ,y t1 ,...,y tn Represents the finger pressure state output by the corresponding time series classification decision layer. The hidden layer network includes hidden layer 1, hidden layer 2, ..., hidden layer n. The output of hidden layer n is connected to the input of the classification decision layer. The value of each hidden layer of the hidden layer network of the previous time series is transmitted to the hidden layer of the same layer in the next time series. For example, the output of hidden layer 1 at time series t0 is transmitted to the hidden layer 1 corresponding to time series t1, and the output of hidden layer 1 corresponding to time series t1 is transmitted to the hidden layer 1 corresponding to time series t2. The content transmitted includes the state vector of the hidden layer output. For example, hidden layer 1 of time series t0 transmits state vector 1 to hidden layer 1 of time series t1, and the output of the next hidden layer will be transmitted to the previous hidden layer. The relationship between hidden layers in the same layer or other layers is similar, as shown in the following example. Figure 5 It is also worth noting that there are weight matrix parameters in the multi-layer recurrent neural network, which will be introduced and explained later when the single-layer recurrent neural network is introduced. The specific number of hidden layers in the multi-layer recurrent neural network can be adjusted, and the model input data size can also be adjusted, which is not limited in the specific embodiments of this application.

[0111] like Figure 6 As shown, as an example, the neural network model includes a second classification decision layer and a single-layer recurrent neural network, the output of which is connected to the second classification decision layer. Since the single-layer recurrent neural network is related to the time series state, it is unfolded according to the time series state. Then the single-layer recurrent neural network includes hidden layer networks corresponding to different time series, and the hidden layer networks represent the network architectures corresponding to different time series. The hidden layer network of the single-layer recurrent neural network only includes one hidden layer, and the output of the hidden layer of the previous time series is transmitted to the hidden layer of the next time series, and the output of the hidden layer is connected to the second classification decision layer. Unlike a multi-layer recurrent neural network, a single-layer recurrent neural network only includes a single hidden layer.

[0112] Specifically, if Figure 6 As shown, in Figure 6 In the example, three time series are used as examples to explain. t-1 ,x t1 ,x t+1 Respectively represent the input data of the hidden layer network corresponding to the three time series, y t-1 ,y t1 ,,y t+1They represent the finger pressing states output by the classification decision layer of the corresponding time series. The hidden layer network includes one hidden layer. The output of the hidden layer 1 corresponding to the time series t-1 is transmitted to the hidden layer at the time series t1, and the output of the hidden layer corresponding to the time series t1 is transmitted to the hidden layer corresponding to the time series t+1.

[0113] In the above single-layer or multi-layer recurrent neural network, the weight matrices U, W, and V are the linear relationship parameters in the recurrent neural network. Figure 6 Taking the single-layer recurrent neural network shown in the figure as an example, the working principle of the neural network model based on the single-layer recurrent neural network is further explained. The network framework of the neural network model based on the single-layer recurrent neural network has the following expression:

[0114] y t =φ(O t )

[0115] O t =g(V·S t )

[0116] S t =f(U·x t +W·S t-1 )

[0117] Wherein, f(*), g(*) represent the activation functions of the corresponding layers, φ(*) represents the classification decision algorithm used in the classification decision layer, and the classification decision algorithm can be a binary classification or multi-classification algorithm, which is not specifically limited, S t represents the value of the hidden layer corresponding to the time series t, that is, the state vector mentioned above, S t-1 represents the value of the hidden layer corresponding to time series t-1, S t+1 represents the value of the hidden layer corresponding to time series t+1, O t represents the output of the hidden layer corresponding to the time series t, O t-1 represents the output of the hidden layer corresponding to time series t-1, O t+1 Represents the output of the classification decision layer corresponding to time series t+1, y t Represents the output of the classification decision layer corresponding to the time series t, y t-1 Represents the output of the classification decision layer corresponding to time series t-1, y t+1 represents the output of the classification decision layer corresponding to time series t+1, the weight matrix U represents the weight matrix from the model input to the hidden layer, the weight matrix V represents the weight matrix from the hidden layer to the classification decision layer, and the weight matrix W represents the weight matrix from the hidden layer of the previous time series to the corresponding hidden layer of the next time series. Therefore, the input vector After the output and decision of the classification decision layer, the finger pressing state of the time sequence is finally output. It can be seen that the output state of each time sequence is not only affected by the current input, but also related to the information of the previous time sequence.

[0118] Furthermore, in one embodiment, in the above-mentioned single-layer or multi-layer recurrent neural network, the number of neuron nodes contained in the hidden layer can also be adjusted, and the training can be performed according to the set parameters, which is not limited in the embodiment of the present application. As an example, taking a single-layer recurrent neural network as an example, the network with nodes can be expanded as follows: Figure 7 As shown. t represents the input vector of the current time series t, h 11 ,h 21 ,...h m1 represents the value of the hidden layer neuron node at time series t, h 12 ,h 21 ,...h m2 Represents the value of the neuron node of the hidden layer node at the previous time sequence t-1, f1, f2, ...h c Represents the activation function of the hidden layer of the current time series t, y t Represents the finger pressing state output by the classification decision layer at time t.

[0119] It should be noted that the above embodiment is introduced by taking the neural network model as an example of a neural network model constructed based on a recurrent neural network. Based on this, on the basis of the neural network model constructed based on a recurrent neural network, the embodiment of the present application also provides at least two detection schemes. The two detection schemes differ in the processing of model input data, and each has its corresponding advantages, which are described in detail below.

[0120] The first solution is the airspace multi-point detection solution

[0121] like Figure 8 As shown, in one embodiment, based on the neural network model constructed based on the above recurrent neural network, in step S10, the ultrasonic echo signal is preprocessed to obtain model input data, including:

[0122] S101A: performing spatial multi-point sampling on the single-frame ultrasonic echo signal to obtain spatial sampling data corresponding to the single-frame ultrasonic echo signal;

[0123] S102A: Convert the spatial domain sampling data into a first data vector;

[0124] S103A: Obtain model input data of the neural network model according to the first data vector.

[0125] This embodiment is one of the embodiments of the above-mentioned spatial multi-point detection solution. After obtaining the single-frame ultrasonic echo signal, the data processing center performs spatial multi-point sampling on the single-frame ultrasonic echo signal, thereby obtaining the spatial sampling data corresponding to the single-frame ultrasonic echo signal.

[0126] like Figure 2 As shown, Figure 2 Taking the screen as the pressing area, the schematic diagram of the ultrasonic signal emission model and ultrasonic echo signal receiving model of the ultrasonic piezoelectric sensor is shown. The ultrasonic array of the ultrasonic piezoelectric sensor transmits ultrasonic signals to the screen at a set fixed frequency, penetrating the screen to reach the ridge and valley position of the finger epidermis. Since the screen fits the ridge position (fingerprint convexity), the finger valley position (fingerprint concave) does not contact the screen. The ultrasonic echo signal in the state of no finger coverage is used as the Base, and the ultrasonic echo signal in the state of finger coverage is used as the Touch. The difference between Touch and Base is the texture image of the ridge, that is, the fingerprint texture.

[0127] In general, the impedance of the screen is much greater than the impedance of the air, which is about 430Rayl. For example, assuming that the impedance of the current screen is about 13MRayl, when the finger is not touching the screen, the impedance of the air can be ignored when the ultrasonic wave is transmitted from the screen to the screen contact surface in the air, and the ultrasonic signal is almost completely reflected; while when the finger is touching the screen, reflection and transmission will occur at the same time, part of the ultrasonic signal will pass through the finger epidermis, and part of the ultrasonic signal will be reflected. Therefore, when the finger is not touching the screen, the ultrasonic echo signal strength is greater, while when the finger is touching the screen, the ultrasonic echo signal strength decreases significantly. Figure 3 As shown, assuming that the acquisition frequency is 50 Hz, the finger cover-lift-cover-lift action is repeated for 20 seconds to collect 1000 frames of data, and the following is obtained: Figure 3 The relationship between the ultrasonic echo signal and the finger pressing state is shown in FIG. 1 , where up indicates the finger is not touching the state, down indicates the finger is touching the state, the signal value is the average of the full-screen intensity, and the horizontal axis indicates the frame number. Figure 3 It can also be seen that when a finger touches, the ultrasonic echo signal intensity value corresponding to the ridge position is low, which is different from the ultrasonic echo signal intensity value at the valley position. Therefore, based on the above characteristics, the spatial multi-point sampling method can be used to extract the ultrasonic echo intensity change characteristics.

[0128] For example, assuming that the spatial domain size of a single-frame ultrasonic echo signal after analog-to-digital conversion is m*n, each point in the spatial domain of the single-frame ultrasonic echo signal represents the ultrasonic echo signal intensity corresponding to the point, the spatial domain multi-point detection scheme is to perform spatial multi-point sampling on the single-frame ultrasonic echo signal, and based on the spatial domain sampling data, a first data vector matching the input size of the recurrent neural network is obtained, and then the model input data of the neural network model is obtained according to the first data vector. As the network input, the model input data is input into the recurrent neural network in the neural network model. After the weight calculation of the recurrent neural network, the output of the recurrent neural network is given to the classification decision layer in the neural network model, so that the classification decision layer outputs the current finger pressure state.

[0129] It can be seen that in this embodiment, a finger pressing state detection method based on a neural network model is provided. In combination with the neural network model constructed based on the recurrent neural network, a single frame of ultrasonic echo signal is subjected to spatial multi-point sampling to form a network input of the recurrent neural network to identify the finger pressing state. In this embodiment, in addition to the non-sampling reference threshold comparison method to improve the applicability and generalization, the use of multi-point spatial sampling can also reduce the size of the model input data and improve processing efficiency.

[0130] In combination with the above embodiments, in one embodiment, in step S101A, that is, performing spatial domain sampling on the collected single frame of ultrasonic echo signal to obtain spatial domain sampling data corresponding to the single frame of ultrasonic echo signal, includes: performing spatial domain sampling on the single frame of ultrasonic echo signal according to preset row and column intervals to obtain spatial domain sampling data corresponding to the single frame of ultrasonic echo signal.

[0131] For example, assuming that the spatial domain size of a single frame of ultrasonic echo signal data is m*n, the Base and Touch frames of size m*n are spatially sampled with a row interval of r1 and a column interval of c1 to obtain an m1*n1 matrix. Finally, the sampling matrix can form a matrix vector of 1*(m1*n1), that is, the first data vector, as the input of the aforementioned recurrent neural network, that is, the input of the neural network model. It should be noted that the above-mentioned row interval of r1 and column interval of c1 can be set according to needs and are not specifically limited.

[0132] In this embodiment, when performing multi-point spatial sampling on the ultrasonic echo signal, sampling can be performed using preset row and column intervals. In addition to reducing the amount of data and improving generalization capabilities, the interval sampling method can effectively reduce the loss of change characteristics of the ultrasonic echo signal to ensure the accuracy of subsequent detection.

[0133] Of course, in other embodiments, the specific sampling interval of the preset row and column intervals can be determined according to actual conditions, and the embodiments of the present application do not limit this. In addition, a spatial multi-point sampling method with non-row and column intervals can also be adopted, and the embodiments of the present application do not limit this.

[0134] In one embodiment, in step S103A, that is, obtaining the model input data of the neural network model based on the first data vector, includes: normalizing the first data vector, and the first data vector after normalization is the model input data of the neural network model.

[0135] In this embodiment, normalizing the first data vector obtained by transformation refers to normalizing all vector elements in the first data vector so that the element values ​​in the first data vector are within a uniform range. Therefore, the first data vector after normalization is the model input data of the neural network model, which can effectively reduce the amount of calculation of the subsequent neural network model, thereby improving the overall finger pressing state detection efficiency. It should be noted that in other embodiments, the transformed first data vector can also be used as model input data, without specific limitation.

[0136] The above-mentioned embodiment describes the spatial multi-point detection solution provided by the embodiment of the present application. The following describes the second solution provided by the embodiment of the application.

[0137] The second solution is the timing detection solution

[0138] like Fig. 9 As shown, in one embodiment, based on the neural network model constructed based on the above recurrent neural network, in step S10, the ultrasonic echo signal is preprocessed to obtain model input data, including:

[0139] S101B: converting the echo intensity characteristic value of the single frame of the ultrasonic echo signal acquired at the current time step and the echo intensity characteristic values ​​corresponding to other time steps into a second data vector, wherein the other time steps are the time steps before the current time step;

[0140] S102B: Obtain model input data of the neural network model according to the second data vector.

[0141] This embodiment is one of the embodiments of the above-mentioned timing detection scheme. After obtaining the ultrasonic echo signals of each time step collected, the data processing center will calculate the ultrasonic echo intensity characteristic value of the single frame ultrasonic echo signal of each time step, which corresponds to the intensity value corresponding to each time step. Taking the data time step sequence N as an example, assuming that the current time step is T N , the current time step TN The corresponding echo intensity characteristic value is Then this embodiment will set the current time step T N The echo intensity characteristic value of the single frame of the ultrasonic echo signal collected, as well as the echo intensity characteristic values ​​corresponding to other time steps, are converted into a second data vector matching the input size of the recurrent neural network The other time steps are the current time step T N The model input data of the neural network model is obtained according to the second data vector at the previous time step.

[0142] like Fig.10 As shown in the figure, it is a schematic diagram of the comparison of the ultrasonic echo intensity changes in the up state and the down state. The horizontal axis represents time and the vertical axis represents the ultrasonic echo intensity. Fig.10 It can be seen that in continuous time, there is an obvious falling edge in the state change from up to down (such as Fig.10 The state change from down to up has a clear rising edge (such as Fig.10 Therefore, the current finger pressing state can be further determined through the continuous timing changes of different frames.

[0143] It can be seen that in this embodiment, a finger pressing state detection method based on a neural network model is provided. In combination with the neural network model constructed based on the recurrent neural network, the echo intensity characteristic value of the single frame of the ultrasonic echo signal collected at the current time step and the echo intensity characteristic values ​​corresponding to other time steps are converted into a network input of the recurrent neural network to identify the finger pressing state. In this embodiment, in addition to the non-sampling reference threshold comparison method to improve the applicability and generalization, due to the changes in the time domain of the finger in the up-down action, the spatial domain sequence of multiple time steps is used as the model input, so that the finger pressing state can be detected by the spatial domain state change in time, and continuous actions can be captured. It can be applied to more scenarios, such as completely covering the changes in ultrasonic echo intensity in high temperature or low temperature scenarios, so as to achieve more accurate or diverse finger pressing state detection.

[0144] In combination with the above embodiment, in one embodiment, the other time steps are the N-1 consecutive time steps before the current time step N. That is to say, in this embodiment, the echo intensity characteristic value of the single frame of the ultrasonic echo signal collected at the current time step N and the echo intensity characteristic values ​​corresponding to the N-1 consecutive time steps before the current time step N are converted into a second data vector, that is:

[0145]

[0146] for example, represents the second data vector, They respectively represent the echo intensity characteristic values ​​corresponding to the current time step N and the N-1 consecutive time steps before the current time step N.

[0147] In this embodiment, the finger pressing state can be detected by the spatial state change in the time continuous step, and more accurate continuous action can be captured, thereby achieving further accurate or diverse finger pressing state detection. Of course, in other embodiments, other time steps can also be the echo intensity characteristic values ​​corresponding to the previous continuous time steps, which is not limited in the specific embodiments of the present application.

[0148] In one embodiment, in step S102B, that is, obtaining the model input data of the neural network model based on the second data vector, includes: normalizing the second data vector, and the second data vector after normalization is the model input data of the neural network model.

[0149] In this embodiment, normalizing the transformed second data vector refers to normalizing all vector elements in the second data vector so that the element values ​​in the second data vector are within a uniform range. Therefore, the normalized second data vector is the model input data of the neural network model, which can effectively reduce the amount of calculation of the subsequent neural network model, thereby improving the overall finger pressing state detection efficiency. It should be noted that in other embodiments, the transformed second data vector can also be used as the model input data, without specific limitation.

[0150] In one embodiment, the echo intensity characteristic value includes the echo intensity mean value of the ultrasonic echo signal of a single frame, or the echo intensity characteristic value includes the minimum echo intensity value of the ultrasonic echo signal of a single frame, and the specific details are not limited. Among them, the echo intensity mean value can more comprehensively take the single frame signal characteristics; while the minimum echo intensity value method simplifies the calculation.

[0151] Illustratively, in the above-mentioned embodiment, the above-mentioned echo intensity mean may be an arithmetic mean or other forms of mean, which is not limited in the specific embodiment of the present application and may be set according to requirements to ensure the flexibility of the solution.

[0152] It should be understood that in the above detection scheme embodiment, the neural network model based on the recurrent neural network is used as an example for explanation. In other embodiments, when other neural network models are used, such as a convolutional neural network, the same detection scheme can also be used to process the model input data, without specific limitation.

[0153] It should be noted that, according to needs, the classification decision layer in the above neural network model can adopt a two-classification algorithm or a three-classification algorithm or other multi-classification situations, without specific limitation.

[0154] In one embodiment, the finger pressing state output by the neural network model includes: a classification determined state or a classification pending state, wherein the classification determined state is any pressing state of at least two categories of pressing states, the classification determined state refers to a state in which the pressing state is already clear and can be used for subsequent touch function applications, such as a finger touch state, a finger non-touch state or other pressing states, etc.; and the classification pending state refers to a pending state in which the pressing state still needs further confirmation.

[0155] In this embodiment, the classification decision layer in the above neural network model can adopt a binary classification algorithm or a ternary classification or other multi-classification situations to improve the adaptability and application scenarios of the solution and increase the application value.

[0156] In one embodiment, the at least two categories of press states include a finger touch state and a finger non-touch state; or; the at least two categories of press states include a finger touch state, a finger non-touch state, a critical state from touch to non-touch, and a critical state from non-touch to touch.

[0157] That is, in one case, the finger pressing state output by the neural network model is the finger touch state, the finger non-touch state or the pending state; in another case, the finger pressing state output by the neural network model is the finger touch state, the finger non-touch state, the critical state from touch to non-touch, the critical state from non-touch to touch, or the pending state.

[0158] Of course, in other embodiments, the finger pressing state output by the neural network model can also have only a classification-determined state. For example, when using binary classification, in one case, the finger pressing state output by the neural network model is a finger touch state or a finger non-touch state; in another case, the finger pressing state output by the neural network model is a finger touch state, a finger non-touch state, a critical state from touch to non-touch, or a critical state from non-touch to touch.

[0159] In combination with the above embodiments, in one embodiment, when the finger pressing state output by the neural network model includes a classification determined state or a classification pending state, after step S20, that is, after inputting the model input data into the trained neural network model so that the neural network model outputs the finger pressing state, the method further includes:

[0160] S101: when the output finger pressing state is the classification determination state, the classification determination state is used as the final finger pressing state, and the output classification determination state is updated to the state memory sequence in a time sequence;

[0161] S102: When the output finger pressing state is the classification pending state, the pressing state recorded last in the state memory sequence is taken as the final finger pressing state, and updated to the state memory sequence in a time sequence.

[0162] For example, assuming that the classification determination state includes the first state and the second state, this embodiment assists the decision of the finger pressing state by setting a continuously updated state memory sequence. The pending state means that the current neural network model cannot output whether it is the first state or the second state, but is in a situation where the final finger pressing state needs to be further determined. In this case:

[0163] When the output finger pressing state is the first type of state, the first type of state is used as the final finger pressing state, and the output first type of state is updated to the state memory sequence in a time sequence;

[0164] When the output finger pressing state is the second type of state, the second type of state is used as the final finger pressing state, and the output second type of state is updated to the state memory sequence in a time sequence;

[0165] When the output finger pressing state is the classification pending state, the last recorded pressing state is taken as the final finger pressing state from the latest state memory sequence and updated to the state memory sequence in a time sequence.

[0166] The first state and the second state are different finger pressing states, and the specific states are not limited. For example, in one embodiment, the first state is a finger touching state, and the second state is a finger non-touching state.

[0167] Taking the first type of state as the finger touch state and the second type of state as the finger non-touch state as an example, it should be understood that the finger may be in the up or down state for a long time. In this case, the data, especially in the timing detection scheme, the data form of the time step sequence may not have a rising edge or a falling edge. In view of the above situation, a state memory sequence is introduced, and a state memory sequence MemSeq of a fixed length is set. The classification decision layer includes multiple classification states, including 0-up, 1-down, and 2-maintain; that is, the classification decision layer outputs 0 to represent the finger non-touch state, output 1 to represent the finger touch state, and output 2 to represent the classification pending state. The output of the neural network model is combined with the state memory sequence MemSeq to comprehensively determine the final finger pressing state.

[0168] Specifically, see Fig.11 As shown in Figure 2, when the finger pressing state output by the neural network model is the finger touching state, that is, the O output by the classification decision layer t =1, the final finger pressing state y t =1, and the finger touch state y is output according to the timing t =1 corresponds to updating to the state memory sequence MemSeq; when the finger pressing state output by the neural network model is the finger non-touch state, that is, O output by the classification decision layer t =0, the final finger pressing state y t =0, and output the finger non-touch state y according to the timing t = 0 corresponds to updating to the state memory sequence MemSeq; when the finger pressing state output by the neural network model is the pending state, that is, O output by the classification decision layer t = 2, then from the latest state memory sequence MemSeq, obtain the finger pressing state whose last recorded state is the final output, that is, y t =MemSeq[-1], and the state is updated to the state memory sequence MemSeq. In general, taking the above three classifications as an example, that is:

[0169] When the classification decision layer outputs an up or down state, the final output finger pressing state is the corresponding classification output state (up or down state), and the state memory sequence MemSeq is updated accordingly;

[0170] When the classification decision layer outputs the maintain state, the previous state recorded in the state memory sequence MemSeq is used to determine whether the final output finger pressing state is up or down, and the state memory sequence MemSeq is updated accordingly.

[0171] In this embodiment, the state memory sequence MemSeq is used to assist decision making, which can improve the richness of the state and make the detection scheme more accurate. It can also provide a corresponding processing mechanism when the neural network model cannot accurately output the up or down state, thereby ensuring the feasibility of the scheme and having a relatively high accuracy even when facing different signal sampling frequencies.

[0172] In combination with the above embodiments, in one embodiment, when the finger pressing state output by the neural network model includes a classification determined state or a classification pending state, after step S20, that is, after inputting the model input data into the trained neural network model so that the neural network model outputs the finger pressing state, the method further includes:

[0173] S201: when the output finger pressing state is the classification determination state, the classification determination state is used as the final finger pressing state, and the output classification determination state is updated to the state memory sequence in a time sequence;

[0174] S202: When the output finger pressing state is the classification pending state, the pressing state with the highest state probability is obtained from the latest state memory sequence as the final output finger pressing state, and updated to the state memory sequence accordingly.

[0175] For example, when three classifications are adopted, the finger pressing state of the neural network model includes a first-class state, a second-class state or a classification pending state, wherein the pending state means that the current neural network model cannot output whether it is a first-class state or a second-class state, but is in a situation where the final finger pressing state needs to be further determined. In this case:

[0176] When the output finger pressing state is the first type of state, the first type of state is used as the final finger pressing state, and the output first type of state is updated to the state memory sequence in a time sequence;

[0177] When the output finger pressing state is the second type of state, the second type of state is used as the final finger pressing state, and the output second type of state is updated to the state memory sequence in a time sequence;

[0178] When the output finger pressing state is the classification pending state, the pressing state with the highest state probability is obtained from the latest state memory sequence as the final finger pressing state, and the state memory sequence is updated accordingly.

[0179] The first state and the second state are different finger pressing states, and the specific states are not limited. For example, in one embodiment, the first state is a finger touching state, and the second state is a finger non-touching state.

[0180] In this embodiment, a fixed-length state memory sequence MemSeq is also set, and the classification decision layer includes multiple classification states, including 0-up, 1-down, and 2-maintain. The final finger pressing state is comprehensively determined by combining the output of the neural network model with the state memory sequence MemSeq record. The difference from the above embodiment is that when the classification decision layer outputs maintain, the highest probability of the state recorded in the state memory sequence MemSeq is combined to determine the final output finger pressing state.

[0181] Taking the above three-three classification as an example, please refer to Fig.12 As shown, when the finger pressing state output by the neural network model is the finger touching state, that is, the O output by the classification decision layer t =1, then the final finger pressing state y t =1, and output the finger touch state y according to the timing t =1 corresponds to updating to the state memory sequence MemSeq; when the finger pressing state output by the neural network model is the finger non-touch state, that is, O output by the classification decision layer t =0, the final finger pressing state y t =0, and output the finger non-touch state y according to the timing t = 0 corresponds to updating to the state memory sequence MemSeq; when the finger pressing state output by the neural network model is the classification pending state, that is, O output by the classification decision layer t =2, then from the state memory sequence MemSeq, the pressing state with the highest state probability is taken as the final finger pressing state. Specifically, as an example, first determine whether the number of times 0 appears in the state memory sequence MemSeq is greater than the number of times 1 appears. If so, then y t =0; if not, then y t =1, and update the last state to the state memory sequence MemSeq. In general, that is:

[0182] When the classification decision layer outputs an up or down state, the final finger pressing state is the corresponding classification output (up or down state), and the state memory sequence MemSeq is updated accordingly;

[0183] When the output of the classification decision layer is the maintain state, the state with the highest probability of 0 or 1 in the state memory sequence MemSeq is taken as the final finger press state, and the state memory sequence MemSeq is updated accordingly.

[0184] In this embodiment, there is another solution that uses the state memory sequence MemSeq to assist in decision-making, which can improve the richness of the state and make the detection solution more accurate. It can also provide a corresponding processing mechanism when the neural network model cannot accurately output the up or down state, thereby ensuring feasibility and being able to have a relatively high accuracy rate even when facing different signal sampling frequencies. In addition, due to the use of previous state probability processing, the fault tolerance rate of the solution in practical applications can also be made higher.

[0185] It should be understood that in the above embodiment, the length of the state memory sequence MemSeq is adjustable, and is not specifically limited. For example, it can be an infinite length sequence, with a length of 1 to infinity; or it can be a fixed sequence, which is set according to the requirements. Moreover, for other multi-classification situations, it is also applicable to use the above state memory sequence to assist decision-making. The above three classifications are only examples and do not limit the embodiments of this application.

[0186] In one embodiment, the method can further determine the user's finger movements through the state recorded in the state memory sequence MemSeq. For example, the state memory sequence MemSeq of [1,1,1,1,1,1] can be regarded as a long press of the finger; the state memory sequence MemSeq of [0,0,1,1,0,0] can be regarded as a single click; the state memory sequence MemSeq of [1,1,0,0,1,1] can be regarded as a double click, thereby enabling more finger movement detection.

[0187] In this embodiment, by further utilizing the state memory sequence MemSeq, finger movements can be quickly detected based on previous finger pressing state detection results.

[0188] In one embodiment, when implementing the above embodiment, the data collection frequency can be adjusted according to demand, not only 50Hz, but also other collection frequencies. The higher the data collection frequency, the more intensity gradient points in the up-down process, which is conducive to improving detection accuracy and efficiency.

[0189] It should be noted that the above embodiment takes the final output result of up or donw as an example. In other embodiments, other final classification states of the output may also be included. Depending on the needs, the neural network model can be set for targeted training, and there is no specific limitation.

[0190] In summary, the embodiments of the present application provide a finger pressure state detection method based on a neural network model, and the technical effects brought about include: the differences between different ultrasonic modules can be eliminated without setting a benchmark threshold comparison method, and the same network model parameters have better generalization capabilities between different ultrasonic modules. Moreover, the differences in various scenarios (temperature, frequency, etc.) and under different finger states can be effectively eliminated, and a variety of special scenarios can be covered, such as high temperature, low temperature, fingers wet with water, etc., with stronger generalization ability and accuracy.

[0191] The above embodiment describes a finger pressure state detection method based on a neural network model provided in an embodiment of the present application. When implementing the above method, it is necessary to train a corresponding neural network model. The following describes the neural network model training method provided in an embodiment of the present application.

[0192] In one embodiment, a neural network model training method is provided, comprising:

[0193] S01: Acquire training sample data, wherein the training sample data includes finger pressing state sample data and its corresponding pressing state classification label, wherein the finger pressing state sample data is constructed based on an ultrasonic echo sample signal, and the ultrasonic echo sample signal is an ultrasonic reflection signal after an ultrasonic sample signal is emitted to a pressing area;

[0194] S02: Based on the training sample data, the neural network model is trained until a neural network model that meets preset model conditions is obtained;

[0195] In this embodiment, it is necessary to first construct a model architecture of the neural network model and train sample data. The training sample data includes finger pressing state sample data and its corresponding pressing state classification label. The finger pressing state sample data is constructed based on the ultrasonic echo sample signal. The ultrasonic echo sample signal is the ultrasonic reflection signal after the ultrasonic sample signal is emitted to the pressing area. Then, based on the training sample data, the neural network model is trained until a neural network model that meets the preset model conditions is obtained. The trained neural network model is used to output the finger pressing state.

[0196] It should be understood that, as an example, the neural network model parameters are obtained through supervised learning of a large amount of training sample data, and the model parameters of the neural network model are obtained by training the finger pressing state data set covering multiple scenes and multiple temperatures constructed in advance, and the information obtained in each scene has a classification label of the pressing. In specific training, the parameter value of the minimization loss function can be solved by the stochastic gradient descent method, and the parameters are updated by back propagation through the chain rule until the optimal model parameters are obtained. The model parameters include the model weight coefficients, and the preset model conditions at least include the above-mentioned optimal model parameter conditions.

[0197] It should be noted that, according to application requirements, the required neural network model can be trained specifically to match the input data during application. In one embodiment, the above-mentioned neural network model includes but is not limited to a neural network model based on a recurrent neural network, or a neural network model based on a convolutional neural network.

[0198] Furthermore, the neural network model includes a first classification decision layer and a multi-layer recurrent neural network, and the output of the multi-layer recurrent neural network is connected to the first classification decision layer;

[0199] or;

[0200] The neural network model includes a second classification decision layer and a single-layer recurrent neural network, and the output of the single-layer recurrent neural network is connected to the second classification decision layer.

[0201] The specific content of the model architecture can be found in the above description and will not be repeated here.

[0202] It should be noted that based on the spatial multi-point detection scheme and the time series detection scheme described above, when training the model, it is necessary to construct corresponding training data in a targeted manner for targeted application, which are described below respectively.

[0203] In one embodiment, the finger pressing state sample data includes a first sample data vector; the first sample data vector is obtained by transforming spatial sampling sample data, and the spatial sampling sample data is obtained after spatial multi-point sampling based on a single-frame ultrasonic echo sample signal.

[0204] In one embodiment, the spatial domain sampling sample data is obtained by performing spatial domain sampling on a single frame of the ultrasonic echo sample signal at a preset interval.

[0205] In one embodiment, the first sample data vector is a normalized sample data vector.

[0206] In this embodiment, for the spatial multi-point detection scheme, the neural network model required for the spatial multi-point detection scheme can be trained specifically. For more relevant content, please refer to the relevant description of the aforementioned embodiment. The difference includes that the model input data at this time is training sample data.

[0207] In one embodiment, the finger pressing state sample data includes a second sample data vector;

[0208] The second sample data vector is obtained by converting echo intensity feature values ​​corresponding to a plurality of single-frame ultrasonic echo sample signals, respectively, and the plurality of single-frame ultrasonic echo sample signals are single-frame ultrasonic echo sample signals corresponding to a plurality of different time steps.

[0209] In one embodiment, the echo intensity characteristic value includes an average echo intensity of a single frame of the ultrasonic echo signal, or the echo intensity characteristic value includes a minimum echo intensity value in a single frame of the ultrasonic echo signal.

[0210] In one embodiment, the plurality of different time steps are continuous time steps.

[0211] In one embodiment, the second sample data vector is a normalized sample data vector.

[0212] In this embodiment, for the timing detection scheme, the neural network model required for the timing detection scheme can be trained specifically. For more relevant content, please refer to the relevant description of the aforementioned embodiment. The difference includes that the model input data at this time is training sample data.

[0213] It should be noted that, according to the detection requirements, the algorithm of the classification decision layer in the neural network model can be set, such as two-classification or three-classification or other multi-classification, and the specific classification type of each classification. When constructing sample data, the corresponding classification label is set according to the algorithm of the classification decision layer, so that the output of the trained neural network model has the corresponding classification state.

[0214] In one embodiment, the finger pressing state output by the neural network model includes: a classification determined state or a classification pending state, and the classification determined state is any one of at least two categories of pressing states.

[0215] In one embodiment, the at least two categories of press states include a finger touch state and a finger non-touch state; or; the at least two categories of press states include a finger touch state, a finger non-touch state, a critical state from touch to non-touch, and a critical state from non-touch to touch.

[0216] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0217] The above mainly describes part of the method provided in the embodiment of the present application, and the embodiments of the relevant devices, media and equipment provided in the embodiment of the present application are described below.

[0218] In one embodiment, a finger pressing state detection device is provided, and the finger pressing state detection device corresponds to the finger pressing state detection method in the above embodiment. Fig.13 As shown, the finger pressing state detection device includes a processing module 101 and an input module 102. Each functional module is described in detail as follows:

[0219] The processing module 101 is used to pre-process the ultrasonic echo signal to obtain model input data, wherein the ultrasonic echo signal is the ultrasonic echo signal corresponding to the ultrasonic array transmitting the ultrasonic signal to the pressing area;

[0220] The input module 102 is used to input the model input data into the trained neural network model so that the neural network model outputs the finger pressing state.

[0221] In combination with the previous embodiment, in one embodiment, the neural network model includes a neural network model constructed based on a recurrent neural network; or a neural network model constructed based on a convolutional neural network.

[0222] In combination with the previous embodiment, in one embodiment:

[0223] The neural network model includes a first classification decision layer and a multi-layer recurrent neural network, and the output of the multi-layer recurrent neural network is connected to the first classification decision layer;

[0224] or;

[0225] The neural network model includes a second classification decision layer and a single-layer recurrent neural network, and the output of the single-layer recurrent neural network is connected to the second classification decision layer.

[0226] In one embodiment, the processing module 101 is used to:

[0227] Performing spatial multi-point sampling on the single-frame ultrasonic echo signal to obtain spatial sampling data corresponding to the single-frame ultrasonic echo signal;

[0228] Converting the spatial domain sampling data into a first data vector;

[0229] According to the first data vector, model input data of the neural network model is obtained.

[0230] In one embodiment, the processing module 101 is used to:

[0231] According to the preset row and column intervals, spatial domain sampling is performed on the single frame of the ultrasonic echo signal to obtain spatial domain sampling data corresponding to the single frame of the ultrasonic echo signal.

[0232] In one embodiment, the processing module 101 is used to:

[0233] The first data vector is normalized, and the first data vector after normalization is model input data of the neural network model.

[0234] In one embodiment, the processing module 101 is used to:

[0235] Converting the echo intensity characteristic value of the single frame of the ultrasonic echo signal acquired at the current time step and the echo intensity characteristic values ​​corresponding to other time steps into a second data vector, wherein the other time steps are the time steps before the current time step;

[0236] According to the second data vector, model input data of the neural network model is obtained.

[0237] In one embodiment, the echo intensity characteristic value includes an average echo intensity of a single frame of the ultrasonic echo signal, or the echo intensity characteristic value includes a minimum echo intensity value in a single frame of the ultrasonic echo signal.

[0238] In one embodiment, the other time steps are the N-1 consecutive time steps before the current time step N.

[0239] In one embodiment, the processing module 101 is used to:

[0240] The second data vector is normalized, and the second data vector after normalization is the model input data of the neural network model.

[0241] In one embodiment, the finger pressing state output by the neural network model includes: a classification determined state or a classification pending state, and the classification determined state is any one of at least two categories of pressing states.

[0242] In one embodiment: the at least two categories of pressing states include a finger touch state and a finger non-touch state; or; the at least two categories of pressing states include a finger touch state, a finger non-touch state, a critical state from touch to non-touch, and a critical state from non-touch to touch.

[0243] In one embodiment, the processing module 101 is further configured to:

[0244] When the output finger pressing state is the classification determination state, the classification determination state is used as the final finger pressing state, and the output classification determination state is updated to the state memory sequence in a time sequence;

[0245] When the output finger pressing state is the classification pending state, the last pressing state recorded in the state memory sequence is taken as the final finger pressing state, or the pressing state with the highest state probability in the state memory sequence is taken as the final output finger pressing state, and it is updated accordingly to the state memory sequence.

[0246] In summary, the embodiments of the present application provide a finger pressure state detection device based on a neural network model, which brings the following technical effects: the differences between different ultrasonic modules can be eliminated without setting a benchmark threshold comparison method, and the same network model parameters have better generalization capabilities between different ultrasonic modules. Moreover, the differences in various scenarios (temperature, frequency, etc.) and under different finger states can be effectively eliminated, and a variety of special scenarios can be covered, such as high temperature, low temperature, fingers wet with water, etc., with stronger generalization ability and accuracy; for more technical effects, please refer to the corresponding description of the aforementioned method embodiment.

[0247] In one embodiment, a neural network model training device is provided, and the neural network model training device corresponds to the neural network model training method in the above embodiment. Fig.14 As shown, the neural network model training device includes an acquisition module 201 and a training module 202. Each functional module is described in detail as follows:

[0248] An acquisition module 201 is used to acquire training sample data, wherein the training sample data includes finger pressing state sample data and its corresponding pressing state classification label, wherein the finger pressing state sample data is constructed based on an ultrasonic echo sample signal, and the ultrasonic echo sample signal is an ultrasonic reflection signal after an ultrasonic sample signal is emitted to a pressing area;

[0249] The training module 202 is used to train the neural network model based on the training sample data until a neural network model that meets the preset model conditions is obtained. The trained neural network model is used to output the finger pressing state.

[0250] In combination with the previous embodiment, in one embodiment, the neural network model includes a neural network model constructed based on a recurrent neural network; or a neural network model constructed based on a convolutional neural network.

[0251] In combination with the previous embodiment, in one embodiment, the neural network model includes a first classification decision layer and a multi-layer recurrent neural network, and the output of the multi-layer recurrent neural network is connected to the first classification decision layer;

[0252] or;

[0253] The neural network model includes a second classification decision layer and a single-layer recurrent neural network, and the output of the single-layer recurrent neural network is connected to the second classification decision layer.

[0254] In one embodiment, the finger pressing state sample data includes a first sample data vector;

[0255] The first sample data vector is obtained by transforming the spatial domain sampling sample data, and the spatial domain sampling sample data is obtained by performing spatial domain multi-point sampling based on a single frame of ultrasonic echo sample signal.

[0256] In one embodiment, the spatial domain sampling sample data is obtained by performing spatial domain sampling on a single frame of the ultrasonic echo sample signal at a preset interval.

[0257] In one embodiment, the first sample data vector is a normalized sample data vector.

[0258] In one embodiment, the finger pressing state sample data includes a second sample data vector;

[0259] The second sample data vector is obtained by converting echo intensity feature values ​​corresponding to a plurality of single-frame ultrasonic echo sample signals, respectively, and the plurality of single-frame ultrasonic echo sample signals are single-frame ultrasonic echo sample signals corresponding to a plurality of different time steps.

[0260] In one embodiment, the plurality of different time steps are continuous time steps.

[0261] In one embodiment, the second sample data vector is a normalized sample data vector.

[0262] In one embodiment, the echo intensity characteristic value includes an average echo intensity of a single frame of the ultrasonic echo signal, or the echo intensity characteristic value includes a minimum echo intensity value in a single frame of the ultrasonic echo signal.

[0263] In one embodiment, the finger pressing state output by the neural network model includes: a classification determined state or a classification pending state, and the classification determined state is any one of at least two categories of pressing states.

[0264] In one embodiment, the at least two categories of press states include a finger touch state and a finger non-touch state; or; the at least two categories of press states include a finger touch state, a finger non-touch state, a critical state from touch to non-touch, and a critical state from non-touch to touch.

[0265] For the specific definition of the above-mentioned device, please refer to the relevant definition of the method above, which will not be repeated here. Each module in the above-mentioned device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the electronic device or computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0266] In one embodiment, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the aforementioned neural network model training methods when executing the computer program.

[0267] In one embodiment, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the aforementioned methods for detecting finger pressure status when executing the computer program.

[0268] Specifically, in one embodiment, if Figure 1 As shown, an electronic device is provided, including an ultrasonic piezoelectric sensor, an analog-to-digital converter, a data processing center and a central controller, wherein the ultrasonic piezoelectric sensor array, the analog-to-digital converter, and the data processing center are all connected to the central controller;

[0269] The ultrasonic piezoelectric sensor is used to respond to the central controller to transmit an ultrasonic signal to the pressing area and receive an ultrasonic echo signal;

[0270] The analog-to-digital converter is used to perform digital-to-analog conversion on the ultrasonic echo signal in response to the central controller;

[0271] The data processing center responds to the central controller to implement the steps of the aforementioned finger pressing state detection method.

[0272] In one embodiment, a readable storage medium is provided, wherein the readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the aforementioned neural network model training method or the finger pressure state detection method are implemented.

[0273] In one embodiment, a computer program product is provided, including a computer program, characterized in that when the computer program is executed by a processor, the steps of the aforementioned neural network model training method or the finger pressure state detection method are implemented.

[0274] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory.

[0275] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0276] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for detecting finger pressing state, characterized in that: The method comprises: Preprocessing the ultrasonic echo signal to obtain model input data, wherein the ultrasonic echo signal is an ultrasonic echo signal corresponding to the ultrasonic array transmitting an ultrasonic signal to the pressing area; The model input data is input into the trained neural network model so that the neural network model outputs the finger pressing state.

2. The finger pressing state detection method according to claim 1, characterized in that: The neural network model includes a first classification decision layer and a multi-layer recurrent neural network, and the output of the multi-layer recurrent neural network is connected to the first classification decision layer; or; The neural network model includes a second classification decision layer and a single-layer recurrent neural network, and the output of the single-layer recurrent neural network is connected to the second classification decision layer.

3. The finger pressing state detection method according to claim 1, characterized in that: The preprocessing of the ultrasonic echo signal to obtain the model input data includes: Performing spatial multi-point sampling on the single-frame ultrasonic echo signal to obtain spatial sampling data corresponding to the single-frame ultrasonic echo signal, and converting the spatial sampling data into a first data vector, so as to obtain model input data of the neural network model according to the first data vector; or; The echo intensity characteristic value of the single frame of the ultrasonic echo signal collected at the current time step and the echo intensity characteristic values ​​corresponding to other time steps are converted into a second data vector to obtain the model input data of the neural network model based on the second data vector, and the other time steps are the time steps before the current time step.

4. The finger pressing state detection method according to claim 3, characterized in that: The performing spatial domain sampling on the single frame of the ultrasonic echo signal to obtain spatial domain sampling data corresponding to the single frame of the ultrasonic echo signal includes: According to the preset row and column intervals, spatial domain sampling is performed on the single frame of the ultrasonic echo signal to obtain spatial domain sampling data corresponding to the single frame of the ultrasonic echo signal.

5. The finger pressing state detection method according to claim 3, characterized in that: The other time steps are the N-1 consecutive time steps before the current time step N.

6. The finger pressing state detection method according to claim 3, characterized in that: The echo intensity characteristic value includes an echo intensity average value of the ultrasound echo signal in a single frame, or the echo intensity characteristic value includes a minimum echo intensity value in the ultrasound echo signal in a single frame.

7. The finger pressing state detection method according to any one of claims 1 to 6, characterized in that: The finger pressing state output by the neural network model includes: a classification determined state or a classification pending state, and the classification determined state is any one of at least two categories of pressing states.

8. The finger pressing state detection method according to claim 7, characterized in that: The at least two types of pressing states include a finger touch state and a finger non-touch state; or; The at least two types of press states include a finger touch state, a finger non-touch state, a touch-to-non-touch critical state, and a non-touch-to-touch critical state.

9. The finger pressing state detection method according to claim 7, characterized in that: After inputting the model input data into the trained neural network model so that the neural network model outputs the finger pressing state, the method further includes: When the output finger pressing state is the classification determination state, the classification determination state is used as the final finger pressing state, and the output classification determination state is updated to the state memory sequence in a time sequence; When the output finger pressing state is the classification pending state, the last pressing state recorded in the state memory sequence is taken as the final finger pressing state, or the pressing state with the highest state probability in the state memory sequence is taken as the final output finger pressing state, and it is updated accordingly to the state memory sequence.

10. A neural network model training method, characterized in that: The method comprises: Acquire training sample data, wherein the training sample data includes finger pressing state sample data and its corresponding pressing state classification label, wherein the finger pressing state sample data is constructed based on an ultrasonic echo sample signal, and the ultrasonic echo sample signal is an ultrasonic echo signal corresponding to an ultrasonic array transmitting an ultrasonic sample signal to a pressing area; Based on the training sample data, the neural network model is trained until a neural network model that meets the preset model conditions is obtained; The trained neural network model is used to output the finger pressing state.

11. The neural network model training method according to claim 10, characterized in that: The neural network model includes a first classification decision layer and a multi-layer recurrent neural network, and the output of the multi-layer recurrent neural network is connected to the first classification decision layer; or; The neural network model includes a second classification decision layer and a single-layer recurrent neural network, and the output of the single-layer recurrent neural network is connected to the second classification decision layer.

12. The neural network model training method according to claim 10, characterized in that: The finger pressing state sample data includes a first sample data vector, the first sample data vector is obtained by transforming the spatial sampling sample data, and the spatial sampling sample data is obtained by performing spatial multi-point sampling based on a single-frame ultrasonic echo sample signal; or; The finger pressing state sample data includes a second sample data vector, which is obtained by converting echo intensity feature values ​​corresponding to multiple single-frame ultrasonic echo sample signals, and the multiple single-frame ultrasonic echo sample signals are single-frame ultrasonic echo sample signals corresponding to multiple different time steps.

13. The neural network model training method according to claim 12, characterized in that: The spatial domain sampling sample data is obtained by performing spatial domain sampling on a single frame of the ultrasonic echo sample signal at a preset interval.

14. The neural network model training method according to claim 12, characterized in that: The multiple different time steps are continuous time steps.

15. The neural network model training method according to claim 12, characterized in that: The echo intensity characteristic value includes an echo intensity average value of the ultrasound echo signal in a single frame, or the echo intensity characteristic value includes a minimum echo intensity value in the ultrasound echo signal in a single frame.

16. The neural network model training method according to any one of claims 10 to 15, characterized in that: The finger pressing state output by the neural network model includes: a classification determined state or a classification pending state, and the classification determined state is any one of at least two categories of pressing states.

17. The neural network model training method according to claim 16, characterized in that: The at least two types of pressing states include a finger touch state and a finger non-touch state; or; The at least two types of press states include a finger touch state, a finger non-touch state, a touch-to-non-touch critical state, and a non-touch-to-touch critical state.

18. An electronic device, characterized in that: The electronic device comprises an ultrasonic piezoelectric sensor, an analog-to-digital converter, a data processing center and a central controller, wherein the ultrasonic piezoelectric sensor array, the analog-to-digital converter and the data processing center are all connected to the central controller; The ultrasonic piezoelectric sensor is used to respond to the central controller to transmit an ultrasonic signal to the pressing area and receive an ultrasonic echo signal; The analog-to-digital converter is used to perform digital-to-analog conversion on the ultrasonic echo signal in response to the central controller; The data processing center responds to the central controller to implement the steps of the finger pressure state detection method as described in any one of claims 1-9.

19. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the neural network model training method as claimed in any one of claims 10 to 17 when executing the computer program.

20. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 17 are implemented.

Images