Press detection method and device, press training method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of signal detection, and discloses a pressing detection method, a training method, a device, a medium and electronic equipment, which are used for realizing generalization ability of pressing degree result detection. The method comprises the steps that multi-time-step sequence data formed by single-point features of a plurality of continuous time steps are obtained, the single-point feature of each time step is an intensity feature value of an ultrasonic echo signal of a single time step, and the ultrasonic echo signal is a corresponding echo signal after an ultrasonic array transmits an ultrasonic signal to a pressing area; preprocessing the multi-time step sequence data to obtain model input data; and inputting the model input data into a trained neural network model to enable the neural network model to output a pressing degree result.

Description

Technical Field

[0001] The present application relates to the field of signal detection technology, and in particular to a pressure detection and training method, device, electronic device and storage medium. Background Art

[0002] Ultrasonic systems transmit ultrasonic waves, receive reflected sound waves, and convert them into digital signals through a series of signal conversions. Because sound wave propagation is affected by acoustic impedance, the acoustic impedance of different reflecting surfaces varies, and the ultrasonic echo intensity also varies. Therefore, the ultrasonic echo intensity between the finger and the touch surface and between the air and the touch surface varies. This difference can be used to detect the user's finger movements and determine the degree of finger pressure.

[0003] In traditional solutions, ultrasonic signals are affected by factors such as transmission frequency, temperature, flight time, and module differences. The signal strength varies greatly under different conditions. Therefore, directly judging the degree of pressure through thresholds requires a large amount of data under different conditions and benchmark thresholds, which makes it difficult to cover all different situations. In actual applications, it is necessary to accurately judge and screen the current state before selecting the appropriate threshold for judgment. Covering all different situations is difficult, resulting in poor generalization ability of traditional pressure degree detection solutions. Summary of the Invention

[0004] The present application provides a pressure detection and training method, device, electronic device and storage medium to solve the technical problem of poor generalization ability of traditional pressure degree detection solutions.

[0005] Based on the above technical problems, the following solutions are provided: In a first aspect, a pressure detection method is provided, the method comprising: Acquire multi-time-step sequence data formed by single-point features of multiple consecutive time steps, where the single-point feature of each time step is an intensity characteristic value of an ultrasonic echo signal of a single time step, and the ultrasonic echo signal is an echo signal corresponding to the ultrasonic array transmitting an ultrasonic signal to the pressing area; Preprocessing the multi-time step sequence data to obtain model input data; The model input data is input into the trained neural network model so that the neural network model outputs a compression degree result.

[0006] Optionally, the single-point feature includes the mean echo intensity of the ultrasonic echo signal in a single time step; or, the single-point feature includes the minimum echo intensity value in the ultrasonic echo signal in a single time step.

[0007] Optionally, the neural network model includes a first-stage network, a second-stage network, a first classifier, and a second classifier; The model input data outputs a first feature vector after passing through the first stage network, and the first feature vector outputs a pressing state feature representing a pressing state result after passing through the first classifier; The first feature vector outputs a second feature vector after passing through the second stage network. The second feature vector is fused with the pressing state feature output by the first classifier and then input into the second classifier, so that the second classifier outputs a pressing degree feature that represents the pressing degree result.

[0008] Optionally, the neural network model includes a first-stage network, a second-stage network, a first classifier, a second classifier, and a target weight module; The model input data is input into the first stage network and then outputs a first feature vector, and the first feature vector is outputted as a pressing state feature representing the pressing state result after passing through the first classifier; The first feature vector outputs a target weight coefficient after passing through the target weight module; The first feature vector is fused with the target weight coefficient and input into the second stage network, so that the second stage network outputs a second feature vector. The second feature vector is fused with the pressing state feature output by the first classifier and input into the second classifier, so that the second classifier outputs a pressing degree feature representing the pressing degree result.

[0009] Optionally: the target weight module uses a self-attention mechanism to output the target weight coefficient corresponding to the first eigenvector; or; the target weight module obtains the target weight coefficient corresponding to the first eigenvector from a preset weight set, and the preset weight set is pre-constructed based on the pressing data, and the pressing data includes the intensity characteristic value sequence and pressure value sequence of the ultrasonic echo signal collected during the pressing process.

[0010] Optionally, preprocessing the multi-time step sequence data to obtain model input data includes: The multi-time step sequence data is normalized using a preset empty sampling calibration value and a pressing calibration value to obtain model input data, wherein the pressing calibration value is related to the pressing degree requirement setting.

[0011] Optionally, the pressing degree result includes a pressing force type or a pressing force range.

[0012] Optionally, the pressing force type includes no pressing, light pressing or heavy pressing.

[0013] In a second aspect, a neural network training method is provided, the method comprising: Acquire training sample data, the training sample data including sample multi-time-step sequence data and its corresponding compression state result label and compression degree result label, the sample multi-time-step sequence data being formed by single-point features of multiple consecutive time steps, each single-point feature of the time step being a signal feature value of a sample ultrasonic echo signal of a single time step, the sample ultrasonic echo signal being an echo signal corresponding to the ultrasonic array transmitting an ultrasonic signal to the compression area; Based on the training sample data, the neural network is trained until a neural network model that meets the preset model conditions is obtained.

[0014] Optionally, the single-point feature includes an echo intensity mean value of the sample ultrasonic echo signal in a single time step; or, the single-point feature includes a minimum echo intensity value in the sample ultrasonic echo signal in a single time step.

[0015] Optionally, the neural network includes a first-stage network, a second-stage network, a first classifier, and a second classifier; The training sample data is used as input to the first-stage network, the output of the first-stage network is used to input the first classifier and the second-stage network, and the first classifier is used to output a pressing state feature representing a pressing state result; The output of the second-stage network is fused with the output of the first classifier and then input into the second classifier. The second classifier is used to output a compression degree feature representing the compression degree result.

[0016] Optionally, the neural network includes a first-stage network, a second-stage network, a first classifier, a second classifier, and a target weight module; The training sample data is used as input to the first-stage network, and the output of the first-stage network is used to input the first classifier and the target weight module, and the first classifier is used to output a pressing state feature representing a pressing state result; The output of the target weight module is fused with the output of the first stage network and input into the second stage network. The output of the second stage network is fused with the output of the first classifier and input into the second classifier. The second classifier is used to output a pressure degree feature representing the pressure degree result.

[0017] Optionally, the target weight module is used to learn the target weight coefficient corresponding to the output of the first stage network using a self-attention mechanism.

[0018] Optionally, the sample multi-time step sequence data is obtained by normalizing using a preset empty sampling calibration value and a press calibration value, and the press calibration value is related to a press degree requirement setting.

[0019] Optionally, the pressing degree result label includes a pressing force type label or a pressing force range label.

[0020] Optionally, the pressure intensity type label includes no press, light press or heavy press.

[0021] In a third aspect, a pressure detection device is provided, the device comprising: An acquisition module is used to acquire multi-time-step sequence data formed by single-point features of multiple consecutive time steps, where the single-point feature of each time step is the intensity characteristic value of the ultrasonic echo signal of a single time step, and the ultrasonic echo signal is the echo signal corresponding to the ultrasonic array transmitting the ultrasonic signal to the pressing area; The processing module is used to preprocess the multi-time step sequence data to obtain model input data; and input the model input data into the trained neural network model so that the neural network model outputs a pressure degree result.

[0022] In a fourth aspect, an electronic device is provided, comprising an ultrasonic array, an analog-to-digital converter, a data processing center, and a central controller, wherein the ultrasonic array, the analog-to-digital converter, and the data processing center are all connected to the central controller, wherein: The ultrasonic array is configured to transmit ultrasonic signals to the pressing area in response to the central controller and receive ultrasonic echo signals; The analog-to-digital converter is configured to perform digital-to-analog conversion on the ultrasonic echo signal in response to the central controller; The data processing center responds to the central controller to implement the steps of any of the aforementioned pressure detection methods.

[0023] In a fifth aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of any of the aforementioned neural network training methods are implemented.

[0024] In a sixth aspect, a readable storage medium is provided, wherein the readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any method described are implemented.

[0025] In a seventh aspect, a computer program product is provided, comprising a computer program, which implements the steps of any of the aforementioned methods when executed by a processor.

[0026] One of the above-mentioned solutions provides a pressure detection method based on a neural network model. Combined with the constructed neural network model, the spatial signal strength characteristic value of a single time step is used as a single-point feature, and the single-point features of multiple consecutive time steps form a multi-time step sequence as the model input data, which is input into the neural network model for detection. The pressure degree result is detected by the change of the spatial state in time. It can detect the pressure degree under the corresponding pressure state, and effectively avoids the use of sampling benchmark threshold comparison method, eliminates the differences between different ultrasound modules, and has stronger generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0028] Figure 1 This is a schematic diagram of a system framework of an ultrasonic processing system or electronic device in one embodiment of the present application; Figure 2 This is a schematic diagram showing the working principle of the signal transmission model and the signal receiving model of the ultrasonic array in one embodiment of the present application; Figure 3 This is a schematic diagram of the relationship between pressing force and ultrasonic echo signal amplitude in an example of the present application; Figure 4 This is a flow chart of a pressure detection method according to an embodiment of the present application; Figure 5 This is a schematic diagram of a model architecture of a neural network model in one embodiment of the present application; Figure 6 This is a schematic diagram of a network architecture of a first-stage network in a neural network model in one embodiment of the present application; Figure 7 This is another network architecture diagram of the first stage network in a neural network model in one embodiment of the present application; Figure 8 yes Figure 6 or Figure 7 Schematic diagram of the relationship between neuron nodes after the hidden layer is expanded; Figure 9 This is a flow chart of using a state memory sequence to assist in making decisions about a pressing state result in one embodiment of the present application; Figure 10 This is another flowchart of using a state memory sequence to assist in making decisions about a pressing state result in one embodiment of the present application; Figure 11 This is a structural diagram of a pressure detection device in one embodiment of the present application; Figure 12 It is a structural diagram of a neural network training device in one embodiment of the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0030] The embodiments of the present application provide a scheme for press detection, which is used to detect and obtain press degree results. For example, press state results include but are not limited to pressed state and unpressed state. Based on the corresponding press degree results under the above-mentioned detected press state results, the scheme can be applied to finger touch scenarios or other applications that include touch function scenarios, including press degree detection for electronic devices (including but not limited to mobile phones, tablets, car-mounted central control and door locks, etc.), using press degree behavior detection to replace physical button functions or other richer functions, and realize different application functions defined by actions of different press degrees, without specific limitation. For example, taking finger detection as an example, when a user uses an electronic device such as a mobile phone or a computer, the user needs to touch the press area (such as the screen or other touchable area, such as a physical contact panel) with their finger to perform an operation or recognition. For example, touch functions include but are not limited to: touch unlock, hard press to confirm, light press to take a screenshot, etc., without specific limitation. Generally, the electronic device needs to first capture the press state result and its press degree result before triggering the corresponding touch function. In addition, the above-mentioned electronic devices include mobile phones, computers, tablets, door locks, vehicle-mounted terminals and other products and equipment that require touch functions. Moreover, the press detection in the embodiment of the present application can be used for detection not limited to fingers or other covering objects with deformation characteristics, such as silicone press and press detection of objects made of other materials, without specific limitation.

[0031] To facilitate understanding of the embodiments of the present application, an ultrasonic processing system framework or an electronic device system framework used in the embodiments of the present application is first introduced. Figure 1 As shown, the ultrasonic processing system framework or electronic device system framework provided by the present application includes a press detection system and a touch function system.

[0032] As an example framework, the compression detection system includes an ultrasonic array, an analog-to-digital converter, a storage unit, a data processing center, and a central controller, wherein the ultrasonic array, analog-to-digital converter, storage unit, and data processing center are respectively connected to the central controller. For example, the ultrasonic array may include an ultrasonic array array, which includes an ultrasonic array and a coupling layer, and may be a PVDF (polyvinylidene fluoride) array, without specific limitation. The analog-to-digital converter may be a high-precision analog-to-digital converter, without specific limitation. The central controller is used to control the ultrasonic array to complete the excitation and recovery of ultrasonic signals. The analog-to-digital converter is used to complete the digital signal conversion of the analog ultrasonic echo signal. The analog-to-digital converted ultrasonic echo signal is transmitted to the data processing center via the storage unit. The data processing center applies the compression detection method provided in the embodiments of the present application based on the ultrasonic echo signal to detect the compression degree result.

[0033] The touch function system can use pressure degree behavior detection to replace physical button functions or other richer functions to realize different application functions defined by actions of different pressure degrees, such as door lock control based on the pressure degree results, and control of the car's central control or mobile phone based on the pressure degree results. The functions corresponding to the pressing actions can be customized, including but not limited to fingerprint registration, recognition, power on and off functions, etc., without specific restrictions.

[0034] It should be noted that Figure 1 This is only an illustration and does not limit the actual ultrasonic processing framework or electronic device system framework.

[0035] Continue reading Figure 2 As shown, Figure 2 Taking the screen as the pressing area, a schematic diagram of the ultrasonic array's ultrasonic signal emission model and ultrasonic echo signal reception model is shown. The ultrasonic array transmits ultrasonic signals to the screen at a fixed signal transmission frequency. The ultrasonic signals penetrate the screen and reach the contact surface with the air or the pressing object (finger or other surface-covered object), and are then reflected back to the ultrasonic array. The inventors have discovered that when sound waves propagate to different contact surfaces, they have different reflectivities due to the different acoustic impedances of the contact surfaces (the reflectivity is calculated as shown in Formula 1, Represent the impedance of the pressing object and the screen respectively. If there is no pressing object, then is the acoustic impedance of air). Due to the significant difference in acoustic impedance between air and a finger (or other surface-covered object), the ultrasonic echo signal intensity differs between when a finger (or other surface-covered object) is touching and when there is no covering object (no pressure). This difference can be used to determine whether a finger (or other surface-covered object) is pressing.

[0036] , Formula-1 The formula for calculating acoustic impedance is shown in Equation 2, where z represents acoustic impedance (Rayl, kg / (m^2*s)), c represents the speed of sound (m / s), and ρ represents density (kg / m^3). For example, taking a finger as an example, when different pressures are applied, the finger will deform to a certain extent, which will change the density of the finger to a certain extent, resulting in a change in its acoustic impedance, and thus a change in reflectivity, such as Figure 3 As shown, Figure 3 The figure shows the correlation between the amplitude of the ultrasonic echo signal and the finger pressure at different levels. Therefore, it can be seen that when the finger presses at different levels, the ultrasonic echo signals received by the corresponding ultrasonic array will also be different, so the degree of pressing can be distinguished. The pressing degree includes the pressing force type or the pressing force range. The pressing force type can include no pressing, light pressing or heavy pressing, etc. The pressing force range represents the specific force range of the pressing. For example, 2N-3N can be one pressing force range or multiple other pressing force ranges.

[0037] , Formula-2 As introduced in the background, in traditional schemes, ultrasonic signals are affected by factors such as transmission frequency, temperature, flight time and module differences. There are large differences in signal strength under different conditions. Therefore, directly judging the degree of pressure through thresholds requires a large amount of data under different conditions and benchmark thresholds. It is difficult to cover all different situations. In actual applications, it is necessary to accurately judge and screen the current state before selecting the appropriate threshold for judgment. It is difficult to cover all different situations, resulting in poor generalization ability of traditional pressure degree detection schemes. For example, the signal strength itself under the same conditions also varies greatly. When the finger is in different states, the change in its ultrasonic echo signal is even greater. For example, when the finger is dry, the ultrasonic echo signal intensity is lower, while when the finger is wet or wet, the ultrasonic echo signal intensity is higher. In addition, the signal strength will also be different at different temperatures. Under high temperature conditions, the ultrasonic echo signal intensity is higher, while under low temperature conditions, the ultrasonic echo signal intensity is smaller. The generalization ability of pressure degree detection is poor and the accuracy is also low.

[0038] In light of the above findings, this application has designed a neural network-based compression detection solution that can detect compression degree results. By learning the characteristics of the ultrasonic echo sound wave intensity changing with compression through the neural network, the network model parameters of the neural network are updated, so that the neural network model after the seat is vacant can be used to determine the compression degree results. The following describes them separately.

[0039] like Figure 4 As shown, Figure 1 Taking the ultrasonic processing system shown in FIG. 1 as an example, the present application provides a pressure detection method based on a neural network model, including the following steps: S10: Acquire multi-time-step sequence data formed by single-point features of multiple consecutive time steps.

[0040] S20: Preprocessing the multi-time step sequence data to obtain model input data.

[0041] S30: Inputting the model input data into the trained neural network model, so that the neural network model outputs a pressing degree result.

[0042] It should be understood that the electronic device includes a pressing area, which can be a touch screen or other touch location, such as a touch panel, etc., without specific limitation. The electronic device uses an ultrasonic array to transmit an ultrasonic signal to the pressing area at a fixed signal transmission frequency, and uses the ultrasonic array of the ultrasonic array to receive the corresponding reflected signal, that is, the ultrasonic echo signal. The ultrasonic echo signal is converted into a digital form by an analog-to-digital converter and then reaches the data processing center through a storage unit. The data processing center processes the ultrasonic echo signal in the storage unit to form multi-time step sequence data, and the single point feature of each time step is the intensity characteristic value of the ultrasonic echo signal in a single time step.

[0043] After collecting the ultrasonic echo signals at each time step, the data processing center will calculate the intensity characteristic value of the ultrasonic echo signal at each time step, that is, the intensity characteristic value of the spatial domain signal at a single time step, which corresponds to the intensity characteristic value corresponding to each time step. For example, taking the data time step sequence N as an example, assuming that the current time step is , the current time step is the corresponding intensity eigenvalue , then this embodiment will set the current time step The intensity characteristic value of the acquired single-frame ultrasonic echo signal and the intensity characteristic values corresponding to other time steps are combined to form a multi-time step characteristic sequence, that is, multi-time step sequence data. As an example, the multi-time step sequence data can be expressed as the following formula-3: , Formula-3 for example, represents multi-time-step sequence data, They represent the intensity characteristic values of the spatial domain signals corresponding to the current time step N and the N-1 consecutive time steps before the current time step N, respectively.

[0044] In addition, in an embodiment of the present application, a neural network model that can output a compression degree result based on multi-time step sequence data is pre-trained. The neural network model is trained based on a large amount of training sample data. The training sample data includes sample multi-time step sequence data and its corresponding compression state result label and compression degree result label. The sample multi-time step sequence data is formed by single-point features of multiple consecutive time steps. The single-point feature of each time step is the signal feature value of the sample ultrasonic echo signal of a single time step. The sample ultrasonic echo signal is the echo signal corresponding to the ultrasonic array transmitting the ultrasonic signal to the compression area. The network parameters of the neural network model are obtained by training on sample multi-time step sequence data covering multiple scenarios, multiple temperatures, and other conditions constructed in advance. The sample multi-time step sequence data obtained in each scenario has compression label information. During model training, the model can be trained using a set model training method until a neural network model that meets the requirements is obtained, that is, a trained neural network model is obtained. Exemplarily, when used, the trained neural network model can be built into an electronic device or deployed on a server side and used in the form of a model call, without specific limitation. For details about model training, please refer to the description of the following embodiments.

[0045] In this step, after steps S10-S20, that is, after the data processing center pre-processes the multi-time step sequence data to obtain model input data, the model input data is input into the trained neural network model so that the neural network model outputs the corresponding pressure degree result.

[0046] It can be seen that in this embodiment, a press detection method based on a neural network model is provided. Combined with the constructed neural network model, the spatial signal strength characteristic value of a single time step is used as a single-point feature, and the single-point features of multiple consecutive time steps form a multi-time step sequence as the model input data, and are input into the neural network model for detection, which can detect the press degree results under the corresponding press state.

[0047] Furthermore, it avoids the traditional method of comparing sampling benchmark thresholds, eliminating differences between different ultrasound modules (for example, different electronic devices use different ultrasonic arrays), allowing the same set of network model parameters to have better generalization capabilities across different ultrasound modules. Furthermore, it can effectively eliminate differences in various scenarios (temperature, frequency, etc.) and different levels of pressure, covering a variety of special scenarios such as high and low temperatures, and wet fingers, with greater generalization and accuracy. Furthermore, due to changes in the time domain during the pressing action, using a multi-time-step spatial sequence as the model input allows the detection of the pressure level results through temporal changes in the spatial state. The introduction of timing information eliminates spatial dependencies, thereby achieving more accurate pressure level results.

[0048] In addition, in one embodiment, the inputting of the model input data into the trained neural network model so that the neural network model outputs the pressure degree result includes: inputting the model input data into the trained neural network model so that the neural network model outputs the pressure degree and its pressure degree result.

[0049] In this embodiment, the neural network model can be made to output the degree of pressure and the result of the degree of pressure at the same time, without any specific limitation.

[0050] In one embodiment, the single-point feature comprises the mean echo intensity of the ultrasonic echo signal at a single time step; alternatively, the single-point feature comprises the minimum echo intensity value of the ultrasonic echo signal at a single time step. The specific embodiment is not limited thereto. Using the mean echo intensity value allows for a more comprehensive assessment of spatial signal characteristics, while using the minimum echo intensity value simplifies calculations.

[0051] For example, in the above embodiment, the echo intensity mean may be an arithmetic mean or other forms of mean, which is not limited in the specific embodiment of the present application and can be set according to requirements to ensure the flexibility of the solution.

[0052] In one embodiment, the trained neural network model can be implemented in a variety of ways, which are not specifically limited. The following describes two main implementation frameworks as examples.

[0053] See also Figure 5 In one embodiment, the neural network model includes a first-stage network, a second-stage network, a first classifier, and a second classifier, wherein: The model input data outputs a first feature vector after passing through the first stage network, and the first feature vector outputs a pressing state feature representing a pressing state result after passing through the first classifier; The first feature vector outputs a second feature vector after passing through the second stage network. The second feature vector is fused with the pressing state feature output by the first classifier and then input into the second classifier, so that the second classifier outputs a pressing degree feature that represents the pressing degree result.

[0054] For example, the above fusion can be multiplication or other feature fusion forms. Figure 5 It is a multiplication and fusion form, and there is no specific limitation.

[0055] In this embodiment, a neural network model architecture is provided, a multi-stage network architecture, the model input data is calculated by the first stage network and then the first feature vector is output. , the first eigenvector After classification by the first classifier, the pressing state feature representing the pressing state result is obtained , Directly used as the input of the second stage network, the second eigenvector is obtained after calculation by the second stage network , the second eigenvector and the pressed state features output by the first classifier Multiply and then classify by the second classifier to output the pressing degree feature representing the pressing degree result , thus obtaining the compression degree result.

[0056] In this embodiment, a model architecture of a neural network model is provided. The structure is relatively simple, which ensures the feasibility of the solution and can realize richer functions. It can not only detect whether there is pressure, but also detect the degree of pressure. Moreover, a variety of pressure degree detections can be realized through a two-stage network structure to meet various pressure degree detection requirements.

[0057] In one embodiment, continue to refer to Figure 5 As shown, the neural network model includes a first-stage network, a second-stage network, a first classifier, a second classifier, and a target weight module; The model input data is input into the first stage network and then outputs a first feature vector, and the first feature vector is outputted as a pressing state feature representing the pressing state result after passing through the first classifier; The first feature vector outputs a target weight coefficient after passing through the target weight module; The first feature vector is fused with the target weight coefficient and input into the second stage network, so that the second stage network outputs a second feature vector. The second feature vector is fused with the pressing state feature output by the first classifier and input into the second classifier, so that the second classifier outputs a pressing degree feature representing the pressing degree result.

[0058] Similarly, the two fusion processes mentioned in this embodiment can both be multiplication or other feature fusion processing methods, and are not specifically limited.

[0059] It can be seen that compared with the model architecture of the aforementioned embodiment, the network architecture of the embodiment of the present application also includes a target weight module, which is mainly based on the target weight coefficient required by the first feature vector output after the first stage network. Specifically, the model input data is calculated by the first stage network and then outputs a set of feature vectors. , The pressing state features are obtained by the first classifier The target weight module uses a learning mechanism or a preset method to obtain a set of target weight coefficients ; As the input of the second stage network, the feature vector is obtained after the second stage network calculation , this feature is combined with the pressed state feature output by the second classifier Multiply and then pass through the second classifier to output the pressing degree feature that represents the pressing degree result , thus obtaining the final compression degree result.

[0060] This embodiment also provides a model architecture of a neural network model and sets a weight processing mechanism to ensure the feasibility of the solution and achieve richer functions. It can not only detect whether there is pressure, but also detect the degree of pressure. Moreover, through the two-stage network structure, various pressure degree detections can be achieved to meet the corresponding pressure degree detection requirements. Moreover, the weight setting is also used to facilitate learning or setting more appropriate feature weights, so that the entire detection process focuses on feature learning, thereby improving detection efficiency and quality.

[0061] It should be noted that, in combination with the above embodiments, the target weight module can implement weight setting in a variety of ways, and the embodiments of the present application provide two of them.

[0062] In one embodiment, the target weight module uses a self-attention mechanism to output the target weight coefficient corresponding to the first eigenvector; or; the target weight module obtains the target weight coefficient corresponding to the first eigenvector from a preset weight set, and the preset weight set is pre-constructed based on the pressing data, and the pressing data includes the intensity characteristic value sequence and pressure value sequence of the ultrasonic echo signal collected during the pressing process.

[0063] In the first method, the target weight module can use the self-attention mechanism to learn the target weight coefficient, that is, the target weight module is a self-attention module. In this processing method, the first eigenvector output by the first stage network is multiplied by the corresponding target weight coefficient. The self-attention module includes a fully connected layer and an activation function softmax. The first eigenvector f1 output by the first stage network is trained and learned by the self-attention module to obtain a set of target weight parameters w1. The sum of the weight parameters is 1. The target weight coefficient can be expressed as shown in the following formula-4: , Formula-4 Where n is the output length of the feature vector output by the fully connected layer of the self-attention module, Represents the features of the first eigenvector f1 output after the fully connected layer.

[0064] It can be seen that in this embodiment, a processing mechanism of a target weight module is provided, which uses the attention module to learn the first feature vector f1 output by the first-stage network, and obtains the feature weight through the attention mechanism learning method. The weights learned autonomously in this way can distinguish the features when only pressing detection is performed, which is more convenient and efficient, and can improve the processing efficiency and accuracy of the model.

[0065] In the second method, the target weight module can use a preset configuration method to obtain the target weight coefficient, that is, the target weight module is a weight preset module, which is used to find the appropriate target weight coefficient of the above-mentioned first eigenvector f1 from the preset weight set. In this processing method, the preset weight set is pre-constructed based on the compression data, and the compression data includes the intensity characteristic value sequence and pressure value sequence of the ultrasonic echo signal collected during the compression process.

[0066] During the construction of the preset weight set, for example, a set of compression data y can be collected in advance, initially in a no-compression state, and the compression force is gradually increased during the compression process until it is fully pressed. The collected ultrasonic echo signal amplitude sequence is normalized to 0-1 to obtain p1, and the corresponding pressure value sequence is normalized to 0-1 to obtain p2. The preset weight set wp is obtained by the following calculation method. When used, the first eigenvector f1 is uniformly sampled in the preset weight set wp to meet the required length, and activated by the softmax function so that the sum of the collected preset weights is 1, and finally the preset weight w is obtained, wherein the preset weight formula is expressed as shown in the following formula-5: , Formula-5 It can be seen that in this embodiment, another processing mechanism of the target weight module is provided, which uses empirical data to construct a preset weight set in advance, and then finds the appropriate target weight coefficient of the above-mentioned first eigenvector f1 from the preset weight set. The target weight found in this way is conducive to distinguishing the features when only press detection is performed, thereby improving the processing efficiency and accuracy of the model; moreover, by using the pre-constructed preset weight set, the weight corresponding to each press detection requirement (different press degrees) can be more correspondingly matched, thereby improving the detection targeting.

[0067] The activation function mentioned above may also be other activation functions, which are not specifically limited.

[0068] It should be noted that in the embodiments of the present application, the first-stage network and the second-stage network can respectively include a recurrent neural network RNN, a convolutional neural network CNN, a gated recurrent unit GRU or a long short-term memory LSTM, without specific limitation.

[0069] Taking the example of a first-stage network comprising a recurrent neural network, in one embodiment, the first-stage network comprises a multi-layer recurrent neural network, the output of which is connected to the corresponding first classifier; alternatively, the first-stage network comprises a single-layer recurrent neural network, the output of which is connected to the first classifier. In other words, the recurrent neural network employed in the first-stage network can be a single-layer recurrent neural network or a multi-layer recurrent neural network.

[0070] The single-layer recurrent neural network or multi-layer recurrent neural network used can be determined according to actual needs and is not specifically limited. Since press detection is a press detection process over a long period of time, and in a recurrent neural network, each current state is related to the previous state or the next state, the embodiment of the present application, through the recurrent neural network-based method, through the association of the previous or previous state, will be conducive to more accurate detection of the press state, thereby obtaining a more accurate press degree result.

[0071] To facilitate understanding of the subsequent detection scheme, the first-stage network constructed based on the recurrent neural network provided in the embodiment of the present application is first introduced below.

[0072] like Figure 6 As shown, as an example, the first stage network includes a multi-layer recurrent neural network, and the output of the multi-layer recurrent neural network is connected to the first classifier. Since the multi-layer recurrent neural network is related to the time series state, it is unfolded according to the time series state, then the multi-layer recurrent neural network includes hidden layer networks corresponding to different time series, and the hidden layer networks respectively represent the network architectures corresponding to different time series. It can be seen that the hidden layer network includes n hidden layers, n≥2, and the n hidden layers in the hidden layer network are connected in sequence, and the output of the hidden layer of the hidden layer network of the previous time series is passed to the same hidden layer of the next time series. The output of the last hidden layer of the hidden layer network, that is, the output of the multi-layer recurrent neural network, is connected to the first classifier.

[0073] For more details, please refer to Figure 6 As shown, in this example, according to the time sequence, Respectively represent different time series in the multi-layer recurrent neural network The corresponding hidden layer network input data, Represents the pressing state output by the corresponding time series classification decision. The hidden layer network includes hidden layer 1, hidden layer 2, ..., hidden layer n. The output of hidden layer n is connected to the input of the classifier. The value of each hidden layer of the hidden layer network of the previous time series is passed to the same hidden layer of the next time series. For example, the time series The output of hidden layer 1 is passed to the time series The corresponding hidden layer 1, time series The output of the corresponding hidden layer 1 is passed to the time series The corresponding hidden layer 1, the content transmitted includes the state vector of the hidden layer output, such as the time series The hidden layer of the 1-way time series The hidden layer 1 of the state vector 1 is passed to the next hidden layer, and the output of the next hidden layer is passed to the previous hidden layer. The relationship between the hidden layers of the same layer or other layers is similar. Figure 5 As shown, detailed description is omitted here. It is also worth noting that there are weight matrix parameters in the multi-layer recurrent neural network, which will be further explained later when introducing the single-layer recurrent neural network. The specific number of hidden layers in the multi-layer recurrent neural network can be adjusted, and the model input data size can also be adjusted, which is not limited by the specific embodiments of this application.

[0074] like Figure 6 As shown, as an example, the first-stage network includes a single-layer recurrent neural network, the output of which is connected to the first classifier. Since the single-layer recurrent neural network is related to the time series state, it is expanded according to the time series state. Then, the single-layer recurrent neural network includes hidden layer networks corresponding to different time series, and the hidden layer networks respectively represent the network architecture corresponding to different time series. The hidden layer network of the single-layer recurrent neural network only contains one hidden layer, and the output of the hidden layer of the previous time series is passed to the hidden layer of the next time series. The output of the hidden layer is connected to the first classifier. Unlike a multi-layer recurrent neural network, a single-layer recurrent neural network only contains a single hidden layer.

[0075] Specifically, if Figure 7 As shown, in Figure 7 In the example, we will take three of the time series as examples to explain. Represents the input data of the hidden layer network corresponding to the three time series, They represent the pressing state results output by the classification decision of the corresponding time series. The hidden layer network includes a hidden layer, and the output of the hidden layer 1 corresponding to the time series t-1 is transmitted to the time series Hidden layer, time series The output of the corresponding hidden layer is passed to the hidden layer corresponding to time sequence t+1.

[0076] In the above single-layer or multi-layer recurrent neural network, the weight matrix It is the linear relationship parameter in the recurrent neural network. Here we can continue with Figure 7 Taking the single-layer recurrent neural network shown in the figure as an example, we continue to explain the working principle of the first-stage network based on the single-layer recurrent neural network. Based on the network framework of the single-layer recurrent neural network, there is the following expression: in, , represents the activation function of the corresponding layer, Indicates the classification decision algorithm used in the first classifier. Exemplarily, the classification decision algorithm can be a binary classification or multi-classification algorithm, which is not specifically limited. Represents the value of the hidden layer corresponding to the time series t, that is, the state vector mentioned above, Represents the value of the hidden layer corresponding to time series t-1, Represents the value of the hidden layer corresponding to time series t+1, represents the output of the hidden layer corresponding to the time series t, represents the output of the hidden layer corresponding to time series t-1, represents the output of the first classifier corresponding to time series t+1, represents the output of the first classifier corresponding to time series t, represents the output of the first classifier corresponding to time series t-1, represents the output of the classifier corresponding to time series t+1, the weight matrix U represents the weight matrix from the model input to the hidden layer, the weight matrix V represents the weight matrix from the hidden layer to the first classifier, and the weight matrix W represents the weight matrix from the hidden layer of the previous time series to the hidden layer of the corresponding next time series. Therefore, after the input is output and decided by the first classifier, the final output is the pressing state feature representing the pressing state result of the time series. It can be seen that the output state of each time series is not only affected by the current input but also related to the information of the previous time series.

[0077] Furthermore, in one embodiment, in the above-mentioned single-layer or multi-layer recurrent neural network, the number of neuron nodes contained in the hidden layer can also be adjusted, and the training can be performed according to the set parameters, which is not limited in the embodiment of the present application. As an example, taking a single-layer recurrent neural network as an example, the network with nodes can be expanded as follows: Figure 8 shown. represents the input vector of the current time series t, Represents the value of the hidden layer neuron node of time series t, Represents the value of the neuron node of the hidden layer node in the previous time sequence t-1, Represents the activation function of the hidden layer of the current time series t, Represents the pressing state result output by the first classifier at time t.

[0078] It should be noted that the above embodiment uses a recurrent neural network as an example for the first-stage network, and is not limited to this. Furthermore, the second-stage network can also employ a recurrent neural network (RNN), a convolutional neural network (CNN), a gated recurrent unit (GRU), or a long short-term memory (LSTM). The specific processing details are not detailed here, and are not limited to this.

[0079] It should be understood that in the above-mentioned detection scheme embodiment, a neural network model based on a recurrent neural network is used as an example for description. In other embodiments, when other neural network models are used, such as a convolutional neural network, the same detection scheme can also be used to process model input data, without specific limitation. In addition, as needed, the classifier in the above-mentioned neural network model can adopt a binary classification algorithm, a three-class classification algorithm, or other multi-class classification scenarios, without specific limitation.

[0080] It should be understood that in order to enable the neural network model to better process data and detect the degree of pressure, this embodiment preprocesses the multi-time step series data to obtain model input data; this model input data is then input into the trained neural network model so that the neural network model outputs the degree of pressure. The above preprocessing can include various aspects of processing, which are not specifically limited.

[0081] In one embodiment, the preprocessing of the multi-time-step series data to obtain the model input data includes normalizing the multi-time-step series data to obtain the model input data.

[0082] In this embodiment, normalizing the acquired multi-time-step sequence data refers to normalizing all characteristic elements of the multi-time-step sequence data so that the element values of the multi-time-step sequence data are within a uniform range. Therefore, the normalized multi-time-step sequence data is the model input data of the neural network model, which can effectively reduce the computational complexity of the subsequent neural network model, thereby improving the overall press detection efficiency. It should be noted that in other embodiments, the multi-time-step sequence data can also be directly used as the model input data, without specific limitation.

[0083] In one embodiment, the normalization method may have multiple modes: The first normalization mode can be understood as 0-1 normalization, and the normalization formula can be expressed as the following formula-6: , Formula-6 Among them, among them, Represents the original data value, Represents the minimum value in the data. Indicates the maximum value of the data. is the normalized data. In the first normalization mode, due to the 0-1 normalization process, the overall detection generalization can be improved, and it is suitable for detection of various temperature scenarios.

[0084] The second normalization mode can be understood as calibration normalization. The multi-time step sequence data is normalized using the preset empty sampling calibration value and the pressing calibration value to obtain the model input data. The pressing calibration value is related to the pressing degree requirement setting. As an example, as shown in Formula-7, the pressing calibration value can be preset and the pressing calibration value can be adjusted according to the pressing requirement. For example, a pressing force greater than 3N is considered to be a hard press. The adjustment value (amp value) at 3N can be set as the pressure calibration value.

[0085] , Formula-7 Among them, among them, Represents the original data value, Indicates the air sampling calibration value, Indicates the press calibration value, is the normalized data.

[0086] In the second normalization mode, since normalization is performed based on the calibration value adjusted according to the pressing requirement, the detection can be made more accurate.

[0087] It should be noted that in the embodiment of the present application, whether 0-1 normalization or calibration normalization is used, the pressure degree detection can be achieved, and the range of pressing objects includes but is not limited to finger pressing, silicone pressing, and pressing by objects of other materials.

[0088] In one embodiment, the pressure degree result output by the neural network model includes a pressure intensity type or a pressure intensity range.

[0089] Exemplarily, the pressure force types include no pressing, light pressing or hard pressing; of course, the pressure force types do not necessarily output only the above three results (no pressing, light pressing, hard pressing), and the corresponding model can also be set to train according to the needs, so that the neural network model can achieve the output of more pressure force types based on the solution provided in the embodiment of the present application. Exemplarily, for example, it can also include force types such as short light pressing and long hard pressing, which are not specifically limited.

[0090] In one embodiment, the neural network model can also output a pressure range corresponding to the force level through training and learning and combining certain strategies. For example, the pressure range is 2N-3N, rather than just outputting the pressure type (such as light press, heavy press), etc. The specific embodiments of this application are not limited to this.

[0091] The above detection results can be used to set corresponding labels for the sample training data according to application requirements to adapt the neural network model and train and update the network parameters to obtain the optimal model parameters and achieve the corresponding press detection requirements.

[0092] In one embodiment, the pressing state result output by the neural network model, that is, the pressing state result output by the first separator includes: a classification determined state or a classification pending state, wherein the classification determined state is any one of at least two categories of pressing states, and the classification determined state refers to a state in which the pressing state is already clear and can be used for subsequent touch function applications, such as a pressed state, an unpressed state, or other pressing states, etc.; and the classification pending state refers to a pending state in which the pressing state still needs further confirmation.

[0093] In one embodiment, the at least two categories of pressed states include a pressed state and a non-pressed state; or; the at least two categories of pressed states include a pressed state, a non-pressed state, a critical state from pressed to non-pressed, and a critical state from non-pressed to pressed.

[0094] That is, in one case, the output pressing state result is a pressed state, an unpressed state, or a pending state; in another case, the output pressing state result is a pressed state, an unpressed state, a critical state from pressed to unpressed, a critical state from unpressed to pressed, or a pending state.

[0095] In combination with the above embodiment, in one embodiment, the method further includes: When the pressing state result output by the first classifier is the classification determination state, the classification determination state is used as the final pressing state result, and the output classification determination state is updated to the state memory sequence in a time sequence; When the pressing state result output by the first classifier is the classification pending state, the pressing state last recorded in the state memory sequence is taken as the final pressing state result, and is updated to the state memory sequence in a time sequence.

[0096] For example, assuming that the classification determines that the state includes a pressed state and a non-pressed state, this embodiment assists in the decision of the pressing state result by setting a continuously updated state memory sequence. The pending state refers to the current neural network model being unable to output whether it is a pressed state or a non-pressed state, but is in a situation where the final pressing state result needs to be further determined.

[0097] It should be understood that the cover may be in an unpressed or pressed state for a long time. In this case, the data, the data form of the time step sequence may not have a rising edge or a falling edge. In view of the above situation, a state memory sequence is introduced, and a fixed-length state memory sequence MemSeq is set. The first classifier includes multiple classification states, including 0-up, 1-down, and 2-maintain; that is, the first classifier outputs 0 to represent an unpressed state, outputs 1 to represent a pressed state, and outputs 2 to represent a classification pending state. The output of the first classifier is combined with the state memory sequence MemSeq to comprehensively determine the final pressing state result.

[0098] Specifically, see Figure 9 As shown, when the pressed state result output by the first classifier is the pressed state, that is, the output When the final pressing state result , and output the pressed state in sequence Corresponding update to the state memory sequence MemSeq; when the pressed state result output by the first classifier is the non-pressed state, that is, the output , the final pressing state result , and output the unpressed state according to the timing Corresponding update to the state memory sequence MemSeq; when the pressing state result output by the first classifier is the pending state, that is, the output , then from the latest state memory sequence MemSeq, obtain the last recorded state as the final output pressing state result, that is, , and the state is updated to the state memory sequence MemSeq.

[0099] In this embodiment, the decision of the pressing state result is assisted by the state memory sequence MemSeq, which can improve the state richness and make the detection scheme more accurate. It can also provide a corresponding processing mechanism when the neural network model cannot accurately output the unpressed or pressed state, thereby ensuring the feasibility of the scheme.

[0100] In combination with the above embodiment, in one embodiment, the method further includes: When the pressing state output by the first classifier is the classification determination state, the classification determination state is used as the final pressing state result, and the output classification determination state is updated to the state memory sequence in a time sequence; When the pressing state output by the first classifier is the pending classification state, the pressing state with the highest state probability is obtained from the latest state memory sequence as the final pressing state result, and is updated to the state memory sequence accordingly.

[0101] This embodiment also uses a fixed-length state memory sequence, MemSeq, and a first classifier with multiple classification states, including 0-up, 1-down, and 2-maintain. The output of the first classifier is combined with the state memory sequence MemSeq records to determine the final compression state. Unlike the previous embodiment, this embodiment uses the highest probability of the state recorded in the state memory sequence MemSeq to determine the final compression state output when the first classifier outputs "maintain."

[0102] Taking the above three categories as an example, please refer to Figure 10 As shown, when the pressing state result output by the first classifier is the pressed state, that is, the output , then the final pressing state result , and output the pressed state according to the timing Corresponding update to the state memory sequence MemSeq; when the pressed state result output by the first classifier is the non-pressed state, that is, the output , then the final pressing state result , and output the unpressed state according to the timing Corresponding update to the state memory sequence MemSeq; when the pressing state result output by the first classifier is the classification pending state, that is, the output , then from the state memory sequence MemSeq, take the pressing state with the highest state probability as the final pressing state result.

[0103] Specifically, as an example, first determine whether the number of times 0 appears in the state memory sequence MemSeq is greater than the number of times 1 appears. If so, then If not, then And update the last state to the state memory sequence MemSeq.

[0104] In this embodiment, there is another solution that uses the state memory sequence MemSeq to assist in the decision of the pressing state result, which can improve the richness of the state, make the detection solution more accurate, and also provide a corresponding processing mechanism when the neural network model cannot accurately output the unpressed or pressed state; in addition, due to the use of previous state probability processing, the fault tolerance rate of the solution in actual application can also be made higher.

[0105] It should be understood that in the above embodiment, the length of the state memory sequence MemSeq is adjustable and not specifically limited. For example, it can be an infinite length sequence, ranging from 1 to infinity, or a fixed sequence, set as needed. Furthermore, the above state memory sequence can also be used to assist decision-making in other multi-classification situations. The above three classifications are merely illustrative and do not limit the present embodiment.

[0106] It should be noted that, in the above embodiment, the state memory sequence is used to assist in the decision-making of the pressing state result, which is also conducive to ensuring the detection of the final pressing degree result.

[0107] In summary, the solution proposed in the embodiments of this application has greater generalization and accuracy than the threshold-based method for determining compression intensity, and can better cover a variety of compression intensity scenarios. It can eliminate the differences between different ultrasound modules and effectively cover the differences in force in various scenarios and under different compression states, thus having stronger generalization and accuracy, and greater applicability.

[0108] The above embodiment describes a press detection method based on a neural network model provided in an embodiment of the present application. When implementing the above method, it is necessary to train a corresponding neural network model. The following describes the neural network model training method provided in an embodiment of the present application.

[0109] In one embodiment, a neural network training method is provided, the method comprising: S101 , obtaining training sample data, wherein the training sample data includes sample multi-time step sequence data and its corresponding pressing state result label and pressing degree result label.

[0110] S102: Based on the training sample data, the neural network is trained until a neural network model that meets preset model conditions is obtained.

[0111] In this embodiment, training sample data must first be constructed. The training sample data includes sample multi-time-step sequence data and its corresponding compression state result label and compression degree result label. The sample multi-time-step sequence data is formed by single-point features of multiple consecutive time steps. The single-point feature of each time step is the signal feature value of the sample ultrasonic echo signal of the single time step. The sample ultrasonic echo signal is the echo signal corresponding to the ultrasonic signal emitted by the ultrasonic array to the compression area.

[0112] Then, based on the training sample data, the neural network model is trained until a neural network model that meets the preset model conditions is obtained. The trained neural network model is used to output the pressure degree result.

[0113] It should be understood that, as an example, the parameters of the neural network model are obtained through supervised learning of a large amount of training sample data. The model parameters of the neural network model are obtained by training a multi-time step sequence data set of samples covering multiple scenarios and multiple temperatures that was constructed in advance, and the information obtained in each scenario has a set label. In specific training, the parameter value of the minimization loss function can be solved by the stochastic gradient descent method, and the parameters are updated by back propagation through the chain rule until the optimal model parameters are obtained. The model parameters include the model weight coefficients, and the preset model conditions at least include the optimal conditions of the above-mentioned model parameters, which are not specifically limited. In this embodiment, the neural network model can output the degree of pressure and the result of the degree of pressure at the same time, without any specific limitation.

[0114] In one embodiment, the single-point feature comprises the mean echo intensity of the sample ultrasonic echo signal at a single time step; alternatively, the single-point feature comprises the minimum echo intensity value of the sample ultrasonic echo signal at a single time step. The specific embodiment is not limited thereto. Using the mean echo intensity value allows for a more comprehensive assessment of spatial signal characteristics, while using the minimum echo intensity value simplifies calculations.

[0115] For example, in the above embodiment, the echo intensity mean may be an arithmetic mean or other forms of mean, which is not limited in the specific embodiment of the present application and can be set according to requirements to ensure the flexibility of the solution.

[0116] In one embodiment, the neural network used for training can be implemented in a variety of ways, which are not specifically limited. The following describes two main implementation frameworks as examples.

[0117] In one embodiment, the neural network includes a first-stage network, a second-stage network, a first classifier, and a second classifier; The training sample data is used as input to the first-stage network, the output of the first-stage network is used to input the first classifier and the second-stage network, and the first classifier is used to output a pressing state feature representing a pressing state result; The output of the second-stage network is fused with the output of the first classifier and then input into the second classifier. The second classifier is used to output a compression degree feature representing the compression degree result.

[0118] In this embodiment, a model architecture of a neural network for training is provided. The structure is relatively simple, which ensures the feasibility of the solution and can realize richer functions. It can not only detect whether there is pressure, but also detect the degree of pressure. Moreover, a two-stage network structure can realize a variety of pressure degree detections to meet corresponding pressure detection requirements.

[0119] In one embodiment, the neural network includes a first-stage network, a second-stage network, a first classifier, a second classifier, and a target weight module; The training sample data is used as input to the first-stage network, and the output of the first-stage network is used to input the first classifier and the target weight module, and the first classifier is used to output a pressing state feature representing a pressing state result; The output of the target weight module is fused with the output of the first stage network and input into the second stage network. The output of the second stage network is fused with the output of the first classifier and input into the second classifier. The second classifier is used to output a pressure degree feature representing the pressure degree result.

[0120] In one embodiment, the target weight module is used to learn the target weight coefficient corresponding to the output of the first stage network using a self-attention mechanism.

[0121] In this embodiment, a model architecture of a neural network for training is also provided, and a weight processing mechanism is set up to ensure the feasibility of the solution and realize richer functions. It can not only detect whether there is pressing, but also detect the degree of pressing. Moreover, a two-stage network structure can realize various pressing degree detections to meet the corresponding pressing detection requirements; and the use of weight setting is conducive to learning or setting more appropriate feature weights, so that the subsequent application detection process can focus on feature learning, thereby improving detection efficiency and quality.

[0122] In one embodiment, the sample multi-time step sequence data is obtained by normalizing using a preset empty sampling calibration value and a press calibration value, where the press calibration value is related to the required press degree setting; or is obtained by normalizing according to a 0-1 normalization method, without specific limitation.

[0123] In this embodiment, during the training process, the acquired sample multi-time-step sequence data is normalized, which means that all characteristic elements of the sample multi-time-step sequence data are normalized so that the element values of the sample multi-time-step sequence data are within a unified range. Therefore, the normalized sample multi-time-step sequence data is used as the model input data of the neural network model for training, which can effectively reduce the subsequent computational load of the neural network, accelerate convergence, and thus improve the overall training efficiency. It should be noted that in other embodiments, the sample multi-time-step sequence data can also be directly used as the model input data for training, without specific limitation.

[0124] It should be noted that in the embodiments of the present application, the first-stage network and the second-stage network can respectively include a recurrent neural network RNN, a convolutional neural network CNN, a gated recurrent unit GRU or a long short-term memory LSTM, without specific limitation.

[0125] Taking the example of a first-stage network comprising a recurrent neural network, in one embodiment, the first-stage network comprises a multi-layer recurrent neural network, the output of which is connected to the corresponding first classifier; alternatively, the first-stage network comprises a single-layer recurrent neural network, the output of which is connected to the first classifier. In other words, the recurrent neural network employed in the first-stage network can be a single-layer recurrent neural network or a multi-layer recurrent neural network. For details on the structure of the model, please refer to the description of the aforementioned embodiment.

[0126] In one embodiment, the pressing degree result label includes a pressing degree type label or a pressing degree range label.

[0127] In one embodiment, the pressure force type label includes no pressure, light pressure or heavy pressure. It is also possible to set the demand training corresponding model so that the neural network model can realize more pressure force type labels based on the solution provided in the embodiment of the present application.

[0128] For example, force type labels such as short light press and long heavy press may also be included, but are not limited to this. Furthermore, the neural network model may also be trained and combined with certain strategies to output pressure range labels corresponding to force levels, for example, a pressure range of 2N-3N, but this is not limited to this embodiment of the present application.

[0129] It is worth noting that the processing during model training corresponds to the processing during model use. For more details or implementation, please refer to the relevant description of the aforementioned embodiments. The difference includes that the model input data at this time is training sample data, which will not be explained in detail here.

[0130] As can be seen, this embodiment provides a neural network model trained based on a neural network training method that can detect the degree of pressure under corresponding pressure states. Furthermore, it can effectively eliminate differences in various scenarios (such as temperature and frequency) and different degrees of pressure, covering a variety of special scenarios such as high and low temperatures and wet fingers, and has stronger generalization and accuracy.

[0131] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0132] The above mainly describes the method part provided in the embodiment of the present application, and then describes the embodiments of the relevant devices, media and equipment provided in the embodiment of the present application.

[0133] In one embodiment, a pressure detection device is provided, which corresponds to the pressure detection method in the above embodiment. Figure 11 As shown, the pressure detection device includes a processing module 101 and an input module 102. The functional modules are described in detail as follows: An acquisition module 101 is configured to acquire multi-time-step sequence data formed by single-point features of multiple consecutive time steps, where the single-point feature of each time step is an intensity characteristic value of an ultrasonic echo signal of a single time step, and the ultrasonic echo signal is an echo signal corresponding to an ultrasonic signal emitted by an ultrasonic array to a pressed area; The processing module 102 is used to pre-process the multi-time step sequence data to obtain model input data; and input the model input data into the trained neural network model so that the neural network model outputs a compression degree result.

[0134] In combination with the above-mentioned device embodiment, in one embodiment, the single-point feature includes the echo intensity mean of the ultrasonic echo signal in a single time step; or, the single-point feature includes the minimum echo intensity value in the ultrasonic echo signal in a single time step.

[0135] In combination with the above device embodiment, in one embodiment, the neural network model includes a first-stage network, a second-stage network, a first classifier, and a second classifier; The model input data outputs a first feature vector after passing through the first stage network, and the first feature vector outputs a pressing state feature representing a pressing state result after passing through the first classifier; The first feature vector outputs a second feature vector after passing through the second stage network. The second feature vector is fused with the pressing state feature output by the first classifier and then input into the second classifier, so that the second classifier outputs a pressing degree feature that represents the pressing degree result.

[0136] In combination with the above device embodiment, in one embodiment, the neural network model includes a first-stage network, a second-stage network, a first classifier, a second classifier, and a target weight module; The model input data is input into the first stage network and then outputs a first feature vector, and the first feature vector is outputted as a pressing state feature representing the pressing state result after passing through the first classifier; The first feature vector outputs a target weight coefficient after passing through the target weight module; The first feature vector is fused with the target weight coefficient and input into the second stage network, so that the second stage network outputs a second feature vector. The second feature vector is fused with the pressing state feature output by the first classifier and input into the second classifier, so that the second classifier outputs a pressing degree feature representing the pressing degree result.

[0137] In combination with the above device embodiment, in one embodiment: The target weight module outputs a target weight coefficient corresponding to the first feature vector using a self-attention mechanism; or; The target weight module obtains the target weight coefficient corresponding to the first eigenvector from a preset weight set, and the preset weight set is pre-constructed based on the compression data, and the compression data includes the intensity characteristic value sequence and pressure value sequence of the ultrasonic echo signal collected during the compression process.

[0138] In combination with the above device embodiment, in one embodiment, the processing module 101 is configured to: The multi-time step sequence data is normalized using a preset empty sampling calibration value and a pressing calibration value to obtain model input data, wherein the pressing calibration value is related to the pressing degree requirement setting.

[0139] In combination with the above device embodiment, in one embodiment, the pressing degree result includes a pressing intensity type or a pressing intensity range.

[0140] In combination with the above device embodiment, in one embodiment, the pressing force type includes no pressing, light pressing, or heavy pressing.

[0141] As can be seen, this embodiment provides a pressure detection device based on a neural network model. Combined with the constructed neural network model, the spatial signal strength feature value of a single time step is used as a single-point feature. The single-point features of multiple consecutive time steps form a multi-time-step sequence as model input data, which is then input into the neural network model for detection. This device can detect the pressure level under corresponding pressure states. Furthermore, it can effectively eliminate differences in various scenarios (such as temperature and frequency) and pressure levels, covering a variety of special scenarios such as high and low temperatures and wet fingers, and has stronger generalization and accuracy.

[0142] In one embodiment, a neural network training device is provided, which corresponds one-to-one to the neural network training method in the above embodiment. Figure 12 As shown, the neural network training device includes an acquisition module 201 and a training module 202. The functional modules are described in detail as follows: An acquisition module 201 is configured to acquire training sample data, wherein the training sample data includes sample multi-time step sequence data and its corresponding pressing state result label and pressing degree result label; The sample multi-time step sequence data is formed by single-point features of multiple consecutive time steps, and the single-point feature of each time step is a signal characteristic value of a sample ultrasonic echo signal of a single time step, and the sample ultrasonic echo signal is an echo signal corresponding to the ultrasonic array transmitting an ultrasonic signal to the pressing area; The training module 202 is used to train the neural network based on the training sample data until a neural network model that meets the preset model conditions is obtained.

[0143] In combination with the above-mentioned training device embodiment, in one embodiment, the single-point feature includes the echo intensity mean of the sample ultrasonic echo signal in a single time step; or, the single-point feature includes the minimum echo intensity value in the sample ultrasonic echo signal in a single time step.

[0144] In conjunction with the above training device embodiment, in one embodiment, the neural network includes a first-stage network, a second-stage network, a first classifier, and a second classifier; The training sample data is used as input to the first-stage network, the output of the first-stage network is used to input the first classifier and the second-stage network, and the first classifier is used to output a pressing state feature representing a pressing state result; The output of the second-stage network is fused with the output of the first classifier and then input into the second classifier. The second classifier is used to output a compression degree feature representing the compression degree result.

[0145] In conjunction with the above training device embodiment, in one embodiment, the neural network includes a first-stage network, a second-stage network, a first classifier, a second classifier, and a target weight module; The training sample data is used as input to the first-stage network, and the output of the first-stage network is used to input the first classifier and the target weight module, and the first classifier is used to output a pressing state feature representing a pressing state result; The output of the target weight module is fused with the output of the first stage network and input into the second stage network. The output of the second stage network is fused with the output of the first classifier and input into the second classifier. The second classifier is used to output a pressure degree feature representing the pressure degree result.

[0146] In combination with the above-mentioned training device embodiment, in one embodiment: the target weight module is used to use the self-attention mechanism to learn the target weight coefficient corresponding to the output of the first stage network.

[0147] In combination with the above-mentioned training device embodiment, in one embodiment, the sample multi-time step sequence data is obtained by normalization processing using a preset empty sampling calibration value and a press calibration value, and the press calibration value is related to the press degree requirement setting.

[0148] In combination with the above-mentioned training device embodiment, in one embodiment, the pressing degree result label includes a pressing degree type label or a pressing degree range label.

[0149] In combination with the above-mentioned training device embodiment, in one embodiment, the pressure intensity type label includes no pressure, light pressure or heavy pressure.

[0150] For the specific limitations of the above-mentioned device, please refer to the relevant limitations of the method above, which will not be repeated here. Each module in the above-mentioned device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the electronic device or computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0151] In one embodiment, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the press detection method or the neural network training method as described above are implemented.

[0152] Specifically, in one embodiment, Figure 1 As shown, an electronic device is provided, which includes an ultrasonic array, an analog-to-digital converter, a data processing center and a central controller. The ultrasonic array, the analog-to-digital converter and the data processing center are all connected to the central controller, wherein: the ultrasonic array is used to respond to the central controller to transmit ultrasonic signals to the pressing area and receive ultrasonic echo signals; the analog-to-digital converter is used to respond to the central controller to perform digital-to-analog conversion on the ultrasonic echo signals; the data processing center is used to respond to the central controller to implement the steps of the pressing detection method as described in any of the above items.

[0153] For more processing details of the data processing center, please refer to the description of each embodiment of the aforementioned pressure detection method, which will not be repeated here.

[0154] In one embodiment, a readable storage medium is provided, wherein the readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the aforementioned neural network training method or the press detection method are implemented.

[0155] In one embodiment, a computer program product is provided, including a computer program, characterized in that when the computer program is executed by a processor, the steps of the aforementioned neural network training method or the press detection method are implemented.

[0156] Those skilled in the art will appreciate that all or part of the processes in the above-described embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-described methods. In particular, any reference to memory, storage, database, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory.

[0157] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0158] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A pressure detection method, characterized in that: The method comprises: Acquire multi-time-step sequence data formed by single-point features of multiple consecutive time steps, where the single-point feature of each time step is an intensity characteristic value of an ultrasonic echo signal of a single time step, and the ultrasonic echo signal is an echo signal corresponding to the ultrasonic array transmitting an ultrasonic signal to the pressing area; Preprocessing the multi-time step sequence data to obtain model input data; The model input data is input into the trained neural network model so that the neural network model outputs a compression degree result.

2. The pressure detection method according to claim 1, wherein: The single-point feature includes an average echo intensity value of the ultrasonic echo signal in a single time step; or, the single-point feature includes a minimum echo intensity value in the ultrasonic echo signal in a single time step.

3. The pressure detection method according to claim 1, wherein: The neural network model includes a first-stage network, a second-stage network, a first classifier and a second classifier; The model input data outputs a first feature vector after passing through the first stage network, and the first feature vector outputs a pressing state feature representing a pressing state result after passing through the first classifier; The first feature vector outputs a second feature vector after passing through the second stage network. The second feature vector is fused with the pressing state feature output by the first classifier and then input into the second classifier, so that the second classifier outputs a pressing degree feature that represents the pressing degree result.

4. The pressure detection method according to claim 1, wherein: The neural network model includes a first-stage network, a second-stage network, a first classifier, a second classifier, and a target weight module; The model input data is input into the first stage network and then outputs a first feature vector, and the first feature vector is outputted by the first classifier to represent a pressing state feature representing a pressing state result; The first feature vector outputs a target weight coefficient after passing through the target weight module; The first feature vector is fused with the target weight coefficient and input into the second stage network, so that the second stage network outputs a second feature vector. The second feature vector is fused with the pressing state feature output by the first classifier and input into the second classifier, so that the second classifier outputs a pressing degree feature representing the pressing degree result.

5. The pressure detection method according to claim 4, wherein: The target weight module outputs a target weight coefficient corresponding to the first feature vector using a self-attention mechanism; or; The target weight module obtains the target weight coefficient corresponding to the first eigenvector from a preset weight set, and the preset weight set is pre-constructed based on the compression data, and the compression data includes the intensity characteristic value sequence and pressure value sequence of the ultrasonic echo signal collected during the compression process.

6. The pressure detection method according to any one of claims 1 to 5, characterized in that: Preprocessing the multi-time step sequence data to obtain model input data includes: The multi-time step sequence data is normalized using a preset empty sampling calibration value and a compression calibration value to obtain model input data, wherein the compression calibration value is related to the compression degree requirement setting.

7. The pressure detection method according to any one of claims 1 to 5, characterized in that: The pressing degree result includes a pressing intensity type or a pressing intensity range.

8. The pressure detection method according to claim 7, wherein: The pressing force type includes no pressing, light pressing, or heavy pressing.

9. A neural network training method, characterized in that: The method comprises: Acquire training sample data, the training sample data including sample multi-time-step sequence data and its corresponding compression state result label and compression degree result label, the sample multi-time-step sequence data being formed by single-point features of multiple consecutive time steps, each single-point feature of the time step being a signal feature value of a sample ultrasonic echo signal of a single time step, the sample ultrasonic echo signal being an echo signal corresponding to the ultrasonic array transmitting an ultrasonic signal to the compression area; Based on the training sample data, the neural network is trained until a neural network model that meets the preset model conditions is obtained.

10. The neural network training method according to claim 9, characterized in that: The single-point feature includes an average echo intensity value of the sample ultrasonic echo signal in a single time step; or, the single-point feature includes a minimum echo intensity value in the sample ultrasonic echo signal in a single time step.

11. The neural network training method according to claim 9, characterized in that: The neural network includes a first-stage network, a second-stage network, a first classifier, and a second classifier; The training sample data is used as input to the first-stage network, the output of the first-stage network is used to input the first classifier and the second-stage network, and the first classifier is used to output a pressing state feature representing a pressing state result; The output of the second-stage network is fused with the output of the first classifier and then input into the second classifier. The second classifier is used to output a compression degree feature representing the compression degree result.

12. The neural network training method according to claim 9, characterized in that: The neural network includes a first-stage network, a second-stage network, a first classifier, a second classifier, and a target weight module; The training sample data is used as input to the first-stage network, and the output of the first-stage network is used to input the first classifier and the target weight module, and the first classifier is used to output a pressing state feature representing a pressing state result; The output of the target weight module is fused with the output of the first stage network and input into the second stage network. The output of the second stage network is fused with the output of the first classifier and input into the second classifier. The second classifier is used to output a pressure degree feature representing the pressure degree result.

13. The neural network training method according to claim 12, wherein: The target weight module is used to learn the target weight coefficient corresponding to the output of the first stage network using the self-attention mechanism.

14. The neural network training method according to any one of claims 9 to 13, characterized in that: The sample multi-time step sequence data is obtained by normalizing using a preset empty sampling calibration value and a press calibration value, and the press calibration value is related to the press degree requirement setting.

15. The neural network training method according to any one of claims 9 to 13, characterized in that: The pressing degree result label includes a pressing force type label or a pressing force range label.

16. The neural network training method according to claim 15, characterized in that: The pressure intensity type label includes no press, light press, or hard press.

17. A pressure detection device, characterized in that: The device comprises: An acquisition module is used to acquire multi-time-step sequence data formed by single-point features of multiple consecutive time steps, where the single-point feature of each time step is the intensity characteristic value of the ultrasonic echo signal of a single time step, and the ultrasonic echo signal is the echo signal corresponding to the ultrasonic array transmitting the ultrasonic signal to the pressing area; The processing module is used to preprocess the multi-time step sequence data to obtain model input data; and input the model input data into the trained neural network model so that the neural network model outputs a pressure degree result.

18. An electronic device, characterized in that: The electronic device includes an ultrasonic array, an analog-to-digital converter, a data processing center, and a central controller, wherein the ultrasonic array, the analog-to-digital converter, and the data processing center are all connected to the central controller, wherein: The ultrasonic array is configured to transmit ultrasonic signals to the pressing area in response to the central controller and receive ultrasonic echo signals; The analog-to-digital converter is configured to perform digital-to-analog conversion on the ultrasonic echo signal in response to the central controller; The data processing center responds to the central controller to implement the steps of the pressure detection method according to any one of claims 1 to 8.

19. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the neural network training method according to any one of claims 9 to 16 when executing the computer program.

20. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 16 are implemented.

Images