Pressing area detection method and device, electronic equipment, storage medium and program product

Abstract

The embodiment of the invention provides a pressing area detection method and device, electronic equipment, a storage medium and a program product. The pressing area detection method comprises the following steps: obtaining pressing data obtained by performing pressing operation on a touch screen by a user; inputting the pressing data into a machine learning model for central point prediction to obtain predicted central coordinates of the pressing area; and performing form prediction according to the pressing data and the prediction center coordinate to obtain a prediction form parameter of the pressing area. According to the scheme, the detection effect on the pressing area of the touch screen is good.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of touch technology, and in particular to a method, device, electronic device, storage medium and program product for detecting a pressed area. Background Art

[0002] With the rapid popularization and continuous development of mobile devices such as smartphones, touch screens have become an indispensable part of them, and pressing the touch screen to touch has become one of the core ways of user interaction. The detection of the pressed area is important for the user experience of the touch screen. For example, touch control can be performed by detecting the parameters of the pressed area. The detection accuracy of the pressed area of ​​the touch screen in the related technology is poor. Therefore, how to improve the accuracy of the pressed area detection of the touch screen has become a problem that needs to be solved urgently. Summary of the invention

[0003] In view of this, embodiments of the present application provide a pressed area detection method, device, electronic device, storage medium, and program product.

[0004] According to the first aspect of an embodiment of the present application, a pressed area detection method is provided, including: obtaining pressed data obtained when a user presses a touch screen; inputting the pressed data into a machine learning model to perform center point prediction to obtain predicted center coordinates of the pressed area; and performing morphological prediction based on the pressed data and the predicted center coordinates to obtain predicted morphological parameters of the pressed area.

[0005] In some optional embodiments, the inputting the press data into a machine learning model for center point prediction to obtain predicted center coordinates of the press area includes: inputting the press data into a machine learning model for feature extraction to obtain feature information of the press data, and performing center point prediction based on the feature information of the press data to obtain predicted center coordinates of the press area; performing morphological prediction based on the press data and the predicted center coordinates to obtain predicted morphological parameters of the press area includes: performing morphological prediction based on the feature information of the press data and the preset center coordinates to obtain predicted morphological parameters of the press area.

[0006] In some optional embodiments, the predicted morphological parameters include at least one of a major axis parameter, a minor axis parameter, a major axis inclination parameter, and a minor axis inclination parameter of a shape corresponding to the pressed area.

[0007] In some optional embodiments, the obtaining of pressing data obtained by a user pressing a touch screen includes: obtaining original capacitance data obtained by a user pressing the touch screen; performing expansion processing on capacitance data corresponding to a pressed area in the original capacitance data to obtain target capacitance data, and determining the target capacitance data as the pressing data.

[0008] In some optional embodiments, the method further includes: displaying a given pressing area on the touch screen to prompt the user to perform a pressing operation on the given pressing area; obtaining a second pressing data sample obtained by the user pressing the given pressing area on the touch screen, wherein the second pressing data sample is used to fine-tune the machine learning model to obtain a machine learning model for the user.

[0009] In some optional embodiments, the method further includes: controlling a terminal device equipped with the touch screen according to the predicted center coordinates of the pressed area and / or the predicted morphological parameters.

[0010] According to a second aspect of an embodiment of the present application, a model training method is provided, including: obtaining a first training sample for training a machine learning model, wherein the first training sample includes a first pressing data sample obtained by a user pressing a touch screen, a first center point true value corresponding to the first pressing data sample, and a first morphological parameter true value corresponding to the first pressing data sample; inputting the first training sample into the machine learning model to perform center point prediction to obtain a first predicted center coordinate sample of the pressed area, and performing morphological prediction based on the first pressing data sample and the first predicted center coordinate sample to obtain a first predicted morphological parameter sample of the pressed area; training the machine learning model based on the first predicted center coordinate sample, the first center point true value, the first predicted morphological parameter sample and the first morphological parameter true value.

[0011] In some optional embodiments, the step of inputting the first training sample into the machine learning model for center point prediction to obtain a first predicted center coordinate sample of the pressed area includes: inputting the first training sample into the machine learning model, performing feature extraction on the first pressing data sample in the first training sample to obtain feature information of the first pressing data sample, performing center point prediction based on the feature information of the first pressing data sample to obtain a first predicted center coordinate sample of the pressed area; the step of performing morphological prediction based on the first pressing data sample and the first predicted center coordinate sample to obtain a first predicted morphological parameter sample of the pressed area includes: performing morphological prediction based on the feature information of the first pressing data sample and the first predicted center coordinate sample to obtain a first predicted morphological parameter sample of the pressed area.

[0012] In some optional embodiments, the machine learning model includes: a feature extraction part, a first perception layer part and a second perception layer part; the feature extraction of the first press data sample in the first training sample to obtain feature information of the first press data sample, and the center point prediction based on the feature information of the first press data sample to obtain a first predicted center coordinate sample of the press area, including: extracting features of the first press data sample in the first training sample through the feature extraction part to obtain feature information of the first press data sample, and predicting the center point of the feature information of the first press data sample through the first perception layer part to obtain a first predicted center coordinate sample of the press area; the morphology prediction based on the feature information of the first press data sample and the first predicted center coordinate sample to obtain a first predicted morphological parameter sample of the press area includes: performing morphology prediction on the feature information of the first press data sample and the first predicted center coordinate sample through the second perception layer part to obtain a first predicted morphological parameter sample of the press area.

[0013] In some optional embodiments, the feature extraction part includes a linear transformation layer, at least one convolution layer and at least one pooling layer; the step of subjecting the first compression data sample in the first training sample to feature extraction by the feature extraction part to obtain feature information of the first compression data sample includes: subjecting the first compression data sample in the first training sample to the linear transformation layer, at least one convolution layer and at least one pooling layer of the feature extraction part to sequentially perform corresponding linear transformation processing, convolution processing and pooling processing to obtain feature information of the first compression data sample.

[0014] In some optional embodiments, the first predicted morphological parameter sample includes at least one of a long-axis parameter sample, a short-axis parameter sample, a long-axis inclination parameter sample, and a short-axis inclination parameter sample of the shape corresponding to the pressed area.

[0015] In some optional embodiments, the first pressing data sample obtained by the user pressing the touch screen is obtained in the following manner: obtaining an original capacitance data sample, the original capacitance data sample being a capacitance data sample used to characterize the capacitance data sample obtained by the user pressing the touch screen; expanding the capacitance data corresponding to the pressed area in the original capacitance data sample to obtain a target capacitance data sample, and determining the target capacitance data sample as the first pressing data sample.

[0016] In some optional embodiments, the method also includes: obtaining a second training sample, wherein the second training sample includes: a second pressing data sample obtained by a user pressing a given pressing area on the touch screen, a second center point true value corresponding to the second pressing data sample, and a second morphological parameter true value corresponding to the second pressing data sample; and using the second training sample to fine-tune the machine learning model to obtain a machine learning model for the user.

[0017] In some optional embodiments, the second center point true value is the center point coordinates of the shape corresponding to the given pressed area, and the second morphological parameter true value is the morphological parameter of the shape corresponding to the given pressed area.

[0018] In some optional embodiments, the use of the second training sample to fine-tune the machine learning model includes: inputting the second training sample into the machine learning model to perform center point prediction to obtain a second predicted center coordinate sample of the given pressing area, and performing morphology prediction based on the second pressing data sample and the second predicted center coordinate sample to obtain a second predicted morphological parameter sample of the given pressing area; fine-tuning the machine learning model based on the second predicted center coordinate sample, the second center point true value, the second predicted morphological parameter sample and the second morphological parameter true value.

[0019] According to the third aspect of an embodiment of the present application, a pressed area detection device is provided, including: an acquisition module, used to obtain pressed data obtained by a user pressing a touch screen; a prediction module, used to input the pressed data into a machine learning model for center point prediction, to obtain predicted center coordinates of the pressed area; and, based on the pressed data and the predicted center coordinates, a morphological prediction is performed to obtain predicted morphological parameters of the pressed area.

[0020] According to a fourth aspect of an embodiment of the present application, a model training device is provided, including: a sample acquisition module, used to acquire a first training sample for training a machine learning model, wherein the first training sample includes a first pressing data sample obtained by a user pressing a touch screen, a first center point true value corresponding to the first pressing data sample, and a first morphological parameter true value corresponding to the first pressing data sample; a parameter prediction module, used to input the first training sample into the machine learning model for center point prediction to obtain a first predicted center coordinate sample of a pressed area, and to perform morphological prediction based on the first pressing data sample and the first predicted center coordinate sample to obtain a first predicted morphological parameter sample of the pressed area; a model training module, used to train the machine learning model based on the first predicted center coordinate sample, the first center point true value, the first predicted morphological parameter sample and the first morphological parameter true value.

[0021] According to the fifth aspect of the embodiments of the present application, there is provided an electronic device, comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; the memory is used to store a computer program; and the processor is used to execute the method described in the first or second aspect above by running the computer program stored in the memory.

[0022] According to a sixth aspect of an embodiment of the present application, a computer storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method described in the first aspect or the second aspect is implemented.

[0023] According to a seventh aspect of an embodiment of the present application, a computer program product is provided, including a computer program, which, when executed by a processor, implements the method described in the first aspect or the second aspect.

[0024] According to the technical solution provided in the embodiments of the present application, since the pressing data obtained by the user pressing the touch screen can be obtained, and then the pressing data is input into the machine learning model for center point prediction to obtain the predicted center coordinates of the pressed area, and the morphology prediction is performed based on the pressing data and the predicted center coordinates to obtain the predicted morphological parameters of the pressed area, therefore, on the one hand, the center point coordinates and morphological parameters of the pressed area of ​​the touch screen by the user can be effectively predicted through the machine learning model, thereby effectively realizing pressed area detection; and, on the other hand, since accurate center point prediction is first performed through the machine learning model to obtain relatively accurate predicted center coordinates, and then the machine learning model is used to perform morphology prediction based on the pressing data and the accurate predicted center point coordinates, the accuracy of the prediction of the morphological parameters of the pressed area can be effectively improved, thereby effectively ensuring the accuracy of the results obtained by pressed area detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0026] Figure 1A The present invention is a workflow diagram of a technical solution for detecting a pressed area of ​​a touch screen in the related art.

[0027] Figure 1B It is a schematic diagram of calculating the elliptical equipotential lines of the pressing area based on the pressing data in the related art.

[0028] Figure 1C A schematic diagram of an exemplary capacitance detection circuit network of a touch screen.

[0029] Figure 2 This is a flowchart of the steps of a pressing area detection method according to an embodiment of the present application.

[0030] Figure 3 This is a flowchart of some optional steps of step S102 in an embodiment of the present application.

[0031] Figure 4 A schematic diagram of obtaining compression data from raw capacitance data is shown.

[0032] Figure 5 An example workflow diagram for implementing pressed area detection through a machine learning model is shown.

[0033] Figure 6 The flowchart is an example application of the press detection solution of the embodiment of the present application.

[0034] Figure 7 A flowchart of the steps of a model training method according to an embodiment of the present application.

[0035] Figure 8 This is a flow chart of an optional method for obtaining a first press data sample according to an embodiment of the present application.

[0036] Fig. 9 A schematic diagram of a prompt area of ​​an example of an embodiment of the present application.

[0037] Fig.10 An example workflow diagram for training a machine learning model is shown.

[0038] Fig.11 A flowchart of an optional method for fine-tuning a machine learning model using a second training sample.

[0039] Fig.12 Schematic diagram of a pressed area detection device according to an embodiment of the present application.

[0040] Fig.13 A schematic diagram of a model training device according to an embodiment of the present application.

[0041] Fig.14 A schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the embodiments of the present application should fall within the scope of protection of the embodiments of the present application.

[0043] With the rapid popularization and continuous development of mobile devices such as smartphones, touch screens have become an indispensable part of them. Pressing the touch screen to touch has become one of the core ways of user interaction. The detection of the pressed area is important for the user experience of the touch screen. For example, touch control can be performed by detecting the parameters of the pressed area.

[0044] The detection accuracy of the pressed area of ​​the touch screen in the related art is poor. For example, the user uses different fingers (such as the thumb or index finger, etc.) and different parts (such as the fingertip, the pulp or the side of the finger, etc.) to press the touch screen at different angles, and each different pressing method may cause the touch screen to form a different pressed area shape. For example, when the user presses with the fingertip, the corresponding pressing data is usually approximately circular (it should be understood that the circle can be regarded as a special ellipse), and when the user presses with the pulp or the side of the finger, the pressing data may be approximately various ellipses. Therefore, the pressed area can usually be treated as an elliptical area to calculate various parameters of the ellipse. However, in the related art, some fixed algorithms are used to calculate the parameters of the pressing area, while the actual pressing data of the user has such a variable form that the pressing area parameters calculated by the fixed algorithm will be deviated. Even if the user tries to press the same position on the touch screen, the calculated pressing area parameters (taking the ellipse as an example, including but not limited to the coordinates of the center of the ellipse, etc.) may deviate from expectations, thus making the pressing area detection effect of the touch screen poor, affecting the user's experience of using the touch screen. In addition, there are individual differences among different users. The finger shapes of different users are different (including differences in length, width and thickness), and the usage habits of pressing the touch screen are also different. This results in that even if the same position is pressed, the calculation results of the pressing area parameters under the fixed algorithm will not be consistent with expectations due to the individual differences of different users, thus further making the pressing area detection effect of the touch screen poor, further affecting the user's experience of using the touch screen. The reduction in the above-mentioned pressed area detection effect will not only affect the accuracy of touch operation, but also lead to a reduction in user experience and satisfaction (for example, some examples that affect user experience may be: when users press the touch screen to input text, they frequently make mistakes and need to re-enter, etc.).

[0045] For example, refer to Figure 1A FIG. 1 shows a workflow diagram of a technical solution for detecting a pressed area of ​​a touch screen in the related art. Figure 1A As shown, taking the case where the pressed area of ​​the touch screen is regarded as an elliptical area and the touch screen is a capacitive touch screen as an example, the technical solution in the related technology can first obtain the pressing data (capacitance data, such as mutual capacitance data) at a certain moment to determine whether there is a pressing event on the touch screen. If so, the pressing data corresponding to the pressed area is determined from the above-mentioned pressing data, and the elliptical equipotential lines are calculated. Then, according to the calculation results of the elliptical equipotential lines, the parameters of the pressed area (for example, including but not limited to the coordinates of the center point of the elliptical area, etc.) are calculated and output.

[0046] Another example Figure 1BAs shown in FIG. 1 , a schematic diagram of calculating elliptical equipotential lines of a pressed area based on pressed data is shown. The elliptical equipotential lines can be considered as fitted elliptical contour lines, which are roughly determined by a set of equipotential points, and the search results of the equipotential point set are mainly determined by the equipotential line threshold. When calculating elliptical equipotential lines, the equipotential point set is usually calculated according to the linear interpolation principle (for example, assuming the equipotential line threshold = 75, then linear interpolation can be used to obtain Figure 1B The “×” in the figure represents equipotential points, which form a set of equipotential points that approximates the elliptic curve (such as Figure 1B In the figure, CD is the major axis of the ellipse, and AB is the minor axis of the ellipse).

[0047] However, on the one hand, the equipotential line threshold in the above scheme is difficult to determine. The equipotential line threshold is usually in a fixed proportional relationship with the peak value in the current pressed area, and this fixed ratio needs to be pre-set by engineers based on the debugging data set and personal experience. However, the debugging data set may be too small, and personal experience usually has a certain subjective influence, which may result in the selection of the equipotential line threshold being not the optimal choice, thereby affecting the performance of ellipse recognition, thereby reducing the accuracy of pressed area detection in the technical scheme of the above-mentioned related technology. On the other hand, in the above scheme, when calculating the elliptical equipotential lines, linear interpolation will also introduce errors. For example Figure 1C As shown, the touch sensing unit of the touch screen may include multiple drivers (TX) and receivers (RX), and the capacitance detection circuit between TX and RX may be approximately simplified into a "grid shape". Usually, there are approximately equal spacings between TX and TX, and between RX and RX, (e.g. Figure 1C As shown, multiple drivers TX are TX0~TXend, multiple receivers RX are RX0~RXend, multiple drivers TX are each connected to a driving signal line (also called a driving electrode), multiple receivers RX are each connected to a receiving signal line (also called a receiving electrode), multiple driving signal lines and multiple receiving signal lines form a capacitance detection circuit network (mutual capacitance type), taking the 5 driving signal lines and 5 receiving signal lines in the figure as an example (only for easy understanding), the changes in mutual capacitance data C11~C55 (only for example) caused by touch in various areas on the touch screen 100 can be detected and collected. ). When an actual pressing event occurs, the event may occur at the junction of TX and RX, or it may occur between the gaps. Due to the uneven distribution of the electric field in the touch sensor layer, the gaps (such as Figure 1C The pressed capacitance data (assuming Touch mid ) is not necessarily equal to the surrounding boundary positions (such as Figure 1C The b1, b2, b3, and b4 positions in the figure are all indicated by “×”) (assuming that the Touch i,i∈{1,2,3,4}) (i.e. ). That is to say, the pressed capacitance data at a certain position is not linearly related to the surrounding data. Because of the existence of this nonlinear relationship, the result of using the linear interpolation method to obtain the equipotential point set in the above solution may not be accurate. Therefore, the technical solution of the above related technology cannot further guarantee the accuracy of pressed area detection. It should be noted that the above Figure 1A and Figure 1B The technical solutions of the related technologies shown may not be prior arts.

[0048] It should be understood that a new pressed area detection scheme is provided in the embodiment of the present application, which can realize the detection of the user's pressed area and the determination of related parameters through a machine learning model, and at least improve the accuracy of the pressed area detection of the touch screen.

[0049] The specific implementation of the embodiment of the present application is further explained below in conjunction with the accompanying drawings of the embodiment of the present application.

[0050] Figure 2 A step flow chart of a method for detecting a pressed area according to an embodiment of the present application. According to a first aspect of an embodiment of the present application, a method for detecting a pressed area is provided, referring to Figure 2 As shown, the method includes steps S102, S104 and S106, specifically:

[0051] S102: Acquire pressing data obtained by the user performing a pressing operation on the touch screen.

[0052] It should be noted that the pressed area detection method in the embodiment of the present application can be applied to the pressed area detection of any touch screen. For example, the touch screen can be a capacitive touch screen, a resistive touch screen, etc. For ease of description, the capacitive touch screen can be taken as an example below.

[0053] Optionally, the pressed area detection method in the embodiment of the present application can be used in the use phase of the touch screen, that is, the phase in which the touch screen is used after it has been manufactured and installed in the terminal device.

[0054] In the embodiment of the present application, the type of the pressed data can be determined according to the type of the touch screen. For example, if the touch screen is a capacitive touch screen, the pressed data can be capacitance data. The capacitance data here can be mutual capacitance data or self capacitance data, depending on whether the touch screen is based on the mutual capacitance principle or the self capacitance principle. Taking the capacitance data as mutual capacitance data as an example, for example, each position on the touch screen can collect mutual capacitance data (which can be combined with the aforementioned Figure 1C The example explanation is for understanding, Figure 1CThe dotted box in the figure roughly shows the shape of the touch screen, which is of course only an example). When the user does not touch the touch screen, the mutual capacitance data collected at various positions on the touch screen fluctuates very little, and when the user presses a part of the touch screen to achieve touch, the mutual capacitance data collected through the part changes, so that the difference in mutual capacitance data compared to when not pressed increases significantly. Therefore, by detecting the difference in mutual capacitance data (for example, the difference with a reference value (for example, 0)), it can be determined whether the touch screen is pressed and touched, thereby facilitating accurate detection of the pressed area.

[0055] In some optional embodiments, referring to Figure 3 In the flowchart shown, step S102 may include steps S1022 and S1024, specifically:

[0056] S1022: Acquire raw capacitance data obtained by the user pressing the touch screen.

[0057] When collecting training samples, the user performs a pressing operation on the touch screen, and the touch screen can detect raw capacitance data according to the pressing operation. The raw capacitance data may at least include capacitance data corresponding to the pressed area.

[0058] S1024: performing expansion processing on the capacitance data corresponding to the pressed area in the original capacitance data to obtain target capacitance data, and determining the target capacitance data as pressed data.

[0059] Optionally, a minimum rectangular frame including the capacitance data corresponding to the pressed area can be determined in the original capacitance data, and then the minimum rectangular frame can be expanded, so as to expand the capacitance data corresponding to the pressed area in the original capacitance data. The capacitance data in the expanded rectangular frame is taken out to obtain the target capacitance data, which is determined as the pressed data.

[0060] It should be understood that the above-mentioned expansion processing can be to expand at least one of the upper, lower, left and right sides of the minimum rectangular frame outward to achieve the expansion processing. For example, if one side of the minimum rectangular frame coincides with the corresponding position on the edge of the touch screen, the remaining three sides can be expanded outward to achieve the expansion processing; if two sides of the minimum rectangular frame coincide with the corresponding positions on the edge of the touch screen, the remaining two sides can be expanded outward to achieve the expansion processing; if the minimum rectangular frame is located in the middle as a whole, the four sides can be expanded outward to achieve the expansion processing. The distance of each side expanding outward can be set as needed, and the distances of each side expanding outward can be the same or different. As long as it can meet the actual needs, no limitation is made here.

[0061] For example Figure 4 FIG. 4 shows a schematic diagram of obtaining compression data from raw capacitance data. Figure 4 The P frame is the minimum rectangular frame including the capacitance data corresponding to the pressed area, and the Q frame is a rectangular frame after the four sides (p1, p2, p3, p4 sides) of the minimum rectangular frame (P frame) are expanded outward. Since the capacitance data in the Q frame is more than that in the P frame, it can effectively avoid the omission of the capacitance data corresponding to the pressed area. Using the target capacitance data obtained by such expansion processing as the pressed data can improve the accuracy of subsequent pressed area detection based on the pressed data.

[0062] Based on this, in the embodiment of the present application, through the optional implementation of the above steps S1022 to S1024, on the one hand, it is possible to effectively avoid the omission of the capacitance data corresponding to the pressed area, and use the target capacitance data obtained by such expansion processing as the pressed data, which can improve the accuracy of subsequent pressed area detection based on the pressed data; on the other hand, since the target capacitance data obtained by the expansion processing is used as the pressed data for processing through the machine learning model, without having to input all of the original capacitance data of the touch screen into the machine learning model for processing, it is also possible to effectively reduce the amount of calculation, thereby improving the efficiency of press detection, and reducing the interference of excessive useless data, further improving the detection accuracy.

[0063] S104: inputting the pressing data into the machine learning model to perform center point prediction to obtain the predicted center coordinates of the pressing area; and

[0064] S106: Perform morphology prediction based on the pressing data and the predicted center coordinates to obtain predicted morphology parameters of the pressing area.

[0065] It should be noted that step S106 is implemented through a machine learning model, just like step S104, that is, the machine learning model performs morphological prediction based on the pressing data and the predicted center coordinates to obtain the predicted morphological parameters of the pressing area. The machine learning model can be considered as a "two-stage prediction model", that is, it can predict the predicted center coordinates and predicted morphological parameters of the pressing area respectively.

[0066] Based on this, in the embodiment of the present application, the pressed area detection scheme through the above steps S102 to S106 can obtain the pressing data obtained by the user pressing the touch screen, and then input the pressing data into the machine learning model for center point prediction to obtain the predicted center coordinates of the pressed area, and perform morphological prediction based on the pressing data and the predicted center coordinates to obtain the predicted morphological parameters of the pressed area. Therefore, on the one hand, the center point coordinates and morphological parameters of the user's pressed area on the touch screen can be effectively predicted through the machine learning model, thereby effectively realizing pressed area detection; and, on the other hand, since accurate center point prediction is first performed through the machine learning model to obtain relatively accurate predicted center coordinates, and then the machine learning model is used to perform morphological prediction based on the pressing data and the accurate predicted center point coordinates, the accuracy of the prediction of the morphological parameters of the pressed area can be effectively improved, thereby effectively ensuring the accuracy of the results obtained by the pressed area detection.

[0067] In addition, due to the use of machine learning models, it is not only convenient to use, but also has better development and debugging efficiency, is conducive to maintenance and optimization, has better versatility, and is also conducive to reducing the extra time and energy invested by R&D personnel before and after development is completed.

[0068] It should be understood that in the technical solution of the embodiment of the present application, first predicting the center point coordinates is crucial for the detection of the pressed area. This is because the predicted center coordinates are first predicted, and then the morphology is predicted based on the pressed data and the predicted center coordinates to obtain the predicted morphology parameters, which is conducive to reducing the negative impact of the nonlinearity of the pressed data on the predicted parameters (which can be combined with the above Figure 1C Related content understanding), thereby effectively improving the accuracy of the prediction of the morphological parameters of the pressed area and improving the accuracy of the pressed area detection.

[0069] The machine learning model in the embodiment of the present application can be a pre-trained machine learning model. The machine learning model is implemented based on artificial intelligence technology. The machine learning model can be of any type. For example, it can be a convolutional neural network model (for example, including but not limited to AlexNet, Inception, VGG, ResNet, MobileNet, FCN, U-Net, DeepLab, BiSeNet, R-CNN, Yolo, SSD, RetinaNet and other models), a deep language model (for example, including but not limited to LSTM, GRU, BiLSTM, Transformer, MarianMT, BERT, RoBERTa, XLNet, GPT, T5, S4, Mamba and other models), or a hybrid variant model based on the above two (for example, including but not limited to ViT, DeiT, ConViT, SwinTransformer, TransUNet, Swin Unet, DETR, Deformable DETR, ConvS2S, ConvBERT and other models). Regarding the training process of the machine learning model, there is no limitation in the embodiment of the present application. For example, the model training method of the second aspect below can be used for training. Regarding the model training part, it will be described in detail below, so I will not go into details here.

[0070] Optionally, when the pressed area is a circular area, the predicted center coordinates may be the center coordinates of the shape corresponding to the pressed area, and the predicted morphological parameters may be circular diameter parameters, radius parameters, etc. It is understandable that a circle may also be considered as a special ellipse with equal major and minor axes.

[0071] When the pressed area is treated as an ellipse area, in some optional embodiments, the predicted center coordinates may be the center coordinates of the ellipse corresponding to the pressed area. The ellipse center is the intersection of the major axis and the minor axis of the ellipse.

[0072] It should be understood that by determining the above predicted center coordinates, the center coordinates of the pressed area can be accurately detected, so as to improve the user experience of the touch screen by using the center coordinates.

[0073] In some optional embodiments, the predicted morphological parameters include at least one of a major axis parameter, a minor axis parameter, a major axis inclination parameter, and a minor axis inclination parameter of a shape corresponding to the pressed area.

[0074] It should be understood that by determining at least one of the above predicted morphological parameters, it is convenient to accurately detect the morphological parameters of the pressed area, so as to improve the user experience of the touch screen by using these morphological parameters.

[0075] Optionally, when the pressed area is treated as an elliptical area for processing, the predicted morphological parameters may include at least one of a major axis parameter, a minor axis parameter, a major axis inclination parameter, and a minor axis inclination parameter of the ellipse.

[0076] The long axis inclination parameter may be the angle between the long axis and the longitudinal axis or the transverse axis of the coordinate system. Generally speaking, the transverse axis and the longitudinal axis of the coordinate system correspond to the length direction or the width direction of the touch screen, or correspond to the width direction or the length direction of the touch screen, respectively. Therefore, the long axis inclination parameter may also be considered as the angle between the long axis and the length direction or the width direction of the touch screen.

[0077] The above-mentioned minor axis inclination parameter may be the angle between the minor axis and the longitudinal axis or the transverse axis of the coordinate system. Similarly, the minor axis inclination parameter may also be considered as the angle between the minor axis and the length direction or the width direction of the touch screen.

[0078] For instance, taking an example, the longitudinal axis of the coordinate system corresponds to the length direction of the touch screen, and the long-axis inclination parameter can be determined as the angle between the long axis and the longitudinal axis of the coordinate system (it can also be considered as the angle between the long axis and the length direction of the touch screen); the short-axis inclination parameter can be determined as the angle between the short axis and the longitudinal axis of the coordinate system (it can also be considered as the angle between the short axis and the length direction of the touch screen).

[0079] In some optional embodiments, the pressed area detection method of the embodiment of the present application further includes: controlling a terminal device equipped with a touch screen according to the predicted center coordinates and / or predicted morphological parameters of the pressed area.

[0080] Therefore, after completing the pressed area detection in the embodiment of the present application and obtaining accurate predicted center coordinates and predicted morphological parameters, the predicted center coordinates and predicted morphological parameters can be utilized, for example, to achieve control of the terminal device so as to improve the user experience of using the touch screen and the terminal device.

[0081] It should be understood that in the embodiments of the present application, the control methods for the terminal device equipped with a touch screen according to the predicted center coordinates and / or predicted morphological parameters of the pressed area can be various, and appropriate control can be implemented as needed. For example, it can include but is not limited to: display control of the touch screen of the terminal device (for example, in some examples of display control, a certain interface element displayed on the touch screen can be controlled to adjust the shape, zoom, rotate, etc., or the touch screen can be controlled to display different light and shadow effects, or the display brightness of the touch screen can be controlled, or the interface of the touch screen can be controlled to slide up and down, etc.), and control of the volume of the terminal device.

[0082] For example, the following examples may be used:

[0083] In some optional application methods, the results of the pressed area detection can be combined with the fingerprint unlocking to realize dynamic feedback of the fingerprint unlocking: for example, in some under-screen fingerprint unlocking systems, a fixed fingerprint entry prompt area (such as a circle or other shape) is usually displayed on the screen (touch screen). The predicted center coordinates and predicted morphological parameters determined above are conducive to a more dynamic and cool unlocking experience. For example, the shape of the fingerprint entry prompt area can be adjusted by the predicted center coordinates and predicted morphological parameters, so that the fingerprint entry prompt area can be deformed as the finger is pressed to match the actual pressing shape. In this way, the display control of the touch screen fingerprint unlocking is realized, which is not only more visually in line with the user operation, but also enhances the overall attractiveness of the product. In addition, in some implementations, when the fingerprint is unlocked, the touch screen can be controlled to display the corresponding light and shadow effects according to the predicted center coordinates and / or predicted morphological parameters, so that the unlocking effect is cooler, thereby improving the user experience (for example, as an example, taking the predicted morphological parameter as the long-axis inclination parameter as an example, when the long-axis inclination parameter is in the range aa1, it corresponds to the light and shadow effect bb1, and when the long-axis inclination parameter is in the range aa2, it corresponds to the light and shadow effect bb2, and the rest can be deduced by analogy. For example, as another example, taking the predicted center point coordinates as an example, when the predicted center point coordinates are in the coordinate range cc1, it corresponds to Light and shadow effect dd1, when the predicted center point coordinates are within the coordinate range cc2, it corresponds to light and shadow effect dd2, and the rest can be deduced by analogy. If a combination of multiple parameters (for example, long-axis inclination parameters and preset center point coordinates) is used to correspond to light and shadow effects, then when the long-axis inclination parameter is within the range aa1 and the predicted center point coordinates are within the coordinate range cc1, it corresponds to light and shadow effect ee1, and when the long-axis inclination parameter is within the range aa2 and the predicted center point coordinates are within the coordinate range cc2, it corresponds to light and shadow effect ee2, and the rest can be deduced by analogy. Of course, the above is only an example and does not constitute any limitation to the embodiments of the present application).

[0084] Some other optional application methods can be combined with the results of the pressed area detection to perform posture estimation: for example, the specific pressing position and pressing posture of the finger can be further inferred by predicting the center coordinates and predicting the morphological parameters, which means that the touch screen and terminal devices (such as mobile phones, etc.) may be able to support richer gesture operations, thereby further optimizing the human-computer interaction experience. For example, some example implementation methods may include: controlling the shape adjustment, scaling, rotation, etc. of a certain interface element displayed on the touch screen, controlling the display brightness of the touch screen of the terminal device, the volume of the terminal device, and controlling the up and down sliding of the touch screen interface. Of course, the above is only an example and does not serve as any limitation to the embodiments of the present application.

[0085] Of course, there are many more application methods, which will not be elaborated here.

[0086] In some optional embodiments, step S104 may include: inputting the press data into a machine learning model for feature extraction to obtain feature information of the press data, performing center point prediction based on the feature information of the press data to obtain predicted center coordinates of the press area; step S106 may include: performing morphology prediction based on the feature information of the press data and preset center coordinates to obtain predicted morphological parameters of the press area.

[0087] Based on this, in the embodiment of the present application, the feature information of the pressed data is obtained by inputting the pressed data into the machine learning model for feature extraction, so that the machine learning model can make accurate center point predictions based on the feature information of the pressed data, and effectively obtain accurate predicted center coordinates. The machine learning model can also make accurate morphological predictions based on the feature information of the pressed data and the accurate predicted center coordinates, and effectively obtain accurate predicted morphological parameters. In addition, since the machine learning model is first used to make accurate center point predictions based on the feature information of the pressed data to obtain relatively accurate predicted center coordinates, and then the machine learning model is used to make morphological predictions based on the feature information of the pressed data and the accurate predicted center point coordinates, the accuracy of the prediction of the morphological parameters of the pressed area can be effectively improved, thereby ensuring the accuracy of the results obtained by the pressed area detection.

[0088] Optionally, the above-mentioned optional implementation manner can be implemented based on the different parts included in the machine learning model. It should be understood that the specific composition structure of the machine learning model in the embodiments of the present application is not limited in any way, as long as it can meet the needs.

[0089] For example, optionally, refer to Figure 5 As shown, the machine learning model may include: a feature extraction part, a first perception layer part, and a second perception layer part. The pressed data may be predicted at the center point through the feature extraction part to obtain feature information of the pressed area, and the feature information of the pressed area may be predicted at the center point through the first perception layer part to obtain predicted center coordinates of the pressed area. The feature information and predicted center coordinates of the pressed data may be predicted at the second perception layer part to obtain predicted morphological parameters of the pressed area.

[0090] It should be understood that such a model structure can accurately implement feature extraction through the feature extraction part, and can accurately perform center point prediction and morphology prediction through the first perception layer part and the second perception layer part, thereby ensuring the accuracy of the results obtained by the pressed area detection.

[0091] The feature extraction part in the embodiment of the present application can be composed of any structure as long as it can meet the requirements. For example, in some optional embodiments, refer to Figure 5As shown, the feature extraction part includes a linear transformation layer, at least one convolution layer and at least one pooling layer; the pressed data can be sequentially subjected to corresponding linear transformation processing, convolution processing and pooling processing through the linear transformation layer, at least one convolution layer and at least one pooling layer of the feature extraction part to obtain feature information of the pressed data.

[0092] It should be understood that by including the feature extraction part of the above-mentioned optional structure, the machine learning model can accurately and effectively extract the feature information of the pressing data to facilitate the detection of the pressing area.

[0093] In the embodiment of the present application, the linear transformation layer can perform some linear transformation processing on the pressed data, and can slightly rotate and / or translate the pressed data. At least one convolution layer and at least one pooling layer perform convolution processing and pooling processing respectively, and can extract high-level features of the pressed data (which can be considered as a sampled expression form of the pressed data).

[0094] Optionally, the feature extraction part of the machine learning model may include multiple convolutional layers and multiple pooling layers to better achieve feature extraction. It should be understood that the specific number can be set according to actual needs and is not limited here.

[0095] Optionally, the first perception layer may include a multilayer perceptron. Optionally, the second perception layer may include a multilayer perceptron. Optionally, the perceptron may be a universal function approximator, which may be used to summarize the features of the first half of the model and output a prediction result. For example, when detecting the pressed area, the multilayer perceptron in the first perception layer may output a prediction result of the center point prediction (i.e., predicted center coordinates), and the multilayer perceptron in the first perception layer may output a prediction result of the morphological prediction (i.e., predicted morphological parameters, including at least one of the major axis parameter, minor axis parameter, major axis inclination parameter, and minor axis inclination parameter of the shape corresponding to the pressed area).

[0096] For example, refer to Figure 5The workflow of the pressed area detection shown in the figure can be understood as follows: first, the original capacitance data can be obtained, the capacitance data corresponding to the pressed area can be determined from the original capacitance data, and then the capacitance data corresponding to the pressed area can be expanded to obtain the target capacitance data, and the target capacitance data can be determined as the pressed data. The pressed data is input into the machine learning model, and a feature extraction part including a linear transformation layer, at least one convolution layer, and at least one pooling layer is used to perform linear transformation processing, convolution processing, and pooling processing in sequence to achieve feature extraction and obtain feature information of the pressed data. The first perception layer part of the machine learning model predicts the center point of the pressed area through the feature information of the pressed data to obtain the predicted center coordinates of the pressed area. The second perception layer part of the machine learning model predicts the morphology of the pressed area through the feature information of the pressed data and the predicted center coordinates to obtain the predicted morphological parameters of the pressed area, which may include at least one of the major axis parameter, the minor axis parameter, the major axis inclination parameter, and the minor axis inclination parameter.

[0097] Alternatively, if Figure 6 The flowchart of the example application of the press detection scheme shown in the figure shows that when the scheme is actually used, the original capacitance data at a certain moment can be obtained first, and then after confirming that there is a press event, the press data can be determined from the original capacitance data, and then the machine learning model can be used to predict the center point and shape of the press area, so as to obtain the predicted center coordinates and predicted shape parameters of the press area (such as an elliptical area) respectively, so as to realize accurate detection of the press area.

[0098] It should be understood that the above Figure 5 and Figure 6 None of them shall be construed as any limitation to the embodiments of the present application.

[0099] In some optional embodiments, the press area detection method in the embodiments of the present application further includes: displaying a given press area on the touch screen to prompt the user to perform a press operation in the given press area; obtaining a second press data sample obtained by the user pressing the given press area on the touch screen, wherein the second press data sample is used to fine-tune the machine learning model to obtain a machine learning model for the user.

[0100] Optionally, the second pressing data sample may be a capacitance data sample, where the capacitance data sample may be mutual capacitance data or self capacitance data, depending on whether the touch screen is based on the mutual capacitance principle or the self capacitance principle.

[0101] Optionally, the touch screen can be used during the use phase when the press area detection method is used. At this time, the machine learning model is in a trained state and has a certain degree of versatility and generalization. However, due to the differences in finger shapes of different users, in practice, in order to adapt to the user's finger shape and further improve the user experience of the touch screen, "registration" can be implemented for specific users in the embodiment of the present application. By obtaining the second press data sample obtained by the user pressing a given press area on the touch screen, it is convenient to use it to fine-tune the machine learning model and obtain the machine learning model for the user, so that the machine learning model after fine-tuning the training can better achieve accurate press area detection for the user.

[0102] The second press data sample, the given press area, and the fine-tuning training are described in detail in the model training method embodiment below. You can refer to the following embodiment for understanding, and will not repeat them here.

[0103] Optionally, the second press data sample can be obtained in the following manner: obtain a second original capacitance data sample, the second original capacitance data sample is a capacitance data sample used to characterize the user's pressing operation in a given pressing area of ​​the touch screen; expand the capacitance data corresponding to the pressing area in the second original capacitance data sample to obtain a second target capacitance data sample, and determine the second target capacitance data sample as the second press data sample (this part of the content can be understood by referring to the content in the model training method embodiment below).

[0104] For example, a finger registration system function can be added to a terminal device (such as a mobile phone, etc.) equipped with a touch screen. Users of the device can register on the registration interface displayed on the touch screen. The registration interface can prompt the user to press a given pressing area (such as an elliptical shape, which can also be understood by referring to the "prompt area" described below) displayed on the touch screen. During registration, a plurality of given pressing areas of different forms can be used. The user can select a given pressing technique (such as the pressing technique in Table 1 below) to perform a pressing operation, completely covering the given pressing area, so as to obtain a second original capacitance data sample for subsequent processing to obtain a second pressing data sample for model fine-tuning training. It should be understood that this part of the content is described in detail in the model training method embodiment below, which can be understood by referring to the following text, and will not be repeated here.

[0105] It can be understood that the above description of the pressed area detection scheme is only an exemplary description of the embodiments of the present application and does not constitute any limitation to the embodiments of the present application.

[0106] To summarize, the pressed area detection scheme provided in the embodiment of the present application can obtain the pressed data obtained by the user pressing the touch screen, and then input the pressed data into the machine learning model to perform center point prediction to obtain the predicted center coordinates of the pressed area, and perform morphological prediction based on the pressed data and the predicted center coordinates to obtain the predicted morphological parameters of the pressed area. Therefore, on the one hand, the center point coordinates and morphological parameters of the pressed area of ​​the touch screen by the user can be effectively predicted through the machine learning model, thereby effectively realizing pressed area detection; and, on the other hand, since the center point is first accurately predicted through the machine learning model to obtain relatively accurate predicted center coordinates, and then the machine learning model is used to perform morphological prediction based on the pressed data and the accurate predicted center point coordinates, the accuracy of the prediction of the morphological parameters of the pressed area can be effectively improved, thereby effectively ensuring the accuracy of the results obtained by the pressed area detection.

[0107] According to a second aspect of the embodiment of the present application, a model training method is provided, referring to Figure 7 As shown in the flowchart, the model training method includes the following steps S202, S204 and S206, specifically:

[0108] S202: Obtain a first training sample for training the machine learning model, wherein the first training sample includes a first pressing data sample obtained by a user pressing the touch screen, a first center point true value corresponding to the first pressing data sample, and a first morphological parameter true value corresponding to the first pressing data sample.

[0109] Optionally, if the touch screen is a capacitive touch screen, the first press data sample in the first training sample may be a capacitive data sample. The capacitive data sample here may be mutual capacitance data or self-capacitance data, depending on whether the touch screen is based on the mutual capacitance principle or the self-capacitance principle. Optionally, the first center point true value corresponding to the first press data sample may include annotation data for annotating the center point of the first press data sample, or may be in other forms. Optionally, the first morphological parameter true value corresponding to the first press data sample may include annotation data for annotating the morphological parameters of the first press data sample (for example, including at least one of the major axis parameter, the minor axis parameter, the major axis inclination parameter, and the minor axis inclination parameter), or may be in other forms.

[0110] In the embodiment of the present application, the machine learning model is trained by obtaining at least a first press data sample, a first center point true value corresponding to the first press data sample, and a first morphological parameter true value, so that the trained machine learning model can accurately predict the center coordinates and morphological parameters of the press area according to the user's actual press data, thereby realizing press area detection, so as to improve the user experience of using the touch screen.

[0111] During model training, the machine learning model can be trained multiple times using multiple first training samples so that the machine learning model can achieve better training results.

[0112] In some optional embodiments, referring to Figure 8 As shown in the flowchart, the first pressing data sample can be obtained by the following steps S2022 to S2024, specifically:

[0113] S2022: Obtaining an original capacitance data sample, where the original capacitance data sample is a capacitance data sample obtained by representing a user pressing operation on the touch screen.

[0114] During model training, raw capacitance data samples obtained by collecting data from different users pressing the touch screen in different pressing techniques may be obtained to determine the first training sample.

[0115] Optionally, when collecting data, users of different groups (e.g., male, female, old, young) can be covered as much as possible. Data with obvious errors can also be screened out manually (e.g., a child's thumb is smaller and may not completely cover the larger prompt area ellipse, so it needs to be removed).

[0116] For example, in an optional implementation, a prompt area can be displayed on the touch screen of a terminal device (such as a mobile phone, etc.), and the prompt area can be, for example, an elliptical area for prompting the user to press an elliptical area with different positions, different major axes, different minor axes, different major axis inclinations, and different minor axis inclinations (at least one of the above). After the prompt area is displayed, the user follows the prompt and selects a pressing technique to press, completely covering the displayed prompt area, and then uses the collected capacitance data as the original capacitance data sample for subsequent processing to obtain the first pressing data sample for model training. Since the relevant parameters of the prompt area displayed on the touch screen can all be known, they can be used as true values, that is, if the prompt area is an elliptical area, the known center point coordinates of the prompt area can be used as the first center point true value, and the known morphological parameters of the prompt area (for example, at least one of the major axis parameter, the minor axis parameter, the major axis inclination parameter, and the minor axis inclination parameter) can be determined as the first morphological parameter true value.

[0117] For example, some example pressing techniques may be shown in Table 1 below:

[0118] Table 1

[0119]

[0120] As shown in Table 1 above, the optional pressing techniques may include pressing with the tip of the thumb, pressing with the pulp of the thumb, pressing with the side of the thumb, pressing with the tip of the index finger, pressing with the pulp of the index finger, pressing with the side of the index finger, etc. There are also other finger or other types of finger pressing techniques, which will not be elaborated here and can be set as needed.

[0121] For example, taking the prompt area displayed on the touch screen as an elliptical area as an example, refer to Fig. 9 The prompt area of ​​the example shown is understood, and its parameter setting method can be set as follows:

[0122] ① Such as Fig. 9 As shown, the center coordinates (x, y) of the prompt area are set according to the following formula:

[0123] (x,y)=(Tx i +a×△Tx,Rx i +b×△Rx)

[0124] The range of a is [0,1), for example, you can choose from the preset values ​​of {0.00, 0.25, 0.50, 0.75}; the range of b is

[0125] [0,1), for example, you can choose from the preset values ​​of {0.00, 0.25, 0.50, 0.75};

[0126] The above △Tx can be the horizontal coordinate unit moment, and △Rx can be the vertical coordinate unit moment. The two moments can be set smaller as needed. The two moments can be understood as follows: For example, assuming that a driver TX i Signal line, receiver RX i The coordinates of the intersection of the signal lines are (Tx i ,Rx i ), adjacent driver TX i+1 , Receiver RX i+1 The coordinates of the signal line junction are (Tx i+1 ,Rx i+1 ), then △Tx=Tx i+1 -Tx i , △Rx=Rx i+1 -Rx i .

[0127] ② If Fig. 9 As shown, the long axis parameter CD of the prompt area can be set arbitrarily according to actual needs. For example, it can be selected from the preset values ​​of {12mm, 14mm, 16mm, 18mm, 20mm}.

[0128] ③ Such as Fig. 9As shown, the major axis parameter AB of the prompt area can be set arbitrarily according to actual needs. For example, it can be selected from the preset values ​​of {10mm, 12mm, 14mm, 16mm}.

[0129] ④ Such as Fig. 9 As shown, the long axis inclination parameter θ of the prompt area, where θ in the example here is the angle between the long axis and the longitudinal axis of the coordinate system (for example, it can correspond to the length direction of the terminal device (such as a mobile phone, etc.)), and θ can be set arbitrarily according to actual needs. For example, you can choose from the preset values ​​of {-90°, -75°, -60°, -45°, -30°, -15°, 0°, 15°, 30°, 45°, 60°, 75°, 90°}, and a negative angle can be considered to correspond to the situation where the long axis slope is less than 0 (such as Fig. 9 In the example of θ, that is, a negative value, the major axis slope is less than 0), and a positive angle can be considered to correspond to the case where the major axis slope is greater than 0.

[0130] Regarding the short axis inclination parameter, it can be inferred by referring to the description of the long axis inclination parameter θ. The short axis inclination parameter may not be used in these examples, and no additional description is given here.

[0131] It should be understood that the above description is only an example and is not intended to limit the embodiments of the present application.

[0132] S2024: performing expansion processing on the capacitance data corresponding to the pressed area in the original capacitance data sample to obtain a target capacitance data sample, and determining the target capacitance data sample as a first pressed data sample.

[0133] Optionally, a minimum rectangular frame including the capacitance data corresponding to the pressed area can be determined in the original capacitance data sample, and then the minimum rectangular frame can be expanded, so as to expand the capacitance data corresponding to the pressed area in the original capacitance data sample. The capacitance data in the expanded rectangular frame is taken out to obtain a target capacitance data sample, which is determined as the first pressed data sample.

[0134] It should be understood that the scope of the above-mentioned expansion processing can be set as needed. For example, at least one of the upper, lower, left, and right sides of the minimum rectangular frame can be expanded outward to achieve expansion processing. For example, if one side of the minimum rectangular frame coincides with the corresponding position of the edge of the touch screen, the remaining three sides can be expanded outward to achieve expansion processing; if two sides of the minimum rectangular frame coincide with the corresponding positions of the edge of the touch screen, the remaining two sides can be expanded outward to achieve expansion processing; if the minimum rectangular frame is located in the middle as a whole, all four sides can be expanded outward to achieve expansion processing. The distance of each side expanding outward can be set as needed, for example, it can be between 1mm and 10mm. And the distances of each side expanding outward can be the same or different. As long as it can meet the actual needs, no limitation is made here.

[0135] It should be understood that the extended processing principle of step S2024 is similar to that of step S1024 above, except that step S1024 above obtains the pressing data from the original capacitance data, while step S2024 obtains the first pressing data sample from the original capacitance data sample. Figure 4 The explanation of is understood by analogy, that is, Figure 4 The P frame is the minimum rectangular frame including the capacitance data corresponding to the pressed area, and the Q frame is a rectangular frame after the four sides (p1, p2, p3, p4 sides) of the minimum rectangular frame (P frame) are expanded outward. Since the capacitance data in the Q frame is more than that in the P frame, the capacitance data corresponding to the pressed area can be effectively avoided from being missed. The target capacitance data sample obtained by such expansion processing is used as the first pressed data sample, which can improve the effect of using the first pressed data sample for model training, so that the trained machine learning model can accurately detect the pressed area.

[0136] Based on this, in the embodiment of the present application, through the optional implementation method of the above steps S2022 to S2024, on the one hand, it is possible to effectively avoid the omission of capacitance data corresponding to the pressed area, and use the target capacitance data sample obtained by such expansion processing as the first pressed data sample, which can improve the effect of model training using the first pressed data sample, so that the trained machine learning model can accurately detect the pressed area; on the other hand, since the target capacitance data sample obtained by the expansion processing is used as the first pressed data sample to realize the training of the machine learning model, without inputting all the original capacitance data samples of the touch screen into the machine learning model for training, it is also possible to effectively reduce the amount of calculation, thereby improving the efficiency of model training, and reducing the interference of excessive useless data, further improving the model training effect, thereby further ensuring the accuracy of the pressed area detection by the trained machine learning model.

[0137] S204: Input the first training sample into the machine learning model to perform center point prediction to obtain a first predicted center coordinate sample of the pressed area, and perform morphology prediction based on the first pressed data sample and the first predicted center coordinate sample to obtain a first predicted morphological parameter sample of the pressed area.

[0138] The machine learning model in the embodiment of the present application can be a pre-trained machine learning model. The machine learning model is implemented based on artificial intelligence technology. The machine learning model can be of any type. For example, it can be a convolutional neural network model (for example, including but not limited to AlexNet, Inception, VGG, ResNet, MobileNet, FCN, U-Net, DeepLab, BiSeNet, R-CNN, Yolo, SSD, RetinaNet and other models), a deep language model (for example, including but not limited to LSTM, GRU, BiLSTM, Transformer, MarianMT, BERT, RoBERTa, XLNet, GPT, T5, S4, Mamba and other models), or a hybrid variant model based on the above two (for example, including but not limited to ViT, DeiT, ConViT, SwinTransformer, TransUNet, Swin Unet, DETR, Deformable DETR, ConvS2S, ConvBERT and other models). Regarding the training process of the machine learning model, there is no limitation in the embodiment of the present application. For example, it can be trained using the model training method of the second aspect described below.

[0139] Optionally, when the pressed area is a circular area, the first predicted center coordinate sample may be a circle center coordinate sample of the shape corresponding to the pressed area, and the first predicted morphological parameter sample may be a circle diameter parameter sample, a radius parameter sample, etc. It is understandable that a circle may also be considered as a special ellipse with equal major and minor axes.

[0140] When the pressed area is treated as an ellipse area for processing, in some optional embodiments, the first predicted center coordinate sample may be an ellipse center coordinate sample of the shape corresponding to the pressed area. The ellipse center is the intersection of the major axis and the minor axis of the ellipse.

[0141] It should be understood that by determining the above first predicted center coordinate sample to train the machine learning model, the trained machine learning model can accurately detect the center coordinates of the pressed area, so as to improve the user experience of the touch screen by using the center coordinates.

[0142] In some optional embodiments, the first predicted morphological parameter sample includes at least one of a long-axis parameter sample, a short-axis parameter sample, a long-axis inclination parameter sample, and a short-axis inclination parameter sample of a shape corresponding to the pressed area.

[0143] It should be understood that by determining at least one of the above first predicted morphological parameter samples, the machine learning model is trained so that the trained machine learning model can facilitate accurate detection of the morphological parameters of the pressed area, so as to utilize these morphological parameters to improve the user experience of the touch screen.

[0144] Optionally, when the pressed area is treated as an elliptical area for processing, the first predicted morphological parameter sample may include at least one of the major axis parameter sample, the minor axis parameter sample, the major axis inclination parameter sample, and the minor axis inclination parameter sample of the ellipse. The relevant contents of the major axis inclination parameter and the minor axis inclination parameter have been explained in the previous text and will not be repeated here.

[0145] In some optional embodiments, the method of obtaining the first predicted center coordinate sample in step S104 may include: inputting the first training sample into the machine learning model, performing feature extraction on the first press data sample in the first training sample to obtain feature information of the first press data sample, performing center point prediction based on the feature information of the first press data sample to obtain a first predicted center coordinate sample of the pressed area; the method of obtaining the first predicted morphological parameter sample in step S104 may include: performing morphological prediction based on the feature information of the first press data sample and the first predicted center coordinate sample to obtain a first predicted morphological parameter sample of the pressed area.

[0146] Based on this, in the embodiment of the present application, by inputting the first training sample into the machine learning model, the feature extraction of the first press data sample in the first training sample is performed to obtain the feature information of the first press data sample, so that the machine learning model can accurately predict the center point based on the feature information of the first press data sample, and effectively obtain the accurate first predicted center coordinate sample, and also enable the machine learning model to accurately predict the morphology based on the feature information of the first press data sample and the accurate first predicted center coordinate sample, and effectively obtain the accurate first predicted morphological parameter sample; in addition, since the machine learning model is first used to accurately predict the center point based on the feature information of the first press data sample to obtain a more accurate first predicted center coordinate sample, and then the machine learning model is used to perform morphological prediction based on the feature information of the first press data sample and the accurate first predicted center coordinate sample, the accuracy of the morphological prediction can be effectively improved; therefore, the machine learning model is trained by using such a more accurate first predicted center coordinate sample and a more accurate first predicted morphological parameter sample, and the training effect is better, so that the machine learning model after the training is completed can more accurately detect the pressed area of ​​the touch screen.

[0147] Optionally, the above-mentioned optional implementation manner can be implemented based on the different parts included in the machine learning model. It should be understood that the specific composition structure of the machine learning model in the embodiments of the present application is not limited in any way, as long as it can meet the needs.

[0148] For example, optionally, refer to Fig.10 As shown, the machine learning model may include: a feature extraction part, a first perception layer part, and a second perception layer part. The first pressing data sample in the first training sample may be subjected to feature extraction by the feature extraction part to obtain feature information of the first pressing data sample, and the feature information of the first pressing data sample may be subjected to center point prediction by the first perception layer part to obtain a first predicted center coordinate sample of the pressing area. The feature information of the first pressing data sample and the first predicted center coordinate sample may be subjected to morphological prediction by the second perception layer part to obtain a first predicted morphological parameter sample of the pressing area.

[0149] It should be understood that through such a model structure, the feature extraction part can accurately realize feature extraction of the first press data sample, and the first perception layer part and the second perception layer part can accurately perform center point prediction and morphology prediction, thereby ensuring the accuracy of the predicted first predicted center coordinate sample and the first predicted morphological parameter sample. The machine learning model is trained with such more accurate first predicted center coordinate samples and more accurate first predicted morphological parameter samples, and the training effect is better, so that the machine learning model after training can more accurately detect the pressed area of ​​the touch screen.

[0150] The feature extraction part in the embodiment of the present application can be composed of any structure as long as it can meet the requirements. For example, in some optional embodiments, refer to Figure 5 As shown, the feature extraction part includes a linear transformation layer, at least one convolution layer and at least one pooling layer; the first compression data sample in the first training sample can be sequentially subjected to corresponding linear transformation processing, convolution processing and pooling processing through the linear transformation layer, at least one convolution layer and at least one pooling layer of the feature extraction part to obtain feature information of the first compression data sample.

[0151] It should be understood that by including the feature extraction part of the above-mentioned optional structure, the machine learning model can accurately and effectively extract the feature information of the first press data sample, so as to realize accurate center point prediction and morphology prediction, thereby ensuring the accuracy of the predicted first predicted center coordinate sample and the first predicted morphology parameter sample. The machine learning model is trained by such more accurate first predicted center coordinate samples and more accurate first predicted morphology parameter samples, and the training effect is better, so that the machine learning model after training can more accurately detect the pressed area of ​​the touch screen.

[0152] Regarding the above-mentioned optional model structure, the feature extraction part, linear transformation layer, convolution layer, pooling layer, first perception layer part, and second perception layer part have been described in detail in the touch area detection method of the first aspect in the previous text, and the relevant content can also be understood by referring to the previous text.

[0153] For example, during the model training phase, the linear transformation layer can perform a certain linear transformation on the first compression data sample, and can slightly rotate and / or translate the compression data. At least one convolution layer and at least one pooling layer perform convolution processing and pooling processing, respectively, and can extract high-level features of the first compression data sample (which can be considered as a sampling expression of the first compression data sample). Optionally, the feature extraction part of the machine learning model may include multiple convolution layers and multiple pooling layers to better achieve feature extraction. It should be understood that the specific number can be set according to actual needs and is not limited here. Optionally, the first perception layer part may include a multi-layer perceptron. Optionally, the second perception layer part may include a multi-layer perceptron. Optionally, the perceptron can be a universal function approximator, which can be used to summarize the features of the first half of the model and output prediction results. For example, the multilayer perceptron in the first perception layer part can output the prediction result of the center point prediction (i.e., the first predicted center coordinate sample), and the multilayer perceptron in the first perception layer part can output the prediction result of the morphology prediction (i.e., the first predicted morphology parameter sample, including at least one of the major axis parameter sample, minor axis parameter sample, major axis inclination parameter sample, and minor axis inclination parameter sample of the shape corresponding to the pressed area).

[0154] S206: Training the machine learning model based on the first predicted center coordinate sample, the first center point true value, the first predicted morphological parameter sample and the first morphological parameter true value.

[0155] After obtaining the first predicted center coordinate sample, the first center point true value, the first predicted morphological parameter sample and the first morphological parameter true value, they can be input into the machine learning model to train the machine learning model until the training end condition is reached, thereby completing the training of the machine learning model. The training can be supervised training. The training end condition can be set as needed, for example, the training can be terminated after the training round reaches a predetermined training round (for example, 1000 times, 5000 times, 10000 times, etc.), or the training can be terminated after the loss value of the loss function of the machine learning model is less than a certain threshold.

[0156] Based on this, in the embodiment of the present application, through the model training scheme of the above steps S202 to S206, since the first training sample for training the machine learning model can be obtained, the first training sample is then input into the machine learning model for center point prediction to obtain the first predicted center coordinate sample of the pressed area, and the morphology is predicted based on the first pressing data sample and the first predicted center coordinate sample in the first training sample to obtain the first predicted morphological parameter sample of the pressed area, and then based on the first predicted center coordinate sample, the first center point true value, the first predicted morphological parameter sample and the first morphological parameter true value, the machine learning model is trained. Therefore, on the one hand, a model capable of realizing touch screen control can be trained. A machine learning model is provided for accurately detecting the pressed area. On the other hand, since the machine learning model is first used to accurately predict the center point to obtain a relatively accurate first predicted center coordinate sample, and then the machine learning model is used to perform morphological prediction based on the first pressed data sample and the accurate first predicted center point coordinate sample, the accuracy of the morphological parameter prediction of the pressed area can be effectively improved, and a relatively accurate first predicted morphological parameter sample can be obtained. Therefore, the machine learning model is trained with such relatively accurate first predicted center coordinate samples and relatively accurate first predicted morphological parameter samples, and the training effect is better, so that the machine learning model after training can more accurately detect the pressed area of ​​the touch screen.

[0157] In addition, since the machine learning model trained by the above-mentioned model training method can be used subsequently, the machine learning model is not only convenient to use, but also has better development and debugging efficiency, is conducive to maintenance and optimization, has better versatility, and is also conducive to reducing the extra time and energy invested by R&D personnel before and after the development is completed.

[0158] It should be understood that in the technical solution of the embodiment of the present application, it is crucial to first predict the first center point coordinate sample for model training. This is because the first center point coordinate sample is predicted first, and then the morphology prediction is performed based on the first press data sample and the first center point coordinate sample to obtain the first predicted morphology parameter sample, which is beneficial to reduce the negative impact of the nonlinearity of the press data on the prediction parameters, thereby better ensuring the model training effect, so that the model can effectively improve the accuracy of the morphology parameter prediction, and enable the machine learning model that has been trained subsequently to more accurately detect the pressed area of ​​the touch screen.

[0159] For example, Fig.10 The model structure shown can be used to train the feature extraction part (linear transformation layer, at least one convolution layer, at least one pooling layer), and adjust the model parameters therein to a suitable state. When training the first perception layer part and the second perception layer part, the first perception layer part can be first subjected to supervised training separately, and its parameters can be adjusted so that the first perception layer part of the machine learning model can achieve a suitable effect on the center point prediction, which can effectively improve the negative impact of the nonlinearity of the pressed data on the pressed area prediction. Thereafter, the second perception layer part can be subjected to supervised training, and its parameters can be adjusted so that the second perception layer part of the machine learning model can achieve a suitable effect on the morphological parameter prediction.

[0160] For example, refer to Fig.10The workflow for training the machine learning model shown in the figure is understood. First, the original capacitance data sample can be obtained (for example, it can be obtained by the user pressing the prompt area on the touch screen when collecting training samples, which can be understood by referring to the previous text), and the capacitance data corresponding to the pressed area is determined from the original capacitance data sample. Then, the capacitance data corresponding to the pressed area is expanded to obtain the target capacitance data sample, and the target capacitance data sample is determined as the first pressed data sample. The first pressed data sample is input into the machine learning model, and a linear transformation processing, a convolution processing, and a pooling processing are performed in sequence through a feature extraction part including a linear transformation layer, at least one convolution layer, and at least one pooling layer to realize feature extraction, and obtain feature information of the first pressed data sample. The machine learning model The first perception layer predicts the center point of the pressed area through the feature information of the first pressed data sample, and obtains the first predicted center coordinate sample of the pressed area. The second perception layer of the machine learning model predicts the morphology of the pressed area through the feature information of the first pressed data sample and the first predicted center coordinate sample, and obtains the first predicted morphological parameter sample of the pressed area (which may include at least one of the major axis parameter, minor axis parameter, major axis inclination parameter, and minor axis inclination parameter). The information of the prompt area (such as the center point coordinates and morphological parameters of the prompt area of ​​the ellipse) is used as the true value (i.e., the first center point true value and the first morphological parameter true value), and the machine learning model is trained through the first pressed data sample, the first predicted morphological parameter sample, the first center point true value, and the first morphological parameter true value. It should be understood that the above Fig.10 None of them shall be construed as any limitation to the embodiments of the present application.

[0161] In some optional embodiments, the model training method of the embodiments of the present application also includes the steps of: obtaining a second training sample, wherein the second training sample includes: a second pressing data sample obtained by the user pressing a given pressing area on the touch screen, a second center point true value corresponding to the second pressing data sample, and a second morphological parameter true value corresponding to the second pressing data sample; and using the second training sample to fine-tune the machine learning model to obtain a machine learning model for the user.

[0162] It should be understood that although the model has a certain degree of universality and generalization after the machine learning model is trained in step S206, due to the differences in finger shapes of different users, in practice, in order to adapt to the user's finger shape and further improve the user experience of using the touch screen, in the embodiment of the present application, "registration" can be implemented for a specific user, by obtaining a second training sample corresponding to the user, and then fine-tuning the machine learning model to obtain a machine learning model for the user, so that the machine learning model after fine-tuning training can better achieve accurate press area detection for the user.

[0163] Optionally, a given pressing area may be displayed on the touch screen to prompt the user to perform a pressing operation in the given pressing area; and a second pressing data sample obtained when the user performs a pressing operation on the given pressing area on the touch screen is acquired.

[0164] Optionally, the second pressing data sample may be a capacitance data sample, where the capacitance data sample may be mutual capacitance data or self capacitance data, depending on whether the touch screen is based on the mutual capacitance principle or the self capacitance principle.

[0165] Optionally, the second press data sample in the second training sample can be obtained in the following manner: obtain a second original capacitance data sample, the second original capacitance data sample is a capacitance data sample used to characterize the user's pressing operation in a given pressing area of ​​the touch screen; expand the capacitance data corresponding to the pressing area in the second original capacitance data sample to obtain a second target capacitance data sample, and determine the second target capacitance data sample as the second press data sample.

[0166] The extended processing here is similar to the relevant content and principle of the original capacitance data sample and the first pressing data sample in the previous steps S2022 to S2024, which can be understood by referring to the previous text and will not be repeated here.

[0167] In some optional embodiments, the second center point true value is the center point coordinates of the shape corresponding to the given pressed area, and the second morphological parameter true value is the morphological parameter of the shape corresponding to the given pressed area.

[0168] The center point coordinates and morphological parameters of the shape corresponding to a given pressed area may be known. Obtaining them as the second center point true value and the second morphological parameter true value is conducive to ensuring the effect of model fine-tuning and implementing model fine-tuning more conveniently and accurately to obtain a machine learning model for the user.

[0169] Optionally, if the given pressed area is an elliptical area, the coordinates of the center point of the shape corresponding to the given pressed area are the center coordinates of the ellipse, and the morphological parameters of the shape corresponding to the given pressed area include, for example, at least one of the major axis parameter, minor axis parameter, major axis inclination parameter, and minor axis inclination parameter of the ellipse. The relevant contents of the major axis inclination parameter and the minor axis inclination parameter have been explained in the previous text and will not be repeated here.

[0170] The setting of the relevant parameters of the given pressing area in the embodiment of the present application can be understood by referring to the relevant parameters of the "prompt area" above, and will not be repeated here.

[0171] For example, a finger registration system function can be added to a terminal device (such as a mobile phone, etc.) equipped with a touch screen. A user using the device can register on the registration interface displayed on the touch screen. The registration interface can prompt the user to press a given pressing area (such as an elliptical shape, which can also be understood by referring to the "prompt area" described above) displayed on the touch screen. During registration, a plurality of given pressing areas of different forms can be used. The user can select a given pressing technique (such as the pressing technique in Table 1 above) to perform a pressing operation, completely covering the given pressing area, so as to obtain a second original capacitance data sample for subsequent processing to obtain a second pressing data sample for model fine-tuning training. Since the relevant parameters of the given pressing area displayed on the touch screen are all known, they can be used as true values, that is, if the given pressing area is an elliptical area, the known center point coordinates of the given pressing area can be used as the second center point true value, and the known morphological parameters of the given pressing area (for example, at least one of the major axis parameter, the minor axis parameter, the major axis inclination parameter, and the minor axis inclination parameter) can be determined as the second morphological parameter true value. It should be understood that the above description is only an example and is not intended to limit the embodiments of the present application.

[0172] In some optional embodiments, referring to Fig.11 As shown in the flowchart, in the embodiment of the present application, the second training sample is used to fine-tune the machine learning model, which can be implemented by the following steps S302 to S304. Specifically:

[0173] S302: Input the second training sample into the machine learning model to perform center point prediction to obtain a second predicted center coordinate sample of the given pressing area, and perform morphology prediction based on the second pressing data sample and the second predicted center coordinate sample to obtain a second predicted morphological parameter sample of the given pressing area.

[0174] The processing method of center point prediction and morphology prediction in this step S302 is similar to the principle of the relevant content of step S204, and can be understood by referring to the relevant content of step S204. The difference is that step S204 inputs the first training sample and outputs the first predicted center coordinate sample and the first predicted morphological parameter sample of the pressed area, while this step S302 inputs the second training sample and outputs the second predicted center coordinate sample and the second predicted morphological parameter sample of the pressed area.

[0175] For example, in some optional embodiments, the second predicted center coordinate sample may be an ellipse center coordinate sample of a shape corresponding to the pressed area.

[0176] For example, in some optional embodiments, the second predicted morphological parameter sample includes at least one of a long axis parameter sample, a short axis parameter sample, a long axis inclination parameter sample, and a short axis inclination parameter sample of a shape corresponding to a given pressing area.

[0177] For example, in some optional embodiments, the method for obtaining the second predicted center coordinate sample in step S302 may include: inputting the first training sample into the machine learning model, performing feature extraction on the second press data sample in the second training sample to obtain feature information of the second press data sample, performing center point prediction based on the feature information of the second press data sample to obtain the second predicted center coordinate sample of the given press area; the method for obtaining the second predicted morphological parameter sample in step S302 may include: performing morphological prediction based on the feature information of the second press data sample and the second predicted center coordinate sample to obtain the second predicted morphological parameter sample of the given press area.

[0178] For example, in some optional embodiments, the machine learning model may include: a feature extraction part, a first perception layer part, and a second perception layer part. The second pressing data sample in the second training sample may be subjected to feature extraction through the feature extraction part to obtain feature information of the second pressing data sample, and the feature information of the second pressing data sample may be subjected to center point prediction through the second perception layer part to obtain a second predicted center coordinate sample of a given pressing area. The feature information of the second pressing data sample and the second predicted center coordinate sample may be subjected to morphological prediction through the second perception layer part to obtain a second predicted morphological parameter sample of a given pressing area.

[0179] For example, in some optional embodiments, the feature extraction part includes a linear transformation layer, at least one convolution layer and at least one pooling layer; the second compression data sample in the second training sample can be sequentially subjected to corresponding linear transformation processing, convolution processing and pooling processing through the linear transformation layer, at least one convolution layer and at least one pooling layer of the feature extraction part to obtain feature information of the second compression data sample.

[0180] S304: Fine-tune the machine learning model based on the second predicted center coordinate sample, the second center point true value, the second predicted morphological parameter sample and the second morphological parameter true value.

[0181] After obtaining the second predicted center coordinate sample, the second center point true value, the second predicted morphological parameter sample and the second morphological parameter true value, they can be input into the machine learning model to fine-tune the machine learning model until the training end condition is reached, thereby completing the fine-tuning training of the machine learning model. The fine-tuning training can be supervised training. The training end condition of the fine-tuning training can be set as needed, for example, the training can be terminated after the training rounds reach a predetermined training round (for example, 10 times, 20 times, 30 times, etc.), or the fine-tuning training can be terminated after the loss value of the loss function of the machine learning model is less than a certain threshold.

[0182] Based on this, through the optional implementation of the above steps S302 to S304, the second training sample can be effectively used to fine-tune the machine learning model, so as to effectively obtain a machine learning model for the user, so that the machine learning model after fine-tuning training can better achieve accurate press area detection for specific users.

[0183] It should be understood that in the technical solution of the embodiment of the present application, first predicting the second center point coordinate sample is crucial for model fine-tuning training. This is because the second center point coordinate sample is first predicted, and then the morphology prediction is performed based on the second press data sample and the second center point coordinate sample to obtain the second predicted morphology parameter sample, which is beneficial to reduce the negative impact of the nonlinearity of the press data on the prediction parameters, thereby better ensuring the model fine-tuning training effect, so that the model can effectively improve the accuracy of morphology parameter prediction, so that the machine learning model completed by subsequent fine-tuning training can more accurately detect the pressed area of ​​the touch screen for a specific user.

[0184] Optionally, for an implementation in which the machine learning model includes a feature extraction part, a first perceptron part, and a second perceptron part, and the feature extraction part includes a linear transformation layer, at least one convolution layer, and at least one pooling layer, during the fine-tuning training in step S304, the at least one convolution layer and at least one pooling layer of the feature extraction part may not be fine-tuned. This is because the convolution layer and the pooling layer are used to extract high-level features of the data and have certain robustness and versatility, so they do not need to be fine-tuned again, which can also improve the efficiency of model fine-tuning. However, the linear transformation layer, the first perceptron part, and the second perceptron part of the feature extraction part all need to be fine-tuned. This is because the linear transformation layer can approximate the user's pressing data in the training phase through some kind of linear transformation, so fine-tuning is required, while the first perception layer and the second perception layer can summarize the features of the first half of the model and output the prediction results (i.e., center coordinates, morphological parameters, etc.). There will be some errors between the prediction results and the true values ​​prompted by the system (i.e., the true values ​​of the center coordinates, the true values ​​of the morphological parameters, etc.) (because different users have different fingers, and each user presses the same area differently, etc.), so fine-tuning is needed to eliminate the error.

[0185] Of course, in other optional embodiments, the linear transformation layer, at least one convolution layer and at least one pooling layer, the first perceptron part and the second perceptron part of the feature extraction part may also be fine-tuned as long as the requirements can be met.

[0186] It should be understood that the above description of the model training scheme is merely an exemplary description of the embodiments of the present application and does not constitute any limitation to the embodiments of the present application.

[0187] It can be seen from the technical solutions of the above embodiments of the present application that the present technical solution aims to realize the detection of the pressed area by using a machine learning model based on artificial intelligence technology, and can accurately determine the center coordinates and morphological parameters (such as the center coordinates of an ellipse and the morphological parameters of an ellipse) of the pressed area corresponding to the pressed data, so as to facilitate the use of accurate detection results when necessary to optimize the user's touch screen and terminal device (such as a mobile phone, etc.) operating experience. Some technical solutions in the embodiments of the present application may include but are not limited to the following key points:

[0188] First, the machine learning model in this technical solution can search for the global optimal solution in a large amount of training data to achieve good generalization and versatility (to a certain extent, it can adapt to the finger states and pressing states of different users), avoiding the problem of engineers adjusting thresholds based on experience and generalization crossover (which may only be able to adapt to the finger states and pressing states of some users) in traditional solutions. In addition, this technical solution collects a variety of training data (covering different pressing positions, different elliptical shapes, and finger characteristics of different groups of people) to pre-train the machine learning model, and the training effect is good. The pressing area is detected by the model, and the pressing area parameters obtained are more accurate.

[0189] Secondly, in order to further adapt to the finger shapes of different users, some optional implementations of this technical solution also introduce a finger registration function, which can collect a specific user's pressing data sample on a given pressing area (elliptical area) on the touch screen, and then feed it back to the machine learning model for fine-tuning, so that the machine learning model can better adapt to the specific user's pressing area detection needs. Using the fine-tuned machine learning model for real-time pressing area detection can adapt to the characteristics of different users and achieve efficient performance.

[0190] Furthermore, in the technical solution of the embodiment of the present application, the machine learning model adopts a "two-stage prediction", which first predicts the predicted center coordinates, and then performs morphological prediction based on the press data and the predicted center coordinates to obtain predicted morphological parameters, which is beneficial to reduce the negative impact of the nonlinearity of the press data on the predicted parameters, thereby effectively improving the accuracy of the prediction of the morphological parameters of the pressed area and improving the accuracy of the pressed area detection.

[0191] Reference Fig.12 As shown, according to a third aspect of an embodiment of the present application, a pressed area detection device 1200 is provided, including:

[0192] An acquisition module 1202 is used to acquire the pressing data obtained by the user pressing the touch screen;

[0193] The prediction module 1204 is used to input the pressing data into the machine learning model to perform center point prediction to obtain the predicted center coordinates of the pressing area; and to perform morphological prediction based on the pressing data and the predicted center coordinates to obtain the predicted morphological parameters of the pressing area.

[0194] According to the technical solution provided in the embodiment of the present application, since the acquisition module 1202 can obtain the pressing data obtained by the user pressing the touch screen, the prediction module 1204 then inputs the pressing data into the machine learning model to perform center point prediction to obtain the predicted center coordinates of the pressed area, and performs morphological prediction based on the pressing data and the predicted center coordinates to obtain the predicted morphological parameters of the pressed area. Therefore, on the one hand, the machine learning model can be used to effectively predict the center point coordinates and morphological parameters of the pressed area of ​​the touch screen by the user, thereby effectively realizing pressed area detection; and, on the other hand, since the machine learning model is first used to perform accurate center point prediction to obtain relatively accurate predicted center coordinates, and then the machine learning model is used to perform morphological prediction based on the pressing data and the accurate predicted center point coordinates, the accuracy of the prediction of the morphological parameters of the pressed area can be effectively improved, thereby effectively ensuring the accuracy of the results obtained by the pressed area detection.

[0195] In some optional embodiments, the prediction module 1204 is specifically used to: input the press data into a machine learning model for feature extraction to obtain feature information of the press data, perform center point prediction based on the feature information of the press data to obtain predicted center coordinates of the press area; and perform morphology prediction based on the feature information of the press data and the preset center coordinates to obtain predicted morphological parameters of the press area.

[0196] In some optional embodiments, the predicted morphological parameters include at least one of a major axis parameter, a minor axis parameter, a major axis inclination parameter, and a minor axis inclination parameter of a shape corresponding to the pressed area.

[0197] In some optional embodiments, the acquisition module 1202 is specifically used to: obtain the original capacitance data obtained by the user pressing the touch screen; expand the capacitance data corresponding to the pressed area in the original capacitance data to obtain target capacitance data, and determine the target capacitance data as the pressing data.

[0198] In some optional embodiments, the pressing area detection device 1200 is also used to: display a given pressing area on the touch screen to prompt the user to perform a pressing operation in the given pressing area; obtain a second pressing data sample obtained by the user pressing the given pressing area on the touch screen, wherein the second pressing data sample is used to fine-tune the machine learning model to obtain a machine learning model for the user.

[0199] In some optional embodiments, the pressed area detection device 1200 is further used to control a terminal device equipped with the touch screen according to the predicted center coordinates of the pressed area and / or the predicted morphological parameters.

[0200] The pressed area detection device 1200 and its optional embodiments in the embodiment of the present application have been described in detail in the aforementioned pressed area detection method embodiment, so their related contents and beneficial effects can be understood by referring to the contents of the various optional embodiments of the above-mentioned method, and will not be repeated here.

[0201] Reference Fig.13 As shown, according to a fourth aspect of an embodiment of the present application, a model training device 1300 is provided, including:

[0202] The sample acquisition module 1302 is used to acquire a first training sample for training the machine learning model, wherein the first training sample includes a first pressing data sample obtained by a user pressing the touch screen, a first center point true value corresponding to the first pressing data sample, and a first morphological parameter true value corresponding to the first pressing data sample;

[0203] A parameter prediction module 1304 is used to input the first training sample into the machine learning model to perform center point prediction to obtain a first predicted center coordinate sample of the pressed area, and perform morphology prediction based on the first pressed data sample and the first predicted center coordinate sample to obtain a first predicted morphological parameter sample of the pressed area;

[0204] The model training module 1306 is used to train the machine learning model based on the first predicted center coordinate sample, the first center point true value, the first predicted morphological parameter sample and the first morphological parameter true value.

[0205] According to the technical solution provided in the embodiment of the present application, since the sample acquisition module 1302 can obtain the first training sample for training the machine learning model, the parameter prediction module 1304 then inputs the first training sample into the machine learning model for center point prediction to obtain the first predicted center coordinate sample of the pressed area, and performs morphological prediction based on the first pressing data sample and the first predicted center coordinate sample in the first training sample to obtain the first predicted morphological parameter sample of the pressed area, and the model training module 1306 then trains the machine learning model based on the first predicted center coordinate sample, the first center point true value, the first predicted morphological parameter sample and the first morphological parameter true value. Therefore, on the one hand, the machine learning model that can be implemented can be trained. A machine learning model is now provided for accurately detecting the pressed area of ​​the touch screen. On the other hand, since the center point is first accurately predicted by the machine learning model to obtain a relatively accurate first predicted center coordinate sample, and then the machine learning model is used to perform morphological prediction based on the first pressed data sample and the accurate first predicted center point coordinate sample, the accuracy of the morphological parameter prediction of the pressed area can be effectively improved, and a relatively accurate first predicted morphological parameter sample can be obtained. Therefore, the machine learning model is trained with such relatively accurate first predicted center coordinate samples and relatively accurate first predicted morphological parameter samples, and the training effect is better, so that the machine learning model after training can more accurately detect the pressed area of ​​the touch screen.

[0206] In some optional embodiments, the parameter prediction module 1304 is specifically used to: input the first training sample into the machine learning model, perform feature extraction on the first press data sample in the first training sample to obtain feature information of the first press data sample, perform center point prediction based on the feature information of the first press data sample to obtain a first predicted center coordinate sample of the press area; perform morphology prediction based on the feature information of the first press data sample and the first predicted center coordinate sample to obtain a first predicted morphological parameter sample of the press area.

[0207] In some optional embodiments, the machine learning model includes: a feature extraction part, a first perception layer part and a second perception layer part; the parameter prediction module 1304 is specifically used to: perform feature extraction on the first press data sample in the first training sample through the feature extraction part to obtain feature information of the first press data sample, perform center point prediction on the feature information of the first press data sample through the first perception layer part to obtain a first predicted center coordinate sample of the press area; perform morphological prediction on the feature information of the first press data sample and the first predicted center coordinate sample through the second perception layer part to obtain a first predicted morphological parameter sample of the press area.

[0208] In some optional embodiments, the feature extraction part includes a linear transformation layer, at least one convolution layer and at least one pooling layer; the parameter prediction module 1304 is specifically used to: subject the first compression data sample in the first training sample to corresponding linear transformation processing, convolution processing and pooling processing in sequence through the linear transformation layer, at least one convolution layer and at least one pooling layer of the feature extraction part to obtain feature information of the first compression data sample.

[0209] In some optional embodiments, the first predicted morphological parameter sample includes at least one of a long-axis parameter sample, a short-axis parameter sample, a long-axis inclination parameter sample, and a short-axis inclination parameter sample of the shape corresponding to the pressed area.

[0210] In some optional embodiments, the sample acquisition module 1302 is specifically used to: obtain an original capacitance data sample, where the original capacitance data sample is a capacitance data sample used to characterize a user pressing operation on a touch screen; perform expansion processing on the capacitance data corresponding to the pressing area in the original capacitance data sample to obtain a target capacitance data sample, and determine the target capacitance data sample as the first pressing data sample.

[0211] In some optional embodiments, the model training device 1300 is also used to: obtain a second training sample, wherein the second training sample includes: a second pressing data sample obtained by a user pressing a given pressing area on the touch screen, a second center point true value corresponding to the second pressing data sample, and a second morphological parameter true value corresponding to the second pressing data sample; and use the second training sample to fine-tune the machine learning model to obtain a machine learning model for the user.

[0212] In some optional embodiments, the second center point true value is the center point coordinates of the shape corresponding to the given pressed area, and the second morphological parameter true value is the morphological parameter of the shape corresponding to the given pressed area.

[0213] In some optional embodiments, the model training device 1300 is specifically used to: input the second training sample into the machine learning model to perform center point prediction to obtain the second predicted center coordinate sample of the given pressing area, and perform morphology prediction based on the second pressing data sample and the second predicted center coordinate sample to obtain the second predicted morphological parameter sample of the given pressing area; based on the second predicted center coordinate sample, the second center point true value, the second predicted morphological parameter sample and the second morphological parameter true value, fine-tune the machine learning model.

[0214] The model training device 1300 and its optional embodiments in the embodiment of the present application have been described in detail in the aforementioned model training method embodiment. Therefore, their relevant contents and beneficial effects can be understood by referring to the contents of the various optional embodiments of the above-mentioned method, and will not be repeated here.

[0215] According to the fifth aspect of the embodiments of the present application, there is provided an electronic device, comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; the memory is used to store a computer program; the processor is used to execute the pressed area detection method described in the first aspect or the model training method described in the third aspect by running the computer program stored in the memory.

[0216] Fig.14 1400 is a block diagram of an optional electronic device in an embodiment of the present application. The present application embodiment does not limit the specific implementation of the electronic device 1400. Fig.14 The electronic device 1400 provided in the embodiment of the present application includes: a processor 1402, a communication interface 1404, a memory 1406, and a communication bus 1408. Among them:

[0217] The processor 1402 , the communication interface 1404 , and the memory 1406 communicate with each other via a communication bus 1408 .

[0218] The communication interface 1404 is used to communicate with other electronic devices or servers.

[0219] The processor 1402 is used to execute the computer program 1410, and specifically can execute the relevant steps in any of the aforementioned pressed area detection methods or model training method embodiments.

[0220] Specifically, the computer program 1410 may include program codes including computer operation instructions.

[0221] The processor 1402 may be a CPU, or a GPU (Graphic Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0222] The memory 1406 is used to store the computer program 1410. The memory 1406 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0223] The computer program 1410 can be specifically used to enable the processor 1402 to execute the pressed area detection method or model training method in any of the aforementioned embodiments.

[0224] The specific implementation of each step in the computer program 1410 can refer to the corresponding description of the corresponding steps and units in any of the aforementioned pressed area detection method or model training method embodiments, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the aforementioned method embodiments, which will not be repeated here.

[0225] The electronic device 1400 in the embodiment of the present application has been described in detail in the aforementioned pressed area detection method or model training method embodiment, so its related content and beneficial effects can be understood with reference to the above-mentioned method embodiment and will not be repeated here.

[0226] According to the sixth aspect of the embodiments of the present application, the embodiments of the present application further provide a computer storage medium on which a computer program is stored, and when the computer program is executed by a processor, the pressed area detection method or model training method described in any of the aforementioned multiple method embodiments is implemented. The computer storage medium includes, but is not limited to: a compact disc read-only memory (CD-ROM), a random access memory (RAM), a floppy disk, a hard disk or a magneto-optical disk, etc.

[0227] According to the seventh aspect of the embodiments of the present application, the embodiments of the present application also provide a computer program product, including a computer program, which, when executed by a processor, implements the pressed area detection method or model training method as described in any one of the above-mentioned multiple method embodiments.

[0228] The pressed area detection device 1200 / model training device 1300 / electronic device 1400 / computer storage medium / computer program product embodiments in the embodiments of the present application have been described in detail in the aforementioned pressed area detection method and model training method embodiments, so their related contents and beneficial effects can be understood with reference to the above-mentioned method embodiments and will not be repeated here.

[0229] It should be understood that the various drawings of the embodiments of the present application are for the convenience of illustrating the structures, and the size ratios of the various structures are not necessarily drawn according to the actual ratios.

[0230] In addition, it should be noted that the user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to sample data used to train the model, data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0231] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0232] The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or implemented as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk or magneto-optical disk), or implemented as a computer code originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded through a network and stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor or programmable or dedicated hardware (such as an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA)). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., a random access memory (RAM), a read-only memory (ROM), a flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown here, the execution of the code converts the general-purpose computer into a dedicated computer for executing the method shown here.

[0233] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for specific applications, but such implementation should not be considered to be beyond the scope of the embodiments of the present application.

[0234] The above implementation methods are only used to illustrate the embodiments of the present application, and are not limitations on the embodiments of the present application. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also belong to the scope of the embodiments of the present application. The scope of patent protection of the embodiments of the present application should be limited by the claims.

Claims

1. A pressing area detection method, comprising: Obtaining the pressing data obtained by the user pressing the touch screen; Inputting the pressing data into a machine learning model to perform center point prediction to obtain predicted center coordinates of the pressing area; Furthermore, a morphology prediction is performed based on the pressing data and the predicted center coordinates to obtain predicted morphology parameters of the pressing area.

2. The method according to claim 1, wherein: The step of inputting the pressing data into a machine learning model to predict the center point and obtain the predicted center coordinates of the pressing domain includes: Inputting the pressing data into a machine learning model to perform feature extraction to obtain feature information of the pressing data, and performing center point prediction based on the feature information of the pressing data to obtain predicted center coordinates of the pressing area; The performing morphology prediction according to the pressing data and the predicted center coordinates to obtain the predicted morphology parameters of the pressing area includes: Morphology prediction is performed according to the characteristic information of the pressing data and the preset center coordinates to obtain predicted morphology parameters of the pressing area.

3. The method according to claim 1, wherein: The predicted morphological parameters include at least one of a major axis parameter, a minor axis parameter, a major axis inclination parameter, and a minor axis inclination parameter of a shape corresponding to the pressed area.

4. The method according to any one of claims 1 to 3, wherein: The step of obtaining the pressing data obtained by the user pressing the touch screen includes: Acquiring raw capacitance data obtained by a user pressing the touch screen; The capacitance data corresponding to the pressed area in the original capacitance data is expanded to obtain target capacitance data, and the target capacitance data is determined as the pressed data.

5. The method according to any one of claims 1 to 3, wherein: The method further comprises: Displaying a given pressing area on the touch screen to prompt the user to perform a pressing operation on the given pressing area; Acquire a second pressing data sample obtained by the user performing a pressing operation on a given pressing area on the touch screen, wherein the second pressing data sample is used to fine-tune the machine learning model to obtain a machine learning model for the user.

6. The method according to any one of claims 1 to 3, wherein: The method further comprises: The terminal device on which the touch screen is installed is controlled according to the predicted center coordinates of the pressed area and / or the predicted morphological parameters.

7. A model training method, comprising: Acquire a first training sample for training a machine learning model, wherein the first training sample includes a first pressing data sample obtained by a user pressing the touch screen, a first center point true value corresponding to the first pressing data sample, and a first morphological parameter true value corresponding to the first pressing data sample; Inputting the first training sample into the machine learning model to perform center point prediction to obtain a first predicted center coordinate sample of the pressed area, and performing morphology prediction based on the first pressed data sample and the first predicted center coordinate sample to obtain a first predicted morphological parameter sample of the pressed area; The machine learning model is trained based on the first predicted center coordinate sample, the first center point true value, the first predicted morphological parameter sample and the first morphological parameter true value.

8. The method according to claim 7, wherein: The step of inputting the first training sample into the machine learning model to perform center point prediction to obtain a first predicted center coordinate sample of the pressed area includes: Inputting the first training sample into the machine learning model, performing feature extraction on a first pressing data sample in the first training sample to obtain feature information of the first pressing data sample, and performing center point prediction based on the feature information of the first pressing data sample to obtain a first predicted center coordinate sample of the pressing area; The performing morphology prediction according to the first pressing data sample and the first predicted center coordinate sample to obtain a first predicted morphology parameter sample of the pressing area includes: Morphology prediction is performed according to the feature information of the first pressing data sample and the first predicted center coordinate sample to obtain a first predicted morphology parameter sample of the pressing area.

9. The method according to claim 8, wherein: The machine learning model includes: a feature extraction part, a first perception layer part and a second perception layer part; The step of extracting features from the first pressing data sample in the first training sample to obtain feature information of the first pressing data sample, and performing center point prediction according to the feature information of the first pressing data sample to obtain a first predicted center coordinate sample of the pressing area includes: Performing feature extraction on the first pressing data sample in the first training sample through the feature extraction part to obtain feature information of the first pressing data sample, and performing center point prediction on the feature information of the first pressing data sample through the first perception layer part to obtain a first predicted center coordinate sample of the pressing area; The performing morphology prediction according to the feature information of the first pressing data sample and the first predicted center coordinate sample to obtain the first predicted morphology parameter sample of the pressing area includes: The feature information of the first pressing data sample and the first predicted center coordinate sample are used to perform morphological prediction through the second perception layer to obtain a first predicted morphological parameter sample of the pressing area.

10. The method according to claim 9, wherein: The feature extraction part includes a linear transformation layer, at least one convolution layer and at least one pooling layer; The step of extracting features of the first pressing data sample in the first training sample by the feature extraction part to obtain feature information of the first pressing data sample includes: The first compression data sample in the first training sample is sequentially subjected to corresponding linear transformation processing, convolution processing and pooling processing through the linear transformation layer, at least one convolution layer and at least one pooling layer of the feature extraction part to obtain feature information of the first compression data sample.

11. The method according to claim 7, wherein: The first predicted morphological parameter sample includes at least one of a long-axis parameter sample, a short-axis parameter sample, a long-axis inclination parameter sample, and a short-axis inclination parameter sample of a shape corresponding to the pressed area.

12. The method according to any one of claims 7 to 11, wherein: The first pressing data sample obtained by the user pressing the touch screen is obtained in the following manner: Acquire an original capacitance data sample, where the original capacitance data sample is a capacitance data sample obtained for representing a user pressing operation on the touch screen; The capacitance data corresponding to the pressed area in the original capacitance data sample is expanded to obtain a target capacitance data sample, and the target capacitance data sample is determined as the first pressed data sample.

13. The method according to any one of claims 7 to 11, wherein: The method further comprises: Obtaining a second training sample, wherein the second training sample includes: a second pressing data sample obtained by a user performing a pressing operation on a given pressing area on the touch screen, a second center point true value corresponding to the second pressing data sample, and a second morphological parameter true value corresponding to the second pressing data sample; The machine learning model is fine-tuned using the second training sample to obtain a machine learning model for the user.

14. The method according to claim 13, wherein: The second center point true value is the center point coordinates of the shape corresponding to the given pressed area, and the second morphological parameter true value is the morphological parameter of the shape corresponding to the given pressed area.

15. The method according to claim 13, wherein: The fine-tuning training of the machine learning model using the second training sample includes: Inputting the second training sample into the machine learning model to perform center point prediction to obtain a second predicted center coordinate sample of the given pressed area, and performing morphology prediction based on the second pressed data sample and the second predicted center coordinate sample to obtain a second predicted morphological parameter sample of the given pressed area; Based on the second predicted center coordinate sample, the second center point true value, the second predicted morphological parameter sample and the second morphological parameter true value, the machine learning model is fine-tuned and trained.

16. A pressure detection device, comprising: An acquisition module, used to acquire the pressing data obtained by the user pressing the touch screen; The prediction module is used to input the pressing data into the machine learning model to perform center point prediction to obtain the predicted center coordinates of the pressing area; and to perform morphological prediction based on the pressing data and the predicted center coordinates to obtain the predicted morphological parameters of the pressing area.

17. A model training device, comprising: A sample acquisition module, used to acquire a first training sample for training a machine learning model, wherein the first training sample includes a first pressing data sample obtained by a user pressing the touch screen, a first center point true value corresponding to the first pressing data sample, and a first morphological parameter true value corresponding to the first pressing data sample; a parameter prediction module, configured to input the first training sample into the machine learning model to perform center point prediction, obtain a first predicted center coordinate sample of the pressed area, and perform morphology prediction based on the first pressed data sample and the first predicted center coordinate sample, obtain a first predicted morphological parameter sample of the pressed area; A model training module is used to train the machine learning model based on the first predicted center coordinate sample, the first center point true value, the first predicted morphological parameter sample and the first morphological parameter true value.

18. An electronic device, comprising: A processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; The memory is used to store computer programs; The processor is configured to execute the method according to any one of claims 1 to 15 by running the computer program stored in the memory.

19. A computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method according to any one of claims 1 to 15.

20. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 15.

Images