Screen touch management method, intelligent terminal, device and readable storage medium

Abstract

This invention discloses a screen touch management method, comprising the following steps: when the front-facing camera is not activated and an object is detected approaching the front-facing camera based on a proximity sensor, the front-facing camera is activated, and it is determined whether the approaching object is a user's finger. Then, upon determining that the approaching object is a user's finger, a tracking mode is activated, and the movement trajectory of the user's finger is acquired based on the front-facing camera and the tracking mode. Next, the four-dimensional coordinates of sampling points on the movement trajectory are acquired based on a preset time interval. Then, the touch type corresponding to the movement trajectory is determined based on the four-dimensional coordinates of each sampling point and a preset touch template, and a touch operation is performed based on the touch type. This invention also discloses a device, a smart terminal, and a readable storage medium. It enables the under-display camera opening to respond to screen touch operations.

Description

Technical Field

[0001] This invention relates to the field of smart terminal technology, and in particular to a screen touch management method, a smart terminal, a device, and a readable storage medium. Background Technology

[0002] With the rapid development of smart terminal technologies such as mobile phones and tablets, the application of smart terminals such as mobile phones and tablets is becoming more and more widespread, and users' requirements for smart terminal screens are also getting higher and higher.

[0003] With the development of smart terminal screen technology, various screens with ultra-high screen-to-body ratios have begun to emerge, including waterdrop and notch designs, making full-screen displays a major trend in smart terminals. To achieve a truly full-screen display, an under-display camera is a crucial technical issue that must be resolved. The under-display camera must be able to respond normally to touch operations, but current technologies such as waterdrop and notch displays have not yet solved this problem. Summary of the Invention

[0004] The main objective of this invention is to provide a screen touch management method, a smart terminal, a device, and a readable storage medium, aiming to solve the technical problem that existing under-display camera openings cannot respond to screen touch operations.

[0005] To achieve the above objectives, the present invention provides a screen touch management method, which includes the following steps:

[0006] When the front-facing camera is not turned on and an object is detected approaching the front-facing camera based on the proximity sensor, the front-facing camera is activated, and it is determined whether the approaching object is the user's finger.

[0007] When it is determined that the approaching object is the user's finger, the tracking mode is activated, and the movement trajectory of the user's finger is obtained based on the front-facing camera and the tracking mode;

[0008] The four-dimensional coordinates of the sampling points on the motion trajectory are obtained based on a preset time interval. The four-dimensional coordinates include the horizontal X-direction coordinates, the vertical Y-direction coordinates, the near and far Z-direction coordinates, and the trajectory time.

[0009] The touch type corresponding to the motion trajectory is determined based on the four-dimensional coordinates of each sampling point and the preset touch template, and the touch operation is performed based on the touch type.

[0010] In addition, to achieve the above objectives, the present invention also provides a smart terminal, the smart terminal comprising: a memory, a processor, and a screen touch management program stored in the memory and executable on the processor, wherein the screen touch management program, when executed by the processor, implements the steps of the screen touch management method described above.

[0011] In addition, to achieve the above objectives, the present invention also provides a readable storage medium storing a screen touch management program, wherein the screen touch management program, when executed by a processor, implements the steps of the screen touch management method described in any of the above claims.

[0012] This invention activates the front-facing camera when it is not activated and an object is detected approaching it by the proximity sensor. It then determines whether the approaching object is a user's finger. Upon confirming the object is a user's finger, it activates the tracking mode and acquires the movement trajectory of the user's finger based on the front-facing camera and the tracking mode. Next, it acquires the four-dimensional coordinates of sampling points along the movement trajectory at preset time intervals. These four-dimensional coordinates include horizontal (X) coordinates, vertical (Y) coordinates, near / far (Z) coordinates, and trajectory time. Finally, it determines the touch type corresponding to the movement trajectory based on the four-dimensional coordinates of each sampling point and a preset touch template, and performs a touch operation based on the touch type. By acquiring the user's finger movement trajectory through the tracking mode of the smart terminal, matching the four-dimensional coordinates of the sampling points along the movement trajectory with a preset touch template, and thus determining the touch type, the under-display camera opening can respond to screen touch operations. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the structure of a smart terminal in the hardware operating environment involved in the embodiments of the present invention;

[0014] Figure 2 This is a flowchart illustrating the first embodiment of the screen touch management method of the present invention;

[0015] Figure 3 This is a flowchart illustrating the second embodiment of the screen touch management method of the present invention;

[0016] Figure 4 This is a flowchart illustrating the third embodiment of the screen touch management method of the present invention;

[0017] Figure 5 This is a flowchart illustrating the fourth embodiment of the screen touch management method of the present invention;

[0018] Figure 6 This is a schematic diagram of the functional modules of an embodiment of the screen touch management device of the present invention.

[0019] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0020] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0021] like Figure 1 As shown, Figure 1 This is a schematic diagram of the structure of a smart terminal in the hardware operating environment involved in the embodiments of the present invention.

[0022] like Figure 1 As shown, the smart terminal may include: a processor 1001, such as a CPU; a network interface 1004; a user interface 1003; a memory 1005; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard. Optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0023] Optionally, the smart terminal may also include a camera, RF (Radio Frequency) circuitry, sensors, audio circuitry, WiFi module, etc. Sensors may include light sensors, motion sensors, and other sensors. Specifically, light sensors may include ambient light sensors and proximity sensors. The ambient light sensor can adjust the brightness of the display screen according to the ambient light level, while the proximity sensor can turn off the display screen and / or backlight when the mobile terminal is moved to the ear. As a type of motion sensor, an attitude sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity, and can be used for applications that identify the mobile terminal's attitude (such as landscape / portrait switching, related games, magnetometer attitude calibration), vibration recognition functions (such as pedometers, tapping), etc. Of course, the smart terminal may also be equipped with other sensors such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, which will not be elaborated here.

[0024] Those skilled in the art will understand that Figure 1 The smart terminal structure shown does not constitute a limitation on the terminal and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0025] like Figure 1As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a screen touch management program.

[0026] exist Figure 1 In the terminal shown, the network interface 1004 is mainly used to connect to the backend server and communicate with the backend server; the user interface 1003 is mainly used to connect to the client (user terminal) and communicate with the client; and the processor 1001 can be used to call the screen touch management program stored in the memory 1005.

[0027] In this embodiment, the smart terminal includes: a memory 1005, a processor 1001, and a screen touch management program stored in the memory 1005 and executable on the processor 1001. When the processor 1001 calls the screen touch management program stored in the memory 1005, it executes the steps of the screen touch management method provided in the various embodiments of this application.

[0028] The present invention also provides a screen touch management method, referring to Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the screen touch management method of the present invention.

[0029] This invention provides an embodiment of a screen touch management method. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0030] In this embodiment, the screen touch management method includes:

[0031] Step S100: When the front camera is not turned on and an object is detected approaching the front camera based on the proximity sensor, the front camera is turned on and it is determined whether the approaching object is the user's finger.

[0032] In this embodiment, current smart terminals all have quasi-full-screen displays, but true full-screen phones will emerge in the future, and full-screen phones will be an inevitable trend. Full-screen phones have already overcome technical hurdles such as under-display fingerprint scanning and screen sound. The remaining technical hurdle is the under-display camera, which requires solving two main problems: first, the display on the screen at the camera's opening location; and second, touch control at the camera's opening location. The technical solution of this invention addresses the second problem: touch control at the camera's opening location.

[0033] Specifically, smart terminals incorporate a proximity sensor at the camera cutout location to detect the presence of an object near the under-display camera, triggering its activation. A proximity sensor is a device capable of sensing the approach of an object. It utilizes the sensitivity of displacement sensors to detect approaching objects and outputs a corresponding switching signal; therefore, it is often referred to as a proximity switch. It is a general term for sensors that replace contact-based detection methods such as switches, aiming to detect objects without physical contact. It can detect the movement and presence of an object and convert it into an electrical signal. Additionally, smart terminals feature a tracking mode, which captures the movement trajectory of the user's finger based on video images captured by the camera. When the front-facing camera is off and the proximity sensor detects an object approaching, the camera is activated to capture an image of the approaching object. A preset image recognition algorithm then determines whether the approaching object is the user's finger. It should be noted that if the front-facing camera is being used for taking photos, facial recognition, video recording, or video calls, and the proximity sensor detects an object approaching, it will not respond to touch events at the corresponding cutout location of the under-display camera.

[0034] Step S200: When it is determined that the approaching object is a user's finger, the tracking mode is activated, and the movement trajectory of the user's finger is obtained based on the front camera and the tracking mode.

[0035] In this embodiment, when the proximity sensor detects an object approaching the front-facing camera, and determines that the approaching object is the user's finger based on the front-facing camera of the smart terminal and a preset image recognition algorithm, the smart terminal's tracking mode is activated. Then, the movement trajectory of the user's finger is captured based on the video image captured by the camera. A movement trajectory refers to the spatial characteristics of an action formed by the path traversed by a part of the body from its starting position to its ending position. A movement trajectory is represented by its direction, form, and amplitude. In this invention, the movement trajectory of the user's finger refers to the spatial characteristics of the action formed by the path traversed by the finger within the shooting area from its starting position to its ending position after the front-facing camera is activated. The direction of the finger's movement trajectory changes continuously, and the form of the movement trajectory is a curve.

[0036] Specifically, moving target tracking is to find the moving target of interest in real time in each image of a sequence of images, including motion parameters such as position, velocity and acceleration. In this application, the tracking mode of the smart terminal uses existing tracking algorithms to identify the user's finger from the video image captured by the front camera. The path taken by the finger is the movement trajectory of the finger.

[0037] Step S300: Obtain the four-dimensional coordinates of the sampling points on the motion trajectory based on a preset time interval, wherein the four-dimensional coordinates include the horizontal X-direction coordinates, the vertical Y-direction coordinates, the near and far Z-direction coordinates, and the trajectory time;

[0038] In this embodiment, the movement trajectory of the finger can be represented by a four-dimensional coordinate system, consisting of a horizontal X-axis, a vertical Y-axis, a near / far Z-axis, and a time T-axis. The origin of the coordinate system can be set according to actual conditions. For ease of description, in this invention, the lower left corner of the opening corresponding to the under-display front camera is taken as the origin. Relative to the origin, the horizontal direction to the right is the positive X-axis, the vertical direction upwards is the positive Y-axis, and the direction perpendicular to the screen of the smart terminal, away from the screen, is the positive Z-axis. The time T-axis represents the actual time. The movement trajectory of the finger is a curve, represented by a four-dimensional coordinate system. Therefore, the movement trajectory can be sampled at preset time intervals to obtain multiple sampling points. Each sampling point is represented by four-dimensional coordinates, including the horizontal X-axis coordinate, the vertical Y-axis coordinate, the near / far Z-axis coordinate, and the trajectory time. These sampling points are used to determine the touch type corresponding to the finger operation, such as determining whether the current finger operation is a leftward swipe or a downward swipe. The preset time interval is determined according to actual conditions, and the preset time interval determines the number of sampling points, ensuring at least two sampling points.

[0039] Step S400: Determine the touch type corresponding to the motion trajectory based on the four-dimensional coordinates of each sampling point and the preset touch template, and perform touch operation based on the touch type.

[0040] In this embodiment, after obtaining the four-dimensional coordinates of sampling points on the motion trajectory according to a preset time interval, the touch type corresponding to the motion trajectory is further determined based on the four-dimensional coordinates of the sampling points and a preset touch template. The touch types include: left swipe touch, right swipe touch, up swipe touch, down swipe touch, single click touch, double click touch, and long press touch. The preset touch template stores left and right swipe threshold data, up and down swipe threshold data, and click threshold data. The left and right swipe threshold data is a three-dimensional array, including X-direction data, Y-direction data, and Z-direction data, used to determine whether it is a left or right swipe touch. Similarly, the up and down swipe threshold data and click threshold data are also three-dimensional arrays, including X-direction data, Y-direction data, and Z-direction data, used to determine whether it is an up and down swipe touch and a click touch, respectively.

[0041] Specifically, step S400 includes:

[0042] Step S410: Determine the maximum X-direction coordinate difference, the maximum Y-direction coordinate difference, and the maximum Z-direction coordinate difference based on the four-dimensional coordinate values ​​of each sampling point;

[0043] In this embodiment, the X-direction coordinate difference, Y-direction coordinate difference, and Z-direction coordinate difference between each sampling point are calculated respectively. The maximum X-direction coordinate difference is obtained from all X-direction coordinate differences, the maximum Y-direction coordinate difference is obtained from all Y-direction coordinate differences, and the maximum Z-direction coordinate difference is obtained from all Z-direction coordinate differences.

[0044] Step S420: Determine the touch type corresponding to the motion trajectory based on the maximum X-direction coordinate difference, the maximum Y-direction coordinate difference, the maximum Z-direction coordinate difference, and the preset touch template.

[0045] In this embodiment, the preset touch template stores left and right swipe threshold data, up and down swipe threshold data, and click threshold data. The maximum X-axis coordinate difference, the maximum Y-axis coordinate difference, and the maximum Z-axis coordinate difference are matched with the preset touch template, and the touch type corresponding to the motion trajectory is determined based on the matching result.

[0046] The screen touch management method proposed in this embodiment activates the front-facing camera when it is not turned on and an object is detected approaching it by the proximity sensor. It then determines whether the approaching object is a user's finger. The method then acquires the movement trajectory of the user's finger based on the front-facing camera and the tracking mode. Next, it acquires the four-dimensional coordinates of sampling points on the movement trajectory at preset time intervals. These four-dimensional coordinates include horizontal X-axis coordinates, vertical Y-axis coordinates, near / far Z-axis coordinates, and trajectory time. Finally, it determines the touch type corresponding to the movement trajectory based on the four-dimensional coordinates of each sampling point and a preset touch template, and performs a touch operation based on the touch type. By acquiring the user's finger movement trajectory through the tracking mode of the smart terminal, matching the four-dimensional coordinates of the sampling points on the movement trajectory with a preset touch template, and thus determining the touch type, the method enables the under-display camera opening to respond to screen touch operations.

[0047] Based on the first embodiment, referring to Figure 3 The present invention provides a second embodiment of the screen touch management method. In this embodiment, step S420 includes:

[0048] Step S421: When it is determined that the maximum X-direction coordinate difference is greater than the maximum Y-direction coordinate difference and the maximum X-direction coordinate difference is greater than the maximum Z-direction coordinate difference, the left and right sliding threshold data is obtained.

[0049] In this embodiment, when a finger touches the screen position of the front-facing camera opening from right to left, and the area of ​​the camera obscured by the finger changes from the right half -> all -> left half, it is determined as a left swipe touch event. The X-axis coordinate of the motion trajectory corresponding to the left swipe touch event changes the most, while the Y and Z axes show slight changes. Similarly, when a finger touches the screen position of the front-facing camera opening from left to right, and the area of ​​the camera obscured by the finger changes from the left half -> all -> right half, it is determined as a right swipe touch event. The X-axis coordinate of the motion trajectory corresponding to the right swipe touch event changes the most, while the Y and Z axes show slight changes. Therefore, when the maximum X-axis coordinate difference is greater than the maximum Y-axis coordinate difference, and the maximum X-axis coordinate difference is greater than the maximum Z-axis coordinate difference, it can be preliminarily determined whether the current touch is a left swipe touch or a right swipe touch, and thus the left and right swipe threshold data in the preset touch template are obtained.

[0050] Step S422: When the maximum X-direction coordinate difference, the maximum Y-direction coordinate difference, and the maximum Z-direction coordinate difference match the left and right swipe threshold data, determine whether the touch type is left swipe touch or right swipe touch.

[0051] In this embodiment, it is further determined whether the maximum X-direction coordinate difference, the maximum Y-direction coordinate difference, and the maximum Z-direction coordinate difference match the left and right swipe threshold data. When the above data match, it can be further determined whether it is a left swipe touch or a right swipe touch. When the above data do not match, the current motion trajectory corresponds to an invalid touch, and the smart terminal does not perform any operation.

[0052] Specifically, step S422 includes:

[0053] Step a, obtain the first sampling point and the second sampling point corresponding to the maximum X-direction coordinate difference, wherein the X-direction coordinate of the first sampling point is greater than the X-direction coordinate of the second sampling point;

[0054] In this embodiment, when the maximum difference in X-direction coordinates, the maximum difference in Y-direction coordinates, and the maximum difference in Z-direction coordinates match the left and right swipe threshold data, it is further determined whether the current swipe is a left or right swipe. Specifically, two sampling points corresponding to the maximum difference in X-direction coordinates are obtained: a first sampling point and a second sampling point. The X-direction coordinate of the first sampling point is set to be greater than the X-direction coordinate of the second sampling point, and the time corresponding to the two sampling points is further compared.

[0055] Step b: When the trajectory time of the first sampling point is later than the trajectory time of the second sampling point, determine that the touch type is right swipe touch;

[0056] In this embodiment, if it is a right swipe touch, that is, swiping along the positive X-axis direction, the X-axis coordinate of the sampling point with an earlier trajectory time is less than the X-axis coordinate of the sampling point with a later trajectory time. Therefore, when the trajectory time of the first sampling point is later than the trajectory time of the second sampling point, it is determined that the touch type corresponding to the current motion trajectory is a right swipe touch.

[0057] Step c: When the trajectory time of the first sampling point is earlier than the trajectory time of the second sampling point, it is determined that the touch type is left swipe touch.

[0058] In this embodiment, if it is a left swipe touch, that is, swiping along the negative X coordinate direction, the X coordinate of the sampling point with an earlier trajectory time is greater than the X coordinate of the sampling point with a later trajectory time. Therefore, when the trajectory time of the first sampling point is earlier than the trajectory time of the second sampling point, it is determined that the touch type corresponding to the current motion trajectory is a left swipe touch.

[0059] The screen touch management method proposed in this embodiment obtains the left and right swipe threshold data when it is determined that the maximum X-direction coordinate difference is greater than the maximum Y-direction coordinate difference and the maximum X-direction coordinate difference is greater than the maximum Z-direction coordinate difference. Then, when it is determined that the maximum X-direction coordinate difference, the maximum Y-direction coordinate difference, and the maximum Z-direction coordinate difference match the left and right swipe threshold data, it determines that the touch type is a left swipe touch or a right swipe touch. This accurately determines whether the touch type is a left swipe touch or a right swipe touch, enabling the under-display camera opening position to respond to screen touch operations.

[0060] Based on the second embodiment, referring to Figure 4 The present invention provides a third embodiment of the screen touch management method. In this embodiment, step S420 further includes:

[0061] Step S423: When it is determined that the maximum Y-direction coordinate difference is greater than the maximum X-direction coordinate difference and the maximum Y-direction coordinate difference is greater than the maximum Z-direction coordinate difference, the up and down sliding threshold data is obtained.

[0062] In this embodiment, when a finger touches the screen position of the front-facing camera opening from bottom to top, and the area of ​​the camera obscured by the finger changes from the lower half -> all -> upper half, it is determined as an upward swipe touch event. The Y-axis coordinate of the motion trajectory corresponding to the upward swipe touch event changes the most, while the X and Z axes show slight changes. Similarly, when a finger touches the screen position of the front-facing camera opening from top to bottom, and the area of ​​the camera obscured by the finger changes from the upper half -> all -> lower half, it is determined as a downward swipe touch event. The Y-axis coordinate of the motion trajectory corresponding to the downward swipe touch event changes the most, while the X and Z axes show slight changes. Therefore, when the maximum difference in the Y-axis coordinate is greater than the maximum difference in the X-axis coordinate, and the maximum difference in the Y-axis coordinate is greater than the maximum difference in the Z-axis coordinate, it can be preliminarily determined whether the current touch is an upward swipe touch or a downward swipe touch. Thus, the upward and downward swipe threshold data in the preset touch template are obtained.

[0063] Step S424: When the maximum X-direction coordinate difference, the maximum Y-direction coordinate difference, and the maximum Z-direction coordinate difference match the up and down sliding threshold data, determine whether the touch type is up-slide touch or down-slide touch.

[0064] In this embodiment, it is further determined whether the maximum X-direction coordinate difference, the maximum Y-direction coordinate difference, and the maximum Z-direction coordinate difference match the vertical swipe threshold data. When the above data match, it can be further determined whether it is an upward swipe touch or a downward swipe touch. When the above data do not match, the current motion trajectory corresponds to an invalid touch, and the smart terminal does not perform any operation.

[0065] Specifically, step S424 includes:

[0066] Step d: Obtain the third and fourth sampling points corresponding to the maximum Y-direction coordinate difference, wherein the Y-direction coordinate of the third sampling point is greater than the Y-direction coordinate of the fourth sampling point;

[0067] In this embodiment, when the maximum X-axis coordinate difference, the maximum Y-axis coordinate difference, and the maximum Z-axis coordinate difference match the vertical swipe threshold data, it is further determined whether the current touch is an upward swipe or a downward swipe. Specifically, two sampling points corresponding to the maximum Y-axis coordinate difference are obtained: the third sampling point and the fourth sampling point. The Y-axis coordinate of the third sampling point is set to be greater than the Y-axis coordinate of the fourth sampling point, and the time corresponding to the two sampling points is further compared.

[0068] Step e: When the trajectory time of the third sampling point is later than the trajectory time of the fourth sampling point, it is determined that the touch type is an up-swipe touch.

[0069] In this embodiment, if it is an upward touch, that is, sliding along the positive Y-axis direction, the Y-axis coordinate of the sampling point with an earlier trajectory time is less than the Y-axis coordinate of the sampling point with a later trajectory time. Therefore, when the trajectory time of the third sampling point is later than the trajectory time of the fourth sampling point, it is determined that the touch type corresponding to the current motion trajectory is an upward touch.

[0070] Step f: When the trajectory time of the third sampling point is earlier than the trajectory time of the fourth sampling point, it is determined that the touch type is a swipe-down touch.

[0071] In this embodiment, if it is a swipe down touch, that is, sliding along the negative Y-axis, the Y-axis coordinate of the sampling point with an earlier trajectory time is greater than the Y-axis coordinate of the sampling point with a later trajectory time. Therefore, when the trajectory time of the third sampling point is earlier than the trajectory time of the fourth sampling point, it is determined that the touch type corresponding to the current motion trajectory is a swipe down touch.

[0072] The screen touch management method proposed in this embodiment obtains the vertical swiping threshold data when it is determined that the maximum Y-direction coordinate difference is greater than the maximum X-direction coordinate difference and the maximum Y-direction coordinate difference is greater than the maximum Z-direction coordinate difference. Then, when it is determined that the maximum X-direction coordinate difference, the maximum Y-direction coordinate difference, and the maximum Z-direction coordinate difference match the vertical swiping threshold data, it determines that the touch type is an upward swipe touch or a downward swipe touch. This accurately determines whether the touch type is an upward swipe touch or a downward swipe touch, enabling the under-display camera opening to respond to screen touch operations.

[0073] Based on the third embodiment, referring to Figure 5 The present invention provides a fourth embodiment of the screen touch management method. In this embodiment, step S420 further includes:

[0074] Step S425: When it is determined that the maximum Z-direction coordinate difference is greater than the maximum X-direction coordinate difference and the maximum Z-direction coordinate difference is greater than the maximum Y-direction coordinate difference, the click threshold data is obtained.

[0075] In this embodiment, when a finger touches the screen position of the front-facing camera opening from a distance and then moves away from it within a short period of time, such as 1 second, it is determined as a single-click touch event. Two consecutive single-click touches within a short period of time are determined as a double-click touch event. When a finger touches the screen position of the front-facing camera opening from a distance, stays for a period of time, and then moves away from it, it is determined as a long-press touch event. Regardless of whether it is a single-click, double-click, or long-press touch, the Z-axis coordinate of the movement trajectory corresponding to these single-click touch events changes the most, while the X and Y axes show slight changes. Therefore, when the maximum Z-axis coordinate difference is greater than the maximum X-axis coordinate difference and the maximum Y-axis coordinate difference, it can be preliminarily determined that the current touch is a single-click touch, and thus the click threshold data in the preset touch template is obtained.

[0076] Step S426: When the maximum X-direction coordinate difference, the maximum Y-direction coordinate difference, and the maximum Z-direction coordinate difference match the click threshold data, the touch type is determined to be click touch.

[0077] In this embodiment, it is further determined whether the maximum X-direction coordinate difference, the maximum Y-direction coordinate difference, and the maximum Z-direction coordinate difference match the click threshold data. When the above data match, it can be further determined whether it is a single-click touch, a double-click touch, or a long-press touch. When the above data do not match, the current motion trajectory corresponds to an invalid touch, and the smart terminal does not perform any operation.

[0078] Step g: Determine the number of times the front camera is blocked and the duration of each blockage based on the motion trajectory.

[0079] In this embodiment, the touch control includes single-click touch, double-click touch, and long-press touch. The duration of obstruction of the front camera by single-click and double-click touch is relatively short, for example, less than 1 second, while the duration of obstruction of the front camera by long-press touch is relatively long, for example, greater than or equal to 2 seconds. Therefore, it is necessary to determine the number of times the front camera is obstructed and the duration of obstruction each time based on the motion trajectory, and then further determine whether it is a single-click touch, double-click touch, or long-press touch.

[0080] Step h: When the duration of occlusion meets the long press condition, determine that the touch type is a long press touch.

[0081] In this embodiment, if there is a long-press occlusion duration greater than a preset duration among all occlusion durations, the occlusion duration is determined to meet the long-press condition. The preset duration is determined based on actual conditions; for example, the preset duration is equal to 2 seconds. When the occlusion duration is determined to meet the long-press condition, the current touch type is determined to be a long-press touch.

[0082] Step i: When the duration of occlusion does not meet the long press condition, and the number of occlusions is equal to the first preset value, the touch type is determined to be single-click touch.

[0083] Step j: When it is determined that the occlusion duration does not meet the long press condition, and the number of occlusions is greater than or equal to the second preset value, the touch type is determined to be double-touch, wherein the second preset value is greater than the first preset value.

[0084] In this embodiment, if there is no long press occlusion duration longer than the preset duration among all occlusion durations, it is determined that the occlusion duration does not meet the long press condition. Then, the current touch type is single-click touch or double-click touch, and the touch type is further determined based on the number of occlusions.

[0085] Specifically, when the number of occlusions equals a first preset value, the current touch type is determined to be single-click touch; when the number of occlusions is greater than or equal to a second preset value, the touch type is determined to be double-click touch. The second preset value is greater than the first preset value. Preferably, the first preset value is equal to 1 and the second preset value is equal to 2.

[0086] The screen touch management method proposed in this embodiment obtains the click threshold data when it is determined that the maximum Z-direction coordinate difference is greater than the maximum X-direction coordinate difference and the maximum Z-direction coordinate difference is greater than the maximum Y-direction coordinate difference. Then, when it is determined that the maximum X-direction coordinate difference, the maximum Y-direction coordinate difference, and the maximum Z-direction coordinate difference match the click threshold data, it is determined that the touch type is click touch. This accurately determines whether the touch type is single-click touch, double-click touch, or long-press touch, enabling the under-display camera opening to respond to screen touch operations.

[0087] The present invention further provides a screen touch management device, with reference to Figure 6 , Figure 6 This is a schematic diagram of the functional modules of an embodiment of the screen touch management device of the present invention.

[0088] The startup module 10 is used to start the front camera when the front camera is not turned on and an object is detected approaching the front camera based on the proximity sensor, and to determine whether the approaching object is the user's finger.

[0089] The acquisition module 20 is used to activate the tracking mode when it is determined that the approaching object is a user's finger, and acquire the movement trajectory of the user's finger based on the front camera and the tracking mode;

[0090] The sampling module 30 is used to obtain the four-dimensional coordinates of the sampling points on the motion trajectory based on a preset time interval, wherein the four-dimensional coordinates include the horizontal X-direction coordinates, the vertical Y-direction coordinates, the near and far Z-direction coordinates, and the trajectory time.

[0091] The determination module 40 is used to determine the touch type corresponding to the motion trajectory based on the four-dimensional coordinates of each sampling point and the preset touch template, and to perform touch operation based on the touch type.

[0092] Furthermore, the determining module 40 is also used for:

[0093] The maximum X-direction coordinate difference, the maximum Y-direction coordinate difference, and the maximum Z-direction coordinate difference are determined based on the four-dimensional coordinate values ​​of each sampling point.

[0094] The touch type corresponding to the motion trajectory is determined based on the maximum X-direction coordinate difference, the maximum Y-direction coordinate difference, the maximum Z-direction coordinate difference, and the preset touch template.

[0095] Furthermore, the determining module 40 is also used for:

[0096] When it is determined that the maximum X-direction coordinate difference is greater than the maximum Y-direction coordinate difference, and the maximum X-direction coordinate difference is greater than the maximum Z-direction coordinate difference, the left and right sliding threshold data is obtained;

[0097] When the maximum X-direction coordinate difference, the maximum Y-direction coordinate difference, and the maximum Z-direction coordinate difference are matched with the left and right swipe threshold data, the touch type is determined to be either left swipe touch or right swipe touch.

[0098] Furthermore, the determining module 40 is also used for:

[0099] Obtain the first sampling point and the second sampling point corresponding to the maximum X-direction coordinate difference, wherein the X-direction coordinate of the first sampling point is greater than the X-direction coordinate of the second sampling point;

[0100] If the trajectory time of the first sampling point is later than the trajectory time of the second sampling point, the touch type is determined to be a right swipe touch.

[0101] If the trajectory time of the first sampling point is earlier than the trajectory time of the second sampling point, the touch type is determined to be a left swipe touch.

[0102] Furthermore, the determining module 40 is also used for:

[0103] When it is determined that the maximum Y-direction coordinate difference is greater than the maximum X-direction coordinate difference, and the maximum Y-direction coordinate difference is greater than the maximum Z-direction coordinate difference, the up and down sliding threshold data is obtained;

[0104] When the maximum X-direction coordinate difference, the maximum Y-direction coordinate difference, and the maximum Z-direction coordinate difference are matched with the vertical swipe threshold data, the touch type is determined to be either vertical swipe touch or vertical swipe touch.

[0105] Furthermore, the determining module 40 is also used for:

[0106] Obtain the third and fourth sampling points corresponding to the maximum Y-direction coordinate difference, wherein the Y-direction coordinate of the third sampling point is greater than the Y-direction coordinate of the fourth sampling point;

[0107] If the trajectory time of the third sampling point is later than the trajectory time of the fourth sampling point, the touch type is determined to be an up-swipe touch.

[0108] If the trajectory time of the third sampling point is less than the trajectory time of the fourth sampling point, the touch type is determined to be a swipe-down touch.

[0109] Furthermore, the determining module 40 is also used for:

[0110] When it is determined that the maximum Z-direction coordinate difference is greater than the maximum X-direction coordinate difference, and the maximum Z-direction coordinate difference is greater than the maximum Y-direction coordinate difference, the click threshold data is obtained;

[0111] When the maximum X-direction coordinate difference, the maximum Y-direction coordinate difference, and the maximum Z-direction coordinate difference match the click threshold data, the touch type is determined to be click touch.

[0112] Furthermore, the determining module 40 is also used for:

[0113] The number of times the front camera is blocked and the duration of each blockage are determined based on the motion trajectory.

[0114] When the duration of occlusion meets the long press condition, the touch type is determined to be a long press touch.

[0115] When the duration of occlusion does not meet the long press condition, and the number of occlusions is equal to the first preset value, the touch type is determined to be a single-click touch.

[0116] When the duration of occlusion does not meet the long press condition, and the number of occlusions is greater than or equal to a second preset value, the touch type is determined to be double-touch, wherein the second preset value is greater than the first preset value.

[0117] Furthermore, the determining module 40 is also used for:

[0118] If there is a long press occlusion duration that is longer than the preset duration among all occlusion durations, it is determined that the occlusion duration meets the long press condition.

[0119] Furthermore, this embodiment of the invention also proposes a readable storage medium storing a screen touch management program, which, when executed by a processor, implements the steps of the screen touch management method in the above embodiments.

[0120] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0121] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0122] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a system device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0123] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A screen touch management method, applied to a smart terminal equipped with a proximity sensor, wherein the proximity sensor is installed within a preset range of the front-facing camera of the smart terminal, and the smart terminal has a tracking mode, characterized in that, The screen touch management method includes the following steps: When the front-facing camera is not turned on and an object is detected approaching the front-facing camera based on the proximity sensor, the front-facing camera is activated, and it is determined whether the approaching object is the user's finger. When it is determined that the approaching object is the user's finger, the tracking mode is activated, and the movement trajectory of the user's finger is obtained based on the front-facing camera and the tracking mode; The four-dimensional coordinates of the sampling points on the motion trajectory are obtained based on a preset time interval. The four-dimensional coordinates include the horizontal X-direction coordinates, the vertical Y-direction coordinates, the near and far Z-direction coordinates, and the trajectory time. The touch type corresponding to the motion trajectory is determined based on the four-dimensional coordinates of each sampling point and a preset touch template, and a touch operation is performed based on the touch type. The preset touch template includes: left-right swipe threshold data, up-down swipe threshold data, and click threshold data. Determining the touch type corresponding to the motion trajectory includes: determining the maximum X-direction coordinate difference, the maximum Y-direction coordinate difference, and the maximum Z-direction coordinate difference based on the four-dimensional coordinate values ​​of each sampling point; and determining the touch type corresponding to the motion trajectory based on the maximum X-direction coordinate difference, the maximum Y-direction coordinate difference, the maximum Z-direction coordinate difference, and the preset touch template. Specifically, when the maximum X-direction coordinate difference is greater than the maximum Y-direction coordinate difference, and the maximum X-direction coordinate difference is greater than the maximum Z-direction coordinate difference, the left and right swipe threshold data is obtained; when the maximum X-direction coordinate difference, the maximum Y-direction coordinate difference, and the maximum Z-direction coordinate difference match the left and right swipe threshold data, the touch type is determined to be either a left swipe touch or a right swipe touch; or... When it is determined that the maximum Y-direction coordinate difference is greater than the maximum X-direction coordinate difference, and the maximum Y-direction coordinate difference is greater than the maximum Z-direction coordinate difference, the vertical swipe threshold data is obtained; when it is determined that the maximum X-direction coordinate difference, the maximum Y-direction coordinate difference, and the maximum Z-direction coordinate difference match the vertical swipe threshold data, the touch type is determined to be either an upward swipe touch or a downward swipe touch; or... When the maximum Z-direction coordinate difference is greater than the maximum X-direction coordinate difference and the maximum Z-direction coordinate difference is greater than the maximum Y-direction coordinate difference, the click threshold data is obtained; when the maximum X-direction coordinate difference, the maximum Y-direction coordinate difference, and the maximum Z-direction coordinate difference match the click threshold data, the touch type is determined to be click touch.

2. The screen touch management method as described in claim 1, characterized in that, The step of determining whether the touch type is a left swipe touch or a right swipe touch when the maximum X-direction coordinate difference, the maximum Y-direction coordinate difference, and the maximum Z-direction coordinate difference match the left and right swipe threshold data includes: Obtain the first sampling point and the second sampling point corresponding to the maximum X-direction coordinate difference, wherein the X-direction coordinate of the first sampling point is greater than the X-direction coordinate of the second sampling point; If the trajectory time of the first sampling point is later than the trajectory time of the second sampling point, the touch type is determined to be a right swipe touch. If the trajectory time of the first sampling point is earlier than the trajectory time of the second sampling point, the touch type is determined to be a left swipe touch.

3. The screen touch management method as described in claim 1, characterized in that, The step of determining whether the touch type is an up-swipe touch or a down-swipe touch when the maximum X-direction coordinate difference, the maximum Y-direction coordinate difference, and the maximum Z-direction coordinate difference match the up-down swipe threshold data includes: Obtain the third and fourth sampling points corresponding to the maximum Y-direction coordinate difference, wherein the Y-direction coordinate of the third sampling point is greater than the Y-direction coordinate of the fourth sampling point; If the trajectory time of the third sampling point is later than the trajectory time of the fourth sampling point, the touch type is determined to be an up-swipe touch. If the trajectory time of the third sampling point is less than the trajectory time of the fourth sampling point, the touch type is determined to be a swipe-down touch.

4. The screen touch management method as described in claim 1, characterized in that, The click-based touch includes single-click touch, double-click touch, and long-press touch. The step of determining that the touch type is a click-based touch when the maximum X-axis coordinate difference, the maximum Y-axis coordinate difference, and the maximum Z-axis coordinate difference match the click threshold data includes: The number of times the front camera is blocked and the duration of each blockage are determined based on the motion trajectory. When the duration of occlusion meets the long press condition, the touch type is determined to be a long press touch. When the duration of occlusion does not meet the long press condition, and the number of occlusions is equal to the first preset value, the touch type is determined to be a single-click touch. When the duration of occlusion does not meet the long press condition, and the number of occlusions is greater than or equal to a second preset value, the touch type is determined to be double-touch, wherein the second preset value is greater than the first preset value.

5. The screen touch management method as described in claim 4, characterized in that, If there is a long press occlusion duration that is longer than the preset duration among all occlusion durations, it is determined that the occlusion duration meets the long press condition.

6. A screen touch management device, characterized in that, The screen touch management device includes: The startup module is used to start the front camera when the front camera is not turned on and an object is detected approaching the front camera based on the proximity sensor, and to determine whether the approaching object is the user's finger. The acquisition module is used to activate the tracking mode when it is determined that the approaching object is a user's finger, and acquire the movement trajectory of the user's finger based on the front camera and the tracking mode; The sampling module is used to obtain the four-dimensional coordinates of the sampling points on the motion trajectory based on a preset time interval, wherein the four-dimensional coordinates include the horizontal X-direction coordinates, the vertical Y-direction coordinates, the near and far Z-direction coordinates, and the trajectory time; The determination module is used to determine the touch type corresponding to the motion trajectory based on the four-dimensional coordinates of each sampling point and a preset touch template, and to perform a touch operation based on the touch type. The preset touch template includes: left-right swipe threshold data, up-down swipe threshold data, and click threshold data. Determining the touch type corresponding to the motion trajectory includes: determining the maximum X-direction coordinate difference, the maximum Y-direction coordinate difference, and the maximum Z-direction coordinate difference based on the four-dimensional coordinate values ​​of each sampling point; and determining the touch type corresponding to the motion trajectory based on the maximum X-direction coordinate difference, the maximum Y-direction coordinate difference, the maximum Z-direction coordinate difference, and the preset touch template. Specifically, when the maximum X-direction coordinate difference is greater than the maximum Y-direction coordinate difference, and the maximum X-direction coordinate difference is greater than the maximum Z-direction coordinate difference, the left and right swipe threshold data is obtained; when the maximum X-direction coordinate difference, the maximum Y-direction coordinate difference, and the maximum Z-direction coordinate difference match the left and right swipe threshold data, the touch type is determined to be either a left swipe touch or a right swipe touch; or... When it is determined that the maximum Y-direction coordinate difference is greater than the maximum X-direction coordinate difference, and the maximum Y-direction coordinate difference is greater than the maximum Z-direction coordinate difference, the vertical swipe threshold data is obtained; when it is determined that the maximum X-direction coordinate difference, the maximum Y-direction coordinate difference, and the maximum Z-direction coordinate difference match the vertical swipe threshold data, the touch type is determined to be either an upward swipe touch or a downward swipe touch; or... When the maximum Z-direction coordinate difference is greater than the maximum X-direction coordinate difference and the maximum Z-direction coordinate difference is greater than the maximum Y-direction coordinate difference, the click threshold data is obtained; when the maximum X-direction coordinate difference, the maximum Y-direction coordinate difference, and the maximum Z-direction coordinate difference match the click threshold data, the touch type is determined to be click touch.

7. A smart terminal, characterized in that, The smart terminal includes: a memory, a processor, and a screen touch management program stored in the memory and executable on the processor. When the screen touch management program is executed by the processor, it implements the steps of the screen touch management method as described in any one of claims 1 to 5.

8. A readable storage medium, characterized in that, The readable storage medium stores the screen touch management program, which, when executed by a processor, implements the steps of the screen touch management method as described in any one of claims 1 to 5.

Images