Input method and electronic device

Abstract

The present application discloses an input method and an electronic device, which obtain a touch operation and a first image on a touch device, wherein the first image is an image obtained by an image acquisition device performing image acquisition on a target range, and the target range is a spatial range that at least includes a touch area of ​​the touch device; an operating body in the first image is identified to obtain an identification result; based on the identification result, an operation attribute is determined, and the operation attribute includes at least an operation type and / or an operation parameter; based on the touch operation and the operation attribute, a target instruction is obtained; and the target instruction is executed.

Description

Technical Field

[0001] The present application relates to the field of touch technology, and in particular to an input method and an electronic device. Background Art

[0002] With the development of technology, the application of touch screens is becoming more and more widespread. For example, in teaching scenarios, teachers provide teaching services to students through touch screens.

[0003] However, the touch screen relies solely on touch detection, which may result in misoperation. Summary of the invention

[0004] In view of this, the present application provides an input method and an electronic device as follows:

[0005] This application provides an input method, including:

[0006] Obtaining a touch operation and a first image on a touch device, wherein the first image is an image obtained by an image acquisition device acquiring an image of a target range, and the target range is a spatial range that at least includes a touch area of ​​the touch device;

[0007] Recognize the operating body in the first image to obtain a recognition result;

[0008] Determine operation attributes according to the recognition result, wherein the operation attributes at least include an operation type and / or an operation parameter;

[0009] Obtaining a target instruction according to the touch operation and the operation attribute;

[0010] Execute the target instruction.

[0011] Preferably, determining the operation attribute according to the recognition result includes:

[0012] An operation type is determined according to the gesture of the operating body in the recognition result, where the operation type is a content modification type or a display control type.

[0013] Preferably, determining the operation type according to the posture of the operating body in the recognition result includes:

[0014] When the operating body is in a first posture, determining the operation type is a display control type;

[0015] Wherein, the first posture is a posture of relative movement of multiple fingers or a stylus on the touch area;

[0016] or

[0017] When the operating body is in a second posture, determining the operation type is a content modification type;

[0018] The second posture is a posture of a single finger or a stylus pen contacting the touch area, or a posture of fingers or handheld styluses of different users moving independently on the touch area.

[0019] Preferably, determining the operation attribute according to the recognition result includes:

[0020] According to the attributes of the operating body in the recognition result, an operating parameter is determined, and the operating parameter includes: a modification parameter of a trajectory or a control parameter of an interface.

[0021] Preferably, determining the operation parameters according to the attributes of the operating body in the recognition result includes:

[0022] Determine, according to the color corresponding to the operating body and / or the input end corresponding to the operating body, a modification parameter of the track in the operation parameter, wherein the modification parameter of the track includes any one or more of the following: a track color parameter, a track size parameter, and a track erasure size;

[0023] or

[0024] According to the contact area between the operating body and the touch area, the modification parameters of the trajectory in the operation parameters and / or the control parameters of the interface are determined, and the control parameters of the interface at least include: interface zoom ratio and / or zoom center position.

[0025] Preferably, determining the operation parameters according to the attributes of the operating body in the recognition result includes:

[0026] In a case where the operating body is a stylus pen and the stylus pen contacts the touch area with a first end, determining the modification parameters of the track in the operating parameters according to the color of the stylus pen and / or a preset track size corresponding to the first end, or determining the modification parameters of the track in the operating parameters according to a preset erasing size corresponding to the first end;

[0027] or

[0028] In a case where the operating body is multiple fingers or styluses belonging to the same hand or different users and the multiple fingers or styluses move relatively on the touch area, determining the control parameters of the interface in the operation parameters according to the contact areas between the multiple fingers or styluses and the touch area;

[0029] The modification parameters of the track include any one or more of the following: track color parameter, track size parameter, track erasure size, and the control parameters of the interface include at least: interface zoom ratio and / or zoom center position.

[0030] Preferably, before determining the operation attribute according to the recognition result, the method further includes:

[0031] Determining whether the recognition result indicates that the operating body in the first image matches a preset input gesture;

[0032] If the recognition result indicates that the operating body matches the input gesture, the steps of: determining the operation attribute according to the recognition result;

[0033] If the recognition result indicates that the operating body does not match the input gesture, the step of determining the operation attribute according to the recognition result is not performed, or the step of obtaining the target instruction according to the touch operation and the operation attribute is not performed.

[0034] Preferably, after identifying the operating body in the first image to obtain a recognition result, the method further includes:

[0035] Determining a first position of the operating body according to the recognition result;

[0036] Controlling a control interface of the touch device to be at a second position on the touch area, the second position corresponding to the first position;

[0037] Wherein, the control interface is in a visible state or a hidden state.

[0038] Preferably, before determining the operation attribute according to the recognition result, the method further includes:

[0039] Determining whether the recognition result indicates that the first image contains an operating body;

[0040] If the recognition result indicates that the first image contains an operating body, the steps of: determining an operation attribute according to the recognition result;

[0041] If the recognition result indicates that the first image does not include an operating body, a target instruction is obtained according to a touch track in the touch operation.

[0042] The present application also provides an electronic device, comprising:

[0043] Touch devices;

[0044] An image acquisition device, used for acquiring images;

[0045] A processor is used to obtain a touch operation and a first image on the touch device, wherein the first image is an image obtained by the image acquisition device performing image acquisition on a target range, and the target range is a spatial range that at least includes a touch area of ​​the touch device; identify an operating body in the first image to obtain a recognition result; determine operation attributes based on the recognition result, wherein the operation attributes include at least an operation type and / or an operation parameter; obtain a target instruction based on the touch operation and the operation attributes; and execute the target instruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 A flowchart of an input method disclosed in this application;

[0048] FIG2( a ) is a schematic diagram of a touch device disclosed in the present application;

[0049] FIG2( b ) is a schematic diagram of the target range disclosed in the present application;

[0050] FIG2( c ) is a schematic diagram of the installation of the image acquisition device disclosed in the present application;

[0051] FIG2( d ) is another schematic diagram of the installation of the image acquisition device disclosed in the present application;

[0052] FIG2( e ) is another installation diagram of the image acquisition device disclosed in the present application;

[0053] FIG2( f ) is a schematic diagram of the image acquisition range of the image acquisition device disclosed in the present application;

[0054] FIG2( g ) is a schematic diagram of another image acquisition range of the image acquisition device disclosed in the present application;

[0055] Figure 3 Another flow chart of an input method disclosed in the present application;

[0056] FIG4( a ) is a schematic diagram of a first posture disclosed in the present application;

[0057] FIG4( b ) is another schematic diagram of the first posture disclosed in the present application;

[0058] FIG5( a ) is a schematic diagram of a second posture disclosed in the present application;

[0059] FIG5( b ) is another schematic diagram of the second posture disclosed in the present application;

[0060] FIG5( c ) is another schematic diagram of the second posture disclosed in the present application;

[0061] FIG5( d ) is another schematic diagram of the second posture disclosed in the present application;

[0062] Figure 6 A schematic diagram of the stylus disclosed in the present application;

[0063] FIG7( a ) is a schematic diagram of a touch area disclosed in the present application;

[0064] FIG7( b ) is another schematic diagram of the touch area disclosed in the present application;

[0065] FIG7( c ) is another schematic diagram of the touch area disclosed in the present application;

[0066] FIG7( d ) is another schematic diagram of the touch area disclosed in the present application;

[0067] FIG7( e ) is another schematic diagram of the touch area disclosed in the present application;

[0068] FIG7( f ) is another schematic diagram of the touch area disclosed in the present application;

[0069] FIG7( g ) is another schematic diagram of the touch area disclosed in the present application;

[0070] Figure 8 is another flow chart of an input method disclosed in this application;

[0071] FIG9( a ) is a schematic diagram of an input gesture disclosed in the present application;

[0072] FIG9( b ) is a schematic diagram of the posture of the operating body disclosed in the present application;

[0073] Fig.10 Another flow chart of an input method disclosed in this application;

[0074] FIG11( a ) is a schematic diagram of a second position disclosed in the present application;

[0075] FIG11( b ) is a schematic diagram of the second position disclosed in the present application;

[0076] Fig.12 A structural block diagram of an electronic device disclosed in this application;

[0077] Fig.13 This is a flow chart of a control scheme for a large touch screen disclosed in this application;

[0078] FIG14( a ) is a schematic diagram of the back posture disclosed in the present application;

[0079] FIG14( b ) is a schematic diagram of the writing posture disclosed in the present application;

[0080] Fig.15 A schematic diagram of the automatic following of the toolbar disclosed in this application;

[0081] Fig.16 Another flow chart of a control scheme for a large touch screen disclosed in this application;

[0082] FIG. 17( a ) is a schematic diagram of small area erasing disclosed in the present application;

[0083] FIG17( b ) is a schematic diagram of large-area erasing disclosed in the present application;

[0084] FIG. 17( c ) is a schematic diagram of the stylus pen disclosed in the present application;

[0085] FIG17( d ) is a schematic diagram of the thin line writing state disclosed in the present application;

[0086] FIG17( e ) is a schematic diagram of the bold line writing state disclosed in the present application;

[0087] Fig.18 This is another flow chart of a control scheme for a large touch screen disclosed in this application;

[0088] Fig.19 A schematic diagram of the handwriting colors disclosed in this application;

[0089] Fig. 20 This is another flow chart of a control scheme for a large touch screen disclosed in this application;

[0090] FIG21( a ) is a schematic diagram of a finger disclosed in the present application;

[0091] FIG21( b ) is a schematic diagram of a one-handed zooming gesture disclosed in the present application;

[0092] FIG. 21( c ) is a schematic diagram of the two-handed zooming gesture disclosed in this application. DETAILED DESCRIPTION

[0093] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0094] See also Figure 1, which shows a flow chart of an input method provided by an embodiment of the present application. The method in this embodiment can be applied to electronic devices capable of controlling touch operations of a touch device, such as a computer or a server. The technical solution in this embodiment is mainly used to avoid misoperation of the touch device to improve the operation accuracy of the touch device.

[0095] Specifically, the input method in this embodiment may include the following steps:

[0096] Step S101: obtaining a touch operation and a first image on a touch device.

[0097] It should be noted that the touch device is a device with a touch function, such as a display device with a touch function such as a large touch screen.

[0098] In the specific implementation of step S101, when a touch operation is detected on the touch device, the touch operation and the first image acquired by the image acquisition device are acquired. The first image may be the first image that is captured by the image acquisition device when the touch operation is detected, or the first image that is captured by the image acquisition device continuously when the touch operation is detected is extracted from the multiple frames of images acquired by the image acquisition device and is in the same time range as the touch operation.

[0099] It can be understood that the touch device includes at least a touch area, and the user can operate in the touch area to trigger the corresponding function. For example, as shown in the schematic diagram of the touch device in Figure 2(a), the touch device includes at least a touch area (the area within the dotted box), and the user can operate in the touch area to trigger the corresponding function.

[0100] In some embodiments, the first image is an image obtained by an image acquisition device performing image acquisition on a target range, and the target range is a spatial range that at least includes a touch area of ​​a touch device, for example, as shown in the schematic diagram of the target range in FIG2(b), the target range includes not only the touch area of ​​the touch device but also a spatial range in front of the touch area. That is, the image acquisition device performs image acquisition on a spatial range that at least includes the touch area of ​​the touch device to acquire the first image.

[0101] In some embodiments, the image acquisition device may be a device with an image acquisition function such as a camera. The image acquisition device may be set on the body of the touch device, or the image acquisition device may be set at a position outside the body of the touch device; for example, as shown in the installation schematic diagram of the image acquisition device shown in FIG2(c), the image acquisition device may be set in the area above the body of the touch device. Similarly, the image acquisition device may also be set in other areas of the touch device: as shown in the installation schematic diagram of the image acquisition device shown in FIG2(d), the image acquisition device may be set in the side area of ​​the body of the touch device; as shown in the installation schematic diagram of the image acquisition device shown in FIG2(e), the image acquisition device may be set in a position in the same space as the touch device, such as the wall directly in front of the touch device. However, the position where the image acquisition device is set needs to meet a preset installation condition, which is: the image acquisition range of the image acquisition device needs to cover the spatial range of the touch area of ​​the touch device; that is, it is necessary to ensure that the image acquisition device can capture images of the target range to obtain the first image.

[0102] For example, as shown in the image acquisition range schematic diagram of FIG2( f ), when the image acquisition device is disposed in the area above the main body of the touch control device, the image acquisition device can acquire an image of the target range to obtain a first image.

[0103] For another example, as shown in the image acquisition range schematic diagram of FIG2(g), when the image acquisition device is disposed at a position outside the touch device body, the image acquisition device can acquire an image of the target range to obtain a first image.

[0104] Step S102: identifying the operating body in the first image to obtain a recognition result.

[0105] It is understandable that when a user operates a touch device, he or she may operate on the touch device with a finger or a stylus (the user holds the stylus) to trigger a corresponding function. In the specific implementation of step S102, the operating body in the first image is identified to obtain a recognition result, and the operating body in the first image is a finger and / or a stylus.

[0106] Step S103: Determine the operation attribute according to the recognition result.

[0107] In the process of implementing step S103, the operation attributes are determined according to the recognition result, and the operation attributes include at least the operation type and / or the operation parameters; wherein the operation type is used to indicate the type of operation to be performed, such as outputting or erasing a track along the operation position, or scaling the interface according to the operation position, etc., and the operation parameters are used to indicate the parameters required to perform the operation corresponding to the operation type, such as the size of the output or erased track, the interface scaling ratio, etc.

[0108] For example, the trajectory of the specified thickness is displayed according to the recognition result, where "display trajectory" is the operation type and "specify thickness" is the operation parameter.

[0109] Another example: according to the recognition result, it is determined to reduce the canvas by 20%, wherein "reduce the canvas" is the operation type and "20%" is the operation parameter.

[0110] Step S104: Obtain a target instruction according to the touch operation and the operation attribute.

[0111] In the process of specifically implementing step S104, after the operation attribute is determined, a target instruction is obtained according to the touch operation and the operation attribute, and the target instruction is used to respond to the touch operation according to the operation attribute to achieve a corresponding operation result.

[0112] Step S105: Execute the target instruction.

[0113] In the specific implementation of step S105 , the target instruction is executed to respond to the touch operation in the touch area, for example: responding to the touch operation in the touch area to display content or delete content.

[0114] For another example, it is determined according to the recognition result that the canvas is to be reduced by 20%. After the target instruction is obtained and executed, the touch operation is responded to so as to reduce the canvas by 20%.

[0115] It can be seen from the above scheme that in an input method disclosed in an embodiment of the present application, by obtaining the touch operation and the first image on the touch device, since the first image is an image obtained by the image acquisition device performing image acquisition on the target range, the recognition result can be obtained by identifying the operating body in the first image, and then the operation attribute is determined according to the recognition result, and then the target instruction can be obtained and executed by combining the operation type and / or operation parameters in the operation attribute with the touch operation. It can be seen that in the present application, the acquisition of the target instruction is achieved by identifying the operating body of the first image, thereby avoiding the situation where the erroneous operation is caused by relying solely on the detection of touch, achieving the purpose of avoiding erroneous operation, and improving the operation accuracy of the touch device.

[0116] It is understandable that the user specifically operates the touch device through an operating body (finger and / or stylus) to trigger a corresponding function. When the first image is captured, the operating body may be blocked (for example, blocked by the user's body or blocked by other objects), resulting in the first image not containing the operating body. In this case, it is necessary to obtain the target instruction according to the touch trajectory in the touch operation; therefore, it is necessary to determine whether the operating body is blocked before determining the operation attribute. The specific implementation process is detailed in Figure 3 The content shown.

[0117] See also Figure 3, which shows another flow chart of an input method provided by an embodiment of the present application. Before step S103, the input method in this embodiment may further include the following steps:

[0118] Step S106: Determine whether the recognition result indicates that the first image contains an operating body. If the recognition result indicates that the first image contains an operating body, execute step S103; if the recognition result indicates that the first image does not contain an operating body, execute step S107.

[0119] In the specific implementation of step S106, it is determined whether the recognition result indicates that the first image contains the operating body, that is, it is determined whether the operating body is blocked. If the first image contains the operating body (the operating body is not blocked), steps S103 to S105 are executed; if the first image does not contain the operating body (the operating body is blocked), steps S107 and S105 are executed.

[0120] Step S107: Obtain a target instruction according to the touch track in the touch operation, and execute step S105.

[0121] In the specific implementation of step S107, if the first image does not contain the operating body, the target instruction is obtained according to the touch track in the touch operation, and step S105 is performed to execute the target instruction. In other words, if the operating body is blocked, the target instruction cannot be obtained using the first image, and the target instruction can be obtained according to the touch track in the touch operation.

[0122] It can be seen from the above scheme that in an input method disclosed in an embodiment of the present application, after identifying the operating body in the first image to obtain the recognition result, it is first determined whether the operating body is blocked according to the recognition result. If the operating body is blocked, the target instruction is obtained according to the touch track in the touch operation. If the operating body is not blocked, the operation attribute is determined according to the recognition result, and then the target instruction can be obtained and executed through the operation attribute and the touch operation. It can be seen that in the present application, when the operating body is not blocked, the target instruction can be obtained by identifying the operation of the first image, thereby avoiding the situation where the erroneous operation is caused by relying solely on the detection of touch, and achieving the purpose of avoiding erroneous operation. In the case where the operating body is blocked, the target instruction can be obtained according to the touch track in the touch operation to avoid the situation where the touch operation cannot be responded to when the operating body is blocked.

[0123] In some embodiments, the operation attribute is determined according to the recognition result obtained by recognizing the operating body of the first image, which is mainly divided into the following situations.

[0124] The first case: only the operation type is determined according to the recognition result (the operation parameters are not determined according to the recognition result); in this case, when obtaining the target instruction, the operation type (determined according to the recognition result) is combined with the trajectory in the touch operation to obtain the target instruction, or the operation type (determined according to the recognition result) and its corresponding preset operation parameters are used to obtain the target instruction.

[0125] The second situation: only the operation parameters are determined according to the recognition results (the operation type is not determined according to the recognition results); in this case, when obtaining the target instruction, the operation type is determined by the trajectory in the touch operation, and the target instruction is obtained using the operation parameters (determined according to the recognition results) and the operation type (determined by the touch operation).

[0126] The third case: determining the operation type and operation parameters according to the recognition result; in this case, when obtaining the target instruction, the target instruction is obtained using the operation type and operation parameters determined by the recognition result.

[0127] The three situations mentioned above are described in detail in the following contents.

[0128] Note on determining only the operation type based on the recognition result:

[0129] It should be noted that different postures of the operating body correspond to different operation types; the operation type is determined based on the posture of the operating body in the recognition result, and the operation type is the type of content modification or the type of display control; the type of content modification includes at least operation types such as trajectory display and content deletion, and the type of display control includes operation types such as zooming in or out.

[0130] In some embodiments, when the operating body is in a first posture, the operation type is determined to be a display control type; wherein the first posture is a posture of relative motion of multiple fingers or a stylus on the touch area. It can be understood that the relative motion posture is: a motion posture of the operating body such as a finger or a stylus moving closer to or farther from a certain center position.

[0131] Specifically, the first gesture is: multiple fingers of a user or a handheld stylus, or fingers of multiple users or handheld stylus, approaching each other along a certain center or moving away from each other along a certain center on the touch area;

[0132] For example, the first posture is shown in FIG4(a), where the thumb and index finger of the right hand of a user are away from each other at a certain center along the touch area. Another example is shown in FIG4(b), where the index finger of the left hand and index finger of the right hand of a user are close to each other at a certain center along the touch area.

[0133] For another example: the first gesture is: the five fingers of a user's left hand (or right hand) are tightened or opened along a certain center on the touch area. For another example: the first gesture is: a user holding a stylus in his right hand and the index finger of the user's left hand are moving away from or approaching each other along a certain center on the touch area. For another example: the first gesture is: the fingers of multiple users or handheld styluses are moving away from or approaching each other along a certain center on the touch area.

[0134] As can be seen from the above content, the type of display control includes operation types such as reduction or enlargement; specifically, the type of display control corresponding to different first postures of the operating body; for example: when the first posture is shown in FIG4(a), that is, when the first posture is that the thumb and index finger of the right hand of a certain user are away from each other at a certain center along the touch area, the type of display control determined is the enlargement type. For another example: when the first posture is shown in FIG4(b), that is, when the first posture is that the index finger and index finger of the left hand of a certain user are close to each other at a certain center along the touch area, the type of display control determined is the reduction type. For another example: when the first posture is that the five fingers of the left hand of a certain user are released at a certain center along the touch area, the type of display control determined is the enlargement type. For another example: when the first posture is that the five fingers of the left hand of a certain user are tightened at a certain center along the touch area, the type of display control determined is the reduction type.

[0135] It is understandable that when zooming in and out on the touch area, the zoomed object may be an object such as a canvas or a window; the zoomed object may be determined specifically by a finger or a stylus. It may be set that when the fingers belonging to the same hand move relatively along a certain center motion on the touch area, the zoomed object is the canvas, or in other words, the zoomed object corresponding to the one-hand zooming gesture is the canvas, and the fingers of a certain hand corresponding to the one-hand zooming gesture are: thumb + index finger, thumb + middle finger, index finger + middle finger, thumb + index finger + middle finger, which are not given examples one by one here. When the fingers belonging to different hands move relatively along a certain center motion on the touch area, the zoomed object is the window, or in other words, the zoomed object corresponding to the two-hand zooming gesture is the window, and the fingers corresponding to the two-hand zooming gesture are: left index finger + right index finger, left thumb + right index finger, which are not given examples one by one here.

[0136] For example, as shown in FIG4(a), when the right thumb and right index finger of a user move away from each other at a center along the touch area, the enlarged object is the canvas; as shown in FIG4(b), when the left index finger and right index finger of a user move closer to each other at a center along the touch area, the shrunk object is the window.

[0137] In some embodiments, when the operating body is in the second posture, the operation type is determined to be a type of content modification; wherein the second posture is a posture of a single finger or stylus touching the touch area, or a posture of fingers or handheld styluses of different users independently moving on the touch area. It can be understood that the posture of independent movement specifically means that the movement between the fingers or styluses is not related to a certain position on the touch area.

[0138] Specifically, the second gesture is: a finger or a handheld stylus moves on the touch area. Alternatively, fingers or handheld styluses of different users move on the touch area. Compared with the relative movement involved in the first gesture, the independent movement corresponding to the second gesture is not related to a certain area of ​​the touch area.

[0139] For example, the second posture is shown in FIG5(a), where a user holds the stylus pen in his right hand and writes on the touch area. Another example is shown in FIG5(b), where a user holds the index finger of his right hand and writes on the touch area.

[0140] For another example, the second gesture is shown in FIG5(c), where user A and user B write on the touch area with their left index finger and right index finger, respectively. For another example, the second gesture is shown in FIG5(d), where user B writes on the touch area with his right index finger and user A writes on the touch area with the stylus in his right hand.

[0141] It should be noted that the input end of the operating body can be a finger, a stylus tip, a stylus end (or a stylus tail end), etc.; from the above content, it can be seen that the types of content modification include operation types such as trajectory display and content deletion; specifically, when the operating body is in the first posture, different input ends of the operating body correspond to different types of content modification; for example: when the user moves the right index finger on the touch area, the determined type of content modification is trajectory display, that is, the user writes on the touch area with the right index finger, and the right index finger is the input end of the operating body.

[0142] For another example, when two users move their right index fingers on the touch area respectively, the determined content modification type is trajectory display, that is, the two users write on the touch area with their right index fingers respectively, wherein the right index fingers of the two users are the input ends of the operating bodies.

[0143] For another example, when a user holds a stylus pen and moves the pen tip or end on the touch area, the type of content modification determined is trajectory display, that is, the user writes on the touch area with the pen tip or end, where the pen tip or end is the input end of the operating body.

[0144] For another example, when the user moves multiple fingers of his right hand on the touch area, the determined type of content modification is content deletion, that is, the user erases the content on the touch area with multiple fingers of his right hand, wherein the multiple fingers of his right hand are the input ends of the operating body.

[0145] For another example, when a user holds a stylus pen and moves the pen tip over a touch area, the type of content modification determined is content deletion, that is, the user erases the content on the touch area with the pen tip, wherein the pen tip of the stylus pen is the input end of the operating body.

[0146] In the above example, when the input end of the operating body is the tip of the stylus pen, the type of content modification determined is track display or content deletion; it should be noted that the user can customize the type of content modification determined when the input end is the tip of the stylus pen as track display or content deletion; that is, the user can customize in advance whether the tip of the stylus pen is used for writing content or for erasing content. It is worth noting that the tip of the pen is usually used for writing content.

[0147] The above is related to determining only the operation type based on the recognition result. After determining the operation type based on the recognition result; the target instruction is obtained using the determined operation type and its corresponding preset operation parameters, or the target instruction is obtained using the determined operation type combined with the touch track in the touch operation, and the target instruction is executed.

[0148] For example: after determining that the operation type is zooming out, the preset operation parameters corresponding to the zooming out operation (such as a specified zooming out percentage) are used to obtain the target instruction (in this case, the zooming out instruction) and execute the target instruction, that is, the user zooms out the zoomed object (window or canvas) by the specified zooming out percentage.

[0149] For example: after determining that the operation type is zooming in, the preset operation parameters corresponding to the zooming in operation (such as a specified zooming in percentage) are used to obtain the target instruction (in this case, the zooming in instruction) and execute the target instruction, that is, the user zooms in the zoomed object (window or canvas) to the specified zooming in percentage.

[0150] For example: after determining that the operation type is track display, the preset operation parameters corresponding to the track display (such as a line of specified thickness and specified color) are used to obtain the target instruction (in this case, an input instruction) and execute the target instruction, that is, the user can write on the touch area with a line of specified thickness and specified color. Another example: after determining that the operation type is content deletion, the preset operation parameters corresponding to content deletion (such as an eraser of specified erasure size) are used to obtain the target instruction and execute the target instruction (in this case, an erase instruction), that is, the user can erase the content on the touch area with an eraser of specified erasure size.

[0151] Note on determining only the operating parameters based on the recognition results:

[0152] According to the attributes of the operating body in the recognition result, the operation parameters are determined, and the operation parameters include: modification parameters of the track and / or control parameters of the interface. The modification parameters of the track include at least display parameters and erasure parameters; the display parameters include at least parameters such as track color, track size, controls corresponding to the input track, and functions corresponding to the start controls; the erasure parameters include at least parameters such as the erasure size; the control parameters of the interface include at least parameters such as the zoom ratio and the zoom center position.

[0153] It can be understood that the color corresponding to the operating body includes: the color of the user's hand or the color of the stylus; wherein the color of the hand includes any one of the skin color, the glove color, or the color of the decorations worn by the user's hand. The input end corresponding to the operating body includes: any one of the finger end, the glove end, the tip of the stylus, or the pen end of the stylus. Among them, the pen shell material of the stylus can be a transparent material and a light bulb is provided in the stylus. By adjusting the light color of the light bulb to change the pen shell color of the stylus, the color corresponding to the operating body is changed; similarly, styluses with different colors can also be used, and the color corresponding to the operating body can be changed by selecting a stylus of the corresponding color.

[0154] In some embodiments, the modification parameters of the trajectory in the operation parameters are determined according to the color corresponding to the operating body and / or the input end corresponding to the operating body, and the modification parameters of the trajectory include any one or more of the following: trajectory color parameter, trajectory size parameter, and trajectory erasure size; wherein the trajectory color parameter and trajectory size parameter are display parameters, and the trajectory erasure size is an erasure parameter. Specifically, the trajectory color parameter can be determined by the color corresponding to the operating body, for example: if the color of the stylus is red, the trajectory color parameter can be determined to be red according to the color of the stylus; the trajectory size parameter and the trajectory erasure size can be determined by the input end corresponding to the operating body, for example: if the input end of the operating body is a finger end, the trajectory size parameter can be determined according to the finger end to be a preset trajectory size corresponding to the finger end, and the trajectory erasure size can be determined according to the finger end to be a preset trajectory erasure size corresponding to the finger end.

[0155] In some other embodiments, the modification parameters of the trajectory and / or the control parameters of the interface in the operation parameters are determined according to the contact area between the operating body and the touch area, and the control parameters of the interface at least include: interface scaling ratio and / or scaling center position. Specifically, the contact area between the operating body and the touch area can be used to determine the modification parameters of the trajectory and / or the control parameters of the interface; for example, the size of the contact area between the finger tip and the touch area is used to determine the trajectory size parameter, the trajectory erasure size, the scaling ratio and the scaling center point.

[0156] After determining the display parameters, if the operation type is determined to be trajectory display (or input) from the trajectory in the touch operation, for example: if the contact area between the input end of the operating body and the touch area is less than a certain threshold, or if the operating body is the tip of a stylus, then the operation type can be determined to be a trajectory display type; at this time, the determined display parameters are used to display the trajectory input by the operating body; if the operation type is determined from the trajectory in the touch operation to be not trajectory display, the determined display parameters are not used.

[0157] After determining the erasing parameters, if the operation type is determined to be content deletion (or erasing) from the trajectory in the touch operation, for example: if the contact area between the input end of the operating body and the touch area is greater than a certain threshold, then the operation type can be determined to be content deletion; at this time, the determined erasing parameters are used to delete the content on the trajectory of the touch operation; if the operation type is determined to be not content deletion from the trajectory in the touch operation (such as determining that the operation type is trajectory display), the determined erasing parameters are not used.

[0158] After determining the control parameters of the interface, if the operation type is determined to be a display control type (such as zooming) from the trajectory in the touch operation, then the interface zoom is performed using the interface zoom ratio and zoom center position in the determined interface control parameters; if the operation type is determined not to be a display control type from the trajectory in the touch operation, such as determining that the operation type is trajectory display, the determined interface control parameters are not used.

[0159] Through the contents in the above embodiments, three operation parameters, namely, display parameters, erasing parameters and interface control parameters, can be determined; in the case where only the operation parameters are determined according to the recognition results, in actual applications, the three operation parameters can be determined in the above manner, and then combined with the operation type determined from the trajectory in the touch operation, the operation parameters corresponding to the operation type are selected and the target instructions are obtained using the selected operation parameters and the operation type; for example: if the operation type determined from the trajectory in the touch operation is trajectory display, the display parameters and the operation type are used to obtain the target instruction, and the target instruction is used to display the corresponding trajectory along the touch position in the touch operation according to the corresponding operation parameters.

[0160] Similarly, the operation type determined from the track in the touch operation can be combined to select and determine the operation parameters corresponding to the corresponding operation type, and use the determined operation parameters and the operation type to obtain the target instruction. For example, if the operation type determined from the track in the touch operation is track display, the display parameters can be selected and determined, and the target instruction can be obtained using the display parameters and the operation type.

[0161] It should be noted that the first end of the stylus is the pen end (that is, the tail end) of the stylus, and the second end of the stylus is the tip of the stylus. For example, the example of the stylus is as follows: Figure 6 As shown, the two ends of the stylus are the tip and the end of the pen respectively; the user can pre-customize whether the tip of the pen is used for writing content or for erasing content. When the tip of the pen is used for writing content, a preset track size corresponding to the tip of the pen is pre-set for the tip of the pen; when the tip of the pen is used for erasing content, a preset erasing size corresponding to the tip of the pen is pre-set for the tip of the pen. It is worth noting that the tip of the stylus is used for writing content, and a preset track size corresponding to the tip of the pen is pre-set for the tip of the pen.

[0162] Regarding the above-mentioned content of determining the modification parameters of the trajectory, in some specific embodiments, when the operating body is a stylus and the stylus contacts the touch area with the first end, the modification parameters of the trajectory in the operating parameters are determined according to the color of the stylus and / or the preset trajectory size corresponding to the first end, or, the modification parameters of the trajectory in the operating parameters are determined according to the preset erasing size corresponding to the first end.

[0163] That is to say, if the pen tip is used to write content, when the operating body is a stylus and the stylus contacts the touch area with the first end, the modification parameters of the trajectory in the operating parameters are determined according to the color of the stylus and / or the preset trajectory size corresponding to the first end, wherein the color of the stylus is used to determine the trajectory color parameter, and the preset trajectory size corresponding to the first end is used to determine the trajectory size parameter; that is, the color corresponding to the trajectory color parameter is the color of the stylus, and the trajectory size corresponding to the trajectory size parameter is the preset trajectory size corresponding to the first end.

[0164] For example: as shown in the schematic diagram of the touch area in Figure 7(a), the user uses the first end of the stylus, i.e., one side of the pen tip, to write on the touch area (determine the operation type as trajectory display based on the trajectory of the touch operation), and determines the trajectory color parameter by the color of the stylus, which is the color of the stylus, and the trajectory size corresponding to the trajectory size parameter is the preset trajectory size corresponding to the first end. Based on this, executing the target instruction is: the color of the trajectory (curved part trajectory) displayed on the touch area is the color of the stylus, and the size of the trajectory displayed on the touch area is the preset trajectory size corresponding to the first end.

[0165] If the pen tip is used to erase content, when the operating body is a stylus and the stylus contacts the touch area with the first end, the modification parameters of the trajectory in the operating parameters are determined according to the preset erasing size corresponding to the first end, wherein the preset erasing size corresponding to the first end is used to determine the trajectory erasing size; that is, the trajectory erasing size is the preset trajectory size corresponding to the first end.

[0166] For example: as shown in the schematic diagram of the touch area in Figure 7(b), the user uses the first end of the stylus, i.e., one side of the pen tip, to erase the content on the touch area (determine the operation type as content deletion based on the trajectory of the touch operation), and determines that the trajectory erasing size in the modification parameters of the trajectory is the preset erasing size corresponding to the first end. Based on this, executing the target instruction is: erasing the content on the trajectory with the preset erasing size on the touch area.

[0167] Regarding the content of determining the control parameters of the interface as mentioned above, in some specific embodiments, when the operating body is multiple fingers or a stylus belonging to the same hand or different users and the multiple fingers or the stylus move relative to each other on the touch area, the control parameters of the interface in the operating parameters are determined according to the contact area between the multiple fingers or the stylus and the touch area.

[0168] For example: when the operating body is multiple fingers belonging to the same hand and the multiple fingers move relatively on the touch area, the control parameters of the interface are determined according to the contact area between the multiple fingers and the touch area. Another example: when the operating body is multiple fingers belonging to two hands of the same user and the multiple fingers move relatively on the touch area, the control parameters of the interface are determined according to the contact area between the multiple fingers and the touch area. Another example: when the operating body is multiple fingers belonging to different users and the multiple fingers move relatively on the touch area, the control parameters of the interface are determined according to the contact area between the multiple fingers and the touch area. Another example: when the operating body is a handheld stylus belonging to different users and the stylus moves relatively on the touch area, the control parameters of the interface are determined according to the contact area between the multiple stylus and the touch area. Another example: when the operating body is a finger of at least one user and a stylus held by at least one user and the finger and the stylus move relatively on the touch area, the control parameters of the interface are determined according to the contact area between the finger and the stylus and the touch area.

[0169] Specifically, the zoom center position can be determined according to the position of the contact area, and the interface zoom ratio can be determined according to the length of the contact area, etc. Taking the operating body shown in Figures 4(a) and 4(b) as an example, the center position of the two contact areas formed by two fingers on the touch area is determined as the zoom center position, and the length of the two contact areas formed by the two fingers on the touch area is used to determine the interface zoom ratio. For example, the longer the contact area, the larger the zoom ratio, and the shorter the contact area, the smaller the zoom ratio.

[0170] Regarding the above-mentioned content of determining the modification parameters of the trajectory, in some specific embodiments, when the operating body is a stylus and the stylus contacts the touch area with the second end, the modification parameters of the trajectory in the operating parameters are determined according to the color of the stylus and / or the preset trajectory size corresponding to the second end.

[0171] That is, when the operating body is a stylus and the second end of the stylus contacts the touch area, the modification parameters of the track in the operating parameters, such as the track color parameter, are determined according to the color of the stylus, and / or, the modification parameters of the track in the operating parameters, such as the track size parameter, are determined according to the preset track size corresponding to the second end of the stylus. The color of the stylus is used to determine the track color parameter, and the preset track size corresponding to the second end is used to determine the track size parameter; that is, the color corresponding to the track color parameter is the color of the stylus, and the track size corresponding to the track size parameter is the preset track size corresponding to the second end.

[0172] For example: as shown in the schematic diagram of the touch area in Figure 7(c), the user uses the second end of the stylus, i.e., the tip of the pen, to write on the touch area (determine the operation type as track display based on the track of the touch operation), and determines the track color parameter by the color of the stylus, which is the color of the stylus, and the track size corresponding to the track size parameter is the preset track size corresponding to the second end; the color of the track (the curved part of the track) displayed on the touch area is the color of the stylus, and the size of the track displayed on the touch area is the preset track size corresponding to the second end.

[0173] Regarding the content of determining the modification parameters of the trajectory as mentioned above, in some specific embodiments, when the operating body is in the form of a single finger moving independently on the touch area, or when the fingers of different users move independently on the touch area, the modification parameters of the trajectory in the operation parameters are determined based on the contact area between the finger and the touch area.

[0174] From the above content, it can be seen that when the operation type is determined as content deletion according to the trajectory of the touch operation, the content on the trajectory is deleted when the finger moves independently on the touch area; when the operation type is determined as trajectory display according to the trajectory of the touch operation, the trajectory is displayed when the finger moves independently on the touch area.

[0175] When a single finger or fingers of different users move independently on the touch area, if the operation type is determined to be a track display according to the track of the touch operation, the modification parameters of the track in the operation parameters are determined using the color of the hand and / or the contact area between the finger and the touch area, wherein the color of the hand is used to determine the track color parameter, and the contact area between the finger and the touch area is used to determine the track size parameter; that is, the color corresponding to the track color parameter is the color of the hand, and the track size corresponding to the track size parameter is: the track size corresponding to the contact area between the finger and the touch area. It is understandable that the color corresponding to the track color parameter can also be a preset color (that is, it can be determined not according to the color of the hand), which is not limited here.

[0176] For example: as shown in the schematic diagram of the touch area in Figure 7(d), the user writes on the touch area with a single finger (the operation type is determined to be track display based on the track of the touch operation), and the track color parameter is determined by the color of the glove worn on the hand, which is the color of the glove, and the track size parameter corresponds to the track size corresponding to the contact area between the finger and the touch area; the color of the track (the curved part of the track) displayed on the touch area is the color of the glove, and the size of the track displayed on the touch area is the track size corresponding to the contact area between the finger and the touch area.

[0177] When a single finger or fingers of different users move independently on the touch area, if the operation type is determined to be content deletion based on the trajectory of the touch operation, the modification parameters of the trajectory in the operation parameters are determined based on the contact area between the finger and the touch area, wherein the contact area between the finger and the touch area is used to determine the trajectory erasing size; that is, the trajectory erasing size is: the trajectory size corresponding to the contact area between the finger and the touch area.

[0178] For example: as shown in the schematic diagram of the touch area in Figure 7(e), the user uses a single finger to erase the content on the touch area (the operation type is determined to be content deletion based on the trajectory of the touch operation), and the trajectory erasing size in the modification parameters of the trajectory is determined as: the erasing size corresponding to the contact area between the finger and the touch area; the content on the trajectory is erased on the touch area with the erasing size corresponding to the contact area between the finger and the touch area.

[0179] Regarding the above-mentioned content of determining the modification parameters of the trajectory, in some specific embodiments, when the operating body is a single hand or multiple hands and the touch area is contacted with multiple fingers or palms combined, the modification parameters of the trajectory in the operation parameters are determined according to the contact area between the combined fingers or palms and the touch area.

[0180] When multiple fingers or palms move independently on the touch area, if the operation type is determined to be a track display according to the track of the touch operation, the color of the hand and / or the contact area between the multiple fingers or palms and the touch area are used to determine the modification parameters of the track in the operation parameters, wherein the color of the hand is used to determine the track color parameter, and the contact area between the multiple fingers or palms and the touch area is used to determine the track size parameter; that is, the color corresponding to the track color parameter is the color of the hand, and the track size corresponding to the track size parameter is: the track size corresponding to the contact area between the multiple fingers or palms and the touch area. It can be understood that the color corresponding to the track color parameter can also adopt a preset color (that is, it can be determined not according to the color of the hand), which is not limited here.

[0181] For example: as shown in the schematic diagram of the touch area in Figure 7(f), the user uses multiple fingers to write on the touch area (the operation type is determined to be track display based on the trajectory of the touch operation), and the track color parameter is determined by the color of the accessories worn on the hand, which is the color of the accessories worn on the hand, and the track size corresponding to the track size parameter is: the track size corresponding to the contact area between the multiple fingers and the touch area; the color of the track (the curved part of the track) displayed on the touch area is the color of the accessories worn on the hand, and the size of the track displayed on the touch area is the track size corresponding to the contact area between the multiple fingers and the touch area.

[0182] When multiple fingers or palms move independently on the touch area, if the operation type is determined to be content deletion based on the trajectory of the touch operation, the modification parameters of the trajectory in the operation parameters are determined based on the contact area between the multiple fingers or palms and the touch area, wherein the contact area between the multiple fingers or palms and the touch area is used to determine the trajectory erasing size; that is, the trajectory erasing size is: the trajectory size corresponding to the contact area between the multiple fingers or palms and the touch area.

[0183] For example: as shown in the schematic diagram of the touch area in Figure 7(g), the user uses multiple fingers to erase content on the touch area (determine the operation type as content deletion based on the trajectory of the touch operation), and the trajectory erasing size in the modification parameters of the trajectory is determined as: the erasing size corresponding to the contact area between the multiple fingers and the touch area; erase the content on the trajectory on the touch area with the erasing size corresponding to the contact area between the multiple fingers and the touch area.

[0184] To summarize the contents of the above specific embodiments, the color corresponding to the operating body is the color of the hand or the color of the pen, wherein the color of the hand includes the skin color, the color of the gloves and the color of the decorations worn; the input ends corresponding to the operating body are: fingers, hands wearing gloves, the tip of the stylus and the end of the stylus.

[0185] In the case where the operating body is a stylus (tip or end) and the operation type is track display (the operation type can be determined based on the touch operation), after determining the color of the stylus, the corresponding track is displayed at the operation position along the touch area, and the color of the track is consistent with the color of the stylus; if the operating body is the tip of the stylus, the size of the displayed track is consistent with the size of the track corresponding to the tip; if the operating body is the end of the stylus, the size of the displayed track is consistent with the size of the track corresponding to the end. It can be understood that if the operating body is the end of the stylus and the operation type is content deletion, the corresponding content is erased at the operation position along the touch area, and the track erasing size is consistent with the preset erasing size corresponding to the end of the pen.

[0186] When the operating body is a finger and the operation type is trajectory display, after determining the color of the hand, the corresponding trajectory is displayed along the operation position on the touch area. The color of the trajectory is consistent with the color of the hand, and the size of the displayed trajectory is consistent with the trajectory size corresponding to the finger.

[0187] When the operating body is a hand wearing a glove and the operation type is track display, after the color of the glove is determined, the corresponding track is displayed along the operation position on the touch area, the color of the track is consistent with the color of the glove, and the size of the displayed track is consistent with the size of the track corresponding to the glove, wherein the size of the track corresponding to the glove can be thicker than the size of the track corresponding to the finger; for example: the user wears a red fingertip to write content on the touch area, and a red track is displayed along the operation position on the touch area, and the size of the displayed track is consistent with the size of the track corresponding to the red fingertip.

[0188] The above is related to determining only the operation parameters according to the recognition result. After the operation parameters are determined according to the recognition result, the target instruction is obtained by using the operation parameters and the operation type determined from the trajectory of the touch operation, and the target instruction is executed.

[0189] For example: after determining that the operation type is track display from the track of the touch operation, the corresponding track is displayed on the touch area using the display parameters determined according to the recognition result. Another example: after determining that the operation type is canvas zoom from the track of the touch operation, the zoom center and zoom ratio determined according to the recognition result are used to control the zoom of the canvas on the touch area.

[0190] Explanation on determining the operation type and operation parameters based on the recognition results:

[0191] The operation type is determined according to the posture of the operating body in the recognition result, and the operation parameter is determined according to the attribute of the operating body in the recognition result.

[0192] In some embodiments, when determining the operation type, it can be implemented in the following manner: when the operating body is in a first posture, the operation type is determined to be a display control type; when the operating body is in a second posture, the operation type is determined to be a content modification type. The first posture is: multiple fingers or handheld stylus of a user, or fingers or handheld stylus of multiple users, approach each other along a certain center on the touch area or move away from each other along a certain center; the second posture is: a finger or handheld stylus moves on the touch area. Alternatively, fingers or handheld stylus of different users move on the touch area separately.

[0193] It should be noted that the specific process of how to determine the operation type based on the recognition result has been described in detail in the above embodiment. Please refer to the above content about only determining the operation type based on the recognition result, which will not be repeated here.

[0194] In some embodiments, when determining the operation parameters after determining the operation type, it can be achieved in the following way: determining the modification parameters of the trajectory in the operation parameters based on the color corresponding to the operating body and / or the input end corresponding to the operating body; or determining the modification parameters of the trajectory in the operation parameters and / or the control parameters of the interface based on the contact area between the operating body and the touch area.

[0195] It should be noted that the specific process of how to determine the operating parameters based on the recognition results has been described in detail in the above embodiments. Please refer to the above content about only determining the operating parameters based on the recognition results, which will not be repeated here.

[0196] Specifically, in this embodiment, the operation type can be first determined according to the posture of the operating body in the recognition result, and then the operation parameters corresponding to the operation type can be determined according to the attributes of the operating body in the recognition result; or, in this embodiment, the operation parameters can be first determined according to the attributes of the operating body in the recognition result, and then the operation type can be determined according to the posture of the operating body in the recognition result; or, in this embodiment, the operation type and the operation parameters are determined at the same time.

[0197] The following describes a specific embodiment by taking the example of first determining the operation type and then determining the operation parameters corresponding to the operation type:

[0198] In some specific embodiments, when the operating body is in the second posture, the operation type is determined to be a content modification type. Further, when the operating body is a stylus and the stylus contacts the touch area with its first end, the modification parameters of the trajectory in the operation parameters are determined according to the color of the stylus and / or the preset trajectory size corresponding to the first end, or the modification parameters of the trajectory in the operation parameters are determined according to the preset erase size corresponding to the first end. Based on this, according to the type of content modification and the modification parameters of the trajectory, the target instruction is obtained and executed to achieve modification of the content in the touch area.

[0199] In other specific embodiments, when the operating body is in the first posture, the operation type is determined to be the type of display control. Further, when the operating body is multiple fingers or a stylus belonging to the same hand or different users and the multiple fingers or the stylus move relatively on the touch area, the control parameters of the interface in the operation parameters are determined according to the contact area between the multiple fingers or the stylus and the touch area; based on this, according to the type of display control and the control parameters of the interface, the target instruction is obtained and executed to realize the display control in the touch area.

[0200] In some other specific embodiments, when the operating body is in the second posture, the operation type is determined to be a content modification type. Further, when the operating body is a stylus pen and the stylus pen contacts the touch area with the second end, the modification parameters of the trajectory in the operation parameters are determined according to the color of the stylus pen and / or the preset trajectory size corresponding to the second end. Based on this, according to the content modification type and the trajectory modification parameters, the target instruction is obtained and executed to modify the content in the touch area.

[0201] In other specific embodiments, when the operating body is in the second posture, the operation type is determined to be a content modification type. Further, when the operating body is in a posture of a single finger contacting the touch area, or when fingers of different users move independently on the touch area, the modification parameters of the trajectory in the operation parameters are determined according to the contact area between the single finger and the touch area. Based on this, according to the type of content modification and the modification parameters of the trajectory, the target instruction is obtained and executed to modify the content in the touch area.

[0202] In other specific embodiments, when the operator is in the second posture, the operation type is determined to be a content modification type. Further, when the operator is a single hand or multiple hands and the touch area is touched with multiple fingers or palms combined, the modification parameters of the trajectory in the operation parameters are determined according to the contact area between the combined fingers or palms and the touch area. Based on this, according to the content modification type and the modification parameters of the trajectory, the target instruction is obtained and executed to modify the content in the touch area.

[0203] It should be noted that the process of how to determine the operation type according to the recognition result and how to determine the operation parameters according to the recognition result has been described in detail in the embodiment. Please refer to the relevant content in the previous text and will not be repeated here.

[0204] That is, when determining the operation type and the operation parameter according to the recognition result, the operation type can be determined by using the recognition result, and then the operation parameter related to the operation type can be determined by using the recognition result.

[0205] For example: if the operation type is determined to be track display according to the recognition result, the display parameters can be determined using the recognition result, and the target instruction can be obtained using the track display operation type and display parameters determined from the recognition result. For another example: if the operation type is determined to be content deletion according to the recognition result, the erase parameters can be determined using the recognition result, and the target instruction can be obtained using the content deletion operation type and erase parameters determined from the recognition result.

[0206] The above embodiments are related instructions for determining the operation type and / or operation parameters according to the recognition results. After determining the operation type and operation parameters, the target instruction is obtained using the determined operation type and operation parameters, and the target instruction is executed to respond to the touch operation. For example: after determining that the operation type is track display and determining the track color and track size, according to the track corresponding to the touch operation, the corresponding content is displayed in the touch area with the determined track color and track size. For another example: after determining that the operation type is content deletion and determining the track erasure size, according to the track corresponding to the touch operation, the corresponding content is erased in the touch area with the determined track erasure size. For another example: after determining that the operation type is zooming (reducing or enlarging) and determining the zoom ratio and zoom center point, according to the track corresponding to the touch operation, the relevant content on the touch area is zoomed with the zoom ratio and zoom center point (the zoom multiple is the zoom ratio). Therefore, the operation type and / or operation parameters can be determined by the recognition results, the target instruction can be obtained using the determined operation type and operation parameters, and the target instruction is executed, and the target instruction indicates to perform corresponding processing (such as track display, content deletion or zooming) in the touch area.

[0207] It can be seen from the above scheme that in an input method disclosed in an embodiment of the present application, by obtaining the touch operation and the first image on the touch device, identifying the operating body in the first image to obtain the recognition result, and then determining the operation attribute according to the recognition result, it is possible to obtain the target instruction through the operation attribute and the touch operation and execute the target instruction. It can be seen that in the present application, the acquisition of the target instruction is achieved by identifying the operating body of the first image, thereby avoiding the situation where the erroneous operation is caused by relying solely on the detection of touch, and achieving the purpose of avoiding erroneous operation.

[0208] It is understandable that in order to further ensure that misoperation can be avoided, the posture of the operator in the first image can be used to determine whether to determine the operation attribute according to the recognition result, that is, the posture of the operator in the first image can be used to determine whether to continue to execute the subsequent steps for responding to the touch operation. For details on how to further ensure that misoperation can be avoided, please refer to Figure 8 , which shows another flow chart of an input method provided by an embodiment of the present application. Before step S103, the input method in this embodiment may further include the following steps:

[0209] Step S108: determining whether the recognition result indicates that the operating body in the first image matches the preset input gesture.

[0210] It should be noted that the recognition result can represent the posture of the operating body. In this embodiment, the posture of the operating body in the recognition result can be compared with the preset input posture to determine whether the recognition result represents that the operating body in the first image matches the input posture.

[0211] In a specific implementation, in this embodiment, a pre-trained recognition model can be used to recognize the operating body in the first image, and the recognition model can recognize information such as the posture of the operating body in the first image, the attributes of the operating body, and the position of the operating body. For example, the recognition model is trained through sample images, and the sample images can be images containing features such as the posture and attributes of the operating body. Therefore, after being trained with the sample images, the recognition model can recognize the posture and / or attributes of the operating body.

[0212] In the process of implementing step S108, the posture of the operating body in the first image represented by the recognition result is determined; according to the posture of the operating body, it is judged whether the operating body in the first image matches the preset input posture; if the recognition result represents that the operating body matches the input posture, steps S103 to S105 are executed; if the recognition result represents that the operating body does not match the input posture, the process ends, that is, step S103 or step S104 is not executed.

[0213] It is understandable that the preset input gesture may be a gesture that requires operating the touch area, such as a writing gesture; for example, as shown in the schematic diagram of the input gesture in FIG9( a ), the input gesture shown in FIG9( a ) is a writing gesture.

[0214] By judging whether the operating body matches the input gesture, it is judged whether the user needs to operate the touch area.

[0215] In a specific implementation, if the recognition result indicates that the operating body matches the input gesture, it is determined that the user needs to operate the touch area. For example, if the gesture of the operating body is shown in FIG. 9(a), it is determined that the user needs to operate the touch area. At this time, step S103 is executed to continue to execute the subsequent steps for responding to the touch operation. In other words, if the recognition result indicates that the operating body matches the input gesture, the contents of steps S103 to S105 are executed, that is, the user's touch operation on the touch area is continued to be responded and processed.

[0216] In another specific implementation, if the recognition result indicates that the operating body does not match the input posture, it is determined that the user does not need to operate the touch area. For example, the posture of the operating body is shown in FIG9(b), and the posture of the operating body does not match the input posture shown in FIG9(a), then it is determined that the user does not need to operate the touch area. At this time, step S103 is not executed, or step S104 is not executed to stop executing subsequent steps for responding to touch operations, that is, stop responding and stop processing the user's touch operations on the touch area (that is, do not respond to touch operations).

[0217] It can be seen from the above scheme that in an input method disclosed in an embodiment of the present application, it is determined whether the recognition result represents that the operating body in the first image matches the preset input gesture. If the recognition result represents that the operating body matches the input gesture, the user's touch operation on the touch area continues to be responded to and processed; if the recognition result represents that the operating body does not match the input gesture, the user's touch operation on the touch area is not responded to. It can be seen that in the present application, the recognition result obtained from the first image recognition is used to determine whether to respond to the user's touch operation on the touch area, further avoiding the situation where erroneous operation is caused by relying solely on touch detection, thereby achieving the purpose of avoiding erroneous operation.

[0218] It is understandable that the touch area of ​​the touch device usually includes a control interface, such as a toolbar and a dialog box. To make it easier for users to operate the control interface, the recognition result obtained from the first image recognition can be used to adjust the position of the control interface so that the user can operate the control interface conveniently. For details on how to adjust the position of the control interface, please refer to Fig.10 , which shows a partial flow chart of an input method provided by an embodiment of the present application, and after step S102, the following steps may also be included:

[0219] Step S109: Determine the first position of the operating body according to the recognition result.

[0220] In the specific implementation of step S109, the first position of the operating body in the first image is determined using the recognition result obtained from the first image; the first image is obtained by performing image acquisition on the target range by an image acquisition device.

[0221] Step S110: controlling the control interface of the touch device to be at a second position on the touch area, where the second position corresponds to the first position.

[0222] It should be noted that if the control interface is in a visible state or a hidden state, the control interface can be set to a visible state or a hidden state through the obtained target instruction, or the content in the control interface can be operated through the target instruction; for example: if the toolbar is currently in a hidden state, the toolbar can be set to a visible state through a target instruction; for another example: if the toolbar is currently in a hidden state, the toolbar can be set to a hidden state through a target instruction.

[0223] In the process of specifically implementing step S110, the position of the control interface of the touch device is controlled according to the first position, and the position of the control interface is adjusted to a second position on the touch area, where the second position corresponds to the first position; the second position is the first position relative to a preset area in the touch area, and the preset area is: the area in the touch area where it is most convenient for the user to operate the control interface.

[0224] By the above method, the position of the control interface can be adjusted so that the control interface is in the second position on the touch area, that is, the position of the control interface changes with the change of the first position, so that the user can conveniently operate the control interface. For example, as shown in the schematic diagram of the second position in Figures 11(a) and 11(b), the control interface is a tool menu, and the control toolbar moves with the user, and the toolbar is always in the second position corresponding to the first position, so that the user can conveniently operate the toolbar at any time.

[0225] It can be seen from the above scheme that in an input method disclosed in an embodiment of the present application, an image acquisition device acquires an image of a target range to obtain a first image. Using the recognition result obtained from the first image, a first position of an operating body in the first image is determined. According to the first position, a control interface for controlling the touch device is at a second position on the touch area, and the second position corresponds to the first position. It can be seen that the present application controls the control interface of the touch device through the image acquisition device to follow the movement of the user, so that the user can conveniently operate the control interface at any time, which can improve the user experience.

[0226] Corresponding to the above method embodiment, the present application embodiment discloses an electronic device, the structure of which is as follows: Fig.12 As shown, it may include: a touch device 1201, an image acquisition device 1202 and a processor 1203;

[0227] The image acquisition device 1202 is used to acquire images.

[0228] The processor 1203 is used to obtain a touch operation and a first image on the touch device 1201, where the first image is an image obtained by the image acquisition device 1202 acquiring an image of a target range, and the target range is a spatial range that at least includes the touch area of ​​the touch device 1201; identify the operating body in the first image to obtain a recognition result; determine operation attributes based on the recognition result, where the operation attributes at least include an operation type and / or an operation parameter; obtain a target instruction based on the touch operation and the operation attributes; and execute the target instruction.

[0229] Taking a large touch screen as an example, in order to avoid misoperation of the large touch screen and to improve the user experience of using the large touch screen, based on the technical solution in this application, a control solution for the large touch screen is proposed in this embodiment, which is as follows:

[0230] See also Fig.13 , which shows a flow chart of a control scheme for a large touch screen, Fig.13 The control scheme provided can prevent accidental touches when the user is facing away from the screen. Fig.13 The following steps may be included:

[0231] Step S1301: Camera detection and identification.

[0232] In the specific implementation of step S1301, a camera (ie, an image acquisition device) is used to detect and identify the user's posture.

[0233] Step S1302: Is it a writing posture? If it is a writing posture, execute step S1303; if it is not a writing posture, execute step S1304.

[0234] Step S1303: Turn on the touch function of the screen.

[0235] Step S1304: the touch function of the screen is disabled.

[0236] Among them, in the process of implementing step S1302, camera recognition and algorithms are used to determine whether the posture is in the writing state. If so, it is determined that the "prepare to write" screen touch function is normally turned on. If not, it is determined as "false touch" and the screen touch function is disabled.

[0237] For example, if the posture is determined not to be a writing posture, such as the posture determined to be a posture with the back facing the screen as shown in FIG14(a), it can be determined as "false touch", and step S1304 is executed to disable the screen touch function. If the posture is determined to be a writing posture, such as the posture determined to be a writing posture as shown in FIG14(b), it can be determined to be "preparing to write", and step S1303 is executed to turn on the screen touch function, that is, the screen touch function is normally turned on.

[0238] The above is an explanation of a control solution for a large touch screen, which uses a camera and an algorithm to determine whether to turn on the screen touch function, thereby preventing the user from accidentally touching the screen when the user is facing away from the screen.

[0239] In some embodiments, the moving position of a person is determined by a camera, and a tool bar or a dialog box is controlled to automatically follow the movement of the person; wherein the control tool and the dialog box are the control interface; for example: Fig.15 The schematic diagram of the toolbar automatically following is shown, where the camera determines the moving position of a person and controls the toolbar to automatically follow the person's movement. By controlling the toolbar or dialog box to automatically follow, when the user needs to operate the toolbar on the touch screen, the toolbar can be easily operated, thereby improving the user experience.

[0240] See also Fig.16 , which shows a flow chart of a control scheme for a large touch screen, Fig.16 The control scheme provided can distinguish the pen tip, pen end, and finger to automatically switch between whiteboard pen and marker pen, and eraser. Fig.16 The following steps may be included:

[0241] Step S1601: Camera detection and identification.

[0242] Step S1602: Is the contact object a hand or a pen? If the contact object is a hand, execute step S1603; if the contact object is a pen, execute step S1604.

[0243] In the specific implementation of step S1602, the camera determines whether the contact object touching the screen is a pen or a hand. If the contact object is determined to be a hand, step S1603 is executed; if the contact object is determined to be a pen, step S1604 is executed.

[0244] Step S1603: Determine whether it is a single-finger contact. If it is a single-finger contact, determine it as a line writing state or a small area erasing state; if it is not a single-finger contact, determine it as a large area erasing state.

[0245] In the process of implementing step S1603, when the contact object is determined to be a hand, it is defaulted to an eraser. The finger contact area is identified by the camera, and the thickness of the writing line in the writing state and the eraser area and the size of the erasing area in the erasing state are determined according to the finger contact area (or the number of fingers). Specifically, if it is a single-finger contact, the line writing state or the small-area erasing state is determined, for example: as shown in Figure 17 (a), if it is a single-finger contact, the line writing state or the small-area erasing state is determined; if it is not a single-finger contact (multi-finger contact), the large-area erasing state is determined, for example: as shown in Figure 17 (b), if it is a multi-finger contact, a large-area erasing state is determined.

[0246] Step S1604: Determine whether the pen tip is in contact. If the pen tip is in contact, determine it as a thin line writing state; if the pen tip is not in contact, determine it as an erasing state or a thick line writing state.

[0247] In the specific implementation of step S1604, when the contact object is determined to be a pen (stylus pen), the camera determines whether the pen tip or the end of the pen is in contact with the screen. For an example of a stylus pen, see the schematic diagram shown in FIG17(c). The stylus pen includes a pen tip (nib) and a pen end (the end of pen). If the pen tip contacts, the pen tip part automatically selects a thin line, and at this time it is determined to be a thin line writing state, for example: as shown in FIG17(d), if the pen tip contacts, it is determined to be a thin line writing state, and the thin line track is displayed. If the pen end contacts, the pen end part automatically selects a thick line or an eraser, and at this time it is determined to be a thick line writing state or an erase state, for example: as shown in FIG17(e), if the pen end contacts, it can be determined to be a thick line writing state, and the thick line track is displayed.

[0248] The above is an explanation of a control solution for a large touch screen. The camera is used to determine whether the contact object is a hand or a pen to further determine the erasing status, writing status and line thickness. The user does not need to call out the toolbar to adjust the line thickness or look for an eraser, thereby improving the user experience.

[0249] See also Fig.18 , which shows a flow chart of a control scheme for a large touch screen, Fig.18 The control scheme provided can automatically adjust the handwriting color according to the color of the pen. Fig.18 The following steps may be included:

[0250] Step S1801: Camera detection and identification.

[0251] Step S1802: Determine whether the contact object is a pen. If the contact object is not a pen, execute step S1802 to continue determination; if the contact object is a pen, execute step S1803.

[0252] Step S1803: The camera detects the color of the pen.

[0253] Step S1804: The system sets the handwriting color to be the same as the pen case color.

[0254] In the specific implementation of step S1804, the displayed handwriting color is set to the pen shell color.

[0255] In some embodiments, there may be multiple pens of different colors. When the contact object is a pen, the handwriting color is set to the color of the pen shell that touches the screen. Similarly, the pen shell can be set to be transparent, and the color of the pen shell can be changed by changing the light color of the bulb set in the pen shell. Different colored pens can write handwriting of different colors, for example: Fig.19 As shown, when the pen shell color is red, red handwriting can be written, when the pen shell color is black, black handwriting can be written, when the pen shell color is yellow, yellow handwriting can be written, and when the pen shell color is blue, blue handwriting can be written.

[0256] The above is a description of a control solution for a large touch screen. The color of the pen is determined by the camera to determine the handwriting color. Different colored pens can write different colors of handwriting. The user does not need to call out the toolbar to adjust the handwriting color, which reduces the complexity of the operation and improves the user experience.

[0257] See also Fig. 20 , which shows a flow chart of a control scheme for a large touch screen, Fig. 20 The control scheme provided can determine the canvas scaling or window scaling, Fig. 20 The following steps may be included:

[0258] Step S2001: Camera detection and identification.

[0259] Step S2002: Identify hand posture, left hand, right hand, which finger is touching, etc.

[0260] In the process of implementing step S2002, the camera is used to identify the object interacting with the screen, whether it is the left hand or the right hand, and the index finger, middle finger and thumb, etc. For example, as shown in Figure 21(a), the right index finger, right middle finger and left index finger, etc. can be identified.

[0261] Step S2003: Determine whether it is a one-hand operation. If it is a one-hand operation, determine it as canvas scaling; if it is not a one-hand operation, determine it as window scaling.

[0262] In the specific implementation of step S2003, it is determined whether it is a one-hand operation, for example, whether it is a one-hand zoom. If it is a one-hand zoom, it is determined to be canvas zoom (or screen zoom), for example: the index finger and thumb of one hand move inward or outward at the same time, it is determined to be screen zoom. For another example: in the one-hand zoom gesture shown in FIG21(b), the index finger and thumb of one hand move outward at the same time, it is determined to be screen zoom.

[0263] If it is not a single-hand zoom (multi-hand zoom), it is determined to be a window zoom. For example, if two hands perform a zoom action, it is determined to be a window zoom. For another example, if the two-hand zoom gesture shown in FIG. 21( c ) is performed, the index fingers of both hands move outward at the same time, which is determined to be a window zoom.

[0264] It should be noted that the one-handed zooming gestures include at least: left hand (right hand): thumb + index finger, thumb + middle finger, thumb + index finger + middle finger, five fingers, etc. The two-handed zooming gestures include at least: left index finger + right index finger, left thumb + right index finger, etc. The specific one-handed zooming gestures and two-handed zooming gestures can be customized and are not limited here.

[0265] The above is a description of a control solution for a large touch screen, which uses the camera to zoom in and out with one hand or with two hands, and then determines whether to zoom the screen or the zoom window. When the screen is high, the user can easily zoom the screen or the window on the screen, avoiding the situation where the user cannot zoom the screen or the window due to the high screen.

[0266] As can be seen from the above, in this embodiment, a camera posture recognition auxiliary interaction algorithm is added, and the camera can be similar to the human eye, making the use of the touch screen more convenient and natural, and closer to the user experience.

[0267] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0268] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0269] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0270] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An input method, comprising: Obtaining a touch operation and a first image on a touch device, wherein the first image is an image obtained by an image acquisition device acquiring an image of a target range, and the target range is a spatial range that at least includes a touch area of ​​the touch device; Recognize the operating body in the first image to obtain a recognition result; Determine operation attributes according to the recognition result, wherein the operation attributes at least include an operation type and / or an operation parameter; Obtaining a target instruction according to the touch operation and the operation attribute; executing the target instruction; Before determining the operation attribute according to the recognition result, the method further includes: Determining whether the recognition result indicates that the operating body in the first image matches a preset input gesture, where the preset input gesture is a gesture of a touch area to be operated; If the recognition result indicates that the operating body matches the input gesture, the steps of: determining the operation attribute according to the recognition result; If the recognition result indicates that the operating body does not match the input gesture, the step of determining the operation attribute according to the recognition result is not performed, or the step of obtaining the target instruction according to the touch operation and the operation attribute is not performed.

2. The method according to claim 1, determining the operation attribute according to the recognition result, comprising: An operation type is determined according to the gesture of the operating body in the recognition result, where the operation type is a content modification type or a display control type.

3. The method according to claim 2, determining the operation type according to the posture of the operating body in the recognition result, comprising: When the operating body is in a first posture, determining the operation type is a display control type; Wherein, the first posture is a posture of relative movement of multiple fingers or a stylus on the touch area; or When the operating body is in a second posture, determining the operation type is a content modification type; The second posture is a posture of a single finger or a stylus pen contacting the touch area, or a posture of fingers or handheld styluses of different users moving independently on the touch area.

4. The method according to claim 1 or 2, determining the operation attribute according to the recognition result, comprising: According to the attributes of the operating body in the recognition result, an operating parameter is determined, and the operating parameter includes: a modification parameter of a trajectory or a control parameter of an interface.

5. The method according to claim 4, determining the operation parameters according to the attributes of the operating body in the recognition result, comprising: Determine, according to the color corresponding to the operating body and / or the input end corresponding to the operating body, a modification parameter of the track in the operation parameter, wherein the modification parameter of the track includes any one or more of the following: a track color parameter, a track size parameter, and a track erasure size; or According to the contact area between the operating body and the touch area, the modification parameters of the trajectory in the operation parameters and / or the control parameters of the interface are determined, and the control parameters of the interface at least include: interface zoom ratio and / or zoom center position.

6. The method according to claim 4, determining the operation parameters according to the attributes of the operation object in the recognition result, comprising: In a case where the operating body is a stylus pen and the stylus pen contacts the touch area with a first end, determining the modification parameters of the track in the operating parameters according to the color of the stylus pen and / or a preset track size corresponding to the first end, or determining the modification parameters of the track in the operating parameters according to a preset erasing size corresponding to the first end; or In a case where the operating body is multiple fingers or styluses belonging to the same hand or different users and the multiple fingers or styluses move relatively on the touch area, determining the control parameters of the interface in the operation parameters according to the contact areas between the multiple fingers or styluses and the touch area; The modification parameters of the track include any one or more of the following: track color parameter, track size parameter, track erasure size, and the control parameters of the interface include at least: interface zoom ratio and / or zoom center position.

7. The method according to claim 1, after identifying the operating body in the first image to obtain a recognition result, the method further comprises: Determining a first position of the operating body according to the recognition result; Controlling a control interface of the touch device to be at a second position on the touch area, the second position corresponding to the first position; Wherein, the control interface is in a visible state or a hidden state.

8. The method according to claim 1, before determining the operation attribute according to the recognition result, the method further comprises: Determining whether the recognition result indicates that the first image contains an operating body; If the recognition result indicates that the first image contains an operating body, the steps of: determining an operation attribute according to the recognition result; If the recognition result indicates that the first image does not include an operating body, a target instruction is obtained according to a touch track in the touch operation.

9. An electronic device, comprising: Touch devices; An image acquisition device, used for acquiring images; A processor is used to obtain a touch operation on the touch device and a first image, wherein the first image is an image obtained by the image acquisition device performing image acquisition on a target range, and the target range is a spatial range that at least includes a touch area of ​​the touch device; identify an operating body in the first image to obtain a recognition result; determine operation attributes according to the recognition result, wherein the operation attributes at least include an operation type and / or an operation parameter; obtain a target instruction according to the touch operation and the operation attributes; and execute the target instruction; Before determining the operation attribute according to the recognition result, the processor is further configured to: Determining whether the recognition result indicates that the operating body in the first image matches a preset input gesture, where the preset input gesture is a gesture of a touch area to be operated; If the recognition result indicates that the operating body matches the input gesture, the steps of: determining the operation attribute according to the recognition result; If the recognition result indicates that the operating body does not match the input gesture, the step of determining the operation attribute according to the recognition result is not performed, or the step of obtaining the target instruction according to the touch operation and the operation attribute is not performed.

Images