Electronic equipment, interaction method thereof and computer readable storage medium

Abstract

The embodiment of the invention discloses electronic equipment, an interaction method thereof and a computer readable storage medium, and the method comprises the steps: the electronic equipment displays a target window, receives a first input operation of a first input mode, and receives a second input operation of a second input mode at the same time; and sending a first reporting event triggered by the first input operation and a second reporting event triggered by the second input operation to a control corresponding to the target window, so that the control corresponding to the target window displays corresponding contents in the target window according to the first reporting event and the second reporting event. Therefore, when the electronic equipment receives the inputs of the two different input modes at the same time, the point reporting events triggered by the two input modes are reported to the control of the application program at the same time, so that the application program can obtain and respond to the point reporting events of the two input modes at the same time, and a user is supported to perform cooperative operation through the two different input modes at the same time; and the user interaction operation experience is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an electronic device and an interaction method thereof, and a computer-readable storage medium. Background Art

[0002] With the increasing popularity of electronic devices, their application in people's work and life is becoming more and more extensive.

[0003] In people's work and life scenarios, when a user performs input operations through input devices such as fingers, stylus or mouse, the electronic device can obtain a reporting event triggered by the input operation of the input device, and perform corresponding operations in response to the reporting event. For example, when a user writes on the screen of a tablet computer with a stylus, the tablet computer can obtain the reporting event of the stylus and display the writing track on the screen in response to the reporting event.

[0004] However, currently electronic devices can only obtain the reporting events of one input device at the same time, and cannot obtain the reporting events of two different input devices at the same time, and thus cannot respond to input operations of two different input devices at the same time, resulting in a low user interaction experience. Summary of the invention

[0005] The embodiments of the present application provide an electronic device and an interaction method thereof, and a computer-readable storage medium, which can solve the problem that existing electronic devices with touch screens can only simultaneously obtain and respond to point events of one input device, resulting in a low user interaction operation experience.

[0006] In a first aspect, an embodiment of the present application provides an interaction method of an electronic device, which is applied to an electronic device. After the electronic device displays a target window, it receives a first input operation of a first input method and a second input operation of a second input method at the same time. The first input method and the second input method can be inputs performed through corresponding input devices. For example, the first input method is an input method based on a first input device, and the second input method is an input method based on a second input device. The first input device and the second input device are two different input devices, that is, input operations of two different input devices are received simultaneously; a first point reporting event and a second point reporting event are sent to a control corresponding to the target window, the first point reporting event is a point reporting event triggered by the first input operation, and the second point reporting event is a point reporting event triggered by the second input operation; the control corresponding to the target window can display corresponding content in the target window according to the first point reporting event and the second point reporting event.

[0007] As can be seen from the above, when the electronic device of the embodiment of the present application receives input operations from two different input devices at the same time, the reporting events triggered by the input operations of the two different input devices are reported to the control of the application at the same time, so that the application running in the foreground can obtain the reporting events of the two different input devices at the same time, and then can respond to the reporting events of the two different input devices at the same time, perform corresponding business operations, and display corresponding content in the window. In this way, the application can obtain the reporting events of the two different input devices at the same time, support users to perform multi-input device collaborative operations through two different input devices, and the multi-input device collaborative operation can complete some complex operations with higher operation efficiency, thereby improving the user's interactive operation experience.

[0008] Exemplarily, the first input device is a finger, that is, the first input method is finger input, and the second input device is a stylus, that is, the second input method is stylus input. In scenarios such as using a stylus for painting, retouching or taking notes, when the user uses the left finger and the stylus (holding the stylus in the right hand) to simultaneously perform touch operations on the touch screen of the tablet computer, the tablet computer simultaneously reports the finger point reporting event and the stylus point reporting event to the painting, note-taking or retouching application, so that the application can simultaneously obtain and respond to the finger point reporting event and the stylus point reporting event, and perform corresponding business operations, thereby supporting users to paint, retouch pictures or take notes through hand-pen collaborative operations, and providing a higher user interactive operation experience.

[0009] In some possible implementations of the first aspect, the target window may be an interface of the first application, or may include an interface of the second application and an interface of the third application. For example, the electronic device displays the interface of the second application and the interface of the third application in split screen on the screen. The second application and the third application may be different applications. Of course, in some other possible implementations, the target window may not be an interface of the application, but a system interface.

[0010] In some possible implementations of the first aspect, if the target window is the interface of the first application, when reporting a point, the electronic device can send the first point reporting event to the control of the first application, and if it is determined that the preset condition is met, the second point reporting event can be sent to the control of the first application, that is, the first point reporting event can be directly reported, and the second point reporting event can be further judged and processed before being reported. In this way, in addition to reporting the first point reporting event normally, the second point reporting event is not discarded immediately, but it is further judged whether the preset condition is met. If the preset condition is met, the second point reporting event is not discarded, but the second point reporting event is also reported, so that the distribution of point reporting events of two different input devices is supported during event distribution, and the point reporting events of two different input devices are reported to the control of the same application at the same time.

[0011] In some possible implementations of the first aspect, if the target window includes the interface of the second application and the interface of the third application. When the electronic device reports a point, it can send the first reporting event to the control of the second application, and if it is determined that the preset conditions are met, it can send the second reporting event to the control of the third application. That is, the reporting events triggered by the two input devices are reported to the corresponding applications respectively. In this way, in addition to reporting the first reporting event to the control of the first application normally, the second reporting event is not discarded immediately, but it is further determined whether the preset conditions are met. If the preset conditions are met, the second reporting event is not discarded, but the second reporting event is also reported to the control of the second application, so that the distribution of reporting events of two different input devices is supported during event distribution, and the reporting of the reporting events of the two different input devices to the controls of the two applications is realized.

[0012] In some possible implementations of the first aspect, if the electronic device receives a fourth input operation while also receiving a third input operation, the third point reporting event is sent to the control corresponding to the target window, and the fourth point reporting event is not sent to the control corresponding to the target window. The third point reporting event is a point reporting event triggered by the third input operation, and the fourth point reporting event is a point reporting event triggered by the fourth input operation. The third input operation and the fourth input operation may be input operations based on different input devices, for example, the third input operation is an input operation of a first input device, and the fourth input operation is an input operation of a second input device, or the third input operation and the fourth input operation are input operations based on input devices different from the first input device and the second input device.

[0013] In some possible implementations of the first aspect, the electronic device directly sends the third point reporting event to the control corresponding to the target window when reporting a point, and does not send the fourth point reporting event to the control corresponding to the target window when it is determined that a preset condition is not met.

[0014] In some possible implementations of the first aspect, the preset condition includes that the application or the system enables the multi-input mode input function. In some cases, after the system enables the multi-input mode function, all installed applications and the system can support the multi-input mode input function.

[0015] In some possible implementations of the first aspect, the preset condition includes that the second reporting point event is not a false reporting point event.

[0016] In some possible implementations of the first aspect, the preset condition includes that the characteristics of the target window conform to the characteristics of the multi-input mode (or multi-input device) collaborative operation scenario. Further, if the target window includes related controls for at least one of document editing, file editing, and image editing, the characteristics of the target window conform to the characteristics of the multi-input device collaborative operation scenario. For example, a canvas control is a type of image editing related control. In this way, in scenarios such as painting, document or file editing, the collaborative operation of multiple input devices is supported, thereby improving the user interaction experience in these scenarios.

[0017] In some possible implementations of the first aspect, the input time of the first first point reporting event triggered by the first input operation is earlier than the input time of the first second point reporting event triggered by the second input operation. That is, the point reporting event of one input device will not interrupt the reporting of the point reporting event of another input device.

[0018] In some possible implementations of the first aspect, the first input operation is an operation acting on any area of ​​the screen of the electronic device, and the second input operation is an operation acting on any area of ​​the screen of the electronic device. That is, each area of ​​the screen of the electronic device can support receiving inputs of the first input device and the second input device at the same time.

[0019] In some possible implementations of the first aspect, the user can turn on and off the multi-input device input function of the application. At this time, after receiving the user's setting operation, the electronic device responds to the user's setting operation and turns on the multi-input mode (or multi-input device) input function. In this way, the user can determine whether to turn on the multi-input device input function as needed, and the user experience is better.

[0020] In some possible implementations of the first aspect, the first input method and the second input method include any one or any two of finger input, stylus input, mouse input, air gesture input, and voice input.

[0021] In some possible implementations of the first aspect, the first input device is a finger, that is, the first input method is finger input, the second input device is a stylus, that is, the second input method is stylus input, and the electronic device includes a touch screen; in this case, the electronic device can receive a first touch operation of the finger on the touch screen, and receive a second touch operation of the stylus on the touch screen. The finger and the stylus are in contact with the touch screen at the same time. In this way, the application can support the user to perform collaborative operations through the finger and the stylus, and the user operation experience is high.

[0022] In some possible implementations of the first aspect, the electronic device includes an input reading module, a first input distribution module, and a second input distribution module. At this time, when reporting a point, the electronic device obtains a first input event triggered by a first input operation, obtains a second input event triggered by a second input operation, and converts the first input event into a first reporting event, and converts the second input event into a second reporting event; the input reading module transmits the first reporting event and the second reporting event to the first input distribution module and the second input distribution module; the first input distribution module determines that the first reporting event is a first type of event and the second reporting event is a second type of event, then distributes the first reporting event to the control corresponding to the target window and discards the second reporting event; the second input distribution module determines that the second reporting event is a second type of event, and determines that the multi-input device input function is in an enabled state, and the characteristics of the target window meet the characteristics of the multi-input device collaborative operation scenario, then distributes the second reporting event to the control corresponding to the target window.

[0023] In this implementation, by adding a second input distribution module, and allowing the input reading module to transmit the first point reporting event and the second point reporting event to the first input distribution module at the same time, the first input distribution module and the second input distribution module obtain the same point reporting event. On this basis, the first input distribution module distributes events according to the normal event distribution process, and the second input distribution module makes further judgments on the discarded events of the first input distribution module. If the multi-input device input function is in an enabled state and the characteristics of the target window meet the characteristics of the multi-input device collaborative operation scenario, the second point reporting event is also reported to the control of the application, thereby realizing the point reporting event distribution that supports two different input devices at the same time during event distribution.

[0024] In some possible implementations of the first aspect, the electronic device includes an input reading module and a third input distribution module. At this time, when the electronic device reports a point, after obtaining a first input event triggered by a first input operation and a second input event triggered by a second input operation, the first input event can be converted into a first reporting event and the second input event can be converted into a second reporting event; the input reading module transmits the first reporting event and the second reporting event to the third input distribution module; the third input distribution module determines that the first reporting event is a first type of event, the second reporting event is a second type of event, and determines that the multi-input device input function is in an enabled state, and the characteristics of the target window meet the characteristics of the multi-input device collaborative operation scenario, and then distributes the first reporting event and the second reporting event to the control corresponding to the target window.

[0025] In this implementation, by modifying the event distribution logic of the third input distribution module, the third input distribution module does not directly discard the events that need to be discarded when distributing events, but further determines whether the input function of the multi-input device is in an enabled state and whether the characteristics of the target window meet the characteristics of the multi-input device collaborative operation scenario. If so, in addition to reporting the first point reporting event normally, the second point reporting event is also reported, thereby realizing the point reporting event distribution that supports two different input devices at the same time during event distribution.

[0026] In some possible implementations of the first aspect, the electronic device may include an anti-false touch algorithm, which can perform false touch judgment on input events. At this time, when the electronic device is in the process of converting the first input event into the first reporting event and the second input event into the second reporting event, if the multi-input device (or multi-input mode) collaborative module is in an enabled state, the preset anti-false touch algorithm in the anti-false touch algorithm set is invalidated, and the remaining anti-false touch algorithms are used to perform false touch processing on the first input event and the second input event, and the remaining anti-false touch algorithms are the anti-false touch algorithms in the anti-false touch algorithm set except the preset anti-false touch algorithms; further, if it is determined that both the first input event and the second input event are not false touch reporting events, and the multi-input device collaborative (or multi-input mode) module determines that the first input event and the second input event are both reporting events in the multi-input device (or multi-input mode) collaborative operation scenario, the first input event is written into the first device node corresponding to the first input device, and the second input event is written into the second device node corresponding to the second input device;

[0027] The input reading module reads the first input event from the first device node and the second input event from the second device node through the event monitoring module, and encapsulates the first input event into a first reporting event and encapsulates the second input event into a second reporting event.

[0028] In this implementation, when the hardware anti-false touch algorithm is processed for the input event, if the multi-input device (or multi-input mode) collaborative module is in an enabled state (i.e., in an on state), the preset anti-false touch algorithm that affects the multi-input device collaborative operation scenario is invalidated, and it is further determined whether the input event is a point reporting event under the multi-input device collaborative operation scenario. If so, the input events of two different input devices are reported at the same time, instead of reporting the input event of one of the input devices, considering the input event of the other input device as a false touch event, and discarding the input event, thereby supporting the reporting of input events of two different input devices at the same time during the hardware anti-false touch processing.

[0029] In some possible implementations of the first aspect, while receiving a first input operation of a first input method and before receiving a second input operation of a second input method, the electronic device may also receive a registration request for an application, and in response to the registration request, add the application to an application whitelist maintained by a multi-input device collaborative operation service; if it is sensed that an application in the application whitelist is switched to the foreground, the multi-input device collaborative operation service enables the second input distribution module and the multi-input device collaborative module, or enables the third input distribution module and the multi-input device collaborative module.

[0030] In this implementation, a new multi-input device collaborative operation service is added to manage the applications and enablement of multi-input device collaborative operation. In this way, through the management and enablement of the application whitelist, applications that need to support the collaborative operation of multiple input devices can simultaneously obtain the reporting events of two different input devices, while other applications are not affected, further improving the user interaction experience.

[0031] For example, for painting, note-taking, or photo editing applications, these applications are added to the whitelist so that they can obtain and respond to the reporting events of two different input devices, supporting users to operate in coordination with multiple input devices. For some other applications, these applications are not added to the whitelist so that they can still obtain the reporting events of only one input device in the original way.

[0032] In a second aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a method as described in any one of the first aspects above when executing the computer program.

[0033] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements any method as described in the first aspect above.

[0034] In a fourth aspect, an embodiment of the present application provides a chip system, the chip system includes a processor, the processor is coupled to a memory, and the processor executes a computer program stored in the memory to implement any method as described in any one of the first aspects above. The chip system can be a single chip, or a chip module composed of multiple chips.

[0035] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the method described in the first aspect above.

[0036] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0038] Figure 2 A schematic block diagram of a software system 200 provided in an embodiment of the present application;

[0039] Figure 3 A schematic diagram of the structure of an electronic device 300 provided in an embodiment of the present application;

[0040] Figure 4A Another schematic block diagram of the software system 200 provided in the embodiment of the present application;

[0041] Figure 4B Another schematic block diagram of the software system 200 provided in the embodiment of the present application;

[0042] Figure 5A A schematic diagram of hand-pen collaborative operation in the note editing process provided in an embodiment of the present application;

[0043] Figure 5B A schematic diagram of hand-pen collaborative operation during the painting process provided in an embodiment of the present application;

[0044] Figure 5C A schematic diagram of hand-pen collaborative operation in the image processing process provided by an embodiment of the present application;

[0045] Figure 5D A schematic diagram of simultaneous hand-pen operation in a split-screen scenario provided in an embodiment of the present application;

[0046] Figure 5E A schematic diagram of simultaneous hand-pen operation in a small window scenario provided by an embodiment of the present application;

[0047] Figure 6 A schematic block diagram of a process of an interaction method of an electronic device provided in an embodiment of the present application;

[0048] Figure 7 Another schematic block diagram of the process of the interaction method of the electronic device provided in the embodiment of the present application;

[0049] Figure 8 This is a schematic block diagram of the structure of an electronic device 300 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures and technologies are provided to facilitate a thorough understanding of the embodiments of the present application.

[0051] See also Figure 1 The schematic diagram of the application scenario provided by the embodiment of the present application is shown, and the application scenario includes a tablet computer 100, a stylus pen 110, a finger 120 and a mouse 130. The mouse 130 can be a wireless mouse (such as a Bluetooth mouse) or a wired mouse. The stylus pen 110 can be exemplarily an active capacitive stylus pen or a passive capacitive stylus pen.

[0052] The tablet computer 100 can support input from a variety of different input devices, such as a stylus 110, a finger 120, and a mouse 130. The user can perform touch operations on the touch screen of the tablet computer 100 through the finger 120 or the stylus 110, or can click or double-click on the touch screen of the tablet computer through the mouse 130, and the tablet computer 100 can respond to the received input operations (such as touch operations and click operations, etc.) and perform corresponding operations.

[0053] Understandably, Figure 1 This is only an exemplary application scenario and does not limit the application scenario of the embodiment of the present application. For example, the tablet computer 100 can be replaced by a smart phone, a laptop computer (with a touch screen), or a car computer, or even any electronic device with a touch screen. In addition, the types and number of input devices supported by the electronic device are not limited to Figure 1 For example, the input device may only have a stylus pen 110 and a finger 120 , and a smart phone generally supports input from the stylus pen 110 and the finger 120 .

[0054] In the related art, although the electronic device can support input from a variety of different input devices, it can only obtain the reporting event of one input device at the same time and can only respond to the input operation of one input device at the same time.

[0055] Generally, when an input device performs an input operation on a touch screen of an electronic device, the electronic device can collect the input position of the input device on the screen, generate a point reporting event of the input device according to the collected input position, and report the point reporting event to the corresponding window of the application. The point reporting event may include position point information and timestamp information, etc.

[0056] The location point information can be expressed as coordinate information. For example, an XY coordinate system is established on the touch screen, and any location point on the touch screen can be represented by X-axis coordinates and Y-axis coordinates. The timestamp information is used to characterize the time when the reporting point occurs. In addition to the location point information and timestamp information, the reporting point event can also include information such as event identifier and reporting rate. The event identifier is used to identify the touch event to which the reporting point belongs. A touch event can include a press (touch down) reporting event and a lift (touch up) reporting event, and may also include a move (touch move) reporting event between the touch down event and the touch up event.

[0057] Among them, when the user's finger or stylus makes initial contact with the touch screen, a touch down event is generated. For example, the user uses a stylus to write on the touch screen; when the user's finger or stylus slides on the touch screen, a touch move event is generated. For example, after the stylus is put on the touch screen, the stylus slides on the touch screen; when the user's finger or stylus leaves the touch screen (that is, the user lifts up the finger or stylus, so that the finger or stylus and the touch screen change from a contact state to a separation state), a touch up event is generated. For example, after the stylus is put on the touch screen and slides for a distance, the user lifts up the stylus, so that the stylus is separated from the touch screen.

[0058] It is understandable that when the input device is a mouse, a point reporting event similar to a touch event will also be generated. For example, when a user uses a mouse to perform a single-click operation on a display screen, a mouse down event is generated when the user presses the mouse; and a mouse up event is generated when the user releases the mouse after pressing it.

[0059] For another example, when a user presses the mouse, moves it a certain distance, and then releases it, a mouse down event is generated when the user presses the mouse; when the user presses the mouse without releasing it and moves it, one or more mouse move events are generated; when the user releases the mouse, a mouse up event is generated. The mouse down event, mouse move event, and mouse up event can all include information such as location information and timestamps.

[0060] In order to better introduce the reporting process of the input device in the related technology, the following is combined with Figure 1 and Figure 2 A schematic block diagram of a software system 200 provided in an embodiment of the present application is shown for introduction and explanation.

[0061] A software system 200 is running on the tablet computer 100. Figure 2As shown, the software system 200 may include a driver layer, a hardware abstraction layer (HAL), an application framework (Framework) layer and an application (Application, App) layer.

[0062] The driver layer includes the driver program of the stylus pen and the driver program of the touch panel (TP). Of course, the driver layer can also include other drivers such as display driver, Bluetooth driver, mouse driver and keyboard driver.

[0063] The hardware abstraction layer includes a hardware anti-false touch algorithm module 210. The hardware anti-false touch algorithm module 210 includes multiple anti-false touch algorithms. The anti-false touch algorithm is used to perform false touch processing on the input event reported by the driver layer to determine whether the input event is a false touch reporting event; if it is a false touch reporting event, the input event is discarded; if it is not a false touch reporting event, the input event is passed to the next layer.

[0064] The application framework layer includes an input management service (IMS) 220 and a window management service (WMS) 230. The IMS 220 may include an input distribution module (InputDispatcher) 221, an input reading module (InputReader) 222, and an event monitoring module (EventHub) 223. The WMS 230 includes window information of multiple windows (Window1 to WindowN). N is a positive integer greater than or equal to 1.

[0065] The application layer includes multiple applications 240. Each application 240 may include multiple controls (View1 to ViewM) and a root node object of a view control tree (such as a ViewRoot object in an Android system, which may be a C++-based function), where M is a positive integer greater than or equal to 1. The view control tree includes multiple controls.

[0066] It should be pointed out that Figure 2 Only the content related to the present application in the software system 200 is shown. In addition, the hardware abstraction layer in the software system 200 is optional. In some embodiments, the software system 200 may not include the hardware abstraction layer. In some embodiments, the software system 200 may include the hardware abstraction layer, but the hardware abstraction layer may not include the hardware false touch prevention algorithm module 210. That is, the software system 200 may not include the hardware false touch prevention algorithm module 210, and may also include the hardware false touch prevention algorithm module 210.

[0067] Based on Figure 2 In the software system 200 shown, when the tablet computer 100 receives an input operation from only one input device, the tablet computer 100 responds to the input operation by reporting one or more reporting events triggered by the input operation to the control of the application.

[0068] Taking the stylus pen 110 as an example, the process of reporting a point event of the stylus pen 110 may be as follows:

[0069] After a wireless connection (eg, Bluetooth connection) is established between the tablet computer 100 and the stylus 110 , when the user uses the stylus 110 to perform a touch operation on the touch screen of the tablet computer 100 , the driver of the stylus 110 receives one or more input events triggered by the touch operation of the stylus.

[0070] The driver layer passes the input event of the stylus 110 to the hardware abstraction layer. The hardware anti-mistouch algorithm module 210 of the hardware abstraction layer performs anti-mistouch processing on the input event of the stylus 110 to determine whether the current input event is a mistouch reporting event. Exemplarily, the mistouch reporting event can be: a touch event of the palm; a mistouch reporting event of the edge of the stylus in the touch down state, that is, when the user is using the stylus 110 to write, the user's hand accidentally touches the back key on the edge (such as touch operations such as left swipe, right swipe and exit).

[0071] At this time, the hardware anti-mistouch algorithm module 210 determines that the input event of the stylus 110 is not a mistouch reporting event, and the hardware abstraction layer writes the input event of the stylus 110 to the stylus device node in the / dev / input directory. The software system 200 includes device nodes of various input devices, for example, the software system 200 may include device nodes of the stylus 110, the mouse 130, and the touch screen, and the device node of the touch screen is different from the device node of the stylus 110.

[0072] The event monitoring module 223 monitors each input device node, and when it monitors that an input event is written to the input device node, it reads the input event from the input device node. At this time, the event monitoring module 223 monitors that an input event is written to the stylus device node, and reads the input event from the stylus device node.

[0073] After the event monitoring module 223 reads the input event of the stylus 110, it transmits the input event to the input reading module 222. The input reading module 222 translates, assembles and encapsulates the input event to obtain a high-level input event (or a reporting event) recognizable by the upper layer, and transmits the reporting event to the input distribution module 221 for event distribution.

[0074] After the input distribution module 221 obtains the reporting event, it determines the target window corresponding to the current reporting event according to the recorded window information of each application, and distributes the reporting event to the target window. After the target window obtains the reporting event of the stylus 110, it can distribute the reporting event of the stylus 110 to the corresponding control along the control tree through the root node object of the view control tree (such as ViewRoot object). The control responds to the received reporting event and performs the corresponding business processing operation.

[0075] In this way, the reporting process of a point reporting event triggered by the input operation of the stylus pen 110 is completed.

[0076] It is understandable that the input operation of the stylus pen 110 on the touch screen may trigger a series of point reporting events, and each point reporting event can be reported according to the above process.

[0077] based on Figure 2 In the software system 200 shown, when the tablet computer 100 receives input operations from at least two different input devices at the same time, since the hardware anti-mistouch algorithm module 210 of the software system 200 does not support simultaneous reporting of multiple devices, and the input distribution module 221 does not support simultaneous distribution of reporting of multiple devices, the software system 200 does not support simultaneous reporting of multiple different input devices, and the application cannot obtain reporting events of multiple different input devices at the same time.

[0078] Among them, when the hardware anti-mistouch algorithm module 210 receives reporting events from multiple different input devices, it will determine that the reporting events of all or part of the input devices are mistouch reporting events, and then discard the reporting events of some or all input devices. It is impossible to report the reporting events of multiple different input devices to the next layer at the same time, so it does not support simultaneous reporting of multiple devices.

[0079] For example, when a user uses both the finger 120 and the stylus 110 to perform touch operations on the touch screen of the tablet computer 100, the hardware anti-mistouch algorithm module 210 simultaneously receives the point reporting events of the stylus 110 and the finger 120, and processes them according to the hardware anti-mistouch algorithm to determine whether to discard at least one of the point reporting events of the stylus 110 and the finger 120.

[0080] When the input distribution module 221 receives point reporting events from multiple different input devices, if one input device is already reporting, the current point reporting event from another input device will be discarded, and the point reporting event from the existing input device will continue to be distributed; or, the point reporting event distribution of the existing input device will be interrupted, the point reporting event from the existing input device will be discarded, and the point reporting event from the existing input device will be distributed to the corresponding window. In this way, the input distribution module 221 cannot distribute the point reporting events from multiple different input devices to the target window at the same time, and does not support the simultaneous distribution of point reporting events from multiple devices.

[0081] For example, Figure 2 In the software system 200 shown, the point reporting event of the stylus pen 110 can interrupt the distribution of the point reporting event of the finger 120. Specifically, when the user uses the finger 120 to perform a touch operation on the touch screen of the tablet computer 100, the input distribution module 221 only receives the point reporting event of the finger 120, and continuously distributes the point reporting events such as the touch down event and the touch move event of the finger 120 to the corresponding window.

[0082] During the process of the user using the finger 120 to perform a touch operation, the user also uses the stylus 110 to perform a touch operation on the touch screen of the tablet computer 100. At this time, the input distribution module 221 is distributing the finger 120's reporting event, that is, there is already an input device (i.e., the finger 120) reporting a point. Since the finger 120 and the stylus 110 are in contact with the touch screen at the same time, the input distribution module 221 receives the finger 120's reporting event and the touch down event of the stylus 110. At this time, the input distribution module 221 interrupts the distribution of the finger 120's reporting event, and discards all the finger 120's reporting events (for example, the subsequent move event and up event of the finger 120 will be discarded), and distributes the touch down event of the stylus 110 to the corresponding window. Therefore, the input distribution module 221 cannot distribute the finger 120's reporting event and the stylus 110's reporting event to the corresponding window at the same time, and the application cannot obtain the finger 120's reporting event and the stylus 110's reporting event at the same time. That is to say, for the upper layer application, the touch down event of the stylus pen 110 will not appear between the touch down event and the touch up event of the finger 120 , thereby making it impossible for the application to obtain the reporting events of both the stylus pen 110 and the finger 120 input devices at the same time.

[0083] That is to say, in the related art, electronic devices do not support multiple different input devices reporting at the same time, and applications cannot obtain the reporting events of multiple different input devices at the same time, which makes it impossible for users to perform collaborative operations through multiple different input devices, affecting the user's interactive operation experience. Especially in scenes such as painting, note-taking and photo editing, there are usually some complex operations such as graphic adjustment, and users can only use one input device to complete these complex operations, which is cumbersome and complicated, and the user's interactive operation experience is low.

[0084] In response to the above-mentioned related issues, the embodiment of the present application proposes an interactive solution for electronic devices, so that when the electronic device receives input operations from multiple different input devices, the electronic device can simultaneously report the reporting events of multiple different input devices to the control of the application, so that the control of the application can respond to the reporting events of multiple different input devices and perform corresponding business operations, thereby supporting users to perform multi-device collaborative input operations through multiple different input devices, thereby improving the user's interactive operation experience. Especially for complex operations in scenes such as painting, note-taking, and photo editing, users can also complete complex operations through multiple different input devices in a collaborative manner, with high operating efficiency and simpler operation.

[0085] The electronic device may be a tablet computer, a smart phone, or a car device, or any electronic device with a touch screen. The specific type and specific hardware structure of the electronic device in the embodiment of the present application are not limited.

[0086] For example, Figure 3 The schematic diagram of the structure of the electronic device 300 provided in the embodiment of the present application is shown. The electronic device 300 may include a processor 310, a memory 320, a display screen 330, a sensor module 340, and a wireless communication module 350. Among them, the sensor module 340 may include a touch sensor 340A and the like.

[0087] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 300. In other embodiments of the present application, the electronic device 300 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0088] The processor 310 may include one or more processing units, for example, the processor 310 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), and a baseband processor, etc. Different processing units may be independent devices or integrated into one or more processors.

[0089] In some embodiments, the processor 310 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), and a general-purpose input / output (GPIO) interface.

[0090] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 310 may include multiple groups of I2C buses. The processor 310 may be coupled to the touch sensor 340A and the like through different I2C bus interfaces. For example, the processor 310 may be coupled to the touch sensor 340A through the I2C interface, so that the processor 310 communicates with the touch sensor 340A through the I2C bus interface, thereby realizing the touch function of the electronic device 300.

[0091] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 310 and the wireless communication module 350. For example, the processor 310 communicates with the Bluetooth module in the wireless communication module 350 through the UART interface to implement the Bluetooth function.

[0092] The MIPI interface can be used to connect the processor 310 with peripheral devices such as the display screen 330. The MIPI interface includes a display serial interface (DSI), etc. In some embodiments, the processor 310 and the display screen 330 communicate via the DSI interface to implement the display function of the electronic device 300.

[0093] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 310 with the display screen 330, the wireless communication module 350, the sensor module 340, etc. The GPIO interface can also be configured as an I2C interface, a UART interface, and a MIPI interface, etc. It can be understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 300. In other embodiments of the present application, the electronic device 300 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0094] The wireless communication function of the electronic device 300 can be implemented through an antenna, a wireless communication module 350, a modem processor, a baseband processor, and the like.

[0095] The antenna is used to transmit and receive electromagnetic wave signals. The modulation and demodulation processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is transmitted to the application processor.

[0096] The wireless communication module 350 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), near field communication (NFC), Bluetooth Low Energy (BLE), and infrared (IR) technology applied to the electronic device 300. The wireless communication module 350 can be one or more devices integrating at least one communication processing module. The wireless communication module 350 receives electromagnetic waves via an antenna, modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 310. The wireless communication module 350 can also receive the signal to be sent from the processor 310, modulate the frequency, amplify it, and convert it into electromagnetic waves for radiation through the antenna. Exemplarily, the electronic device 300 can connect input devices such as a stylus and a Bluetooth mouse via Bluetooth.

[0097] In some embodiments, the antenna of the electronic device 300 is coupled to the wireless communication module 350, so that the electronic device 300 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include BT, WLAN, NFC, BLE and / or IR technology, etc.

[0098] The electronic device 300 implements the display function through a GPU, a display screen 330, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 330 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 310 may include one or more GPUs, which execute program instructions to generate or change display information.

[0099] The display screen 330 is used to display images, videos, etc. The display screen 330 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 300 may include 1 or N display screens 330, where N is a positive integer greater than 1.

[0100] The memory 320 can be used to store computer executable program codes, which include instructions. The memory 320 may include a program storage area and a data storage area. Among them, the program storage area may store software systems, applications required for at least one function (such as sound playback function, image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 300 (such as audio data, phone book, etc.), etc. In addition, the memory 320 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 310 executes various functional applications and data processing of the electronic device 300 by running instructions stored in the memory 320, and / or instructions stored in a memory provided in the processor.

[0101] The touch sensor 340A is also called a "touch control device". The touch sensor 340A can be set on the display screen 330, and the touch sensor 340A and the display screen 330 form a touch screen, also called a "touch control screen". The touch sensor 340A is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 330. In other embodiments, the touch sensor 340A can also be set on the surface of the electronic device 300, which is different from the position of the display screen 330.

[0102] After the hardware structure of the electronic device 300 is exemplarily introduced, the software system of the electronic device 300 is introduced below.

[0103] The software system of the electronic device 300 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes a layered architecture software system as an example to exemplify the software structure of the electronic device 300.

[0104] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the software system is divided into four layers, from top to bottom: application layer, application framework layer, hardware abstraction layer, and driver layer.

[0105] In order to allow the electronic device 300 to support simultaneous reporting of multiple different input devices, the present application embodiment provides two solutions, which are described below.

[0106] The first solution is to add an input dispatcher module to the application framework layer and a multi-input device coordination module to the hardware anti-false touch algorithm module. In addition, the logic of the input reader module is modified so that the input reader module transmits point events not only to the original input dispatcher module, but also to the newly added input dispatcher module.

[0107] The original input distribution module distributes events according to the original event distribution strategy. After the newly added input distribution module determines the discarded reporting events according to the original event distribution strategy, it also makes further judgments on the discarded reporting events. If the discarded reporting events meet the requirements, the discarding operation is not performed, but the originally discarded reporting events are distributed to the target window.

[0108] The multi-input device collaboration module can invalidate some anti-mistouch algorithms that affect the multi-input device collaborative operation scenarios, and can determine whether the point reporting event needs to be reported. If it is determined that the point reporting event needs to be reported, the point reporting events of multiple input devices will be reported to the next layer at the same time.

[0109] Optionally, a multi-input device collaboration service may be newly added to the application framework layer. The multi-input device collaboration service is used to sense the running state and life cycle of the application, and enable the multi-input device collaboration module and the newly added input distribution module according to the running state of the application.

[0110] The running status of an application may include foreground running and background running, and the life cycle refers to information such as application opening and application closing.

[0111] The multi-input device collaborative service can maintain an application whitelist. The applications in the application whitelist are applications that need to support multi-input device collaborative operations. The application actively sends a registration request to the multi-input device collaborative service. In response to the registration request, the multi-input device collaborative service adds the application that initiated the active registration to the maintained application whitelist. For specific applications, the application can actively register with the multi-input device collaborative service when it is opened or installed. For example, the information on whether the handwriting collaborative service is supported is written into the configuration file of the application in advance. When the application is installed, the software system of the electronic device determines whether the installed application supports the handwriting collaborative service by reading the configuration file. When the relevant information of the configuration file indicates that the application supports the handwriting collaborative service, it is considered that the application actively initiated the service registration, and the handwriting collaborative service adds the application to the application whitelist.

[0112] When the multi-input device collaboration service senses that an application in the application whitelist is switched to the foreground, the multi-input device collaboration module and the newly added input distribution module are enabled. After the multi-input device collaboration module and the newly added input distribution module are enabled, the electronic device 300 enables the multi-input device input function, and the application running in the foreground can support simultaneous input of multiple input devices, that is, support simultaneous reporting of multiple different input devices.

[0113] For example, taking the scenario of collaborative operation of a stylus and a finger as an example, at this time, the multi-input device collaborative service may refer to a stylus collaborative service, and the multi-input device collaborative module may refer to a stylus collaborative module.

[0114] See also Figure 4A Another schematic block diagram of a software system 200 provided in an embodiment of the present application is shown. The software system 200 may include a driver layer, a hardware abstraction layer, an application framework layer, and an application program layer.

[0115] The driver layer includes drivers for the stylus and the touch screen. The hardware anti-mistouch algorithm module 210 of the hardware abstraction layer includes a stylus coordination module 211 .

[0116] The application framework layer includes an input management service 220, a window management service 230, a handwriting coordination service 250, an activity management service (AMS) 260, and a handwriting coordination input distribution module 270. The input management service 220 includes an event monitoring module 223, an input reading module 222, and an input distribution module 221.

[0117] The application layer may include a painting application 241. The painting application 241 may include a canvas control 242, other interactive controls 243, and a hand event processing module 244. It is understood that in other embodiments, the painting application 241 may be replaced with other types of applications, such as a note application or an image editing application.

[0118] Compared to Figure 2 The software system 200 shown, Figure 4A The software system 200 newly adds: a handwriting coordination module 211 , a handwriting coordination service 250 , an activity management service 260 , a handwriting coordination input distribution module 270 , and a handwriting event processing module 244 .

[0119] also, Figure 4A The event transmission logic of the input reading module 222 is also modified so that Figure 4A When the input reading module 222 transmits a point reporting event to the input distribution module 221, it also transmits a point reporting event to the handwriting collaborative input distribution module 270. That is, the point reporting events received by the input distribution module 221 and the handwriting collaborative input distribution module 270 are the same. For example, the input reading module 222 obtains the point reporting event of the stylus pen and the point reporting event of the finger, transmits the point reporting event of the stylus pen and the point reporting event of the finger to the input distribution module 221, and also transmits the point reporting event of the stylus pen and the point reporting event of the finger to the handwriting collaborative input distribution module 270.

[0120] It can be understood that the input distribution module 221, the input reading module 222, and the event monitoring module 223 are all threads of the software system 200. That is, the input distribution module 221 is the Input Dispatcher thread, the input reading module 222 is the InputReader thread, and the event monitoring module 223 is the EventHub thread. The newly added handwriting collaborative input distribution module 270 can be regarded as a newly added Input Dispatcher thread.

[0121] based on Figure 4A In the software system 200, the reporting process of the stylus 110 and the finger 120 can be as follows:

[0122] After the painting application 241 is started or installed, it registers with the handwriting coordination service 250 and starts handwriting coordination. The handwriting coordination service 250 adds the painting application 241 to the application whitelist. The applications in the application whitelist are applications that need to support handwriting coordination operations.

[0123] The handwriting coordination service 250 monitors the running state and life cycle of the application through the activity management service 260. When monitoring that the application in the application whitelist (i.e., the painting application 241) switches to the foreground, the handwriting coordination service 250 enables the handwriting coordination input distribution module 270 and the handwriting coordination module 211.

[0124] It is understandable that the handwriting collaborative input distribution module 270 and the handwriting collaborative module 211 are initially in a closed state, and when receiving an enable signal from the handwriting collaborative service 250 , they are changed from the closed state to the open state.

[0125] based on Figure 1 The scenes shown and Figure 4A In the software system 200 shown, when a user performs input operations on the touch screen of the tablet computer 100 using the stylus 110 and the finger 120, the driver of the stylus 110 receives input events from the stylus, and the driver of the touch screen receives input events from the finger.

[0126] The driver layer transfers the input events of the stylus pen 110 and the input events of the finger 120 to the hardware abstraction layer.

[0127] The hardware anti-false touch algorithm module 210 of the hardware abstraction layer may include two types of anti-false touch algorithms, one is a general anti-false touch algorithm, and the other is an anti-false touch algorithm that affects hand-pen collaborative reporting. The anti-false touch algorithm that affects hand-pen collaborative reporting may refer to an algorithm that determines at least one of the input events of the stylus and the input events of the finger as a false touch reporting event. The general anti-false touch algorithm will not determine the input events of the stylus and the input events of the finger as false touch reporting events.

[0128] When the hand-pen coordination module 211 is not turned on, that is, when it is not in the enabled state, the hardware anti-false touch algorithm module 210 runs the general anti-false touch algorithm and the anti-false touch algorithm that affects the hand-pen coordination reporting, and performs the anti-false touch algorithm judgment on the input event. Therefore, it may regard at least one of the input events of the stylus 110 and the input events of the finger 120 as a false touch reporting event, and discard the false touch reporting event, and cannot realize the reporting of the input events of the stylus 110 and the input events of the finger 120 to the next layer.

[0129] After the handwriting coordination module 211 is turned on, the general anti-false touch algorithm in the hardware anti-false touch algorithm module 210 remains valid, while the anti-false touch algorithm affecting the handwriting coordination reporting becomes invalid. In other words, the hardware anti-false touch algorithm module 210 only runs the general anti-false touch algorithm to perform anti-false touch algorithm judgment on the input events of the stylus 110 and the input events of the finger 120, and does not run the anti-false touch algorithm affecting the handwriting coordination reporting to perform anti-false touch algorithm judgment on the input events of the stylus 110 and the input events of the finger 120.

[0130] The hardware anti-mistouch algorithm module 210 performs an anti-mistouch algorithm judgment on the input events of the stylus pen 110 and the input events of the finger 120 through a general anti-mistouch algorithm, and determines that the input events of the stylus pen 110 and the input events of the finger 120 are not mistouch reporting events, and then transmits the input events of the stylus pen 110 and the input events of the finger 120 to the stylus collaboration module 211.

[0131] The handwriting coordination module 211 further judges the input events of the stylus 110 and the finger 120 to determine whether the input events of the stylus 110 and the finger 120 are point reporting events that need to be reported in the handwriting coordination scenario. If it is recognized that the input events of the stylus 110 and the finger 120 are both point reporting events in the handwriting coordination scenario, the input events of the stylus 110 are written to the stylus device node in the / dev / input directory, and the input events of the finger 120 are written to the touch screen device node in the / dev / input directory.

[0132] In short, when the stylus collaboration module 211 is not turned on, the hardware anti-mistouch algorithm module 210 judges the input events obtained according to the original anti-mistouch algorithm processing flow, and discards the mistouch reporting events according to the original flow. After the stylus collaboration module 211 is turned on, it ignores (or disables) some of the anti-mistouch algorithms that affect the stylus collaboration scene, identifies the reporting events in the stylus collaboration scene, and reports the stylus input events and finger input events at the same time.

[0133] When the event monitoring module 223 monitors that new input events are written into the stylus device node and the touch screen device node, it reads the input events of the stylus 110 from the stylus device node and reads the input events of the finger 120 from the touch screen device node.

[0134] The input reading module 222 obtains the input events of the stylus pen 110 and the input events of the finger 120 read by the event monitoring module 223, and performs operations such as translation, assembly and packaging on the obtained input events to obtain high-level input events (or point reporting events) that can be recognized by the upper layer. At this time, the point reporting events of the stylus pen 110 and the point reporting events of the finger 120 can be obtained through operations such as translation and packaging.

[0135] The input reading module 222 transmits the point reporting events of the stylus pen 110 and the finger 120 to the input distribution module 221 . In particular, the input reading module 222 also transmits the point reporting events of the stylus pen 110 and the finger 120 to the hand-pen coordinated input distribution module 270 .

[0136] After acquiring the stylus pen pointing event and the finger pointing event, the input distribution module 221 distributes the stylus pen 110 pointing event and the finger pointing event according to the original event distribution strategy.

[0137] According to the original event distribution strategy, a discard event and a report event are determined from the reporting event of the stylus 110 and the reporting event of the finger 120. The discard event may be the reporting event of the stylus 110 or the reporting event of the finger 120, and the report event is the other of the two reporting events.

[0138] For example, suppose the original event distribution strategy is: if there is an input device currently reporting a point, the reporting event of another different input device is discarded. At this time, according to the event distribution strategy, the input distribution module 221 determines that the current reporting event is the reporting event of the finger 120. After obtaining the reporting event of the finger 120 and the reporting event of the stylus 110, since the stylus 110 is different from the finger 120 and belongs to a different input device, it is determined that the reporting event of the finger 120 is a reporting event, that is, the reporting event of the finger 120 continues to be distributed to the corresponding window, and it is determined that the reporting event of the stylus 110 is a discarded event, that is, the reporting event of the stylus 110 is discarded, and the reporting event of the stylus 110 is not distributed to the corresponding window.

[0139] After the input distribution module 221 determines the reporting events that need to be reported and the reporting events that need to be discarded, it discards the corresponding reporting events, and determines the target window corresponding to the reporting events that need to be reported based on the recorded window information of each application, and distributes the reporting events to the target window.

[0140] That is, when the input distribution module 221 distributes events according to the original event distribution strategy, only one of the point reporting event of the stylus pen 110 and the point reporting event of the finger 120 is distributed to the target window.

[0141] The handwriting collaborative input distribution module 270 obtains the point reporting events of the stylus and the finger, and after determining the point reporting events that need to be discarded according to the event distribution strategy of the input distribution module 221 (i.e., the original event distribution strategy), further judgment is made on the point reporting events that need to be discarded; if the current handwriting collaborative input distribution module 270 is in an enabled state (or the handwriting collaborative system is turned on, and the application running in the foreground has registered the handwriting collaborative service, etc.), and the characteristics of the current target window meet the characteristics of the handwriting collaborative scene, the point reporting event that needs to be discarded is not discarded, but the target window corresponding to the point reporting event that needs to be discarded is determined based on the recorded window information of each application, and the point reporting event that needs to be discarded is distributed to the target window.

[0142] The target window may be a focus window. The focus window may refer to a window where the focus is located, which is usually a window of an application running in the foreground. The target window may be an interface of an application, or may be an interface of a software system of the electronic device 300. Further, the target window may be an interface of an application; or may be an interface including two applications, for example, the interfaces of two applications are displayed in split screen on the screen of the electronic device, in which case the target window may include the interfaces of the two applications; or may be an interface including an application and an interface of a software system, for example, an interface of an application is displayed superimposed on the system interface, in which case the target window may include the system interface and the application interface.

[0143] When the target window includes the target control, it is considered that the characteristics of the target window meet the characteristics of the hand-pencil collaboration scene; when the focus window does not include the target control, it is considered that the characteristics of the target window do not meet the characteristics of the hand-pencil collaboration scene.

[0144] Exemplarily, the target control may refer to a control related to at least one of image editing, document editing, and file editing. For example, a canvas control is a control related to image editing. When the target window contains only static information, or controls such as buttons, numbers, paths, and clicks, it is considered that the characteristics of the target window do not conform to the characteristics of the handwriting collaboration scene. Click controls may, for example, include a display page of a user privacy agreement and a homepage list; when the target window includes a canvas control, etc., it is determined that the characteristics of the target window conform to the characteristics of the handwriting collaboration scene.

[0145] After the target window obtains the reporting events of the stylus pen 110 and the finger 120, the reporting events of the stylus pen 110 and the finger 120 are dispatched to the corresponding controls along the control tree through the root node object of the view control tree (for example, the ViewRoot object). The control responds to the received reporting events and performs corresponding business processing operations to display the corresponding content in the target window.

[0146] like Figure 4A As shown, the painting application 241 receives the reported point reporting events of the stylus pen 110 and the finger 120. The hand event processing module 244 processes the point reporting events of the stylus pen 110 and the finger 120 and performs corresponding service processing operations.

[0147] In this way, when the painting application 241 is in the foreground running state, when the user uses the stylus 110 and the finger 120 to perform hand-pen collaborative operations on the touch screen of the tablet computer 100 at the same time, Figure 4A The software system 200 can report the point reporting event of the stylus pen 110 and the point reporting event of the finger 120 to the painting application 241 at the same time, so that the painting application 241 can respond to the point reporting event of the stylus pen 110 and the point reporting event of the finger 120 at the same time and perform corresponding business processing operations.

[0148] As can be seen from the above, by adding a new handwriting collaborative input distribution module 270 in the application framework layer, the point reporting events of the stylus 110 and the point reporting events of the finger 120 are simultaneously distributed to the corresponding application controls; by adding a new handwriting collaborative module 211 in the hardware abstraction layer, the point reporting events of the stylus 110 and the point reporting events of the finger 120 are simultaneously reported to the next layer.

[0149] In addition, a new handwriting collaborative service 250 is added to manage the handwriting collaborative module 211 and the handwriting collaborative input distribution module 270. In this way, through the management and enabling of the application whitelist, applications that need to support collaborative operations of multiple input devices can simultaneously obtain point events from two different input devices, while other applications are not affected, further improving the user interaction experience. For example, for applications such as painting, note-taking, or photo editing, these applications are added to the whitelist so that these applications can obtain and respond to point events from two different input devices, supporting users to use collaborative operations with multiple input devices. For some other applications, these applications are not added to the whitelist, so that these applications can still only obtain point events from one input device in the original way.

[0150] In addition, a new module is added to the application framework layer and the hardware abstraction layer so that the software system 200 supports hand-pen collaborative operations, that is, a system-level hand-pen collaborative operation solution is provided, so that all applications that turn on the hand-pen collaborative service can implement hand-pen collaborative operations based on the system-level solution, while other applications that do not turn on the hand-pen collaborative service will not be affected.

[0151] It should be noted that, in addition to managing the opening and closing of the multi-input device input function of the application by adding the handwriting collaborative service 250, the electronic device 300 can also provide a setting interface to the user, so that the user can independently set the opening and closing of the multi-input device input function of the application through the setting interface. The setting interface can be a system setting interface, in which the user can uniformly set the opening and closing of the multi-input device input function of the system installed applications in batches; it can also be an application setting interface, that is, based on the setting interface in each application, the user can enter the setting interface of each application and set the opening and closing of the multi-input device input function of the application separately.

[0152] The second solution is to add a multi-input device collaborative distribution strategy to the input distribution module, and to add a multi-input device collaborative module to the hardware anti-mistouch algorithm module.

[0153] The input distribution module includes the original event distribution strategy and the newly added multi-input device collaborative distribution strategy.

[0154] According to the original event distribution strategy, the events that need to be reported and those that need to be discarded can be determined. The newly added multi-input device collaborative distribution strategy makes further judgments on the events that need to be discarded. If the events that need to be discarded meet the requirements, the events that need to be discarded will not be discarded, but will be distributed to the target window.

[0155] The multi-input device collaboration module can invalidate some anti-mistouch algorithms that affect the multi-input device collaborative operation scenarios, and can determine whether the point reporting event needs to be reported. If it is determined that the point reporting event needs to be reported, the point reporting events of multiple input devices will be reported to the next layer at the same time.

[0156] Optionally, a multi-input device collaboration service can be added to the application framework layer. The multi-input device collaboration service is used to sense the running state and life cycle of the application, and enable the multi-input device collaboration module and the newly added multi-input device collaboration distribution strategy according to the running state of the application.

[0157] For example, taking the scenario of stylus and finger collaborative operation as an example, at this time, the multi-input device collaborative service may refer to the stylus collaborative service, the multi-input device collaborative module may refer to the stylus collaborative module, and the multi-input device collaborative distribution strategy may refer to the stylus collaborative distribution strategy.

[0158] See also Figure 4B Another schematic block diagram of a software system 200 provided in an embodiment of the present application is shown. The software system 200 may include a driver layer, a hardware abstraction layer, an application framework layer, and an application program layer.

[0159] The driver layer includes drivers for the stylus and the touch screen. The hardware anti-mistouch algorithm module 210 of the hardware abstraction layer includes a stylus coordination module 211 .

[0160] The application framework layer includes an input management service 220 , a window management service 230 , a handwriting coordination service 250 , an activity management service 260 , and a handwriting coordination distribution strategy 280 . The input management service 220 includes an event monitoring module 223 , an input reading module 222 , and an input distribution module 221 .

[0161] The application layer may include a painting application 241. The painting application 241 may include a canvas control 242, other interactive controls 243, and a pen event processing module 244.

[0162] Compared to Figure 2 The software system 200 shown, Figure 4B The software system 200 newly adds: a handwriting coordination module 211 , a handwriting coordination service 250 , an activity management service 260 , a handwriting coordination distribution strategy 280 , and a handwriting event processing module 244 .

[0163] based on Figure 4B In the software system 200, the reporting process of the stylus point reporting event and the finger point reporting event can be as follows:

[0164] After the painting application 241 is started or installed, it registers with the handwriting coordination service 250 and starts the handwriting coordination. The handwriting coordination service 250 adds the painting application 241 to the application whitelist.

[0165] The handwriting coordination service 250 monitors the running state and life cycle of the application through the activity management service 260. When monitoring that the application in the application whitelist (ie, the painting application 241) switches to the foreground, the handwriting coordination service 250 enables the handwriting coordination distribution strategy 280 and the handwriting coordination module 211.

[0166] based on Figure 1 The scenes shown and Figure 4B In the software system 200 shown, when a user performs input operations on the touch screen of the tablet computer 100 using the stylus 110 and the finger 120 , the driver of the stylus 110 receives the input event of the stylus 110 , and the driver of the touch screen receives the input event of the finger 120 .

[0167] The driver layer transfers the input events of the stylus pen 110 and the input events of the finger 120 to the hardware abstraction layer.

[0168] After the handwriting coordination module 211 is turned on, the general anti-false touch algorithm in the hardware anti-false touch algorithm module 210 remains valid, while the anti-false touch algorithm affecting handwriting coordination reporting becomes invalid. In other words, the hardware anti-false touch algorithm module 210 only runs the general anti-false touch algorithm to perform anti-false touch algorithm judgment on the input events of the stylus and the input events of the finger, and does not run the anti-false touch algorithm affecting handwriting coordination reporting to perform anti-false touch algorithm judgment on the input events of the stylus and the input events of the finger.

[0169] The hardware anti-mistouch algorithm module 210 performs an anti-mistouch algorithm judgment on the input events of the stylus pen 110 and the input events of the finger 120 through a general anti-mistouch algorithm, and determines that the input events of the stylus pen 110 and the input events of the finger 120 are not mistouch reporting events, and then transmits the input events of the stylus pen 110 and the input events of the finger 120 to the stylus collaboration module 211.

[0170] The handwriting coordination module 211 further judges the input events of the stylus 110 and the finger 120 to determine whether the input events of the stylus 110 and the finger 120 are point reporting events that need to be reported in the handwriting coordination scenario. If it is recognized that the input events of the stylus 110 and the finger 120 are both point reporting events in the handwriting coordination scenario, the input events of the stylus 110 are written to the stylus device node in the / dev / input directory, and the input events of the finger 120 are written to the touch screen device node in the / dev / input directory.

[0171] When the event monitoring module 223 monitors that new input events are written into the stylus device node and the touch screen device node, it reads the input events of the stylus 110 from the stylus device node and reads the input events of the finger 120 from the touch screen device node.

[0172] The input reading module 222 obtains the input events of the stylus 110 and the input events of the finger 120 read by the event monitoring module 223, and translates, assembles and packages the obtained input events to obtain the point reporting events of the stylus 110 and the point reporting events of the finger 120, and transmits the point reporting events of the stylus 110 and the point reporting events of the finger 120 to the input distribution module 221.

[0173] After the input distribution module 221 obtains the point reporting events of the stylus pen 110 and the point reporting events of the finger 120, it determines the point reporting events that need to be discarded and the point reporting events that need to be reported from the point reporting events of the stylus pen 110 and the point reporting events of the finger 120 according to the original event distribution strategy. The point reporting event that needs to be discarded may be the point reporting event of the stylus pen 110 or the point reporting event of the finger 120, and correspondingly, the point reporting event that needs to be reported is the other of the two point reporting events.

[0174] After the input distribution module 221 determines the reporting events that need to be reported and the reporting events that need to be discarded, it does not immediately perform a discard operation on the reporting events that need to be discarded, but makes further judgments on the reporting events that need to be discarded according to the handwriting collaborative distribution strategy 280; if the handwriting collaborative distribution strategy 280 is in an enabled state, and the characteristics of the current target window meet the characteristics of the handwriting collaborative scene, the reporting events that need to be discarded are not discarded, but the target window corresponding to the reporting events that need to be discarded is determined based on the recorded window information of each application, and the reporting events that need to be discarded are distributed to the target window. In addition, the target window corresponding to the reporting events that need to be reported is determined based on the recorded window information of each application, and the reporting events that need to be reported are distributed to the target window.

[0175] After the target window obtains the reporting events of the stylus 110 and the finger 120, the root node object of the view control tree (such as the ViewRoot object) dispatches the reporting events of the stylus 110 and the finger 120 to the corresponding controls along the control tree. The controls respond to the received reporting events and perform corresponding business processing operations.

[0176] like Figure 4B As shown, the painting application 241 receives the reported point reporting events of the stylus pen 110 and the finger 120. The hand event processing module 244 processes the point reporting events of the stylus pen 110 and the finger 120 and performs corresponding service processing operations.

[0177] In this way, when the painting application 241 is in the foreground running state, when the user uses the stylus 110 and the finger 120 to perform hand-pen collaborative operations on the touch screen of the tablet computer 100 at the same time, Figure 4B The software system 200 can report the point reporting event of the stylus pen 110 and the point reporting event of the finger 120 to the painting application 241 at the same time, so that the painting application 241 can respond to the point reporting event of the stylus pen 110 and the point reporting event of the finger 120 at the same time and perform corresponding business processing operations.

[0178] As can be seen from the above, by adding a new handwriting collaborative distribution strategy 280 to the input distribution module, the point reporting events of the stylus 110 and the point reporting events of the finger 120 are distributed to the corresponding application controls at the same time; by adding a new handwriting collaborative module 211 to the hardware abstraction layer, the point reporting events of the stylus 110 and the point reporting events of the finger 120 are reported to the next layer at the same time.

[0179] In addition, a new handwriting coordination service 250 is added to manage the handwriting coordination module 211 and the handwriting coordination input distribution module 270. In this way, through the management and enabling of the application whitelist, applications that need to support the coordinated operation of multiple input devices can simultaneously obtain the reporting events of two different input devices, while other applications are not affected, further improving the user interaction experience.

[0180] In addition, a new handwriting collaborative distribution strategy 280 is added to the input distribution module, and a new handwriting collaborative module 211 is added to the hardware abstraction layer, so that the software system 200 supports handwriting collaborative operations, that is, a system-level handwriting collaborative operation solution is provided, so that applications that open the handwriting collaborative service can implement handwriting collaborative operations based on the system-level solution, while other applications that do not open the handwriting collaborative service will not be affected.

[0181] It should be noted that in the related art, since the electronic device does not support simultaneous reporting of multiple different input devices, the reporting event of one input device will not appear between the time period of the pressing reporting event and the lifting reporting event of another input device. However, through the first solution and the second solution provided in the embodiment of the present application, the electronic device can support simultaneous reporting of multiple different input devices, so the reporting event of one input device can appear between the time period of the pressing reporting event and the lifting reporting event of another input device.

[0182] For example, in the scenario of collaborative operation of a stylus pen and a finger, the user first uses the stylus pen to perform input operations on the touch screen, and in the process of using the stylus pen to perform input operations, the user then uses the finger to perform input operations on the touch screen. In this process, the stylus pen contacts the touch screen first, and then the finger contacts the touch screen.

[0183] Before the finger is not in contact with the touch screen and the stylus is not lifted, the application can obtain the stylus's touch down event and a series of touch move events. When the finger is in contact with the touch screen, since the finger and the stylus are in contact with the touch screen at the same time, the stylus's touch move event will still be generated, and the finger's touch down event will also be generated.

[0184] based on Figure 2In the software system shown, when a touch move event of the stylus pen and a touchdown event of the finger are generated, when passing through the hardware anti-false touch algorithm module 210, since the anti-false touch algorithm affecting the hand-pen collaborative operation scenario has not failed, one of the touch move event of the stylus pen and the touch down event of the finger may be determined as a false touch event, so the touch move event of the stylus pen or the touch down event of the finger will be discarded, so the touch down event of the finger will not appear in the time period between the touch down event and the touch up event of the stylus pen.

[0185] In addition, if the touch move reporting event of the stylus pen and the touch down reporting event of the finger are not discarded after passing through the hardware false touch prevention algorithm module 210, or a certain software system does not include the hardware false touch prevention algorithm module 210, Figure 2 The input distribution module 221 can obtain the touch move reporting event of the stylus pen and the touch down reporting event of the finger. According to the original event distribution strategy, since the input device currently reporting is the stylus pen, the input distribution module 221 will discard the touch down reporting event of the finger and report the touch move reporting event of the stylus pen to the application. Alternatively, the current stylus reporting is interrupted, the touch move reporting event of the stylus pen is discarded, and the touchdown reporting event of the finger is reported to the control of the application. In this way, the touch down reporting event of the finger will not appear in the time period between the touch down reporting event and the touch up reporting event of the stylus pen.

[0186] Based on Figure 4A and Figure 4B In the software system shown in the figure, when passing through the hardware anti-mistouch algorithm module 210, the anti-mistouch algorithm that affects the hand-pen collaborative operation scene fails, so the touch move reporting event of the stylus pen and the touchdown reporting event of the finger will not be identified as mistouch reporting events, and will be reported to the next layer. When the touch move reporting event of the stylus pen and the touch down reporting event of the finger are in the event distribution stage, Figure 4A and Figure 4BThe software system can report the touch move event of the stylus pen and the touch down event of the finger to the application program at the same time. Therefore, the touch down event of the finger can appear in the time period between the touch down event of the stylus pen and the touch up event of the stylus pen. In addition, the input time of the first event triggered by the stylus pen input operation (i.e., the touch down event of the stylus pen) is earlier than the input time of the first event triggered by the finger input operation (i.e., the touch down event of the finger).

[0187] Through the first and second schemes mentioned above, the electronic device 300 can report the reporting events of multiple different input devices to the application at the same time, so that the application can obtain the reporting events of multiple different input devices at the same time, and then the user can perform collaborative operations through multiple different input devices at the same time.

[0188] For example, taking the scenario of stylus and finger collaborative operation as an example, see Figure 5A The schematic diagram of the hand-pen collaborative operation in the note editing process provided by the example of the present application is shown, and the tablet computer 100 displays the note editing interface 101 on the touch screen. In the initial state, an ellipse 102 is displayed on the note editing interface 101, and the leftmost point of the ellipse 102 is at the position point 105, and the rightmost point is at the position point 104. At this time, if the user wants to adjust the shape of the ellipse 102, he presses the position point 104 with the finger 120, and slides from the position point 105 to the position point 106 with the stylus 110.

[0189] In response to the input operation of the finger 120 continuously pressing the position point 104, the tablet computer 100 obtains the finger 120 reporting event (including the touch don event). In addition, in the process of the finger 120 continuously pressing the position point 104, in addition to obtaining the finger 120 reporting event, the tablet computer 100 also responds to the input operation of the stylus 110 sliding from the position point 105 to the position point 106, and obtains a series of stylus 110 reporting events (including the touch down event and the touch move event).

[0190] Tablet 100 based Figure 4A and Figure 4B The software system 200 reports the finger 120 and the stylus 110 point reporting events to the note application at the same time. The note application displays an ellipse 103 on the note editing interface 101 in response to the finger 120 and the stylus 110 point reporting events. The leftmost point of the ellipse 103 is located at the position point 106, and the rightmost point is located at the position point 104.

[0191] That is, the user uses the finger 120 and the stylus 110 to adjust the ellipse 102 to the ellipse 103 (ie, adjusts the shape of the ellipse).

[0192] exist Figure 5A In the present invention, the tablet computer 100 supports simultaneous reporting of points by multiple input devices, so that the user can use the finger 120 and the stylus 110 to adjust the shape during the note editing process, which makes the operation easier, more efficient, and provides a better user interactive operation experience.

[0193] See also Figure 5B The schematic diagram of the hand-pen coordinated operation in the painting process provided by the embodiment of the present application is shown, and the tablet computer 100 displays a painting interface 107 on the touch screen. The painting interface 107 displays a line thickness adjustment control 108 and a painting line 109. The line thickness adjustment control 108 is used to adjust the thickness of the painting line 109. Figure 5B As shown, when the sliding button on the line thickness adjustment control 108 slides upward, the drawing line 109 output by the stylus 110 becomes thicker; when the sliding button on the line thickness adjustment control 108 slides downward, the drawing line 109 output by the stylus 110 becomes thinner.

[0194] At this time, the user wants to draw a gradually thicker line, so he uses the finger 120 to press the sliding button on the line thickness adjustment control 108, and presses the touch screen of the tablet computer 100 with the stylus 110; while the finger 120 slides the sliding button on the line thickness adjustment control 108 upward, the stylus 110 performs a sliding operation on the touch screen of the tablet computer 100, so that as the finger 120 slides and the stylus 110 slides on the touch screen, the drawing interface 107 will display the drawing line 109. The drawing line 109 is a gradually thicker line.

[0195] In short, during the drawing process, while the stylus 110 is drawing a line, the user can use the finger 120 to adjust the thickness of the line.

[0196] In this process, the tablet computer 100 can simultaneously receive the sliding operation of the finger 120 on the line thickness adjustment control 108 and the sliding operation of the stylus 110 on the touch screen; in response to the input operation of the finger 120, obtain the point reporting event of the finger 120; in response to the input operation of the stylus 110, obtain the input operation of the stylus 110; and report the point reporting event of the finger 120 and the point reporting event of the stylus 110 to the painting application. The painting application simultaneously responds to the point reporting event of the finger 120 and the point reporting event of the stylus 110, and displays a gradually thickening line on the painting interface 107.

[0197] exist Figure 5BIn the drawing, the tablet computer 100 supports simultaneous reporting of points by multiple input devices, so that the user can use the finger 120 to adjust the thickness of the line drawn by the stylus 110 during the drawing process, which makes the operation easier, more efficient, and provides a better user interactive operation experience.

[0198] See also Figure 5C The schematic diagram of the hand-pen collaborative operation in the picture processing process provided by the embodiment of the present application is shown, and the tablet computer 100 displays a picture editing interface 111 on the touch screen. An object 112 is displayed on the picture editing interface 111.

[0199] At this time, the user wants to zoom the object 112 , so the user uses the stylus 110 to select the object 112 , and after the object 112 is selected, the stylus 110 continues to contact the touch screen, and the user uses the finger 120 to make a zoom gesture on the touch screen.

[0200] In this process, the tablet computer 100 can simultaneously receive the zoom gesture input of the finger 120 and the operation of the stylus 110 on the touch screen; in response to the input operation of the finger 120, obtain the finger 120 reporting event; in response to the input operation of the stylus 110, obtain the stylus 110 input operation; report the finger 120 reporting event and the stylus 110 reporting event to the image processing program. The image processing program simultaneously responds to the finger 120 reporting event and the stylus 110 reporting event, and displays the reduced object 112 on the image editing interface 111.

[0201] That is to say, when processing an image, the user can use his fingers to make corresponding gestures to cooperate with the stylus to translate and zoom the canvas, making it easier to process the details of a specified position in the image.

[0202] exist Figure 5C In the present invention, the tablet computer 100 supports simultaneous reporting of multiple input devices, so that the user can use the finger 120 to make corresponding gestures during the image processing to cooperate with the stylus to translate and zoom the canvas, which makes the operation easier, more efficient, and provides a better user interactive operation experience.

[0203] See also Figure 5D The schematic diagram of simultaneous pen operation in a split-screen scenario provided by an embodiment of the present application is shown, and the touch screen of the tablet computer 100 displays an interface 113 and an interface 114 in a split-screen manner. The interface 113 may be a news interface or an e-book interface, and the interface 114 is a note editing interface, that is, the user uses the interface 113 to browse news or read e-books, and uses the interface 114 to take notes. The interface 113 and the interface 114 are interfaces of two different applications.

[0204] At this time, the user wants to turn the page up or slide up on the interface 113 and take notes on the interface 114 at the same time, so the user uses the stylus 110 to continuously perform input operations on the interface 114 to edit the notes. While the user continues to edit the notes using the stylus 110, the user uses the finger 120 to turn the page up on the interface 114.

[0205] In this process, before the finger 120 touches the touch screen, the tablet computer 100 can continuously obtain the reporting event of the stylus 110. At a certain moment or a period of time, the finger 120 and the stylus 110 touch the touch screen at the same time and perform input operations. At this time, the tablet computer 100 can report the reporting event triggered by the input operation of the stylus 110 to the application corresponding to the interface 114, and report the reporting event triggered by the input operation of the finger 120 to the application corresponding to the interface 113.

[0206] In a specific application, the tablet computer 100 can determine the application to which the point reporting event belongs based on the position point coordinates of the point reporting event. For example, when the position point information of a point reporting event indicates that the input position point is located in the display area of ​​the interface 113, the point reporting event is distributed to the application of the interface 113; similarly, when the position point information of a point reporting event indicates that the input position point is located in the display area of ​​the interface 114, the point reporting event is distributed to the application of the interface 114.

[0207] That is, the tablet computer 100 can support simultaneous reporting of points by the stylus pen 110 and the finger 120 , and can report the reporting events of the stylus pen 110 and the finger 120 to two different application programs respectively.

[0208] The hardware abstraction layer's anti-false touch algorithm processing process can be found in Figure 4A and Figure 4B The relevant introduction will not be repeated here.

[0209] After the application corresponding to the interface 113 obtains the finger 120's reporting event, it responds to the reporting event and performs a page turning or page sliding operation. After the application corresponding to the interface 114 obtains the stylus 110's reporting event, it responds to the reporting event and displays the corresponding content on the interface 114.

[0210] In this way, in a split-screen scenario, users can use their fingers and stylus simultaneously to perform input operations on the interfaces of two applications respectively, so as to further improve the input interaction operation experience in the split-screen scenario.

[0211] See also Figure 5EIn the schematic diagram of simultaneous pen operation in the small window scenario provided by the embodiment of the present application, a system interface 115 and a small window interface 116 are displayed on the touch screen of the tablet computer 100. The small window interface 116 is superimposed and displayed on the system interface 115. The small window interface 116 can be a projection interface, such as a mobile phone projection interface in a multi-screen collaboration scenario, or it can be an interface of an application installed on the tablet computer 100.

[0212] At this time, the user uses the stylus 110 to perform input operations (such as writing or drawing, etc.) on the small window interface 116, and uses the finger 120 to control the system interface to slide to the right.

[0213] In this process, the tablet computer 100 can simultaneously receive the sliding operation input of the finger 120 and the input of the stylus 110 on the small window interface 116; in response to the input operation of the finger 120, obtain the point reporting event of the finger 120; in response to the input operation of the stylus 110, obtain the input operation of the stylus 110; report the point reporting event of the stylus 110 to the application corresponding to the small window interface 116, and report the point reporting event of the finger 120 to the system view control. In response to the point reporting event of the finger 120, the system view control slides the system interface to the right; in response to the point reporting event of the stylus 110, the small window interface 116 displays the corresponding content on the interface 116.

[0214] In this way, in the small window scenario, users can input with their fingers and stylus at the same time, which makes the operation easier, more efficient and provides a better user interactive experience.

[0215] It should be pointed out that the above article takes the stylus and finger collaborative operation scenario as an example to introduce the first and second solutions in detail. The software systems in other multi-input device operation scenarios are similar. Figure 4A and Figure 4B For example, in the scenario where the stylus pen and mouse work together, Figure 4A and Figure 4B The driver layer in may include a mouse driver.

[0216] In order to better introduce the technical solution provided by the embodiment of the present application, an introduction and explanation is given below in conjunction with a flow chart.

[0217] See also Figure 6 A schematic flow chart of an interaction method of an electronic device provided in an embodiment of the present application is shown, and the method may include the following steps:

[0218] Step S601: The electronic device 300 displays a target window.

[0219] The target window may be an interface of an application program or an interface of a software system of the electronic device 300. Further, the target window may include interfaces of two application programs, for example Figure 5D The split-screen scene shown can also include the application interface and system interface, such as Figure 5E The small window scene shown.

[0220] The target window may be a focus window or may not be a focus window. Typically, the target window is the interface of an application running in the foreground. The target window may include controls of one or more applications, such as canvas controls, click controls, and button controls.

[0221] Step S602: The electronic device 300 receives a first input operation of a first input method and a second input operation of a second input method at the same time.

[0222] It is understandable that the electronic device 300 displays the target window on the screen. The user can use the first input device and the second input device to perform input operations on the screen. The first input method may refer to a method of inputting with the first input device, and the second input method may refer to a method of inputting with the second input device. In some places of this document, the first input method may be equivalent to the first input device, and the second input method may be equivalent to the second input device.

[0223] Exemplarily, the first input method may be finger input, stylus input, mouse input, air gesture input, or voice input. Similarly, the second input method may also be finger input, stylus input, mouse input, air gesture input, or voice input. Finger input may refer to a method of inputting on the screen using a finger as an input device; stylus input may refer to a method of inputting on the screen using a stylus as an input device.

[0224] Typically, the first input method and the second input method are different, that is, the two input methods are methods of inputting using different input devices respectively.

[0225] The first input device and the second input device are two different input devices, that is, the electronic device 300 can receive input operations from the two different input devices at the same time.

[0226] For example, the first input device may be a finger, or a touch screen acted on by a finger, and the second input device may be a stylus. In this case, the user uses the stylus and the finger to perform input operations on the touch screen of the electronic device 300 at the same time, that is, the stylus and the finger are in contact with the touch screen at the same time, and the user uses the finger to perform a first touch operation (i.e., a first input operation) on the touch screen, and uses the stylus to perform a second touch operation (i.e., a second input operation) on the touch screen. Figure 5B As shown, the first touch operation may be a sliding operation of the finger 120 on the line thickness adjustment control 108, and the second touch operation may be a line drawing operation of the stylus 110. Figure 5A As shown, the first touch operation may be a pressing operation of the finger 120 at the position point 104 , and the second touch operation may be an operation of sliding the stylus pen 110 from the position point 105 to the position point 106 .

[0227] It should be noted that the stylus and the finger are in contact with the touch screen at the same time means that at a certain point in time or within a time period, the stylus and the touch screen are in contact, and the finger is also in contact with the touch screen. However, it is not necessarily required that the stylus and the finger generate touch down events at the same time, that is, it is not necessarily required that the stylus and the finger perform the pressing operation at the same time. The stylus can be in contact with the touch screen first, and then the finger, or the finger can be in contact with the touch screen first, and then the stylus can be in contact with the touch screen, and at a certain point in time or within a time period, the stylus and the finger are in contact with the touch screen at the same time. For example, Figure 5A As shown, the finger 120 first presses the position point 104, and then the stylus 110 slides from the position point 105 to the position point 106, that is, the finger 120 first contacts the touch screen, and then the stylus 110 contacts the touch screen, and because the finger 120 continues to press the position point 104, when the stylus 110 slides from the position point 105 to the position point 106, the stylus 110 and the finger 120 contact the touch screen at the same time.

[0228] Of course, in some cases, the stylus and the finger may also generate a touch down event at the same time, that is, the finger and the stylus perform a pressing operation at the same time.

[0229] For example, the first input device may be a finger or a stylus, and the second input device may be a mouse. In this case, the first input operation may be a touch operation of a finger or a stylus on the touch screen, and the second input operation may be a click operation of a mouse on the touch screen.

[0230] Step S603: The electronic device 300 sends a first point reporting event and a second point reporting event to the control corresponding to the target window, wherein the first point reporting event is a point reporting event triggered by a first input operation, and the second point reporting event is a point reporting event triggered by a second input operation.

[0231] The first input operation may trigger one or more first point reporting events. Similarly, the second input operation may also trigger one or more second point reporting events.

[0232] For example, Figure 5BAs shown, the first input operation may be a sliding operation of the finger 120 on the line thickness adjustment control 108, and the second input operation may be a line drawing operation of the stylus 110. At this time, when the finger 120 slides on the line thickness adjustment control 108 (i.e., the first input operation), the following first reporting events may be triggered: a pressing reporting event (touch down) of the finger 120, one or more moving reporting events (touch move) of the finger 120, and a lifting reporting event (touch up) of the finger 120.

[0233] When the stylus pen 110 draws a line on the touch screen (i.e., the second input operation), the following second reporting events may be triggered: a touch down reporting event of the stylus pen 110 , one or more move reporting events of the stylus pen 110 , and a touch up reporting event of the stylus pen 110 .

[0234] The electronic device 300 can be operated as follows: Figure 4A and Figure 4B In the process shown, the reporting event of the first input device and the reporting event of the second input device are reported to the control of the application. Figure 4A and Figure 4B In the example, after the stylus pen reporting event and the finger reporting event are distributed to the target window, the target window can dispatch the stylus pen reporting event and the finger reporting event to the canvas control 242 of the painting application 241 along the control tree.

[0235] Each first reporting point event and each second reporting point event can be similar to Figure 4A and Figure 4B The event reporting process is reported so that the application can obtain the first reporting event and the second reporting event at the same time. The process of reporting the first reporting event and the second reporting event at the same time can be found in the above Figure 4A and Figure 4B The relevant content will not be repeated here.

[0236] It should be noted that the "simultaneous" in "simultaneously reporting the reporting events of multiple input devices" and "the application simultaneously obtains the reporting events of multiple input devices" in the embodiments of the present application does not mean that the reporting time points of the two reporting events or the time points when the application obtains the reporting events are consistent. Usually, due to the limitations of the event reporting mechanism, multiple reporting events are reported in a certain order.

[0237] The "simultaneous" in "simultaneously reporting the reporting events of multiple input devices" and "the application simultaneously obtains the reporting events of multiple input devices" in the embodiments of the present application can be understood as: the reporting event of one input device can appear between the time period of the pressing reporting event and the lifting reporting event of another input device. That is, when the reporting event of one input device is reported, the reporting event of another input device will not be discarded, so that the reporting events of two different input devices can be reported to the application "simultaneously (with a certain order)".

[0238] The "inability to realize simultaneous reporting of two different input devices" mentioned in the related art can be understood as: the reporting event of one input device will not appear in the time period between the pressing reporting event and the lifting reporting event of another input device. In other words, when the reporting event of one input device is reported, the reporting event of the other input device will be discarded.

[0239] Step S604: The control corresponding to the target window of the electronic device 300 displays corresponding content in the target window according to the first point reporting event and the second point reporting event.

[0240] After the application of the electronic device 300 obtains the first and second reporting events, it performs corresponding business processing operations according to the first and second reporting events to change the visual output on the touch screen, that is, to change the display content.

[0241] For example, Figure 5A As shown, in the initial state, the touch screen displays an ellipse 102. In response to the coordinated operation of the finger 120 and the stylus 110, the tablet computer 100 displays an ellipse 103 on the touch screen. The ellipse 103 is different from the ellipse 102, that is, after the coordinated operation of the finger and the stylus 110, the display content of the tablet computer 100 has changed.

[0242] As can be seen from the above, when the electronic device 300 receives input operations from two different input devices at the same time, it reports the point events triggered by the input operations of the two different input devices to the control of the application at the same time, so that the application can obtain the point events of the two different input devices at the same time, and then can respond to the point events of the two different input devices at the same time, execute corresponding business operations to refresh the window display content, support users to perform multi-input device collaborative operations through two different input devices, improve operation efficiency, and thus improve the user's interactive operation experience.

[0243] In a possible implementation, if the target window is the interface of the first application, the electronic device 300 displays the interface of the first application; when the user simultaneously performs input operations on the screen of the electronic device 300 through the first input device (i.e., the first input method) and the second input device (i.e., the second input method), a first reporting event triggered by the first input operation and a second reporting event triggered by the second input erasing are obtained; by Figure 4A and Figure 4B In the event reporting process shown, the first point reporting event and the second point reporting event are reported to the control of the first application; the control of the first application responds to the first point reporting event and the second point reporting event and displays corresponding content in the target window. In this implementation, the first point reporting event and the second point reporting event are both reported to the same application.

[0244] Correspondingly, the control of the first application can display corresponding content in the target window according to the first point reporting event and the second point reporting event.

[0245] Furthermore, during the event reporting process, the electronic device 300 can directly report the first point reporting event to the control of the first application. As for the second point reporting event, it is necessary to first determine whether the preset conditions are met. If the preset conditions are met, the second point reporting event is reported to the control of the first application; if the preset conditions are not met, the second point reporting event may not be reported.

[0246] Normally, the first reporting event that is directly reported can be a reporting event of an existing input device, while the second reporting event that needs to be judged whether it meets the preset conditions before being reported is a reporting event of a new input device. For example, the user first uses a stylus (i.e., the first input device) to perform input operations, and then uses a finger (i.e., the second input device) to perform input operations during the process of using the stylus input operation. At this time, the stylus is an existing input device, and when the finger is not in contact with the touch screen, the reporting event of the stylus is continuously reported. When the finger is in contact with the touch screen, the electronic device 300 obtains the reporting event of the stylus and the reporting event of the finger, and directly reports the reporting event of the stylus. As for the reporting event of the finger, it is necessary to determine whether it meets the preset conditions before reporting.

[0247] In this case, the input time of the first first reporting event (i.e., down event) triggered by the first input operation is earlier than the input time of the first second reporting event (i.e., down event) triggered by the second input operation. Alternatively, the input time of the first second reporting event is between the input time of the first first reporting event and the input time of the last second reporting event (i.e., up event or cancel event).

[0248] In a possible implementation, if the target window includes the interface of the second application and the interface of the third application, that is, includes the interfaces of two applications. At this time, the electronic device 300 displays the interface of the second application and the interface of the third application on the screen (for example, Figure 5D If two input devices simultaneously perform input operations on an application's interface, the reporting events of the two input devices are reported to the application corresponding to the interface; if the two input devices respectively perform input operations on the interfaces of two applications, that is, the first input device performs input operations on the interface of the second application, and the second input device performs input operations on the interface of the third application, the first reporting event triggered by the first input operation is reported to the control of the second application, and the second reporting event triggered by the second input operation is reported to the control of the third application.

[0249] Correspondingly, the control of the second application displays corresponding content on the interface of the second application according to the first reporting event; the control of the third application displays corresponding content on the interface of the third application according to the second reporting event.

[0250] Furthermore, during the event reporting process, the electronic device 300 can directly report the first point reporting event to the control of the second application. As for the second point reporting event, it is necessary to first determine whether the preset conditions are met. If the preset conditions are met, the second point reporting event is reported to the control of the third application; if the preset conditions are not met, the second point reporting event may not be reported.

[0251] Similarly, the input time of the first first reporting event (ie, down event) triggered by the first input operation must be earlier than the input time of the first second reporting event (ie, down event) triggered by the second input operation.

[0252] In a possible implementation, the target window may also be a system interface, or include a system interface and an application program interface (eg Figure 5E small window scene).

[0253] If the target window is a system interface, the system can be regarded as an "application" at this time, and the first point reporting event and the second point reporting event are reported to the corresponding control of the system at the same time, and the control responds to the first point reporting event and the second point reporting event.

[0254] If the target window includes a system interface and an application interface, and one input device performs input operations in the system interface and another input device performs input operations in the application interface, the reporting events triggered by the two input devices can be reported to the system interface and the application interface respectively.

[0255] In a possible implementation, the electronic device 300 may also receive a fourth input operation while receiving the third input operation; at this time, the third reporting event is sent to the control corresponding to the target window, and the fourth reporting event is not sent to the control corresponding to the target window. The third reporting event is a reporting event triggered by the third input operation, and the fourth reporting event is a reporting event triggered by the fourth input operation.

[0256] It can be understood that the third input operation and the fourth input operation can be input by the first input device or the second input device, or can be input by other input devices.

[0257] Furthermore, the electronic device 300 may directly report the third point reporting event, and not report the fourth point reporting event after determining that the non-preset condition is met.

[0258] Similar to the process of reporting the first and second point reporting events to the controls corresponding to the target window, the relevant contents of the event reporting process of the third and fourth point reporting events can be found above and will not be repeated here.

[0259] In a possible implementation, the preset condition may include that the application enables the multi-input mode input function. For example, in the hand-pen collaboration scenario, after the hand-pen collaboration is enabled, it is considered that the multi-input device (or multi-input mode) input function is enabled. The system can also be regarded as an "application", so if it is for the system interface, it can be considered that the system has enabled the multi-input device input function.

[0260] In a possible implementation, the preset condition may include that the second reporting event is not an accidental reporting event. Figure 4A and Figure 4B In the embodiment, the anti-mistouch algorithm of the hardware abstraction layer is used to determine whether the input event is a mistouch reporting event. If it is not a mistouch reporting event, the reporting event is not discarded, but the reporting events of the two input devices are reported to the next layer. Furthermore, the preset condition may also include that the second reporting event is not a mistouch reporting event, and the second reporting event is an event in a multi-input device collaborative operation scenario, such as a reporting event in a hand-pen collaborative operation scenario.

[0261] In a possible implementation, the preset condition may include that the characteristics of the target window meet the characteristics of the multi-input device collaborative operation scenario. Exemplarily, when the target window includes a target control, it is considered that the characteristics of the target window meet the characteristics of the multi-input device collaborative operation scenario, otherwise it does not meet.

[0262] In a possible implementation, an input time of a first first point reporting event triggered by a first input operation is earlier than an input time of a first second point reporting event triggered by a second input operation.

[0263] In a possible implementation, the first input operation is an operation acting on any area of ​​the screen of the electronic device 300, and the second input operation is an operation acting on any area of ​​the screen of the electronic device 300. That is, the full screen of the electronic device can support simultaneous input of two input devices, and report the reporting events of the two input devices.

[0264] In a possible implementation, the electronic device 300 may also receive a setting operation of the user; in response to the setting operation of the user, the multi-input device input function of the application or system is enabled. That is, the user can autonomously enable and disable the multi-input device input function as needed. For example, the tablet computer 100 may provide a setting interface for the user to autonomously enable and disable the hand-pen collaborative input of the installed application or system.

[0265] In some possible implementations, the first input method and the second input method include any one or any two of finger input, stylus input, mouse input, air gesture input, and voice input.

[0266] In a possible implementation, the first input device may be a finger, that is, the first input mode is finger input, and the second input device may be a stylus, that is, the second input mode is stylus input. In this case, the first input operation may be a touch operation of a finger on the touch screen, and the second input operation may be a touch operation of the stylus on the touch screen. The finger and the stylus are in contact with the touch screen at the same time at a certain time point or time period.

[0267] In a possible implementation, the electronic device 300 may add an input distribution module, and report the second point reporting event through the newly added input distribution module. At this time, in the event reporting process, the electronic device 300 first obtains the first input event triggered by the first input operation, and obtains the second input event triggered by the second input operation; converts the first input event into the first point reporting event, and converts the second input event into the second point reporting event; the input reading module transmits the first point reporting event and the second point reporting event to the first input distribution module and the second input distribution module; the first input distribution module determines that the first point reporting event is a first type of event and the second point reporting event is a second type of event, then distributes the first point reporting event to the control corresponding to the target window, and discards the second point reporting event.

[0268] The second input distribution module determines that the second point reporting event is a second type of event, and determines that the multi-input device input function is in an enabled state, and the characteristics of the target window meet the characteristics of the multi-input device collaborative operation scenario, and then distributes the second point reporting event to the control corresponding to the target window.

[0269] The first type of event may refer to an event that is directly reported, and the second type of event refers to an event that needs to be determined whether it meets the preset conditions before determining whether to report. Usually, the first type of event is an event that needs to be reported according to the original event distribution strategy, and the second type of event is an event that needs to be discarded according to the original event distribution strategy.

[0270] In a possible implementation, the electronic device 300 may also modify the event distribution strategy of the original input distribution module to report the second point reporting event and the first point reporting event at the same time. At this time, in the event reporting process, the electronic device 300 first obtains the first input event triggered by the first input operation, and obtains the second input event triggered by the second input operation; converts the first input event into the first point reporting event, and converts the second input event into the second point reporting event; the input reading module transmits the first point reporting event and the second point reporting event to the third input distribution module; the third input distribution module determines that the first point reporting event is a first type of event, the second point reporting event is a second type of event, and determines that the multi-input device input function is in an enabled state, and the characteristics of the target window meet the characteristics of the multi-input device collaborative operation scenario, and then distributes the first point reporting event and the second point reporting event to the control corresponding to the target window.

[0271] In one possible implementation, when the electronic device 300 converts an input event into a point reporting event, if the electronic device 300 does not include an anti-mistouch algorithm module of the hardware abstraction layer, then after obtaining the first input event and the second input event, the first input event is written into the device node of the first input device, and the second input event is written into the device node of the second input device; after the input reading module reads the input event from the device node, the input event is converted into a point reporting event.

[0272] If the electronic device 300 includes an anti-false touch algorithm module of the hardware abstraction layer, when the multi-input device collaboration module of the electronic device 300 is in an enabled state, the preset anti-false touch algorithm in the anti-false touch algorithm set is disabled, and the remaining anti-false touch algorithms are used to perform anti-false touch processing on the first input event and the second input event. The remaining anti-false touch algorithms are the anti-false touch algorithms in the anti-false touch algorithm set except the preset anti-false touch algorithm.

[0273] If neither the first input event nor the second input event is an accidental touch reporting event, and the multi-input device collaborative module determines that the first input event and the second input event are both reporting events in the multi-input device collaborative operation scenario, then the first input event is written into the first device node corresponding to the first input device, and the second input event is written into the second device node corresponding to the second input device; the input reading module reads the first input event from the first device node through the event monitoring module, and reads the second input event from the second device node, and encapsulates the first input event into a first reporting event, and encapsulates the second input event into a second reporting event.

[0274] In a possible implementation, a multi-input device coordination service can be added to the application framework to manage the applications and enablement of multi-input device coordination. By managing and enabling the application whitelist, applications that need to enable multi-input device coordination can simultaneously obtain the reporting events of two different input devices, while other applications are not affected.

[0275] At this time, the electronic device 300 receives a registration request for the application, and in response to the registration request, adds the application to the application whitelist maintained by the multi-input device collaborative operation service; if it senses that the application in the application whitelist is switched to the foreground, the multi-input device collaborative operation service enables the second input distribution module and the multi-input device collaborative module, or enables the third input distribution module and the multi-input device collaborative module.

[0276] See also Figure 7 Another schematic block diagram of the process of an interaction method of an electronic device provided in an embodiment of the present application is shown, and the method may include the following steps:

[0277] Step S701: The electronic device 300 receives a first input operation from a first input device and, at the same time, receives a second input operation from a second input device, wherein the first input device and the second input device are two different input devices.

[0278] Step S702: The electronic device 300 reports a first point reporting event and a second point reporting event to the target window of the application running in the foreground in response to the first input operation and the second input operation, wherein the first point reporting event is a point reporting event triggered by the first input operation, and the second point reporting event is a point reporting event triggered by the second input operation.

[0279] Step S703: The electronic device 300 changes the display content of the target window in response to the first point reporting event and the second point reporting event.

[0280] In some possible implementations, the simultaneous reporting process of the electronic device 300 may be as follows:

[0281] The electronic device 300 generates a first input event of the first input operation in response to the first input operation, and generates a second input event of the second input operation in response to the second input operation. The first input event and the second input event may be original input events. The upper layer cannot recognize the original input event, and needs to convert the original input event into an advanced input event recognizable by the upper layer, so the electronic device 300 may convert the first input event into a first point reporting event and convert the second input event into a second point reporting event. The first point reporting event and the second point reporting event are advanced input events recognizable by the upper layer.

[0282] If the electronic device 300 determines the reporting event and the discarding event from the first reporting event and the second reporting event according to the first event distribution strategy (the original event distribution strategy), and determines that the discarding event complies with the second event distribution strategy, the first reporting event and the second reporting event are distributed to the target window. The reporting event is the first reporting event, and the discarding event is the second reporting event; or, the reporting event is the second reporting event, and the discarding event is the first reporting event.

[0283] The first event distribution strategy may refer to the original event distribution strategy of the input distribution module. For example, the original event distribution strategy means: if there is currently an input device A reporting a point, when the reporting events of input device A and input device B are obtained, the reporting event of another input device (i.e., input device B) is determined as a discarded event, and a discard operation is performed, and the reporting event of input device A is determined to be a reporting event, and the reporting event of input device A continues to be distributed.

[0284] The second event distribution strategy is an event distribution strategy in a multi-input device collaborative operation scenario. Exemplarily, the second event distribution strategy may mean that the multi-input device collaborative operation state is in an enabled state, and the characteristics of the target window (i.e., the focus window) meet the characteristics of the multi-input device collaborative operation scenario.

[0285] For example, Figure 4A As shown, the input distribution module 221 distributes events according to the first event distribution strategy. The handwriting collaborative input distribution module 270 distributes events according to the second event distribution strategy. After the handwriting collaborative input distribution module 270 determines the discarded event according to the first event distribution strategy (i.e., the original event distribution strategy), it determines whether the discarded event complies with the second event distribution strategy. If the current handwriting collaborative input distribution module 270 is in an enabled state and the characteristics of the current focus window meet the handwriting collaborative scenario, it is determined that the discarded event complies with the second event distribution strategy, and the determined discarded event is not discarded.

[0286] For example, Figure 4BAs shown, the input distribution module 221 determines the discarded event and the reported event according to the first event distribution strategy. Further, it is determined whether the discarded event complies with the second event distribution strategy. If the handwriting collaborative distribution strategy 280 is in an enabled state and the characteristics of the current focus window comply with the handwriting collaborative scenario, it is determined that the discarded event complies with the second event distribution strategy.

[0287] In this way, after determining the reporting event and discarding event according to the first event distribution strategy, in addition to distributing the reporting event to the target window normally, the determined discarding event is not discarded immediately, but it is further determined whether the discarding event complies with the event distribution strategy (i.e., the second event distribution strategy) in the multi-input device collaborative operation scenario; if it is determined that the discarding event is an event that needs to be reported in the multi-input device collaborative operation scenario, the event is not discarded, but the discarding event is also distributed to the target window, so that the target window can obtain the reporting events of two different input devices at the same time, thereby realizing the simultaneous reporting of the reporting events of two different input devices to the target window.

[0288] Furthermore, in some embodiments, an input distribution module may be added to the application framework layer, and the input distribution module may be used to further judge the discard event so as to distribute the discard event to the target window, thereby realizing simultaneous reporting of multiple different input devices.

[0289] At this time, the electronic device 300 may include a first input distribution module and a second input distribution module. The input reading module transmits the first point reporting event and the second point reporting event to the first input distribution module and also to the second input distribution module.

[0290] The first input distribution module determines the reporting event and the discarding event from the first reporting event and the second reporting event according to the first event distribution strategy, distributes the reporting event to the target window, and discards the discarding event. The second input distribution module determines the reporting event and the discarding event from the first reporting event and the second reporting event according to the first event distribution strategy, and determines that the discarding event complies with the second event distribution strategy, and distributes the discarding event to the target window.

[0291] For example, Figure 4A As shown, the first input distribution module may be the input distribution module 221 , and the second input distribution module may be the handwriting collaborative input distribution module 270 .

[0292] In other embodiments, the event distribution strategy of the input distribution module may be modified so that after determining a discard event, the input distribution module further determines whether the discard event needs to be reported, so as to achieve simultaneous reporting of multiple different input devices.

[0293] At this time, the input reading module of the electronic device 300 transmits the first point reporting event and the second point reporting event to the third input distribution module; the third input distribution module determines the reporting event and the discarding event from the first point reporting event and the second point reporting event according to the first event distribution strategy, and determines that the discarding event complies with the second event distribution strategy, and then distributes the reporting event and the discarding event to the target window.

[0294] For example, Figure 4B As shown, the third input distribution module may refer to the input distribution module 221. The input distribution module 221 may determine to discard events according to the original event distribution strategy, and further determine whether the current state is in a handwriting collaboration enabled state according to the handwriting collaboration distribution strategy 280, and whether the characteristics of the focus window meet the window characteristics in the handwriting collaboration scenario.

[0295] In some possible implementations, the electronic device 300 may not have a hardware false-touch prevention algorithm module, or may be provided with a hardware false-touch prevention algorithm module.

[0296] When the electronic device 300 includes a hardware anti-false touch algorithm module, the electronic device 300 needs to perform hardware anti-false touch algorithm processing in the process of converting the original input event into a high-level input event that can be recognized by the upper layer. In order to allow the hardware anti-false touch algorithm module to support multiple devices reporting points at the same time, the anti-false touch algorithm that affects the multi-input device collaborative operation scenario can be disabled.

[0297] At this time, if the multi-input device collaboration module is in an enabled state, the electronic device 300 will disable the preset anti-false touch algorithm in the anti-false touch algorithm set, and use the remaining anti-false touch algorithms to perform anti-false touch processing on the first input event and the second input event. The remaining anti-false touch algorithms are the anti-false touch algorithms in the anti-false touch algorithm set except the preset anti-false touch algorithm.

[0298] The preset anti-false touch algorithm may refer to an anti-false touch algorithm that affects the multi-input device collaborative operation scenario, and the remaining anti-false touch algorithms may be general anti-false touch algorithms. For example, see Figure 4A and Figure 4B The multi-input device collaboration module may be the hand-pen collaboration module 211. The preset anti-mis-touch algorithm refers to an anti-mis-touch algorithm that affects the hand-pen collaboration scenario.

[0299] If neither the first input event nor the second input event is an accidental touch reporting event, and the multi-input device collaborative module determines that the first input event and the input event are both reporting events in a multi-input device collaborative operation scenario, the electronic device 300 writes the first input event into the first device node corresponding to the first input device, and writes the second input event into the second device node corresponding to the second input device.

[0300] For example, Figure 4A and Figure 4BAs shown, the handwriting coordination module 211 determines whether the input event of the stylus pen and the input event of the finger are point reporting events that need to be reported in the handwriting coordination scenario. If it is recognized that the input event of the stylus pen and the input event of the finger are both point reporting events in the handwriting coordination scenario, the input event of the stylus pen is written to the stylus pen device node in the / dev / input directory, and the input event of the finger is written to the touch screen device node in the / dev / input directory.

[0301] The input reading module reads the first input event from the first device node and the second input event from the second device node through the event monitoring module; the input reading module encapsulates the first input event into a first reporting event and encapsulates the second input event into a second reporting event.

[0302] In this way, the preset anti-mistouch algorithm that affects the collaborative operation scenario of multiple input devices is invalidated, and it is further determined whether the input event is a point reporting event in the collaborative operation scenario of multiple input devices. If so, the input events of two different input devices are reported at the same time, thereby supporting the reporting of input events of two different input devices at the same time during the hardware anti-mistouch processing.

[0303] In some possible implementations, if the electronic device 300 receives a registration request from an application, the electronic device 300 adds the application to the application whitelist maintained by the multi-input device collaborative operation service in response to the registration request. The registration request may be a request for registering the multi-input device collaborative operation service that is actively sent when the application is opened or installed; if the application in the application whitelist is sensed to switch to the foreground, the multi-input device collaborative operation service enables the second input distribution module and the multi-input device collaborative module, or enables the second event distribution strategy and the multi-input device collaborative module of the third input distribution module.

[0304] For example, Figure 4A and Figure 4B As shown, the multi-input device collaborative operation service refers to the handwriting collaborative service 250. The multi-input device module may refer to the handwriting collaborative module 211. The second input distribution module may refer to the handwriting collaborative input distribution module 270.

[0305] Corresponding to the method described in the above embodiment, Figure 8 The structure schematic block diagram of the electronic device 300 provided in the embodiment of the present application is shown. For the convenience of description, only the parts related to the embodiment of the present application are shown.

[0306] Reference Figure 8 , the electronic device 300 may include: an input system 810 and an application 820.

[0307] The input system 810 is used to: receive a first input operation of a first input device and a second input operation of a second input device at the same time, wherein the first input device and the second input device are two different input devices; in response to the first input operation and the second input operation, report a first point reporting event and a second point reporting event to a target window of an application program running in the foreground, wherein the first point reporting event is a point reporting event triggered by the first input operation and the second point reporting event is a point reporting event triggered by the second input operation;

[0308] The application 820 is used to change the display content of the target window in response to the first point reporting event and the second point reporting event.

[0309] In some possible implementations, the input system 810 specifically includes an input event generating module 811 , an input reading module 812 , and an input distributing module 813 .

[0310] The input event generation module 811 is used to: generate a first input event of the first input operation in response to a first input operation; generate a second input event of the second input operation in response to a second input operation;

[0311] The input reading module 812 is used to: convert the first input event into a first reporting event, and convert the second input event into a second reporting event;

[0312] The input distribution module 813 is used for: if according to the first event distribution strategy, the reporting event and the discarding event are determined from the first reporting event and the second reporting event, and it is determined that the discarding event complies with the second event distribution strategy, then the first reporting event and the second reporting event are distributed to the target window, and the second event distribution strategy is an event distribution strategy under the scenario of collaborative operation of multiple input devices; the reporting event is the first reporting event, and the discarding event is the second reporting event; or, the reporting event is the second reporting event, and the discarding event is the first reporting event.

[0313] In some possible implementations, the input distribution module 813 includes a first input distribution module and a second input distribution module;

[0314] The input reading module 812 is specifically used to: transmit the first reporting event and the second reporting event to the first input distribution module and the second input distribution module;

[0315] The first input distribution module is used to: determine the reporting event and the discarding event from the first reporting event and the second reporting event according to the first event distribution strategy, distribute the reporting event to the target window, and discard the determined discarding event;

[0316] The second input distribution module is used to: determine the reporting event and the discarding event from the first reporting event and the second reporting event according to the first event distribution strategy, and if the discarding event complies with the second event distribution strategy, distribute the discarding event to the target window.

[0317] In some possible implementations, the input distribution module 813 includes a third input distribution module;

[0318] The input reading module 812 is specifically used to: transmit the first reporting event and the second reporting event to the third input distribution module;

[0319] The third input distribution module is used to: determine the reporting event and the discarding event from the first reporting point event and the second reporting point event according to the first event distribution strategy, and if the discarding event complies with the second event distribution strategy, distribute the reporting event and the discarding event to the target window.

[0320] In some possible implementations, the second input distribution module or the third input distribution module is specifically used to: if the multi-input device collaborative operation state is in an enabled state and the characteristics of the target window meet the characteristics of the multi-input device collaborative operation scenario, then determine that the discarded event meets the second event distribution strategy.

[0321] In some possible implementations, the electronic device 300 further includes a hardware false touch prevention algorithm module 830, and the hardware false touch prevention algorithm module 830 includes a multi-input device coordination module 831;

[0322] The hardware anti-false touch algorithm module 830 is used to: if the multi-input device coordination module 831 is in an enabled state, disable the preset anti-false touch algorithm in the anti-false touch algorithm set, and use the remaining anti-false touch algorithms to perform false touch processing on the first input event and the second input event, and the remaining anti-false touch algorithms are the anti-false touch algorithms in the anti-false touch algorithm set except the preset anti-false touch algorithm;

[0323] The multi-input device coordination module 831 is used to: if the first input event and the second input event are not mis-touch reporting events, and it is determined that the first input event and the input event are both reporting events in the multi-input device coordination operation scenario, then write the first input event into the first device node corresponding to the first input device, and write the second input event into the second device node corresponding to the second input device;

[0324] The input reading module 812 is specifically used to: read the first input event from the first device node through the event monitoring module, and read the second input event from the second device node; encapsulate the first input event into a first reporting event, and encapsulate the second input event into a second reporting event.

[0325] In some possible implementations, the first input device is a finger, the second input device is a stylus, and the electronic device includes a touch screen; the input system 810 is specifically used to: receive a first touch operation of the finger on the touch screen, and receive a second touch operation of the stylus on the touch screen; the finger and the stylus are in contact with the touch screen at the same time, the first input operation is a first touch operation, and the second input operation is a second touch operation.

[0326] In some possible implementations, the electronic device 300 further includes a multi-input device collaborative operation service 840 .

[0327] The multi-input device collaborative operation service 840 is used to: if a registration request from an application is received, then in response to the registration request, add the application to the application whitelist maintained by the multi-input device collaborative operation service 840; if it is sensed that an application in the application whitelist is switched to the foreground, then enable the second input distribution module and the multi-input device collaborative module, or enable the second event distribution strategy and the multi-input device collaborative module of the third input distribution module.

[0328] The above-mentioned electronic device 300 has the function of implementing the interaction method of the above-mentioned electronic device. This function can be implemented through hardware, or through hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned function, and the module can be software and / or hardware.

[0329] It should be noted that, since the information interaction, execution process and other contents between the above-mentioned electronic devices 300 are based on the same concept as the method embodiment of the present application, their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0330] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0331] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0332] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0333] In the embodiments provided in the present application, it should be understood that the disclosed devices, electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0334] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0335] The electronic device provided in the embodiments of the present application may include a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, a method as described in any one of the above method embodiments is implemented.

[0336] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0337] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0338] The present application also provides a chip system, which includes a processor coupled to a memory, and the processor executes a computer program stored in the memory to implement the methods described in the above method embodiments. The chip system can be a single chip or a chip module composed of multiple chips.

[0339] In the above embodiments, the description of each embodiment has its own emphasis. For the part that is not described or recorded in detail in a certain embodiment, please refer to the relevant description of other embodiments. It should be understood that the size of the sequence number of each step in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. In addition, in the description of the present application specification and the attached claims, the terms "first", "second", "third", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first", "second", "third", and "fourth" can explicitly or implicitly include one or more of the features. In addition, it should be understood that at least one of the embodiments of the present application includes one or more; wherein, multiple means greater than or equal to two. In the embodiment of the present application, "and / or" is only a kind of association relationship describing the associated objects, indicating that there can be three relationships, for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character “ / ” in this article generally indicates that the previous and next associated objects are in an “or” relationship.

[0340] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways.

[0341] Finally, it should be noted that the above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An interactive method for an electronic device, characterized in that: Applied to electronic equipment, the method comprises: Display the target window; While receiving a first input operation in a first input mode, receiving a second input operation in a second input mode; Sending a first point reporting event and a second point reporting event to a control corresponding to a target window, wherein the first point reporting event is a point reporting event triggered by the first input operation, and the second point reporting event is a point reporting event triggered by the second input operation; The control corresponding to the target window displays corresponding content in the target window according to the first point reporting event and the second point reporting event.

2. The method according to claim 1, characterized in that The target window is the interface of the first application, or includes the interface of the second application and the interface of the third application.

3. The method according to claim 2, characterized in that The target window is the interface of the first application; Sending the first point reporting event and the second point reporting event to the control corresponding to the target window includes: Sending the first reporting event to a control of the first application; If the preset condition is met, the second point reporting event is sent to the control of the first application.

4. The method according to claim 2, characterized in that: The target window includes an interface of the second application and an interface of the third application; Sending the first point reporting event and the second point reporting event to the control corresponding to the target window includes: Sending the first reporting event to the control of the second application; If the preset condition is met, the second point reporting event is sent to the control of the third application.

5. The method according to claim 1, characterized in that The method further comprises: While receiving the third input operation, receiving a fourth input operation; The third point reporting event is sent to the control corresponding to the target window, and the fourth point reporting event is not sent to the control corresponding to the target window, the third point reporting event is a point reporting event triggered by the third input operation, and the fourth point reporting event is a point reporting event triggered by the fourth input operation.

6. The method according to claim 5, characterized in that Sending the third point reporting event to a control corresponding to the target window, and not sending the fourth point reporting event to the control corresponding to the target window, comprises: Sending the third point reporting event to the control corresponding to the target window; If the preset condition is not met, the fourth point reporting event is not sent to the control corresponding to the target window.

7. The method according to any one of claims 3 to 6, characterized in that: The preset condition includes enabling a multi-input mode input function.

8. The method according to any one of claims 3 to 7, characterized in that: The preset condition includes that the second reporting event is not an accidental reporting event.

9. The method according to any one of claims 3 to 8, characterized in that: The preset condition includes that the characteristics of the target window conform to the characteristics of a multi-input mode collaborative operation scenario.

10. The method according to claim 9, characterized in that If the target window includes controls related to at least one of document editing, file editing, and image editing, the characteristics of the target window conform to the multi-input mode collaborative operation scenario.

11. The method according to claim 1, characterized in that: An input time of a first first point reporting event triggered by the first input operation is earlier than an input time of a first second point reporting event triggered by the second input operation.

12. The method according to claim 1, characterized in that The first input operation is an operation acting on any area of ​​the screen of the electronic device, and the second input operation is an operation acting on any area of ​​the screen of the electronic device.

13. The method according to claim 1, characterized in that The method further comprises: Receive user's setting operation; In response to a setting operation by the user, the multi-input mode input function is enabled.

14. The method according to claim 1, characterized in that The first input method and the second input method include any one or any two of finger input, stylus input, mouse input, air gesture input, and voice input.

15. The method according to claim 1, characterized in that The first input mode is finger input, the second input mode is stylus input, and the electronic device includes a touch screen; Receiving a first input operation in a first input mode and receiving a second input operation in a second input mode at the same time include: receiving a first touch operation of a finger on the touch screen, and receiving a second touch operation of a stylus on the touch screen; The finger and the stylus are in contact with the touch screen at the same time.

16. The method according to claim 1, characterized in that The electronic device includes an input reading module, a first input distribution module and a second input distribution module, wherein the step of sending the first point reporting event and the second point reporting event to the control corresponding to the target window includes: Acquire a first input event triggered by the first input operation, and acquire a second input event triggered by the second input operation; Convert the first input event into the first point reporting event, and convert the second input event into the second point reporting event; The input reading module transmits the first reporting point event and the second reporting point event to the first input distribution module and the second input distribution module; The first input distribution module determines that the first point reporting event is a first type of event and the second point reporting event is a second type of event, distributes the first point reporting event to the control of the target window, and discards the second point reporting event; The second input distribution module determines that the second point reporting event is the second type of event, and determines that the multi-input mode input function is in an enabled state, and the characteristics of the target window meet the characteristics of the multi-input mode collaborative operation scenario, then distributes the second point reporting event to the control of the target window.

17. The method according to claim 1, characterized in that The electronic device includes an input reading module and a third input distribution module, wherein sending the first point reporting event and the second point reporting event to the control corresponding to the target window includes: Acquire a first input event triggered by the first input operation, and acquire a second input event triggered by the second input operation; Convert the first input event into the first point reporting event, and convert the second input event into the second point reporting event; The input reading module transmits the first reporting event and the second reporting event to the third input distribution module; The third input distribution module determines that the first point reporting event is a first type of event, the second point reporting event is a second type of event, and determines that the multi-input mode input function is in an enabled state, and the characteristics of the target window meet the characteristics of the multi-input mode collaborative operation scenario, then distributes the first point reporting event and the second point reporting event to the controls of the target window.

18. The method according to claim 17 or 16, characterized in that Converting the first input event into the first point reporting event, and converting the second input event into the second point reporting event, comprises: If the multi-input mode collaboration module is in an enabled state, a preset anti-false touch algorithm in the anti-false touch algorithm set is disabled, and the remaining anti-false touch algorithms are used to perform false touch processing on the first input event and the second input event, and the remaining anti-false touch algorithms are the anti-false touch algorithms in the anti-false touch algorithm set except the preset anti-false touch algorithm; If the first input event and the second input event are not accidental touch reporting events, and the multi-input mode collaboration module determines that the first input event and the second input event are both reporting events in a multi-input mode collaborative operation scenario, then the first input event is written into a first device node corresponding to the first input mode, and the second input event is written into a second device node corresponding to the second input mode; The input reading module reads the first input event from the first device node through the event monitoring module, and reads the second input event from the second device node; The input reading module encapsulates and converts the first input event into the first point reporting event, and encapsulates and converts the second input event into the second point reporting event.

19. The method according to any one of claims 16 to 18, characterized in that While receiving the first input operation in the first input mode, and before receiving the second input operation in the second input mode, the method further includes: receiving a registration request of an application, and in response to the registration request, adding the application to an application whitelist maintained by the multi-input mode collaborative operation service; If it is sensed that an application in the application whitelist is switched to the foreground, the multi-input mode collaborative operation service enables the second input distribution module and the multi-input mode collaborative module, or enables the third input distribution module and the multi-input mode collaborative module.

20. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 19 is implemented.

21. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 19 is implemented.

Images