Response method, system and device for multi-finger operation in application, equipment and medium

Abstract

The invention discloses a response method, system and device of multi-finger operation in application, equipment and a medium. The method is executed by an application side in an OpenHarmony system, and comprises the following steps: when an initialization event reported by a bottom layer module is received each time, obtaining an event type in the initialization event, and obtaining gesture information in the initialization event according to the event type; according to the event type and a default gesture operation list pre-stored in the northbound application service, determining an interface response operation matched with the event type; and determining a target application operation of the user according to the interface response operation and the gesture information, and responding to the target application operation on the application side. By adopting the technical scheme, each application in the OpenHarmony system can perform different operation responses aiming at the same multi-finger operation, and the operation responses in the applications can be customized by updating the default gesture operation list, so that flexible configuration of the multi-finger operation responses is realized.

Description

Technical Field

[0001] The present invention relates to the field of open source Hongmeng system development technology, and in particular to a response method, system, device, equipment and medium for multi-finger operations in an application. Background Art

[0002] As capacitive touch screen technology matures and is widely used, most existing smart devices can accurately identify multi-touch operations, and can effectively improve the use efficiency of smart devices by providing users with portable and fast multi-touch operations.

[0003] However, most existing PC (personal computer) devices only support multi-finger operations at the system desktop layer, for example, swiping up with three fingers can open the multi-tasking view, swiping down with three fingers can display the desktop, etc. There are few multi-finger operations involving applications, and generally only two-finger pinching and two-finger sliding up and down operations can make the application respond. In addition, the multi-finger operation function of existing PC devices is generally bound and fixed after the system is initialized, and users cannot actively change it. Summary of the invention

[0004] The present invention provides a method, system, device, equipment and medium for responding to multi-finger operations in applications. Each application in the OpenHarmony system can perform different operation responses for the same multi-finger operation. The operation response in the application can be customized by updating the default gesture operation list, thereby achieving flexible configuration of the multi-finger operation response.

[0005] According to one aspect of the present invention, a method for responding to a multi-finger operation in an application is provided, which is executed by an application side in an OpenHarmony system and includes:

[0006] Whenever an initialization event reported by the bottom module is received, an event type in the initialization event is obtained, and gesture information is obtained in the initialization event according to the event type;

[0007] Determine an interface response operation matching the event type according to the event type and a default gesture operation list pre-stored in the northbound application service;

[0008] According to the interface response operation and the gesture information, a target application operation of the user is determined, and the target application operation is responded to on the application side.

[0009] According to another aspect of the present invention, there is provided an OpenHarmony system, including an application side, a multi-mode input subsystem, a northbound application service, Libinput, and a driver layer;

[0010] The driver layer is used to analyze the electrical signal uploaded by the touch device, generate input events according to the analysis results and send them to Libinput;

[0011] The Libinput is used to process each data in the input event, generate an original event according to the data processing result, and send it to the multi-mode input subsystem;

[0012] The multi-mode input subsystem is used to perform standardization processing on the original event, generate a standardized event, and forward the standardized event to the application side through the northbound application service;

[0013] The northbound application service is used to store a standard mapping list and a default gesture operation list corresponding to each application side in the OpenHarmony system;

[0014] The application side is used to execute the method for responding to a multi-finger operation in an application as described in any embodiment of the present invention.

[0015] According to another aspect of the present invention, a device for responding to a multi-finger operation in an application is provided, which is executed by an application side in an OpenHarmony system and includes:

[0016] An information acquisition module, configured to acquire an event type in the initialization event whenever an initialization event reported by the bottom layer module is received, and acquire gesture information in the initialization event according to the event type;

[0017] An interface response operation matching module, used to determine an interface response operation matching the event type according to the event type and a default gesture operation list pre-stored in the northbound application service;

[0018] The application operation response module is used to determine the user's target application operation according to the interface response operation and the gesture information, and respond to the target application operation on the application side.

[0019] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0020] at least one processor; and

[0021] a memory communicatively connected to the at least one processor; wherein,

[0022] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for responding to a multi-finger operation in an application as described in any embodiment of the present invention.

[0023] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for responding to multi-finger operations in an application as described in any embodiment of the present invention when executed.

[0024] The technical solution of the embodiment of the present invention is to obtain gesture information in the initialization event according to the event type in the initialization event whenever an initialization event reported by the underlying module is received, determine the interface response operation that matches the event type according to the event type and the default gesture operation list pre-stored in the northbound application service, determine the user's target application operation according to the interface response operation and the gesture information, and respond to the target application operation on the application side. It can flexibly customize the multi-finger operation in each application according to user needs, so that the multi-finger operation in the application is not restricted by the multi-finger operation initially configured by the system, and the multi-finger operation can produce different response effects in different applications, which greatly improves the flexibility, rationality and portability of the multi-finger operation.

[0025] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0027] Figure 1 is a flowchart of a method for responding to a multi-finger operation in an application according to a first embodiment of the present invention;

[0028] Figure 2 is a flowchart of another method for responding to multi-finger operations in an application provided in Embodiment 2 of the present invention;

[0029] Figure 3 is a schematic diagram of the structure of a response device for multi-finger operation in an application provided according to a third embodiment of the present invention;

[0030] Figure 4 is a schematic diagram of the structure of an OpenHarmony system provided according to Embodiment 4 of the present invention;

[0031] Figure 5 is a schematic diagram of the structure of another OpenHarmony system provided according to an embodiment of the present invention;

[0032] Figure 6 It is a structural schematic diagram of an electronic device for implementing the response method of multi-finger operation in an application according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] Embodiment 1

[0036] Figure 1 This is a flowchart of a method for responding to multi-finger operations in an application provided in the first embodiment of the present invention. This embodiment is applicable to the case of responding to multi-finger operations of a user in an application of an OpenHarmony system. The method can be executed by a device for responding to multi-finger operations in an application. The device for responding to multi-finger operations in an application can be implemented in the form of hardware and / or software and can generally be configured in the application side of an OpenHarmony system server. Figure 1 As shown, the method includes:

[0037] S110. Whenever an initialization event reported by a bottom-level module is received, an event type in the initialization event is obtained, and gesture information is obtained in the initialization event according to the event type.

[0038] Optionally, the technical solution proposed in the present invention can be mainly used in personal computers installed with the OpenHarmony system to solve the problem that existing personal computers can only support multi-finger operations on the system interface but cannot support personalized multi-finger operations in applications.

[0039] Optionally, in the OpenHarmony system, the underlying modules on the application side may include a driver layer, Libinput, a multi-mode input subsystem, and a northbound application service. The driver layer may interact with the touch device, process the electrical signal uploaded by the touch device, and report the processing result to Libinput. Libinput also processes the information uploaded by the driver layer, and reports the processing result to the multi-mode input subsystem. The multi-mode input subsystem processes again, generates an initialization event, and sends it to the northbound application service, which is then forwarded to the application side by the northbound application service.

[0040] Optionally, the event type can be used to describe the user's multi-finger operation content, such as: two-finger click, two-finger right slide, three-finger click, three-finger upward slide, etc., which are not listed here one by one, but only illustrate the form of the event type by giving examples.

[0041] Optionally, when the event type is a click event, the gesture information obtained in the initialization event includes the logical coordinates of each finger; when the event type is a slide event, the gesture information obtained in the initialization event includes the logical coordinates of each finger and the operation trajectory.

[0042] Optionally, the logical coordinates may refer to the coordinates of each finger on the display screen; and the operation trajectory may refer to the sliding trajectory of each finger on the display screen.

[0043] It is understandable that when the event type is a click event, the operation trajectory cannot be generated in the initialization event reported by the underlying module. Only when the event type is a sliding event can the underlying module generate the operation trajectory of each finger by collecting the position of the finger at different time points.

[0044] S120: Determine an interface response operation that matches the event type according to the event type and a default gesture operation list pre-stored in the northbound application service.

[0045] Optionally, the default gesture operation list corresponds to the application side. For different applications in the OpenHarmony system, each application side corresponds to a different default gesture operation list. The northbound application service stores the default gesture operation list corresponding to each application side.

[0046] Optionally, the default gesture operation list includes multiple customized event types and customized interface response operations having a mapping relationship with each customized event type, and the customized event type and the customized interface response operation are in a one-to-one mapping relationship.

[0047] In a specific implementation, for a video playback application, when the custom event type configured by the user is a three-finger click, the user can configure the custom interface response operation to switch applications, but for a music playback application, when the custom event type configured by the user is a three-finger click, the user can also configure the custom interface response operation corresponding to the three-finger click to display the desktop. That is, for the same custom event type, the corresponding custom interface response operations in different applications may be different, which is only used as an example here.

[0048] Optionally, in the default gesture operation list, each custom event type has only a unique corresponding custom interface response operation. After the event type is determined in the initialization event, the interface response operation matching the event type can be determined through the default gesture operation list.

[0049] S130: Determine the user's target application operation according to the interface response operation and gesture information, and respond to the target application operation on the application side.

[0050] Optionally, when the event type is a click event, the position of each finger in the application interface can be determined based on the logical coordinates in the gesture information, and the user's target application operation can be determined based on the configuration information of the application interface and the interface response operation.

[0051] It can be understood that, taking two-finger clicking as an example, the interface response operation that may be matched is the online query function. When the application is a video application, the target person to be selected can be determined according to the logical coordinates of the fingers, and an online query can be performed on the face of the target person. This is only an exemplary explanation.

[0052] Optionally, when the event type is a sliding event, the sliding position of each finger in the application interface can be determined based on the logical coordinates in the gesture information and the trajectory of each finger, and the user's target application operation can be determined based on the configuration information of the application interface and the interface response operation.

[0053] It can be understood that, taking two-finger sliding as an example, the interface response operation that may be matched is selecting the target area. When the application is a drawing application, the current position of the finger on the screen can be determined according to the logical coordinates of the finger, and the historical curve drawn by the finger in the current sliding event can be determined according to the operation trajectory. Based on the historical curve and the current position of the finger on the screen, the target area selected by the user can be tracked. This is only an exemplary explanation.

[0054] The technical solution of the embodiment of the present invention is to obtain gesture information in the initialization event according to the event type in the initialization event whenever an initialization event reported by the underlying module is received, determine the interface response operation that matches the event type according to the event type and the default gesture operation list pre-stored in the northbound application service, determine the user's target application operation according to the interface response operation and the gesture information, and respond to the target application operation on the application side. It can flexibly customize the multi-finger operation in each application according to user needs, so that the multi-finger operation in the application is not restricted by the multi-finger operation initially configured by the system, and the multi-finger operation can produce different response effects in different applications, which greatly improves the flexibility, rationality and portability of the multi-finger operation.

[0055] Embodiment 2

[0056] Figure 2 This is a flowchart of a method for responding to a multi-finger operation in an application provided by Embodiment 2 of the present invention. Based on the above embodiments, this embodiment specifically describes a method for responding to a multi-finger operation in an application. Figure 2 As shown, the method includes:

[0057] S210: Whenever at least one customized gesture operation sent by the user to the application side is received, the legitimacy of each customized gesture operation is verified according to the standard mapping list pre-stored in the northbound application service.

[0058] Optionally, the standard mapping list includes a plurality of standard event types and at least one standard interface response operation having a mapping relationship with each standard event type;

[0059] The default gesture operation list includes customized event types and customized interface response operations that have a mapping relationship with the customized event types.

[0060] Optionally, the legitimacy of each custom gesture operation is verified, including:

[0061] The custom event type is matched with the standard event types in the standard mapping list. When the target standard event type can be matched and it is determined that the custom interface response operation belongs to any standard interface response operation that has a mapping relationship with the target standard event type, the custom gesture operation is determined to be legal; otherwise, the custom gesture operation is determined to be illegal.

[0062] Optionally, for different application sides, users can independently configure interface response operations corresponding to each event type. Among the customized gesture operations sent by the user, there is only a unique corresponding customized gesture operation for the target customized event.

[0063] Optionally, after the application is started, the application side monitors the customized gesture operation configuration event in real time, and when receiving the customized gesture operation sent by the user to the application side, performs a legality check on the customized gesture operation.

[0064] Optionally, a standard mapping list may be stored in the northbound application service, which stores a variety of standard event types. Each standard event type may correspond to one or more interface response operations. The standard mapping list is only intended to specify the legal operations corresponding to each standard event type to avoid situations where completely independent configuration by the user may cause conflicts in the system.

[0065] In an optional example, in the standard mapping list, standard event types may include two-finger click, two-finger slide, three-finger click, three-finger slide, four-finger click, and four-finger slide. Taking two-finger click as an example, its corresponding standard interface response operations may be interface response operation 1, interface response operation 2, and interface response operation 3. When the user performs a customized gesture operation on the target application side, if the customized event type sent by the user is two-finger click, determine whether the customized interface response operation corresponding to the two-finger click belongs to one of interface response operation 1, interface response operation 2, and interface response operation 3. If so, determine that the customized gesture operation is legal; otherwise, it is illegal. If the customized event type sent by the user is five-finger slide, but the labeled event type stored in the standard mapping list does not include five-finger slide, it is also determined to be illegal.

[0066] S220: When it is determined that the customized gesture operation is legal, update the default gesture operation list in the northbound application service according to the customized gesture operation.

[0067] Among them, the default gesture operation list matches the application side.

[0068] Optionally, for each application, a unique corresponding default gesture operation list is stored in the northbound application service. When the user re-customizes the corresponding customized interface response operation for the target gesture operation in the default gesture operation list, if it is confirmed to be legal, the currently issued customized interface response operation is used to replace the original interface response operation corresponding to the target gesture operation.

[0069] S230. Whenever an initialization event reported by a bottom-level module is received, an event type in the initialization event is obtained, and gesture information is obtained in the initialization event according to the event type.

[0070] Wherein, according to the event type, acquiring gesture information in the initialization event includes any of the following:

[0071] When the event type is a click event, obtaining the logical coordinates of each finger in the initialization event;

[0072] When the event type is a sliding event, the logical coordinates and operation trajectory of each finger are obtained in the initialization event.

[0073] S240: Determine an interface response operation that matches the event type according to the event type and a default gesture operation list pre-stored in the northbound application service.

[0074] S250: Determine the user's target application operation according to the interface response operation and gesture information, and respond to the target application operation on the application side.

[0075] The technical solution of the embodiment of the present invention is to obtain gesture information in the initialization event according to the event type in the initialization event whenever an initialization event reported by the underlying module is received, determine the interface response operation that matches the event type according to the event type and the default gesture operation list pre-stored in the northbound application service, determine the user's target application operation according to the interface response operation and the gesture information, and respond to the target application operation on the application side. It can flexibly customize the multi-finger operation in each application according to user needs, so that the multi-finger operation in the application is not restricted by the multi-finger operation initially configured by the system, and the multi-finger operation can produce different response effects in different applications, which greatly improves the flexibility, rationality and portability of the multi-finger operation.

[0076] Embodiment 3

[0077] Figure 3 This is a schematic diagram of a structure of a response device for multi-finger operation in an application provided by Embodiment 3 of the present invention. Figure 3 As shown, the device includes: an information acquisition module 310, an interface response operation matching module 320 and an application operation response module 330.

[0078] The information acquisition module 310 is used to obtain the event type in the initialization event whenever receiving the initialization event reported by the bottom module, and obtain gesture information in the initialization event according to the event type.

[0079] The interface response operation matching module 320 is used to determine the interface response operation matching the event type according to the event type and a default gesture operation list pre-stored in the northbound application service.

[0080] The application operation response module 330 is used to determine the user's target application operation according to the interface response operation and the gesture information, and respond to the target application operation on the application side.

[0081] The technical solution of the embodiment of the present invention is to obtain gesture information in the initialization event according to the event type in the initialization event whenever an initialization event reported by the underlying module is received, determine the interface response operation that matches the event type according to the event type and the default gesture operation list pre-stored in the northbound application service, determine the user's target application operation according to the interface response operation and the gesture information, and respond to the target application operation on the application side. It can flexibly customize the multi-finger operation in each application according to user needs, so that the multi-finger operation in the application is not restricted by the multi-finger operation initially configured by the system, and the multi-finger operation can produce different response effects in different applications, which greatly improves the flexibility, rationality and portability of the multi-finger operation.

[0082] Based on the above embodiments, the information acquisition module 310 can be used to perform any of the following:

[0083] When the event type is a click event, obtaining the logical coordinates of each finger in the initialization event;

[0084] When the event type is a sliding event, the logical coordinates and operation trajectory of each finger are obtained in the initialization event.

[0085] On the basis of the above embodiments, a finger operation customization module may also be included, including:

[0086] A legitimacy verification unit, configured to verify the legitimacy of each customized gesture operation according to a standard mapping list pre-stored in the northbound application service whenever receiving at least one customized gesture operation issued by the user to the application side;

[0087] The default gesture operation list updating unit is used to update the default gesture operation list in the northbound application service according to the customized gesture operation when it is determined that the customized gesture operation is legal; wherein the default gesture operation list matches the application side.

[0088] Based on the above embodiments, the standard mapping list includes multiple standard event types and at least one standard interface response operation having a mapping relationship with each standard event type;

[0089] The default gesture operation list includes customized event types and customized interface response operations that have a mapping relationship with the customized event types.

[0090] Based on the above embodiments, the legality verification unit can be specifically used for:

[0091] The custom event type is matched with the standard event types in the standard mapping list. When the target standard event type can be matched and it is determined that the custom interface response operation belongs to any standard interface response operation that has a mapping relationship with the target standard event type, the custom gesture operation is determined to be legal; otherwise, the custom gesture operation is determined to be illegal.

[0092] The device for responding to multi-finger operations in an application provided by an embodiment of the present invention can execute the method for responding to multi-finger operations in an application provided by any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.

[0093] Embodiment 4

[0094] Figure 4 This is a schematic diagram of the structure of an OpenHarmony system provided in Embodiment 4 of the present invention. Figure 4 As shown, the OpenHarmony system includes an application side 410 , a multi-mode input subsystem 420 , a northbound application service 430 , Libinput 440 , and a driver layer 450 .

[0095] The driver layer 450 is used to analyze the electrical signal uploaded by the touch device, generate an input event according to the analysis result, and send it to the Libinput 440 .

[0096] Libinput 440 is used to process each data in the input event, generate an original event according to the data processing result, and send it to the multi-mode input subsystem 420.

[0097] The multi-mode input subsystem 420 is used to perform standardization processing on the original event, generate a standardized event, and forward the standardized event to the application side through the northbound application service 430.

[0098] The northbound application service 430 is used to store a standard mapping list and a default gesture operation list corresponding to each application side in the OpenHarmony system.

[0099] The application side 410 is used to execute the method for responding to a multi-finger operation in an application as described in any embodiment of the present invention.

[0100] Figure 5 FIG. 1 is a schematic diagram of another OpenHarmony system provided in an embodiment of the present invention. Figure 5As shown, Driver is the driver layer 450, multimodalinput is the multimodal input subsystem 420, and the hap application side is the application side 410. The northbound application service and the multimodal input subsystem 420 can be regarded as the same level. The standardized events received by the application side 410 are actually sent by the multimodal input subsystem 420, but the multimodal input subsystem 420 needs to be forwarded through the northbound application service. The northbound application service does not process the standardized events and only plays a forwarding role.

[0101] Optionally, the capacitive sensor on the touch device can convert physical touch into an electrical signal and send it to the driver layer 450. After receiving the electrical signal, the driver layer 450 identifies the electrical signal, obtains the number of fingers, pressure, and physical coordinates, and encapsulates the number of fingers, pressure, and physical coordinates as an input event and sends it to Libinput 440.

[0102] Optionally, the input event includes the number of fingers, the position of the first finger, the finger action and the event identifier.

[0103] Optionally, the first finger position may refer to the physical coordinates of the finger, that is, the position of the finger on the touch device. The event identifier may be used to mark the start and end of an event, and whether it is a click event or a sliding event. For example, when the finger touches the device for the first time, the event identifier may be down. When the finger is lifted, it indicates the end of the event, and the event identifier may be up. When it is a click event, it may be press. When it is a sliding event, it may be move.

[0104] Optionally, after receiving the input event, Libinput 440 may convert the physical coordinates into logical coordinates, and identify the corresponding event type through an internally integrated gesture recognition algorithm according to the event flag, the number of fingers and the logical coordinates.

[0105] Optionally, Libinput 440 can be used specifically for:

[0106] Determining the physical coordinates of each finger in the touch device according to the first finger position;

[0107] The scaling ratio between the display screen and the touch device is determined according to the display screen size and the touch device size, and the logical coordinates of each finger are calculated according to the physical coordinates of each finger in the touch device and the scaling ratio.

[0108] Optionally, Libinput 440 can also be used specifically for:

[0109] When it is determined that the finger action is a sliding action, each target input event belonging to the current sliding event is determined according to the event identifier, and a first movement trajectory of each finger is constructed according to the first finger position in each target input event;

[0110] The first movement trajectory is smoothed to obtain a second finger trajectory, and the operation trajectory of each finger on the display screen is calculated according to the zoom ratio.

[0111] Optionally, the second finger trajectory may refer to the trajectory of each finger on the display screen in a sliding event.

[0112] Optionally, Libinput 440 may also be used to provide the multi-mode input subsystem 420 with interfaces for acquiring key information, so that the multi-mode input subsystem 420 can acquire various information in the original event through various interfaces.

[0113] Optionally, the multi-mode input subsystem 420 performs standardization processing on the original time, which can be understood as performing information format conversion, converting it into an information format that can be read by the application side 410 and encapsulating it into an initialization event. In addition to the operation trajectory, logical coordinates and event type parsed and obtained by Libinput 440, the initialization event also includes the number of fingers, the first finger position, the finger action and the event identifier provided by the driver layer 450.

[0114] The technical solution of the embodiment of the present invention can accurately collect user event information and respond to user multi-finger operation events according to the default gesture operation list of the application by configuring the application side, multi-mode input subsystem, northbound application service, Libinput and driver layer in the OpenHarmony system, and can realize flexible customization of multi-finger operation in each application according to user needs, so that the multi-finger operation in the application is not restricted by the multi-finger operation initially configured by the system, and the multi-finger operation can produce different response effects in different applications, which greatly improves the flexibility, rationality and portability of the multi-finger operation.

[0115] Embodiment 5

[0116] Figure 6A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0117] like Figure 6 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0118] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0119] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as the response method of multi-finger operation in an application as described in an embodiment of the present invention. That is:

[0120] Whenever an initialization event reported by the bottom module is received, an event type in the initialization event is obtained, and gesture information is obtained in the initialization event according to the event type;

[0121] Determine an interface response operation matching the event type according to the event type and a default gesture operation list pre-stored in the northbound application service;

[0122] According to the interface response operation and the gesture information, a target application operation of the user is determined, and the target application operation is responded to on the application side.

[0123] In some embodiments, the method for responding to multi-finger operations in an application may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the method for responding to multi-finger operations in an application described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the method for responding to multi-finger operations in an application in any other appropriate manner (e.g., by means of firmware).

[0124] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0125] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0126] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0127] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0128] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0129] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0130] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0131] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for responding to multi-finger operations in an application, executed by an application side in an OpenHarmony system, characterized in that: include: Whenever an initialization event reported by the bottom module is received, an event type in the initialization event is obtained, and gesture information is obtained in the initialization event according to the event type; Determine an interface response operation matching the event type according to the event type and a default gesture operation list pre-stored in the northbound application service; According to the interface response operation and the gesture information, a target application operation of the user is determined, and the target application operation is responded to on the application side.

2. The method according to claim 1, characterized in that According to the event type, acquiring gesture information in the initialization event includes any of the following: When the event type is a click event, obtaining the logical coordinates of each finger in the initialization event; When the event type is a sliding event, the logical coordinates and operation trajectory of each finger are obtained in the initialization event.

3. The method according to claim 1, characterized in that Also includes: Whenever receiving at least one customized gesture operation sent by the user to the application side, verify the legitimacy of each customized gesture operation according to the standard mapping list pre-stored in the northbound application service; When it is determined that the customized gesture operation is legal, the default gesture operation list in the northbound application service is updated according to the customized gesture operation; wherein the default gesture operation list matches the application side.

4. The method according to claim 3, characterized in that The standard mapping list includes a plurality of standard event types and at least one standard interface response operation having a mapping relationship with each standard event type; The default gesture operation list includes customized event types and customized interface response operations having a mapping relationship with the customized event types; Verify the legitimacy of each custom gesture operation, including: The custom event type is matched with the standard event type in the standard mapping list. When the target standard event type can be matched and it is determined that the custom interface response operation belongs to any standard interface response operation that has a mapping relationship with the target standard event type, the custom gesture operation is determined to be legal; Otherwise, it is determined that the custom gesture operation is illegal.

5. An OpenHarmony system, characterized in that: Includes the application side, multi-mode input subsystem, northbound application service, Libinput, and driver layer; The driver layer is used to analyze the electrical signal uploaded by the touch device, generate input events according to the analysis results and send them to Libinput; The Libinput is used to process each data in the input event, generate an original event according to the data processing result, and send it to the multi-mode input subsystem; The multi-mode input subsystem is used to perform standardization processing on the original event, generate a standardized event, and forward the standardized event to the application side through the northbound application service; The northbound application service is used to store a standard mapping list and a default gesture operation list corresponding to each application side in the OpenHarmony system; The application side is used to execute the method for responding to a multi-finger operation in an application as described in any one of claims 1 to 4.

6. The system according to claim 5, characterized in that The input event includes the number of fingers, the first finger position, the finger action and the event identifier; The Libinput is specifically used for: Determining the physical coordinates of each finger in the touch device according to the first finger position; The scaling ratio between the display screen and the touch device is determined according to the display screen size and the touch device size, and the logical coordinates of each finger are calculated according to the physical coordinates of each finger in the touch device and the scaling ratio.

7. The system according to claim 6, characterized in that The Libinput is also specifically used for: When it is determined that the finger action is a sliding action, each target input event belonging to the current sliding event is determined according to the event identifier, and a first movement trajectory of each finger is constructed according to the first finger position in each target input event; The first movement trajectory is smoothed to obtain a second finger trajectory, and the operation trajectory of each finger on the display screen is calculated according to the zoom ratio.

8. A response device for multi-finger operation in an application, executed by the application side of the OpenHarmony system, characterized in that: include: An information acquisition module, configured to acquire an event type in the initialization event whenever an initialization event reported by the bottom layer module is received, and acquire gesture information in the initialization event according to the event type; An interface response operation matching module, used to determine an interface response operation matching the event type according to the event type and a default gesture operation list pre-stored in the northbound application service; The application operation response module is used to determine the user's target application operation according to the interface response operation and the gesture information, and respond to the target application operation on the application side.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for responding to a multi-finger operation in an application according to any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for responding to a multi-finger operation in an application according to any one of claims 1 to 4 when the instructions are executed.

Images