Human-computer interaction method, electronic device and storage medium

Abstract

The embodiment of the present application provides a human-computer interaction method, an electronic device, and a storage medium. The method is applied to an electronic device. In the method, when the user presses, moves, and lifts the external input device, the electronic device obtains the coordinate data corresponding to each operation and saves it in a target data group; the electronic device determines at least one target data in the target data group, and calculates the data difference according to each target data, so that when the data difference meets the multi-selection mode condition, it enters the multi-selection mode and determines the multi-selection range, so that all interactive objects within the multi-selection range are marked as selected. In this way, the electronic device can respond to user operations, select all interactive objects within the multi-selection range, and realize human-computer interaction, thereby simplifying the interaction method between the user and the electronic device, making the human-computer interaction more in line with user habits, and improving the user experience.

Description

Technical Field

[0001] The present application relates to the technical field of intelligent terminals, and in particular to a human-computer interaction method, an electronic device and a storage medium. Background Art

[0002] With the development of science and technology, people's use of electronic devices with touch screens, such as mobile phones and tablet computers, is no longer limited to directly controlling these electronic devices by touching the screen, but can also control these electronic devices through external input devices. For example, a stylus pen, a mouse, etc. can be externally connected to electronic devices such as mobile phones and tablet computers to control the electronic devices using the stylus pen or the mouse.

[0003] However, when a user uses an external input device to operate an electronic device, if he wants to select multiple files in the electronic device, the user needs to click and hold the touch screen, and after the electronic device enters the multiple selection mode, the user moves the input device to select multiple files. However, the above interaction method of multiple file selection does not conform to user habits, resulting in a poor user experience. Summary of the invention

[0004] In order to solve the above technical problems, the embodiments of the present application provide a human-computer interaction method, an electronic device and a storage medium. In this method, when a user operates an external input device, the electronic device can save the coordinate data corresponding to the operation, and when entering the multi-selection mode, the multi-selection range is determined according to the saved coordinate data, so as to select all interactive objects within the multi-selection range, realize human-computer interaction, and further simplify the interaction mode between the user and the electronic device, make the human-computer interaction more in line with the user's habits, and improve the user experience.

[0005] In the first aspect, the embodiment of the present application provides a human-computer interaction method, which is applied to an electronic device. The method includes: in response to a user's pressing operation, moving operation, and lifting operation on an external input device, the electronic device obtains coordinate data corresponding to each operation, and saves each coordinate data into a target data group; the electronic device determines at least one target data in the target data group, and calculates at least one data difference based on each target data; when the electronic device determines that each data difference satisfies a multi-selection mode condition, the electronic device determines the target range corresponding to the external input device based on each coordinate data in the target data group, and marks all interactive objects within the target range as selected.

[0006] The electronic device includes a touch screen and is connected to an external input device.

[0007] The external input device and the electronic device are connected in communication. For example, the external input device and the electronic device can be connected wirelessly, etc., which is not limited in this application.

[0008] The target data group may be a data group for storing coordinate data of an external input device when a user operates the external input device.

[0009] The target data may be any data in the target data group.

[0010] Specifically, the target data may be any horizontal coordinate data or any vertical coordinate data, and this application does not limit this.

[0011] The data difference may be a difference between each target data. The multiple selection mode condition may be a condition that needs to be satisfied when the electronic device enters the multiple selection mode.

[0012] The target range may be a range corresponding to a motion track formed after the user lifts up the external input device.

[0013] For example, the target range may be Figure 5a or Figure 5b The range corresponding to the selected rectangle shown.

[0014] The interactive object may be an object with which the user interacts with the electronic device.

[0015] Exemplarily, the interactive object may be Figure 4a Document files in Figure 4b Chat history in Figure 4c The calendar date in the file, or it can be a picture file, audio or video file, etc.

[0016] In this way, the electronic device can simplify the interaction mode between the user and the electronic device, make the human-computer interaction more in line with the user's habits, and improve the user experience.

[0017] According to the first aspect, the data difference value comprises a first data difference value and a second data difference value;

[0018] The electronic device determines that each data difference value satisfies the multiple selection mode condition, which may include: the electronic device determines that each data difference value satisfies the multiple selection mode condition when the first data difference value is greater than a first threshold value and / or the second data difference value is greater than a second threshold value.

[0019] The first data difference may be a data difference between the horizontal coordinates, and the second data difference may be a data difference between the vertical coordinates.

[0020] The first threshold may be a threshold determined by the horizontal axis size of the touch screen. For example, the first threshold may be 1 / 5 of the horizontal axis size of the touch screen.

[0021] The second threshold may be a threshold determined by the longitudinal axis size of the touch screen. For example, the second threshold may be 1 / 5 of the longitudinal axis size of the touch screen.

[0022] In this way, after the user operates the external input device and the movement range of the external input device is large enough, the electronic device can enter the multiple selection mode, thereby simplifying the interaction method when multiple selections are made on the interactive objects.

[0023] Specifically, when the first data difference is less than 1 / 5 of the horizontal axis size of the touch screen and the second data difference is less than 1 / 5 of the vertical axis size of the touch screen, the electronic device can determine not to enter the multiple selection mode and ignore the user's operation on the external input device.

[0024] According to the first aspect, or any implementation of the first aspect above, the external input device is a mouse;

[0025] The electronic device determines at least one target data in a target data group and calculates at least one data difference based on each target data, which may include: the electronic device determines the horizontal coordinate data corresponding to the pressing operation in the target data group as the first target data; determines the vertical coordinate data corresponding to the pressing operation in the target data group as the second target data; determines the horizontal coordinate data corresponding to the lifting operation in the target data group as the third target data; and determines the vertical coordinate data corresponding to the lifting operation in the target data group as the fourth target data; the electronic device determines the difference between the first target data and the third target data as the first data difference, and determines the difference between the second target data and the fourth target data as the second data difference.

[0026] Specifically, when the external input device is a mouse, the pressing operation may be an operation in which the user clicks the left button of the mouse, and the lifting operation may be an operation in which the user releases the left button of the mouse.

[0027] The first target data may be the horizontal coordinate of the position coordinates where the mouse pointer is located when the user clicks the left button of the mouse, and the second target data may be the vertical coordinate of the position coordinates where the mouse pointer is located when the user clicks the left button of the mouse.

[0028] Specifically, the first target data may be the horizontal coordinate in the mouse click coordinates, and the second target data may be the vertical coordinate in the mouse click coordinates.

[0029] The third target data may be the horizontal coordinate of the position coordinate of the mouse pointer when the user releases the left mouse button. The fourth target data may be the vertical coordinate of the position coordinate of the mouse pointer when the user releases the left mouse button.

[0030] Specifically, the third target data may be the horizontal coordinate of the mouse release coordinates, and the fourth target data may be the vertical coordinate of the mouse release coordinates.

[0031] In this way, the electronic device can determine the target data in the target data group and the data difference when the external input device is a mouse.

[0032] According to the first aspect, or any implementation of the first aspect above, the electronic device determines the target range corresponding to the external input device based on the coordinate data in the target data group, which may include: the electronic device uses the first target data and the third target data as the first horizontal coordinates, respectively; and uses the second target data and the fourth target data as the first vertical coordinates, respectively; the electronic device determines the coordinate data of four first target points based on each first horizontal coordinate and each first vertical coordinate, and determines the target range corresponding to the external input device based on the coordinate data of each first target point.

[0033] The first target point may be a point in the motion trajectory of the external input device.

[0034] Specifically, the target range may be a rectangular range, and correspondingly, the first target points may be four vertices constituting the target range.

[0035] In this way, when the external input device is a mouse, the electronic device can determine the coordinate data of four points according to the horizontal and vertical coordinates corresponding to the pressing operation and the horizontal and vertical coordinates corresponding to the lifting operation, thereby determining a rectangular range according to the coordinate data of the four points, and further accurately determining the multiple selection range of the interactive object.

[0036] According to the first aspect, or any implementation of the first aspect above, the external input device is a stylus pen;

[0037] The electronic device determines at least one target data in the target data group and calculates at least one data difference based on each target data, which may include: the electronic device determines the horizontal coordinate data and the vertical coordinate data of each coordinate data in the target data group; the electronic device determines the maximum horizontal coordinate in the horizontal coordinate data of each coordinate data as the fifth target data; and determines the minimum horizontal coordinate in the horizontal coordinate data of each coordinate data as the sixth target data; and determines the maximum vertical coordinate in the vertical coordinate data of each coordinate data as the seventh target data; and determines the minimum vertical coordinate in the vertical coordinate data of each coordinate data as the eighth target data; the electronic device determines the difference between the fifth target data and the sixth target data as the first data difference; and determines the difference between the seventh target data and the eighth target data as the second data difference.

[0038] Specifically, when the external input device is a stylus pen, the pressing operation may be an operation in which the user touches the screen with the stylus pen, and the lifting operation may be an operation in which the user lifts the stylus pen to make it leave the touch screen.

[0039] Specifically, X d =Xmax -X min ; Among them, X d is the first data difference; X max is the maximum horizontal coordinate among the horizontal coordinate data of each coordinate data; min It is the minimum horizontal coordinate among the horizontal coordinate data of each coordinate data.

[0040] Specifically, Y d =Y max -Y min ; Among them, Y d is the second data difference; Y max Y is the maximum ordinate among all ordinate data; min It is the minimum vertical coordinate among the vertical coordinate data of each coordinate data.

[0041] In this way, the electronic device can determine the target data in the target data group and the data difference when the external input device is a stylus.

[0042] According to the first aspect, or any implementation method of the first aspect above, the electronic device determines the target range corresponding to the external input device according to the coordinate data in the target data group, which may include: the electronic device determines the motion trajectory corresponding to the stylus according to the coordinate data in the target data group, and determines the coordinate data of the first point in each coordinate data of the target data group; the electronic device divides the motion trajectory into regions according to the coordinate data of the first point, and determines the coordinate data of the second point in each coordinate data of the target data group; the electronic device determines the target area in the motion trajectory according to the coordinate data of the second point, and determines the coordinate data of the third point in the target area in each coordinate data of the target data group; the electronic device determines the target range corresponding to the external input device according to the coordinate data of the first point, the coordinate data of the second point and the coordinate data of the third point.

[0043] The target area may be an area where the second point does not exist after the motion trajectory is divided into areas.

[0044] For example, the first point can be Fig.10b The second point can be P1, P2, P3, P4. Fig.10c Q1, Q2, Q3, Q4 in . The target area can be Fig.10c Area 23 and area 25 in the third point can be Fig.10d R1 and R2 in .

[0045] In this way, the electronic device can determine the first point, the second point, and the third point in the motion trajectory of the stylus, and determine the selection rectangular range of the stylus based on the first point, the second point, and the third point, so as to more accurately determine the selection range corresponding to the motion trajectory of the stylus, thereby improving the accuracy of human-computer interaction.

[0046] According to the first aspect, or any implementation of the first aspect above, the electronic device determines the coordinate data of the first point in each coordinate data of the target data group, which may include: the electronic device determines the sum of the fifth target data and the sixth target data as the first data sum; and determines the sum of the seventh target data and the eighth target data as the second data sum; the electronic device determines the center point coordinate data of the motion trajectory according to the first data sum and the second data sum; the electronic device determines the coordinate data of the first point in each coordinate data of the target data group according to the center point coordinate data.

[0047] For example, the center point coordinate data can be Fig.10b The coordinates of point O shown (X ave ,Y ave ).

[0048] Specifically, X ave =(X max +X min ) / 2,Y ave =(Y max +Y min ) / 2; where X ave Y is the horizontal coordinate of the center point coordinate data; ave The vertical coordinate of the center point coordinate data.

[0049] In this way, the electronic device can determine the center point coordinate data to determine the coordinate data of the first point according to the center point coordinate data.

[0050] According to the first aspect, or any implementation of the first aspect above, the number of the first point is at least one;

[0051] Correspondingly, the electronic device determines the coordinate data of the first point in each coordinate data of the target data group based on the center point coordinate data, which may include: the electronic device determines the coordinate data with the same horizontal coordinate as the horizontal coordinate of the center point coordinate data in each coordinate data of the target data group as the first coordinate data; the electronic device determines the coordinate data with the same vertical coordinate as the vertical coordinate of the center point coordinate data in each coordinate data of the target data group as the second coordinate data; the electronic device determines the first coordinate data with the largest vertical coordinate in each first coordinate data as the coordinate data of the first point; and determines the first coordinate data with the smallest vertical coordinate in each first coordinate data as the coordinate data of the first point; the electronic device determines the second coordinate data with the largest horizontal coordinate in each second coordinate data as the coordinate data of the first point; and determines the second coordinate data with the smallest horizontal coordinate in each second coordinate data as the coordinate data of the first point.

[0052] Therein, the number of the first coordinate data is at least one; and the number of the second coordinate data is at least one.

[0053] Specifically, the number of the first points may be four, the number of the first coordinate data may be two, and the number of the second coordinate data may be two.

[0054] For example, the first coordinate data may be (X ave ,Y1)、(X ave ,Y2)、…、(X ave ,Y i ), Y a =max(Y1,Y2,…,Y i ), Y b =min(Y1,Y2,…,Y i ), then (X ave ,Y a ) is the first coordinate data with the largest ordinate among all the first coordinate data, (X ave ,Y b ) is the first coordinate data with the smallest ordinate among the first coordinate data, then the coordinate data of the first point may include (X ave ,Y a ), (X ave ,Y b ).

[0055] For example, the second coordinate data may be (X1, Y ave )、(X2,Y ave ),…,(X j ,Y ave ), X a =max(X1,X2,…,X j ), X b=min(X1,X2,…,X j ), then (X a ,Y ave ) is the second coordinate data with the largest horizontal coordinate among all the second coordinate data, (X b ,Y ave ) is the second coordinate data with the smallest horizontal coordinate among the second coordinate data, then the coordinate data of the first point may include (X a ,Y ave ), (X b ,Y ave ).

[0056] In this way, the electronic device can determine the coordinate data of the first point based on the coordinate data of the center point.

[0057] According to the first aspect, or any implementation of the first aspect above, the number of the second point is at least one;

[0058] Correspondingly, the electronic device determines the coordinate data of the second point in each coordinate data of the target data group, which may include: the electronic device determines the coordinate data corresponding to the fifth target data, the sixth target data, the seventh target data and the eighth target data as the coordinate data of the second point.

[0059] Specifically, the number of the second points may be four.

[0060] For example, among the horizontal coordinate data of each coordinate data, the maximum horizontal coordinate X max The corresponding coordinate data is (X max ,Y d ), among the horizontal coordinate data of each coordinate data, the minimum horizontal coordinate X min The corresponding coordinate data is (X min ,Y c ), among the ordinate data of each coordinate data, the maximum ordinate Y max The corresponding coordinate data is (X c ,Y max ), among the ordinate data of each coordinate data, the minimum ordinate Y min The corresponding coordinate data is (X d ,Y min ), thus, the coordinate data of the second point includes (X max ,Y d )、(X min ,Y c )、(X c ,Y max )、(X d ,Y min ).

[0061] In this way, the electronic device can determine the coordinate data of the second point according to the maximum value and the minimum value in each horizontal coordinate data and the maximum value and the minimum value in each vertical coordinate data.

[0062] According to the first aspect, or any implementation method of the first aspect above, the electronic device determines the target area in the motion trajectory based on the coordinate data of the second point, which may include: the electronic device determines the area where the second point in the motion trajectory is located based on the coordinate data of the second point, and determines the area in the motion trajectory where the second point does not exist as the target area.

[0063] The number of target areas may be one or more.

[0064] According to the first aspect, or any implementation of the first aspect above, the electronic device determines the coordinate data of the third point in the target area from each coordinate data of the target data group, which may include: the electronic device determines the third coordinate data in the target area from each coordinate data of the target data group; the electronic device determines the difference between the horizontal coordinate of the third coordinate data and the horizontal coordinate of the center point coordinate data as the first value, and determines the difference between the vertical coordinate of the third coordinate data and the vertical coordinate of the center point coordinate data as the second value; the electronic device determines the third coordinate data as the coordinate data of the third point when the difference between the first value and the second value is less than a third threshold value.

[0065] The third threshold may be a threshold between pre-set differences. Specifically, the third threshold may be a preset difference threshold, for example, a number close to zero.

[0066] For example, assuming that the third coordinate data is (X e ,Y e ), then in |X e -X ave ∣-∣Y e -Y ave When |X≈0, the third coordinate data is determined as the coordinate data of the third point. e -X ave ∣-∣Y e -Y ave When ∣≈0, the coordinate data of the third point is (X e ,Y e ).

[0067] In this way, the electronic device can determine a coordinate data in the target area of ​​the motion trajectory as the coordinate data of the third point, thereby improving the accuracy of selecting the rectangular range.

[0068] According to the first aspect, or any implementation method of the first aspect above, after the electronic device obtains the coordinate data corresponding to each operation in response to the user's pressing operation and moving operation on the external input device, and saves each coordinate data to the target data group, it may also include: when the electronic device has not received the user's lifting operation on the external input device, the horizontal coordinate data corresponding to the moving operation in the target data group is determined as the ninth target data; and the vertical coordinate data corresponding to the moving operation in the target data group is determined as the tenth target data; when the ninth target data is different from the first target data, and the tenth target data is different from the second target data, the electronic device generates a target event corresponding to the mouse and responds to the target event.

[0069] The target event may be a mouse click and move event generated by the electronic device in response to the user's operation after the user presses the mouse and moves the mouse.

[0070] For example, the target event may be Figure 5a The click move event shown.

[0071] In this way, the electronic device can generate an event corresponding to the user operation and respond to the event when the user has not released the left mouse button and the horizontal and vertical coordinates corresponding to the pressing operation are different from the horizontal and vertical coordinates corresponding to the moving operation, thereby improving the user experience during human-computer interaction.

[0072] According to the first aspect, or any implementation method of the first aspect above, after the electronic device generates a target event corresponding to the mouse, it may also include: the electronic device determines the current rectangular range corresponding to the target event when it does not respond to the target event; and draws a shadow within the current rectangular range.

[0073] The current rectangular range may be a rectangular range corresponding to a motion track formed after the user moves the mouse.

[0074] Specifically, the current rectangular range may be a moving rectangular range of the mouse.

[0075] In this way, the electronic device can draw the shadow of the moving rectangle when it does not respond to the mouse click and move event, thereby prompting the user to select multiple interactive objects in the shadow after releasing the left mouse button at the current position, thereby improving the user experience during human-computer interaction.

[0076] According to the first aspect, or any implementation of the first aspect above, the electronic device determines the current rectangular range corresponding to the target event, which may include: the electronic device uses the first target data and the ninth target data as the second horizontal coordinates, respectively; and uses the second target data and the tenth target data as the second vertical coordinates, respectively; the electronic device determines the coordinate data of four second target points based on each second horizontal coordinate and each second vertical coordinate, and determines the current rectangular range corresponding to the target event based on the coordinate data of each second target point.

[0077] The second target point may be a point in the motion trajectory of the external input device.

[0078] Specifically, the second target point may be four vertices constituting the current rectangular range.

[0079] In this way, when the external input device is a mouse, the electronic device can determine the coordinate data of four points according to the horizontal and vertical coordinates corresponding to the pressing operation and the horizontal and vertical coordinates corresponding to the moving operation, thereby determining a rectangular range according to the coordinate data of the four points, thereby improving the accuracy of shadow drawing.

[0080] In a second aspect, an embodiment of the present application provides an electronic device. The electronic device includes: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and when the computer programs are executed by the one or more processors, the electronic device executes the human-computer interaction method in the first aspect and any one of the first aspects.

[0081] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.

[0082] In a third aspect, an embodiment of the present application provides a computer-readable storage medium. The computer-readable storage medium includes a computer program, and when the computer program is executed on an electronic device, the electronic device executes the human-computer interaction method of the first aspect and any one of the first aspects.

[0083] The third aspect and any implementation of the third aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the third aspect and any implementation of the third aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.

[0084] In a fourth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed, enables a computer to execute a human-computer interaction method as in the first aspect or any one of the first aspects.

[0085] The fourth aspect and any implementation of the fourth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fourth aspect and any implementation of the fourth aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.

[0086] In a fifth aspect, the present application provides a chip, the chip comprising a processing circuit and a transceiver pin, wherein the transceiver pin and the processing circuit communicate with each other through an internal connection path, and the processing circuit executes the human-computer interaction method as in the first aspect or any one of the first aspects to control the receiving pin to receive a signal, and to control the sending pin to send a signal.

[0087] The fifth aspect and any implementation of the fifth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fifth aspect and any implementation of the fifth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 is a schematic diagram of the hardware structure of an electronic device shown as an example;

[0089] Figure 2 A schematic diagram of the software structure of an electronic device is shown as an example;

[0090] Figure 3 An application scenario is shown as an example;

[0091] Figure 4a-4c Different human-computer interaction scenarios are shown as examples;

[0092] Figure 5a-5b A schematic diagram of module interaction of a human-computer interaction method is shown as an example;

[0093] Figure 6 A schematic diagram of module interaction of a human-computer interaction method is shown as an example;

[0094] Figure 7a-7e A schematic diagram of a user interface for human-computer interaction is shown as an example;

[0095] Figure 8-Figure 9 A schematic diagram of module interaction of another human-computer interaction method is shown as an example;

[0096] Fig.10a is a schematic diagram of another user interface for human-computer interaction shown as an example;

[0097] Figure 10b-10e is a schematic diagram showing an exemplary motion trajectory of a stylus;

[0098] Fig.10f The figure is a schematic diagram of another user interface for human-computer interaction. DETAILED DESCRIPTION

[0099] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0100] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0101] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects rather than to describe a specific order of objects. For example, a first target object and a second target object are used to distinguish different target objects rather than to describe a specific order of target objects.

[0102] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0103] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more than two. For example, multiple processing units refer to two or more processing units; multiple systems refer to two or more systems.

[0104] The human-computer interaction method provided in the embodiment of the present application can be applied to electronic devices with touch screens. Optionally, the electronic devices in the embodiment of the present application can be mobile phones with touch screens, sports cameras (GoPro), digital cameras, tablet computers, handheld computers, vehicle-mounted devices, ultra-mobile personal computers (UMPC), netbooks, cellular phones, personal digital assistants (PDA), augmented reality (AR) / virtual reality (VR) devices, etc., and the embodiment of the present application does not impose any special restrictions on the specific form of the electronic devices.

[0105] like Figure 1 The figure shows a schematic diagram of the structure of the electronic device 100. Optionally, the electronic device 100 may be a terminal, which may also be referred to as a terminal device, and the terminal may be a device with a touch screen such as a cellular phone or a tablet computer (pad), which is not limited in the present application.

[0106] It should be understood that Figure 1 The electronic device 100 shown is only one example of an electronic device, and the electronic device 100 may have more or fewer components than shown in the figure, may combine two or more components, or may have a different configuration of components. Figure 1 The various components shown in the EMBODIMENTS 2000 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0107] The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor, a gyroscope sensor, an acceleration sensor, a temperature sensor, a motion sensor, an air pressure sensor, a magnetic sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0108] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0109] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0110] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory.

[0111] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0112] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0113] The mobile communication module 150 may provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc.

[0114] The wireless communication module 160 can provide wireless communication solutions for application in the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication technology (NFC), infrared technology (IR), etc.

[0115] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, so that electronic device 100 can communicate with the network and other devices through wireless communication technology.

[0116] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0117] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0118] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0119] The ISP is used to process the data fed back by the camera 193. The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0120] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement a data storage function.

[0121] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121, such as enabling the electronic device 100 to implement the human-computer interaction method in the embodiment of the present application. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0122] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0123] The pressure sensor is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor can be arranged on the display screen 194. The electronic device 100 can also calculate the touch position according to the detection signal of the pressure sensor.

[0124] The touch sensor is also called a "touch panel". The touch sensor can be set on the display screen 194. The touch sensor and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor is used to detect a touch operation on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event.

[0125] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.

[0126] Figure 2 It is a software structure block diagram of the electronic device 100 according to an embodiment of the present application.

[0127] The layered architecture of the electronic device 100 divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime (Android Runtime) and the system library, the hardware abstraction layer (HAL) and the kernel layer.

[0128] The application layer can include a series of application packages.

[0129] like Figure 2 As shown, the application package may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0130] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer, including various components and services to support developers' Android development. The application framework layer includes some predefined functions.

[0131] like Figure 2 As shown, the application framework layer may include a window manager, a content provider, a view system, a resource manager, an activity manager, a notification manager, an interaction optimization service, etc.

[0132] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0133] The window manager can provide window management services (Window Manager Service, WMS).

[0134] In an embodiment of the present application, the window management service can be used to send received input events to the interaction optimization service, and can also be used to receive events fed back by the interaction optimization service and send them to the application.

[0135] The interactive optimization service can be used to receive input events sent by WMS, process the input events, and generate new events. The interactive optimization service can also be used to send the generated new events to WMS.

[0136] Content providers are used to store and retrieve data and make it accessible to applications. The data can include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0137] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0138] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0139] The activity manager is responsible for managing activities, as well as starting, switching, and scheduling components in the system and managing and scheduling applications.

[0140] The activity manager can provide activity management services (Activity Manager Service, AMS).

[0141] In the embodiment of the present application, the activity management service may be used to send the received input event to the window management service.

[0142] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0143] Android Runtime includes core libraries and virtual machines. Android Runtime is responsible for scheduling and management of the Android system.

[0144] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0145] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0146] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0147] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0148] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0149] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0150] A 2D graphics engine is a drawing engine for 2D drawings.

[0151] The HAL layer is an interface layer between the operating system kernel and the hardware circuit. The HAL layer includes but is not limited to: a camera HAL module and an audio HAL module. The camera HAL module is used to process the image stream, and the audio HAL module is used to process the audio stream (for example, noise reduction, directional enhancement, etc.).

[0152] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, audio driver, touch screen driver, mouse driver, etc.

[0153] The hardware includes at least a processor, a display screen, a mouse, a stylus, etc. It is understandable that Figure 2 The layers in the software structure shown and the components included in each layer do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer layers than shown, and each layer may include more or fewer components, which is not limited in the present application.

[0154] It is understandable that, in order to implement the human-computer interaction method in the embodiment of the present application, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiment disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of the present application.

[0155] Figure 3 FIG. 1 is a schematic diagram of an exemplary application scenario. Figure 3 As shown in (1), the application scenario includes a tablet computer and a stylus. Exemplarily, after the tablet computer and the stylus are connected, the user can control the tablet computer through the stylus. It is understandable that the tablet computer and the stylus can be connected wirelessly, such as Bluetooth connection or other wireless connection methods, which is not limited in this application.

[0156] like Figure 3 As shown in (2), the application scenario includes a tablet computer and a mouse. Exemplarily, after the tablet computer is connected to the mouse, the user can control the tablet computer through the mouse. It is understandable that the tablet computer and the mouse can be connected wirelessly, such as Bluetooth connection, or other wireless connection methods, which is not limited in this application.

[0157] It should be noted that the human-computer interaction method in the embodiment of the present application can also be applied to interaction scenarios of other electronic devices. For example, the human-computer interaction method in the embodiment of the present application can also be applied to electronic devices with touch screens such as mobile phones, wearable devices, notebooks, and televisions. Figure 3 The device types shown are only illustrative examples and are not limited in this application.

[0158] In the embodiment of the present application, the user can use an external input device (such as Figure 3 stylus or mouse) to an electronic device (such as Figure 3 Multiple selections of interactive objects in the tablet computer (as shown) are performed to further process the selected multiple interactive objects.

[0159] Figure 4a-4c FIG. 1 is a schematic diagram of a scenario in which multiple selections of interactive objects are exemplified. Figure 4aAs shown, the interactive object may be a document file. The user may read, modify, etc. the document file in the "Document" application through the "Document" application of the tablet computer. The user may also select multiple documents in the "Document" application at the same time to move, delete, rename, etc. the selected multiple document files at the same time.

[0160] like Figure 4b As shown, the interaction object may be a chat record. The user may communicate and interact with friends through the "WeChat" application in the tablet computer. The user may also select multiple chat records in the "WeChat" application at the same time to forward, delete, etc. the selected multiple chat records at the same time.

[0161] like Figure 4c As shown, the interactive object may be a calendar date. The user may view dates, schedules, etc. through the "Calendar" application of the tablet computer. The user may also select multiple dates in the "Calendar" application at the same time to mark the selected multiple dates, add schedules, etc. at the same time.

[0162] It should be noted that the multiple selection of interactive objects in the human-computer interaction method of the embodiment of the present application can also be applied to the interactive scenarios of other applications. For example, the multiple selection of interactive objects in the human-computer interaction method of the embodiment of the present application can also be multiple selection of interactive objects such as picture files, film and television files, etc. Figure 4a-4c The interactive object multiple selection scenario shown is only an illustrative example and is not limited in this application.

[0163] For current electronic devices, when a user uses a stylus to select multiple interactive objects in the electronic device, the user needs to first click and long press the touch screen with the stylus. After the electronic device responds to the user's operation and enters the multiple selection mode, the user can then slide the stylus to select multiple interactive objects. Correspondingly, when a user uses a mouse to select multiple interactive objects in the electronic device, the user needs to first long press the left mouse button. After the electronic device responds to the user's operation and enters the multiple selection mode, the user can select multiple interactive objects by moving the mouse while holding down the left mouse button. As a result, the above-mentioned interactive method of multiple selection of interactive objects is relatively complicated and does not conform to user habits, resulting in a poor user experience.

[0164] In order to solve the above technical problems, an embodiment of the present application provides a human-computer interaction method. In this method, the electronic device can respond to the user's operation on the external input device (such as pressing operation, moving operation and lifting operation), and save the coordinate data corresponding to each operation into a data group. Thus, when the user lifts the external input device, the electronic device can identify the circle selection event according to the coordinate data in the data group, so as to enter the multi-selection mode when it is identified as a circle selection event, and then after entering the multi-selection mode, the multi-selection range is determined according to the coordinate data in the data group, and then the interactive object within the multi-selection range is selected to realize the multi-selection of the interactive object. In this way, the electronic device can simplify the interaction mode between the user and the electronic device, make the human-computer interaction more in line with the user's habits, and thus improve the user experience.

[0165] The following uses "the external input device is a mouse" as an example to explain the specific process of the human-computer interaction method of the embodiment of the present application. Figure 5a When a user operates an electronic device with a mouse, the electronic device can respond to the user operation through the mouse driver of the kernel layer and generate an input event. After the mouse driver of the kernel layer generates an input event in response to the user operation, it can send the input event to the WMS of the system framework layer.

[0166] In the embodiment of the present application, the user operation may include the operation of the user clicking the left button of the mouse, the operation of the user moving the mouse, and the operation of the user releasing the mouse. It is understandable that the user moving the mouse may be the user clicking the left button of the mouse and moving the mouse at the same time.

[0167] It should be noted that the embodiment of the present application only illustrates key steps by way of example, and does not illustrate in detail all the steps of "the mouse driver sends the input event to the WMS". For example, after generating an input event, the mouse driver of the kernel layer may send the input event to the input dispatcher. The input dispatcher processes the input event and sends the processed input event to the AMS, so as to pass the event to the WMS through the AMS.

[0168] Continue to refer to Figure 5a After receiving the input event, the WMS may send the input event to the interactive optimization service. After receiving the input event, the interactive optimization service identifies the input event.

[0169] If the user operation corresponding to the input event is the user clicking the left mouse button, the input event received by the interaction optimization service is a mouse click event. When the interaction optimization service determines that the input event is a mouse click event, it can start recording the trajectory coordinates of the mouse operation.

[0170] Exemplarily, the interaction optimization service may identify the input event as a mouse click event based on the parameters MotionEvent#isFromSource(SOURCE_MOUSE) and MotionEvent#ACTION_DOWN in the input event.

[0171] If the user operation corresponding to the input event is the user moving the mouse (that is, the user operation is the user clicking the left mouse button and moving the mouse at the same time), the input event received by the interactive optimization service is a mouse moving event. When the interactive optimization service determines that the input event is a mouse moving event, it can record the track coordinates of the mouse operation and generate a mouse click moving event based on the recorded track coordinates of the mouse operation.

[0172] Exemplarily, the interaction optimization service may identify the input event as a mouse movement event according to the parameters MotionEvent#isFromSource(SOURCE_MOUSE) and MotionEvent#ACTION_MOVE in the input event.

[0173] It is understandable that, when the user moves the mouse, the interaction optimization service may receive multiple mouse movement events.

[0174] Continue to refer to Figure 5a The interactive optimization service can also send the mouse click and move event to the WMS, and the WMS sends the mouse click and move event to the application. After receiving the mouse click and move event, the application can respond to the mouse click and move event or not.

[0175] It should be noted that the embodiments of the present application do not limit the specific implementation method of "application responding to mouse click movement event".

[0176] If the user operation corresponding to the input event is the user releasing the left mouse button, the input event received by the interactive optimization service is a mouse release event. When the interactive optimization service determines that the input event is a mouse release event, it can record the track coordinates of the mouse operation and generate a mouse selection event based on the recorded track coordinates of the mouse operation.

[0177] Exemplarily, the interaction optimization service may identify the input event as a mouse release event based on parameters MotionEvent#isFromSource(SOURCE_MOUSE) and MotionEvent#ACTION_UP in the input event.

[0178] It can be understood that the operation of the user releasing the mouse may be an operation after the user clicks the left button of the mouse, or an operation after the user moves the mouse.

[0179] Continue to refer to Figure 5a The interactive optimization service can also send the mouse circle selection event to the WMS, which then sends the mouse circle selection event to the application. After receiving the mouse circle selection event, the application can respond to the mouse circle selection event to enter the multi-selection mode and mark the interactive objects in the rectangular area corresponding to the mouse circle selection event as selected to achieve multi-selection of interactive objects.

[0180] The following uses "the external input device is a stylus" as an example to explain the specific process of the human-computer interaction method of the embodiment of the present application. Figure 5b When a user operates an electronic device with a stylus, the electronic device can respond to the user operation through the touch screen driver of the kernel layer and generate an input event. After the touch screen driver of the kernel layer generates an input event in response to the user operation, it can send the input event to the WMS of the system framework layer.

[0181] In the embodiment of the present application, the user operation may include an operation of the user touching the screen, an operation of the user moving the stylus, and an operation of the user lifting the stylus. It is understandable that the operation of the user touching the screen may be an operation of the user touching the screen with the stylus. The user moving the stylus may be an operation of the user moving the stylus while touching the screen with the stylus.

[0182] It should be noted that the embodiment of the present application only illustrates key steps by way of example, and does not illustrate in detail all the steps of "the touch screen driver sends the input event to the WMS". For example, after generating an input event, the touch screen driver of the kernel layer may send the input event to the input distributor. The input distributor processes the input event and sends the processed input event to the AMS, so as to pass the event to the WMS through the AMS.

[0183] Continue to refer to Figure 5b After receiving the input event, the WMS may send the input event to the interactive optimization service. After receiving the input event, the interactive optimization service identifies the input event.

[0184] If the user operation corresponding to the input event is a user touch screen operation, the input event received by the interaction optimization service is a stylus click event. When the interaction optimization service determines that the input event is a stylus click event, it can start recording the trajectory coordinates of the stylus operation.

[0185] Exemplarily, the interaction optimization service may identify the input event as a stylus click event based on the parameter MotionEvent#isFromSource(SOURCE_STYLUS) and the parameter MotionEvent#ACTION_DOWN in the input event.

[0186] If the user operation corresponding to the input event is the user moving the stylus (that is, the user operation is the user touching the screen with the stylus and moving the stylus), the input event received by the interaction optimization service is a stylus movement event. When the interaction optimization service determines that the input event is a stylus movement event, it can record the trajectory coordinates of the stylus operation.

[0187] Exemplarily, the interaction optimization service may identify the input event as a stylus movement event according to the parameters MotionEvent#isFromSource(SOURCE_STYLUS) and MotionEvent#ACTION_MOVE in the input event.

[0188] It is understandable that, when the user moves the stylus, the interaction optimization service may receive multiple stylus movement events.

[0189] If the user operation corresponding to the input event is the user lifting the stylus, the input event received by the interactive optimization service is a stylus lift event. When the interactive optimization service determines that the input event is a stylus lift event, it can record the trajectory coordinates of the stylus operation and generate a stylus circle event based on the recorded trajectory coordinates of the stylus operation.

[0190] Exemplarily, the interaction optimization service may identify the input event as a stylus up event based on parameters MotionEvent#isFromSource(SOURCE_STYLUS) and MotionEvent#ACTION_UP in the input event.

[0191] It can be understood that the operation of the user lifting the stylus pen may be an operation after the user touches the screen, or may be an operation after the user moves the stylus pen.

[0192] Continue to refer to Figure 5b The interactive optimization service can also send the stylus circle selection event to the WMS, and the WMS sends the stylus circle selection event to the application. After receiving the stylus circle selection event, the application can respond to the stylus circle selection event to enter the multi-selection mode, and mark the interactive objects in the rectangular area corresponding to the stylus circle selection event as selected, so as to achieve multiple selection of interactive objects.

[0193] The following uses a specific example to explain the "human-computer interaction method in the scenario where the external input device is a mouse". Figure 6 FIG. 1 is a schematic diagram showing the interaction flow of each module. Figure 6 , when the external input device is a mouse, the human-computer interaction method specifically includes:

[0194] S101. The mouse driver sends a mouse click event to the WMS. After receiving the mouse click event, the WMS sends the mouse click event to the application and the interaction optimization service.

[0195] After the user clicks the mouse, the mouse driver generates a mouse click event in response to the user's mouse click operation, and sends the mouse click event to the WMS. After receiving the mouse click event, the WMS can send the mouse click event to the application and the interaction optimization service respectively.

[0196] It should be noted that the embodiment of the present application only illustrates key steps by way of example, and does not illustrate in detail all the steps of “the mouse driver sends a mouse click event to the WMS”.

[0197] After receiving the mouse click event, the application may respond to the mouse click event or not, which is not limited in this application. It should be noted that the specific implementation method of "the application responds to the mouse click event" is not limited in this application embodiment.

[0198] It is understandable that before the mouse driver sends the mouse click event to the WMS, the application has been started, and after the application is started, it can send a request to initialize the interface to the AMS, and the AMS sends a request to load the layout to the WMS according to the request to initialize the interface, and the WMS sends an interface drawing request to the application interface (View) according to the request to load the layout, and the View completes the initialization of the interface. For the relevant processing flow of the above application interface initialization, please refer to the existing technology, which will not be repeated here.

[0199] S102: After receiving the mouse click event sent by the WMS, the interactive optimization service determines the mouse click coordinates according to the mouse click event, and records the mouse click coordinates into a mouse coordinate data group.

[0200] The mouse click coordinates may be the coordinates of the position of the mouse pointer when the user clicks the left button of the mouse. It is understandable that the mouse click event may include parameters corresponding to the mouse click coordinates.

[0201] For example, Figure 7aAs shown, in the "Document" application of the electronic device, the user can move the mouse pointer 11 to point A and click the left mouse button at point A. The electronic device responds to the user's operation of clicking the left mouse button, and the mouse driver generates a mouse click event, and sends the mouse click event from the WMS to the interactive optimization service. In this way, after receiving the mouse click event, the interactive optimization service can determine the coordinate data (X1, Y1) of point A as the mouse click coordinates, and record the coordinate data of point A in the mouse coordinate data group.

[0202] It should be noted that the user clicking the left mouse button can also be an operation of the user in other applications, such as gallery applications, audio and video applications, calendar applications, social applications, etc. Figure 7a The "document" application shown is only an illustrative example and is not limited in this application. Figure 7a-7e The interface shown schematically only shows part of the display interface in the "Documents" application.

[0203] The mouse coordinate data set may be a data set for recording all track coordinate data during the mouse movement process. It is understandable that the mouse click coordinates may be the first set of data in the mouse coordinate data set.

[0204] After receiving the mouse click event, the interactive optimization service determines the mouse click coordinates according to the mouse click event, starts recording the trajectory coordinate data during the mouse movement, and records the mouse click coordinates in the mouse coordinate data group.

[0205] S103: The mouse driver sends at least one mouse movement event to the WMS. After receiving the mouse movement event, the WMS sends the mouse movement event to the interaction optimization service.

[0206] The user moves the mouse while clicking the left mouse button. The mouse driver generates a mouse move event in response to the user operation and sends the mouse move event to the WMS.

[0207] It is understandable that during the mouse movement process, the mouse driver can generate one or more mouse movement events. For example, when the coordinate data of the mouse pointer changes, the mouse driver can generate a mouse movement event, or when a preset time interval arrives, the mouse driver can generate a mouse movement event, which is not limited in this application.

[0208] Specifically, after the mouse driver generates a mouse motion event, it can send it to the WMS. That is, the mouse driver can send the generated multiple mouse motion events to the WMS in sequence. Similarly, after the WMS receives the mouse motion event, it can send it to the interactive optimization service. That is, the WMS sends the received multiple mouse motion events to the interactive optimization service in sequence.

[0209] It should be noted that this embodiment does not limit the execution order of step S102 and step S103.

[0210] S104. After receiving each mouse movement event, the interactive optimization service determines the mouse movement coordinates corresponding to each mouse movement event according to each mouse movement event, and records each mouse movement coordinate in a mouse coordinate data group in sequence.

[0211] The mouse movement coordinates may be the position coordinates obtained when the mouse pointer moves when the user moves the mouse. It is understandable that the mouse movement event may include parameters corresponding to the mouse movement coordinates.

[0212] After receiving each mouse movement event, the interactive optimization service can determine the mouse movement coordinates corresponding to the mouse movement event according to the mouse movement event, and record the mouse movement coordinates into the mouse coordinate data group. Thus, the interactive optimization service can record all the trajectory coordinate data of the mouse movement during the mouse movement process into the mouse coordinate data group.

[0213] For example, Figure 7b As shown, the user can click the left mouse button and move the mouse in the direction of arrow 12. When the mouse pointer 11 moves to point B, the mouse driver can respond to the user's operation of moving the mouse to point B, generate a mouse move event, and send the mouse move event to the interactive optimization service through the WMS. In this way, after receiving the mouse move event, the interactive optimization service can determine the coordinate data (X2, Y2) of point B as the mouse move coordinates, and record the coordinate data of point B in the mouse coordinate data group.

[0214] It should be noted that the mouse driver generates a mouse movement event when the mouse pointer 11 moves to point B, which is only an illustrative example. It is understandable that in the process of the mouse pointer 11 moving to point B, the mouse driver can respond to the movement of the mouse pointer 11 and generate multiple mouse movement events.

[0215] S105. The interactive optimization service generates mouse click movement events corresponding to each mouse movement coordinate according to the mouse click coordinates and each mouse movement coordinate, and sends each mouse click movement event to the application through the WMS in sequence.

[0216] The mouse click move event includes coordinate parameters of the moving rectangle.

[0217] After receiving each mouse movement event, the interactive optimization service can generate a mouse click movement event according to the mouse movement coordinates and mouse click coordinates corresponding to the mouse movement event, and send the mouse click movement event to the application through WMS.

[0218] Specifically, the interactive optimization service may determine a mouse movement rectangle according to the mouse movement coordinates and the mouse click coordinates, and generate a mouse click movement event according to the coordinates of the movement rectangle.

[0219] The moving rectangle may be a rectangle formed during the movement of the mouse. It is understandable that the coordinates of the moving rectangle may be the coordinates of the four vertices of the moving rectangle.

[0220] For example, Figure 7c As shown, when the mouse pointer 11 moves to point B, the interactive optimization service can determine the coordinates of the four points (X1, Y1), (X1, Y2), (X2, Y1), and (X2, Y2) based on the mouse movement coordinates (X2, Y2) and the mouse click coordinates (X1, Y1), thereby determining the moving rectangle 13 based on the coordinates of the four points.

[0221] It can be understood that each mouse movement event can correspond to a mouse movement rectangle.

[0222] S106: When the application does not respond to the mouse click movement event, the application sends an event unprocessed feedback to the WMS, and the WMS sends the event unprocessed feedback to the interactive optimization service.

[0223] Event not handled feedback can be a feedback generated when the application does not respond to the mouse click and move event.

[0224] After receiving the mouse click and move event, the application may not respond to the mouse click and move event. If the application does not respond to the mouse click and move event, a feedback indicating that the event is not processed may be sent to the WMS.

[0225] It is understandable that after receiving the mouse click movement event, the application may also respond to the event. It should be noted that the embodiment of the present application does not limit the specific implementation of "the application responding to the mouse click movement event".

[0226] For example, Figure 7d As shown, in response to the mouse click movement event, the application may mark the icons of the documents 8 and 9 within the moving rectangle as icons 14 with shadows.

[0227] S107: After receiving the unprocessed event feedback sent by the WMS, the interactive optimization service draws the shadow of the moving rectangle according to the coordinate parameters of the moving rectangle.

[0228] For example, Figure 7e As shown, when the application does not respond to the mouse click move event, the interactive optimization service can draw the shadow of the moving rectangle according to the coordinate parameters of the moving rectangle, such as Figure 7e The shaded rectangle 15 in FIG.

[0229] S108: The mouse driver sends a mouse release event to the WMS. After receiving the mouse release event, the WMS sends the mouse release event to the interaction optimization service.

[0230] After the user releases the mouse, the mouse driver generates a mouse release event in response to the user's operation of releasing the mouse, and sends the mouse release event to the WMS. After receiving the mouse release event, the WMS can send the mouse release event to the interaction optimization service.

[0231] It should be noted that this embodiment does not limit the execution order of steps S104-S107 and step S108.

[0232] S109: After receiving the mouse release event, the interactive optimization service determines the mouse release coordinates according to the mouse release event, and records the mouse release coordinates into a mouse coordinate data group.

[0233] The mouse release coordinates may be the coordinates of the position of the mouse pointer when the user releases the left button of the mouse. It is understandable that the mouse release event may include parameters corresponding to the mouse release coordinates.

[0234] After receiving the mouse release event, the interactive optimization service can determine the mouse release coordinates and record them in the mouse coordinate data group. Specifically, the mouse release coordinates are the last set of data in the mouse coordinate data group.

[0235] S110. The interactive optimization service determines a mouse selection event according to the mouse click coordinates and the mouse release coordinates, and sends the mouse selection event to the application through the WMS.

[0236] The mouse selection event includes coordinate parameters of the selection rectangle.

[0237] The interactive optimization service may determine a rectangle according to the mouse click coordinates and the mouse release coordinates, and the rectangle may be a selection rectangle of the mouse.

[0238] For example, Figure 7c As shown, if the user releases the mouse at point B, the coordinates of point B are the coordinates of the mouse release, and the rectangle 13 can be determined according to the coordinates of point A and point B. Therefore, the rectangle 13 is also the circled rectangle of the mouse.

[0239] Specifically, the interactive optimization service can obtain the first set of data and the last set of data in the mouse coordinate data group, and determine the mouse selection rectangle based on the two sets of data. The interactive optimization service can generate a mouse selection event based on the coordinates of the mouse selection rectangle, and send the mouse selection event to the WMS, so that the WMS sends it to the application.

[0240] S111. After receiving the mouse selection event, the application determines the mouse selection rectangle range according to the mouse selection event, and marks the interactive object within the selection rectangle range as selected.

[0241] The selection rectangle of the mouse is the rectangular range formed after the user releases the left mouse button.

[0242] In the embodiment of the present application, all interactive objects within the circled rectangle may be marked as selected.

[0243] For example, Figure 7c As shown, assuming that the user releases the mouse at point B, rectangle 13 is the selection rectangle of the mouse. Although the icon of document 8 is not completely within rectangle 13, document 8 will also be marked as selected by the application.

[0244] The following explains the “human-computer interaction method in a scenario where the external input device is a stylus” through another specific example. Figure 8 FIG. 1 is a schematic diagram showing the interaction flow of each module. Figure 8 In the scenario where the external input device is a stylus, the human-computer interaction method specifically includes:

[0245] S201: The stylus driver sends a stylus click event to the WMS. After receiving the stylus click event, the WMS sends the stylus click event to the application and interaction optimization service.

[0246] After the user touches the screen with a stylus, the stylus driver generates a stylus click event in response to the user operation and sends the stylus click event to the WMS. After receiving the stylus click event, the WMS can send the stylus click event to the application and the interaction optimization service respectively.

[0247] It should be noted that the embodiment of the present application only illustrates key steps by way of example, and does not illustrate in detail all the steps of “the stylus driver sending a stylus click event to the WMS”.

[0248] After receiving the stylus click event, the application may respond to the stylus click event or not, which is not limited in this application. It should be noted that the specific implementation method of "the application responding to the stylus click event" is not limited in this application embodiment.

[0249] It is understandable that before the stylus driver sends the stylus click event to the WMS, the application has been started, and after the application is started, it can send a request to initialize the interface to the AMS, and the AMS sends a request to load the layout to the WMS according to the request to initialize the interface, and the WMS sends an interface drawing request to the application interface (View) according to the request to load the layout, and the View completes the initialization of the interface. For the relevant processing flow of the above application interface initialization, please refer to the existing technology, which will not be repeated here.

[0250] S202: After receiving the stylus click event sent by the WMS, the interactive optimization service determines the stylus click coordinates according to the stylus click event, and records the stylus click coordinates into a stylus coordinate data group.

[0251] The stylus click coordinates may be the coordinates of the position where the stylus touches the screen when the user touches the screen with the stylus. It is understandable that the stylus click event may include parameters corresponding to the stylus click coordinates.

[0252] The stylus coordinate data set may be a data set for recording all trajectory coordinate data during the stylus movement process. It is understandable that the stylus click coordinates may be the first set of data in the stylus coordinate data set.

[0253] After the interactive optimization service determines the stylus click coordinates according to the stylus click event, it can start recording the trajectory coordinate data during the stylus movement and record the stylus click coordinates into the stylus coordinate data group.

[0254] S203: The stylus driver sends at least one stylus movement event to the WMS. After receiving the stylus movement event, the WMS sends the stylus movement event to the interaction optimization service.

[0255] The user moves the stylus while the stylus touches the screen, and the stylus driver generates a stylus movement event in response to the user operation and sends the stylus movement event to the WMS.

[0256] It is understandable that during the stylus movement process, the stylus driver can generate one or more stylus movement events. For example, when the coordinate data of the stylus touching the screen changes, the stylus driver can generate a stylus movement event, or when a preset time interval arrives, the stylus driver can generate a stylus movement event, which is not limited in this application.

[0257] Specifically, after the stylus driver generates a stylus movement event, it can send it to the WMS. That is, the stylus driver can send the generated multiple stylus movement events to the WMS in sequence. Similarly, after the WMS receives the stylus movement event, it can send it to the interactive optimization service. That is, the WMS sends the received multiple stylus movement events to the interactive optimization service in sequence.

[0258] It should be noted that this embodiment does not limit the execution order of step S202 and step S203.

[0259] S204. After receiving each stylus movement event, the interactive optimization service determines the stylus movement coordinates corresponding to each stylus movement event according to each stylus movement event, and records each stylus movement coordinate in a stylus coordinate data group in sequence.

[0260] The stylus movement coordinates may be the position coordinates obtained when the stylus moves on the screen when the user moves the stylus. It is understandable that the stylus movement event may include parameters corresponding to the stylus movement coordinates.

[0261] After receiving each stylus movement event, the interactive optimization service can determine the stylus movement coordinates corresponding to the stylus movement event according to the stylus movement event, and record the stylus movement coordinates into the stylus coordinate data group. Thus, the interactive optimization service can record all trajectory coordinate data of the stylus movement during the stylus movement process into the stylus coordinate data group.

[0262] S205: The stylus driver sends a stylus lift event to the WMS. After receiving the stylus lift event, the WMS sends the stylus lift event to the interactive optimization service.

[0263] After the user lifts the stylus, the stylus driver generates a stylus lift event in response to the user's operation of lifting the stylus, and sends the stylus lift event to the WMS. After receiving the stylus lift event, the WMS may send the stylus lift event to the interaction optimization service.

[0264] It should be noted that this embodiment does not limit the execution order of step S204 and step S205.

[0265] S206: After receiving the stylus lift event, the interactive optimization service determines the stylus lift coordinates according to the stylus lift event, and records the stylus lift coordinates into a stylus coordinate data group.

[0266] The stylus lift coordinates may be the position coordinates of the stylus when the user lifts the stylus and the stylus leaves the screen. It is understandable that the stylus lift event may include parameters corresponding to the stylus lift coordinates.

[0267] After receiving the stylus lift event, the interactive optimization service can determine the stylus lift coordinates and record them in the stylus coordinate data set. Specifically, the stylus lift coordinates are the last set of data in the stylus coordinate data set.

[0268] S207: The interactive optimization service calculates the X-axis coordinate difference X according to each coordinate data in the stylus coordinate data group. d and the Y-axis coordinate difference Y d .

[0269] The interactive optimization service determines the maximum value X in the X-axis coordinate among the coordinate data of the stylus coordinate data set. max and minimum value X min , and the maximum value Y in the Y-axis coordinate max and minimum value Y min .

[0270] For example, it is assumed that the stylus coordinate data set includes (X1, Y1), (X2, Y2), ..., (X n ,Y n ), then X max =max(X1,X2,…,X n );X min =min(X1,X2,…,X n );Y max =max(Y1,Y2,…,Y n );X max =max(Y1,Y2,…,Y n ).

[0271] The interactive optimization service can be based on the maximum value X in the X-axis coordinate max and minimum value X min Calculate the X-axis coordinate difference X d , and according to the maximum value Y in the Y-axis coordinate max and minimum value Y min Calculate the Y axis coordinate difference Y d .

[0272] Specifically, X d =X max -X min , Y d =Y max -Y min .

[0273] S208, interactive optimization service determines the X-axis coordinate difference X d Less than 1 / 5 of the screen's X-axis size, and the Y-axis coordinate difference Y d Is it smaller than 1 / 5 of the screen Y-axis size? If not, execute S209; if so, execute S211.

[0274] The interactive optimization service calculates X d and Y d After that, you can first determine the X-axis size of the screen 屏幕 And the Y axis size Y 屏幕 , and then judge X d With X 屏幕、 Y d With Y 屏幕 The size relationship between them.

[0275] If X d <(1 / 5)*X 屏幕 , and Y d <(1 / 5)*Y 屏幕 , indicating that the stylus pen has a small movement range, and the movement of the stylus pen can be ignored.

[0276] If X d >(1 / 5)*X 屏幕 or Y d >(1 / 5)*Y 屏幕 , or X d >(1 / 5)*X 屏幕 And Y d >(1 / 5)*Y 屏幕 The interactive optimization service can determine the stylus circle selection event based on each coordinate data in the stylus coordinate data group.

[0277] S209: The interactive optimization service determines a stylus circle selection event according to each coordinate data in the stylus coordinate data group, and sends the stylus circle selection event to the application through the WMS.

[0278] The stylus pen circle selection event includes coordinate parameters of the circle selection rectangle.

[0279] The difference in X-axis coordinates X d Less than 1 / 5 of the screen's X-axis size, and the Y-axis coordinate difference Y d When the value smaller than 1 / 5 of the screen Y-axis size is not true, the interactive optimization service can determine the stylus circle selection event based on each coordinate data in the stylus coordinate data group.

[0280] Reference Fig. 9 The interactive optimization service determines the process of the stylus circle selection event according to each coordinate data in the stylus coordinate data group, which specifically includes:

[0281] S2091, calculate the coordinates (X) of the center point of the stylus motion trajectory ave ,Y ave ).

[0282] The interactive optimization service can calculate the stylus motion trajectory based on the coordinate data in the stylus coordinate data group.

[0283] After determining the motion trajectory of the stylus pen, the interactive optimization service may first determine the maximum value X in the X-axis coordinates among the coordinate data of the stylus pen coordinate data group. max and minimum value X min , and the maximum value Y in the Y-axis coordinate max and minimum value Y min ; Then according to the maximum value X in the X-axis coordinate max and minimum value X min , and the maximum value Y in the Y-axis coordinate max and minimum value Y min , calculate the coordinates of the center point (X ave ,Y ave ).

[0284] In the present embodiment, X ave =(X max +X min ) / 2,Y ave =(Y max +Y min ) / 2.

[0285] For example, Fig.10a As shown, in the "Document" application of the electronic device, the user can touch the screen with a stylus 21. After the user lifts the stylus, the interactive optimization service can determine the stylus motion trajectory 22 according to the coordinate data in the stylus coordinate data group. Fig.10b As shown, the interactive optimization service can calculate the coordinates (X ave ,Y ave ).

[0286] It should be noted that the user touching the screen with the stylus pen can also be an operation of the user in other applications, such as gallery applications, audio and video applications, calendar applications, social applications, etc. Fig.10a The "document" application shown is only an illustrative example and is not limited in this application. Fig.10a The interface shown schematically only shows part of the display interface in the "Documents" application. Figure 10b-10e The interface shown only schematically shows the movement trajectory of the stylus pen, and does not show the user interface of the electronic device.

[0287] S2092: Determine the coordinates of the first point (X ave ,Y a ), (X ave ,Y b ), (X a ,Y ave ), (X b ,Y ave ).

[0288] The first point may be a point in the motion trajectory of the stylus.

[0289] In the embodiment of the present application, the number of first points in the stylus pen motion trajectory may be four, that is, there may be four first points in the stylus pen motion trajectory.

[0290] Specifically, the interactive optimization service can first determine the horizontal coordinate as X in the stylus coordinate data. ave It can be understood that the horizontal axis is X ave There can be multiple points. Then, the interactive optimization service is used in multiple horizontal axes with X ave Among the points, select the point with the largest ordinate (that is, the point with coordinate (X ave ,Y a ) and the point with the smallest ordinate (that is, the point with coordinate (X ave ,Y b ) point).

[0291] Correspondingly, in the stylus coordinate data, first determine the ordinate as Y ave It can be understood that the ordinate is Y ave There can be multiple points. Then, in multiple ordinates with Y ave Among the points, select the point with the largest horizontal coordinate (that is, the point with coordinate (X a ,Y ave ) and the point with the smallest horizontal coordinate (that is, the point with coordinate (X b ,Y ave ) point).

[0292] For example, Fig.10b As shown, in the stylus motion trajectory, the coordinates of P1, P2, P3, and P4 can be determined according to the coordinates of the center point, and the coordinates of P1, P2, P3, and P4 are determined as the coordinates of the first point. Among them, the coordinates of P1 are (X a ,Y ave ), the coordinates of P2 are (X ave ,Y b ), the coordinates of P3 are (X b ,Y ave ), the coordinates of P4 are (X ave ,Ya ).

[0293] S2093: Divide the motion trajectory of the stylus pen into regions according to the first point.

[0294] In an embodiment of the present application, the interactive optimization service may divide the stylus motion trajectory into four areas according to the four first points.

[0295] For example, Fig.10b As shown, in the stylus pen motion trajectory, the stylus pen motion trajectory can be divided into area 23, area 24, area 25, and area 26 according to P1, P2, P3, and P4.

[0296] S2094: Determine the maximum value X in the X-axis coordinates in the stylus coordinate data. max and minimum value X min , and the maximum value Y in the Y-axis coordinate max and minimum value Y min The corresponding second point respectively, and the area in the stylus motion trajectory where the second point is located.

[0297] The second point may be another point in the stylus motion trajectory.

[0298] In the embodiment of the present application, the number of second points in the stylus pen motion trajectory may be four, that is, there may be four second points in the stylus pen motion trajectory.

[0299] Specifically, the interactive optimization service can store the stylus coordinate data, where the horizontal coordinate is X max The point with the horizontal coordinate X min The point with the vertical coordinate as Y max The point with the vertical coordinate as Y min The point is determined as the second point.

[0300] For example, Fig.10c As shown, in the stylus motion trajectory, Q1, Q2, Q3, Q4 can be determined, and Q1, Q2, Q3, Q4 are determined as the second point. Among them, the coordinates of Q1 are (X min ,Y c ), the coordinates of Q2 are (X d ,Y min ), the coordinates of Q3 are (X max ,Y d ), the coordinates of Q4 are (X c ,Y max ).

[0301] like Fig.10cAs shown, the area where Q1 and Q2 are located is area 24 , and the area where Q3 and Q4 are located is area 26 , that is, the areas in the stylus motion trajectory where the second point is located are area 24 and area 26 .

[0302] S2095. Determine, in the stylus coordinate data, a third point in an area where the second point does not exist; wherein a difference between the abscissa difference and the ordinate difference between the third point and the center point is less than a preset difference threshold.

[0303] The third point may be another point in the stylus motion trajectory.

[0304] The number of third points in the stylus motion trajectory is consistent with the number of regions where the second point does not exist. For example, if there is one region where the second point does not exist, the number of third points is one. If there are two regions where the second point does not exist, the number of third points is two.

[0305] Specifically, assuming that the difference between the abscissa of a point in the stylus motion trajectory and the abscissa of the center point is the abscissa difference, and the difference between the ordinate of the point and the ordinate of the center point is the ordinate difference. When the difference between the abscissa difference and the ordinate difference is less than a preset threshold between the differences, the point can be determined as the third point.

[0306] In the embodiment of the present application, the preset difference threshold may be a number close to 0, that is, when the horizontal coordinate difference and the vertical coordinate difference are close to being equal, the point may be determined as the third point.

[0307] For example, Fig.10d As shown, the regions where the second point does not exist are regions 23 and 24, then R1 may be determined in region 23, R2 may be determined in region 25, and R1 and R2 may be determined as the third point.

[0308] The coordinates of R1 are (X e ,Y e ), and |X e -X ave ∣-∣Y e -Y ave ∣≈0. The coordinates of R2 are (X f ,Y f ), and ∣X f -X ave ∣-∣Y f -Y ave ∣≈0.

[0309] S2096: Determine coordinate parameters of a circle selection rectangle in the stylus circle selection event according to the first point, the second point, and the third point.

[0310] For example, Fig.10eAs shown, the interactive optimization service can determine multiple rectangles based on the first points P1, P2, P3, P4, the second points Q1, Q2, Q3, Q4, and the third points R1, R2. After determining the multiple rectangles, the interactive optimization service can also determine the outer boundaries 27 of the multiple rectangles.

[0311] The interactive optimization service may also determine the area formed by the outer boundaries 27 of the multiple rectangles as a circle selection rectangle, and determine the coordinate parameters of the circle selection rectangle in the stylus circle selection event according to the vertex coordinates of each rectangle.

[0312] S210 . After receiving the stylus pen circle selection event, the application determines the circle selection rectangle range of the stylus pen according to the stylus pen circle selection event, and marks the data within the circle selection rectangle range as selected.

[0313] The selection rectangle of the stylus pen, that is, the rectangular range formed after the user lifts the stylus pen.

[0314] In the embodiment of the present application, all interactive objects within the circled rectangle may be marked as selected.

[0315] For example, Fig.10f As shown, after receiving the stylus circle selection event, the application can mark all interactive objects within the outer boundaries 27 of the multiple rectangles, such as Document 3, Document 4, Document 5, Document 8, Document 9, Document 10 and Document 11, as selected.

[0316] S211: Ignore the operation of the stylus pen.

[0317] If X d <(1 / 5)*X 屏幕 , and Y d <(1 / 5)*Y 屏幕 , indicating that the moving range of the stylus pen is small, and the user's operation of moving the stylus pen can be ignored.

[0318] This embodiment further provides a computer storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the human-computer interaction method in the above-mentioned embodiment.

[0319] This embodiment further provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the above-mentioned related steps to implement the human-computer interaction method in the above-mentioned embodiment.

[0320] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer execution instructions, and when the device is running, the processor can execute the computer execution instructions stored in the memory so that the chip executes the human-computer interaction method in the above-mentioned method embodiments.

[0321] Among them, the electronic device (such as a mobile phone, etc.), computer storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be repeated here.

[0322] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0323] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0324] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A human-computer interaction method, characterized in that: Applied to an electronic device, the electronic device includes a touch screen, and the electronic device is connected to an external input device; the external input device is a stylus; The method comprises: In response to a user's pressing operation, moving operation and lifting operation on the external input device, acquiring coordinate data corresponding to each operation, and saving each coordinate data into a target data group; In the target data group, at least one target data is determined, and at least one data difference is calculated according to each target data; When it is determined that each of the data differences satisfies a multiple selection mode condition, determining a motion trajectory corresponding to the stylus according to the coordinate data in the target data group; Determine the coordinate data of a first point in each coordinate data of the target data group; Dividing the motion trajectory into regions according to the coordinate data of the first point; Determine the coordinate data of the second point in each coordinate data of the target data group; Determine a target area in the motion trajectory according to the coordinate data of the second point; wherein the target area is an area where the second point does not exist; Determine the coordinate data of a third point in the target area from among the coordinate data of the target data group; Determine a target range corresponding to the external input device according to the coordinate data of the first point, the coordinate data of the second point, and the coordinate data of the third point; wherein the target range is the range of the circled rectangle in the circle selection event; All interactive objects within the target range are marked as selected.

2. The method according to claim 1, characterized in that The data difference includes a first data difference and a second data difference; The determining that each of the data differences meets the multiple selection mode condition includes: When the first data difference value is greater than a first threshold value, and / or the second data difference value is greater than a second threshold value, it is determined that each of the data difference values ​​satisfies a multiple selection mode condition.

3. The method according to claim 2, characterized in that Determining at least one target data in the target data group, and calculating at least one data difference according to each target data, comprises: Determine the horizontal coordinate data and the vertical coordinate data of each coordinate data in the target data group; The maximum horizontal coordinate among the horizontal coordinate data of each coordinate data is determined as the fifth target data; Determine the minimum horizontal coordinate among the horizontal coordinate data of each coordinate data as the sixth target data; The maximum ordinate among the ordinate data of each coordinate data is determined as the seventh target data; Determine the minimum ordinate among the ordinate data of each coordinate data as the eighth target data; determining a difference between the fifth target data and the sixth target data as a first data difference; The difference between the seventh target data and the eighth target data is determined as a second data difference.

4. The method according to claim 3, characterized in that Determining the coordinate data of the first point among the coordinate data of the target data group includes: determining a sum value between the fifth target data and the sixth target data as a first data sum value; determining a sum value between the seventh target data and the eighth target data as a second data sum value; Determine the coordinate data of the center point of the motion trajectory according to the first data and value and the second data and value; The coordinate data of the first point is determined in each coordinate data of the target data group according to the center point coordinate data.

5. The method according to claim 4, characterized in that The number of the first point is at least one; The step of determining the coordinate data of the first point in each coordinate data of the target data group according to the center point coordinate data includes: Among the coordinate data of the target data group, the coordinate data having the same abscissa as the abscissa of the center point coordinate data is determined as the first coordinate data; wherein the number of the first coordinate data is at least one; Among the coordinate data of the target data group, the coordinate data having the same ordinate as the ordinate of the center point coordinate data is determined as the second coordinate data; wherein the number of the second coordinate data is at least one; Determine the first coordinate data with the largest ordinate among the first coordinate data as the coordinate data of the first point; Determine the first coordinate data with the smallest ordinate among the first coordinate data as the coordinate data of the first point; Determine the second coordinate data with the largest horizontal coordinate among the second coordinate data as the coordinate data of the first point; The second coordinate data with the smallest horizontal coordinate among the second coordinate data is determined as the coordinate data of the first point.

6. The method according to claim 3, characterized in that The number of the second point is at least one; Determining the coordinate data of the second point in each coordinate data of the target data group includes: The coordinate data corresponding to the fifth target data, the sixth target data, the seventh target data and the eighth target data, respectively, are determined as the coordinate data of the second point.

7. The method according to claim 1, characterized in that The step of determining the target area in the motion trajectory according to the coordinate data of the second point includes: determining, according to the coordinate data of the second point, an area in the motion trajectory where the second point is located; An area in the motion trajectory where the second point does not exist is determined as the target area.

8. The method according to claim 4, characterized in that Determining the coordinate data of the third point in the target area from among the coordinate data of the target data group comprises: Determine third coordinate data in the target area among the coordinate data of the target data group; Determine the difference between the abscissa of the third coordinate data and the abscissa of the center point coordinate data as a first value; Determine the difference between the ordinate of the third coordinate data and the ordinate of the center point coordinate data as a second value; In a case where the difference between the first value and the second value is smaller than a third threshold, the third coordinate data is determined as the coordinate data of the third point.

9. An electronic device, characterized in that: include: one or more processors; Memory; And one or more computer programs, wherein the one or more computer programs are stored in the memory, and when the computer programs are executed by the one or more processors, the electronic device executes the human-computer interaction method as described in any one of claims 1-8.

10. A computer-readable storage medium comprising a computer program, characterized in that: When the computer program runs on an electronic device, the electronic device executes the human-computer interaction method according to any one of claims 1 to 8.

Images