Method for displaying annotations and electronic equipment

Abstract

The invention provides an annotation display method and electronic equipment, and the method comprises the steps: receiving a first user operation in a preset scene, the first user operation being an annotation operation on a first content on a first application interface, and the preset scene comprising screen sharing, homologous screen projection or screen recording; in response to the first user operation, a first annotation handwriting is displayed, the first annotation handwriting is located on a canvas, and a control corresponding to the canvas is injected into a parent control of a target scroll control of the first application interface; in the preset scene, a second user operation is received, and the second user operation is a sliding operation on the first application interface; and in response to a second user operation, controlling the canvas and the first application interface to synchronously roll, so that the first annotation handwriting and the first content synchronously roll. According to the method provided by the invention, anchoring of the annotation note and the annotated content can be realized, and the user experience of using the annotation function is improved.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a method for displaying annotations and an electronic device. Background Art

[0002] With the rise of automated office, more and more users are accustomed to using conference software installed on electronic devices (for example, DingTalk, Tencent Meeting, etc.) for business training, online classes, or solution presentations, etc. In these processes, they often need to annotate the currently displayed text, pictures, or videos, such as highlighting key points and answering questions.

[0003] At present, the annotation function of the conference software cannot anchor the annotation notes with the annotated content. That is, when the user slides the currently annotated page, the annotation handwriting disappears or the annotation handwriting does not slide synchronously with the annotated content, resulting in a poor user experience. Summary of the invention

[0004] The present application provides a method and electronic device for displaying annotations, which can realize anchoring of annotation notes and annotated content, thereby improving the user experience of using the annotation function.

[0005] On the first aspect, the present application provides a method for displaying annotations, which can be executed by an electronic device, or by a component configured in the electronic device (such as a chip, a chip system, etc.), or by a logic module or software that can implement all or part of the functions of the electronic device, and the present application does not limit this.

[0006] Exemplarily, in a preset scenario, a first user operation is received, and the first user operation is an annotation operation on a first content on a first application interface, and the preset scenario includes screen sharing, same-source screen projection, or screen recording; in response to the first user operation, a first batch of annotation handwriting is displayed, and the first batch of annotation handwriting is located on a canvas, and the control corresponding to the canvas is injected into the parent control of the target scroll control of the first application interface; in the preset scenario, a second user operation is received, and the second user operation is a sliding operation on the first application interface; in response to the second user operation, the canvas is controlled to scroll synchronously with the first application interface, so that the first batch of annotation handwriting scrolls synchronously with the first content.

[0007] Optionally, the method also includes: in a preset scenario, receiving a third user operation, which is a zoom operation on the first application interface; in response to the third user operation, controlling the canvas to be zoomed synchronously with the first application interface, so that the first batch of annotations are zoomed synchronously with the first content.

[0008] Optionally, the method also includes: in a preset scenario, receiving a fourth user operation, which is a rotation operation on the first application interface; in response to the fourth user operation, controlling the canvas to rotate synchronously with the first application interface, so that the first batch of annotations rotate synchronously with the first content.

[0009] Since the parent control of the scroll control is used to control the scrolling of the application interface when the electronic device receives a user operation, the present application can realize the synchronous scrolling of the application interface and the canvas when the application interface scrolls by injecting the control corresponding to the canvas used to display the annotation handwriting into the parent control of the scroll control of the application interface. In this way, after the user annotates the first content in the application interface, and then slides the annotated application interface, the synchronous scrolling of the annotation handwriting and the annotated content can be realized, thereby improving the user experience of using the annotation function.

[0010] In combination with the first aspect, in certain implementations of the first aspect, a first batch of annotation traces after sliding synchronously with the first content are displayed.

[0011] In combination with the first aspect, in certain implementations of the first aspect, the canvas is provided by a speech annotation service, which is a service provided by the operating system of the electronic device for providing annotation functions; the method also includes: when entering the preset scene is detected, starting the speech annotation service; injecting the control corresponding to the canvas into the parent control of the target scroll control of the first application interface.

[0012] The changes of the preset scenes in this application can be monitored by monitoring events. For example, when the electronic device is monitored to enter at least one of the screen sharing scene, the same source screen projection scene or the screen recording scene, the speech annotation service is started to provide the annotation function service.

[0013] The method of starting the speech annotation service provided by this application can pull up the speech annotation service after any software is started, without relying on third-party conference software for adaptation, or relying on the system's preset software whitelist, etc. It has good adaptability and more stable functions.

[0014] In combination with the first aspect, in certain implementations of the first aspect, there are multiple scroll controls in the first application interface, and the target scroll control is the largest scroll control in the first application interface; injecting the control corresponding to the canvas into the parent control of the target scroll control of the first application interface includes: injecting the control corresponding to the canvas into the parent control of the largest scroll control of the first application interface.

[0015] In combination with the first aspect, in certain implementations of the first aspect, the target scroll control includes each scroll control of the first application interface; injecting the control corresponding to the canvas into the parent control of the target scroll control in the first application interface includes: injecting the control corresponding to the canvas into the parent control of each scroll control in the first application interface.

[0016] When a canvas control is injected into the parent control corresponding to each scroll control in the first application interface, multiple canvases may be covered on the first application interface. The maximum coverage of each of the multiple canvases should not exceed the range of the first application interface scrolling that can be controlled by the corresponding parent control, and the corresponding parent control refers to the parent control into which the control corresponding to the canvas is injected.

[0017] In combination with the first aspect, in certain implementations of the first aspect, in response to the second user operation, the canvas and the first application interface are scrolled synchronously so that the first batch of annotations slide synchronously with the first content, including: in response to the second user operation, according to the sliding distance of the maximum scroll control, the canvas and the first application interface are controlled to scroll synchronously so that the first batch of annotations slide synchronously with the first content, wherein the ratio of the sliding distance of the canvas to the sliding distance of the scroll control is equal to the ratio of the sliding distance of the first content to the sliding distance of the scroll control.

[0018] The manner of controlling the synchronous scrolling of the canvas and the first application interface according to the sliding distance can further ensure the synchronous scrolling of the annotation pen and the annotated content.

[0019] In combination with the first aspect, in certain implementations of the first aspect, before injecting the control corresponding to the canvas into the parent control of the target scroll control of the first application interface, the method also includes: obtaining the package name and interface name of the application to which the first application interface belongs; based on the package name and the interface name, obtaining the activity instance corresponding to the first application interface, the activity instance is used to search for all controls in the first application interface, including the target scroll control; based on the activity instance, determining the target scroll control of the first application interface.

[0020] The package name of the application is used to identify different applications, and the interface name is used to identify different interfaces in the same application. According to the package name and interface name of the application, a unique application interface can be determined. In other words, one application interface corresponds to a unique package name and interface name of the application.

[0021] In combination with the first aspect, in some implementations of the first aspect, the method further includes: upon detecting that the first application interface is switched to a second application interface, injecting the canvas into a scroll control of the second application interface.

[0022] In other words, in the preset scenario, every time the electronic device detects a change in the interface running in the foreground, the speech annotation service needs to re-traverse all controls in the switched application interface to obtain the target scrolling control, and add the control corresponding to the canvas provided by the speech annotation service to the parent control of the obtained target scrolling control.

[0023] In combination with the first aspect, in some implementations of the first aspect, the method further includes: deleting a control corresponding to the canvas from a parent control of the template scroll control of the first application interface.

[0024] In combination with the first aspect, in certain implementations of the first aspect, after receiving the first user operation, the method further includes: when the first user operation is performed by a stylus, determining that the first user operation is the annotation operation; or, when the first user operation is performed by a finger, determining that the first user operation is not the annotation operation.

[0025] In combination with the first aspect, in certain implementations of the first aspect, after receiving the second user operation, the method includes: when the second user operation is performed by a finger, determining that the second user operation is the sliding operation; or, when the second user operation is performed by a stylus, determining that the second user operation is not the sliding operation.

[0026] In this way, electronic devices can process different user operations differently, further improving the user experience.

[0027] In combination with the first aspect, in some implementations of the first aspect, the method further includes: closing the speech annotation service when it is detected that the preset scene is exited.

[0028] In some possible implementations, the electronic device may be in more than two preset scenarios at the same time. For example, the electronic device is in the screen sharing and screen recording scenarios at the same time, and the electronic device will start the speech annotation service in both scenarios. When the electronic device is in two preset scenarios at the same time, if the speech annotation service needs to be turned off, it is necessary to exit all preset scenarios, that is, exit the screen recording and screen sharing scenarios.

[0029] In a second aspect, the present application provides an electronic device, including a processor coupled to a memory, the memory being used to store code instructions; the processor being used to execute the code instructions so that the electronic device executes the method described in the first aspect or any one of the implementations of the first aspect.

[0030] Optionally, the electronic device further comprises a memory for storing a computer program and / or a configuration file of the logic circuit.

[0031] Optionally, the electronic device further includes a communication interface, and the processor is coupled to the communication interface for inputting and / or outputting signals.

[0032] In a third aspect, the present application provides a processor, comprising: an input circuit, an output circuit, and a processing circuit. The processing circuit is used to receive a signal through the input circuit and transmit a signal through the output circuit, so that the processor executes the method in any possible implementation of the first aspect above.

[0033] In the specific implementation process, the above-mentioned processor can be a chip, the input circuit can be an input pin, the output circuit can be an output pin, and the processing circuit can be a transistor, a gate circuit, a trigger, and various logic circuits. The input signal received by the input circuit can be, for example, but not limited to, received and input by a receiver, and the signal output by the output circuit can be, for example, but not limited to, output to a transmitter and transmitted by the transmitter, and the input circuit and the output circuit can be the same circuit, which is used as an input circuit and an output circuit at different times. This application does not limit the specific implementation of the processor and various circuits.

[0034] In a fourth aspect, the present application provides a computer-readable storage medium, which stores instructions. When the instructions are executed, the computer executes the method described in the first aspect or any implementation of the first aspect.

[0035] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed, enables a computer to execute the method described in the first aspect or any one of the implementations of the first aspect.

[0036] It should be understood that the second to fifth aspects of the present application correspond to the technical solution of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic diagram of the structure of an electronic device;

[0038] Figure 2 It is a software architecture diagram of an electronic device;

[0039] Figure 3 It is a schematic diagram of a scenario applicable to the method provided in the embodiment of the present application;

[0040] Figure 4 is a schematic flow chart of a method for displaying annotations provided in an embodiment of the present application;

[0041] Figure 5 is a schematic diagram of an interface for displaying annotations provided in an embodiment of the present application;

[0042] Figure 6 This is a schematic diagram of an interface for enabling the annotation function provided in an embodiment of the present application;

[0043] Figure 7 is a schematic flow chart of a method for starting a speech annotation service provided in an embodiment of the present application;

[0044] Figure 8 It is a schematic diagram of the interface for starting screen sharing;

[0045] Fig. 9 This is a schematic diagram of the principle of same-source screen projection provided in an embodiment of the present application;

[0046] Fig.10 This is a schematic diagram of the interface for starting same-source screen projection;

[0047] Fig.11 It is a schematic diagram of a processing flow of injecting into a canvas provided in an embodiment of the present application;

[0048] Fig.12 is a schematic flow chart of a method for obtaining an activity instance provided in an embodiment of the present application;

[0049] Fig.13 This is a method of displaying annotation handwriting provided by an embodiment of the present application;

[0050] Fig.14 It is a schematic diagram of the interface sliding process provided by an embodiment of the present application;

[0051] Fig.15 is a schematic flow chart of a method for determining a device type provided in an embodiment of the present application;

[0052] Fig.16 It is a schematic diagram of a method for transmitting annotation handwriting provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0054] To facilitate understanding of the embodiments of the present application, the following description is first made:

[0055] First, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first" and "second" are used to distinguish the same items or similar items with basically the same functions and effects. For example, the first application interface and the second application interface are only used to distinguish different interfaces, and their order is not limited. Those skilled in the art can understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not necessarily limit them to be different.

[0056] Second, in the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship, but it does not exclude the situation where the previous and next associated objects are in an "and" relationship. The specific meaning can be understood in combination with the context. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c; a and b; a and c; b and c; or a and b and c. Where a, b, c can be single or multiple.

[0057] Third, the electronic devices mentioned in the embodiments of the present application may be any type of device, including but not limited to: mobile phones, personal computers (PCs) (such as desktop computers, laptops, PDAs, tablet computers, etc.), wireless terminals in smart homes (such as smart TVs), vehicle-mounted devices, wearable devices, etc.

[0058] The following will be combined Figure 1 Describe in detail the possible hardware structure of the electronic device.

[0059] Figure 1 It is a schematic diagram of the structure of an electronic device. Figure 1As shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a sensor module 180, a button 190, an indicator 192, a camera 193, and a display screen 194, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0060] It is understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device. It should be understood that the electronic device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented by hardware, software, or a combination of software and hardware.

[0061] The processor 110 may include one or more processing units. Different processing units may be independent devices or integrated into one or more processors. The processor 110 may also be provided with a memory for storing instructions and data.

[0062] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and improves the efficiency of the system.

[0063] The processor 110 can perform different operations to achieve different functions by executing instructions. The instructions can be, for example, instructions pre-stored in the memory before the device leaves the factory, or instructions read from the APP after the user installs a new application (APP) during use, which is not limited in the embodiments of the present application.

[0064] The universal serial bus (USB) interface 130 is an interface that complies with the USB standard specification. The USB interface 130 can be used to connect a charger to charge the terminal device, and can also be used to transmit data between the terminal device and peripheral devices. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices.

[0065] The charging management module 140 is used to receive charging input from a charger, which may be a wireless charger or a wired charger. The power management module 141 is used to connect the charging management module 140 to the processor 110 .

[0066] The wireless communication function of the electronic device can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor.

[0067] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. The antenna in the terminal device can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antenna.

[0068] The mobile communication module 150 can provide wireless communication solutions including the second generation wireless telephone technology (2G) / the third generation mobile communication technology (3G) / the fourth generation mobile communication technology (4G) / the fifth generation mobile communication technology (5G) applied to the terminal device. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0069] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), etc. applied to the terminal device. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0070] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, so that electronic device 100 can communicate with the network and other devices through wireless communication technology. For example, when electronic devices are performing screen sharing or same-source screen projection, the source electronic device can send data to the destination electronic device through antenna 1 or antenna 2.

[0071] The electronic device can realize the display function through a graphics processing unit (GPU), a display screen 194, and an application processor. The GPU connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering.

[0072] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In some embodiments, the electronic device may include one or more display screens 194.

[0073] The electronic device can realize the shooting function through image signal processing (ISP), camera 193, video codec, GPU, display screen 194 and application processor.

[0074] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. In some embodiments, the electronic device may include one or more cameras 193.

[0075] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as music and videos can be stored in the external memory card.

[0076] The internal memory 121 can be used to store computer executable program codes, and the executable program codes include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0077] The electronic device can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, and the application processor.

[0078] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0079] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0080] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0081] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to microphone 170C to input the sound signal into microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0082] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130, or a 3.5 mm (millimeter, mm) open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0083] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. The gyroscope sensor 180B can be used to determine the motion posture of the terminal device. The air pressure sensor 180C is used to measure the air pressure. The magnetic sensor 180D includes a Hall sensor. The acceleration sensor 180E can detect the magnitude of the acceleration of the terminal device in all directions (generally three axes). The distance sensor 180F is used to measure the distance. The proximity light sensor 180G can include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The ambient light sensor 180L is used to sense the brightness of the ambient light. The fingerprint sensor 180H is used to collect fingerprints. The temperature sensor 180J is used to detect the temperature. The touch sensor 180K, also known as a "touch device". The bone conduction sensor 180M can obtain a vibration signal.

[0084] The touch sensor 180K may be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, or a "touch screen". In the embodiment of the present application, a capacitive sensing node grid (hereinafter referred to as a capacitive sensor) may be disposed in the touch screen. When the terminal device determines that the value of the capacitance in at least one grid received by the capacitive sensor exceeds the capacitance threshold, it may be determined that a touch operation has occurred; further, the terminal device may determine the touch area corresponding to the touch operation based on the area occupied by at least one grid exceeding the capacitance threshold.

[0085] The key 190 includes a power key (or power key), a volume key, etc. The key 190 can be a mechanical key or a touch key. The electronic device 100 can receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0086] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0087] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power changes, messages, missed calls, notifications, etc.

[0088] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the electronic device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195. The electronic device 100 can support one or more SIM card interfaces. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 100 uses an eSIM, that is, a pre-set SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0089] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. Figure 2 The software structure of the electronic device 100 is exemplified. The layered architecture refers to dividing the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces, which are used to manage the chip downward and provide application programming interface (API) services upward.

[0090] Figure 21 is a schematic diagram of a software architecture of the electronic device 100. Figure 2 As shown, the electronic device can divide the Android system into four layers, which are, from top to bottom, the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.

[0091] The application layer can include a series of application packages. Figure 2 As shown, the application layer may include lock screen applications, desktop applications, etc. Desktop applications may further include calendars, maps, music, short messages, calls, photo galleries, smart homes, etc.

[0092] In the present application, the application layer may also include, for example, conference apps.

[0093] The application framework layer provides API interfaces and programming frameworks for applications in the application layer. The application framework layer includes some predefined functions. Figure 2 As shown, the application framework layer may include activity management services, process management services, window managers, notification management services, and the like.

[0094] The window manager is used to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0095] The activity management service is used to manage the life cycle of each application and the navigation fallback function. It is responsible for the creation of the Android main thread and the maintenance of the life cycle of each application. The activity management service includes a process list, a remote callback interface, a process priority controller, etc.

[0096] The notification management service enables applications to display notification information in the status bar. It can be used to convey notification-type messages, and can stay for a short time and then disappear automatically without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0097] The Android runtime can include core libraries and virtual machines. The Android runtime is responsible for scheduling and management of the Android system.

[0098] The core library can include two parts: one part is the function that needs to be called by the Java language, and the other part is the core library of the Android system.

[0099] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0100] The system library can include multiple functional modules, such as surface manager, media libraries, 3D graphics processing library, 2D graphics engine, etc. The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.

[0101] The kernel layer is a layer between hardware and software. The kernel layer at least includes power management services, sensor drivers, display drivers, camera drivers, audio drivers, etc. The present application embodiment does not impose any restrictions on this.

[0102] In the embodiment of the present application, the core layer also includes a stylus and a touch screen driver.

[0103] like Figure 2 As shown, the system library and kernel layer below the application framework layer can be called the bottom system. The bottom system includes a bottom display system for providing display services, for example, the bottom display system includes a display driver in the kernel layer and a surface manager in the system library.

[0104] Below the underlying system is the hardware, which provides the basis for software operation. Figure 2 As shown, the hardware of the terminal may include, but is not limited to, a power button, a sensor, a display screen, a fingerprint, a camera, etc.

[0105] To facilitate understanding of the method for displaying annotations provided in the embodiment of the present application, the application scenario of the method provided in the embodiment of the present application is described below. It is understandable that the application scenario described in the embodiment of the present application is to more clearly illustrate the technical solution of the embodiment of the present application, and does not constitute a limitation on the technical solution provided in the embodiment of the present application.

[0106] Figure 3 Schematic diagram of a scenario applicable to the method provided in the embodiment of the present application. Figure 3 As shown in (a), the electronic device is sharing a document 1 through a conference APP, and the user uses the annotation function service provided by the conference APP to use a stylus pen to Figure 3 The content 1 in the document 1 shown in (a) is annotated to obtain the annotation handwriting 1, which is displayed Figure 3In the interface shown in (b) of FIG. 1 ; in response to the user's upward sliding operation on the document 1, the annotation handwriting may disappear or the annotation handwriting 1 may not slide synchronously with the content 1, for example, Figure 3 The interface shown in (c) in the figure; or the user Figure 3 When the document 1 shown in (b) is scaled, the annotation handwriting may not be scaled synchronously with the content 1. For example, Figure 3 The interface shown in (d) provides a poor user experience.

[0107] In view of this, the present application provides a method for displaying annotations, by injecting the control corresponding to the canvas used to display annotation handwriting into the parent control of the scroll control of the current annotation interface, so that the annotation handwriting and the annotated content can scroll synchronously, which is conducive to improving the user experience of using the annotation function.

[0108] The method for displaying annotations provided by the present application will be described in detail below in conjunction with specific embodiments.

[0109] It should be understood that the embodiments shown below describe the method from the perspective of electronic devices, but should not constitute any limitation on the execution subject of the method. As long as the computer program that can record the method provided by the embodiment of the present application can be run, the method provided by the embodiment of the present application can be executed. For example, the electronic device can also be replaced by a component (such as a chip, a chip system, etc.) configured in the electronic device, or other functional modules that can call and execute the program, and the embodiment of the present application does not limit this.

[0110] Figure 4 FIG. 4 is a schematic flow chart of a method 400 for displaying annotations provided in an embodiment of the present application. Figure 4 As shown, the method 400 may include S401 to S404. The various steps shown in the method 400 are described in detail below.

[0111] S401: In a preset scenario, a first user operation is received.

[0112] The first user operation is an annotation operation on the first content on the first application interface. The preset scenarios include screen sharing, same-source screen projection, or screen recording.

[0113] Same-source screen projection refers to a source device projecting its screen to at least one destination device, and the content displayed on the source device is completely synchronized with the content displayed on the destination device. When the preset scenario is the same-source screen projection scenario, the electronic device that receives the first user's operation is the source device.

[0114] Screen sharing refers to a screen sharing scenario provided by a conference APP. For example, an electronic device enters a screen sharing scenario in response to a user turning on a screen sharing operation in a conference APP.

[0115] The first application interface is an interface running in the foreground of the electronic device. That is, the activity of the first application interface is the top activity in the electronic device. The first application interface can be the desktop of the electronic device, the interface of the system application, or the interface of a third-party application installed on the electronic device, which is not limited in this application.

[0116] S402: In response to a first user operation, display a first batch of annotation handwritings.

[0117] The first batch of annotations are located on the canvas, and the control corresponding to the canvas is injected into the parent control of the target scrolling control of the first application interface. The scrolling range of the first application interface controlled by the parent control of the target scrolling control includes the first content.

[0118] The first batch of notes being located on the canvas can be understood as the first batch of notes belonging to the canvas, or the canvas includes the first batch of notes, or the first batch of notes being displayed on the canvas.

[0119] The canvas in this application is provided by a speech annotation service, which is a service provided by the operating system of an electronic device for providing an annotation function. In other words, the speech annotation service can provide a system-level annotation function service.

[0120] It is understandable that the number of scroll controls in the first application interface can be one or more. Each scroll control in the first application interface corresponds to a parent control, that is, N scroll controls correspond to N parent controls, where N is a positive integer. The scroll range of the first application interface controlled by each parent control is different.

[0121] Example 1: There is one scroll control in the first application interface, and the target scroll control is the scroll control.

[0122] Example 2: There are multiple scroll controls in the first application interface, and the target scroll controls include all scroll controls in the first application interface. In this case, injecting the canvas into the parent control of the target scroll control in the first application interface can be replaced by injecting the canvas into the parent control of each scroll control in the first application interface.

[0123] When canvas controls are injected into the N parent controls corresponding to the N scroll controls in the first application interface, multiple canvases may be covered on the first application interface. The maximum coverage of each of the multiple canvases should not exceed the range of the scrolling of the first application interface that can be controlled by the corresponding parent control, and the corresponding parent control refers to the parent control into which the control corresponding to the canvas is injected.

[0124] Example 3: There are multiple scroll controls in the first application interface, and the target scroll control is the largest scroll control in the first application interface. In this case, injecting the canvas into the parent control of the target scroll control in the first application interface can be replaced by injecting the canvas into the parent control of the largest scroll control in the first application interface.

[0125] For example, the parent control can be a container, such as a scrollview. When a scroll event arrives, the child controls in the container will receive the scroll event at the same time, thereby achieving synchronous scrolling. Therefore, the present application can achieve synchronous scrolling of the canvas control and the scroll control by injecting the control corresponding to the canvas into the parent control of the scroll control of the first application interface.

[0126] S403: In a preset scenario, a second user operation is received.

[0127] The second user operation is a sliding operation on the first application interface, and the sliding operation may be an up-down sliding operation or a left-right sliding operation on the first application interface.

[0128] S404, in response to the second user operation, controlling the canvas and the first application interface to scroll synchronously, so that the first batch of annotations scroll synchronously with the first content.

[0129] It is understandable that the sliding amplitude of each sliding operation performed by the user may be different, so when the electronic device responds to each sliding operation of the user, the content included in the first application interface may be different.

[0130] For example, in response to the second user operation, the electronic device may display the first batch of annotations after sliding synchronously with the first content. Alternatively, in response to the second user operation, the electronic device displays the second content in the first application interface, but does not display the first content and the first batch of annotations.

[0131] Optionally, the method 400 also includes: in a preset scenario, receiving a third user operation, which is a zoom operation on the first application interface; in response to the third user operation, controlling the canvas to be zoomed synchronously with the first application interface so that the first batch of annotations are zoomed synchronously with the first content.

[0132] Similarly, the zooming amplitude of each zooming operation performed by the user may be different, so when the electronic device responds to each zooming operation performed by the user, the content included in the first application interface may be different.

[0133] Optionally, the method 400 also includes: in a preset scenario, receiving a fourth user operation, which is a rotation operation on the first application interface; in response to the fourth user operation, controlling the canvas to rotate synchronously with the first application interface, so that the first batch of annotations rotate synchronously with the first content.

[0134] Similarly, the rotation angle of each sliding operation performed by the user may be different. Therefore, when the electronic device responds to each rotation operation performed by the user, the directions of the first content included in the first application interface and the first batch of handwriting may be different.

[0135] Since the parent control of the scroll control is used to control the scrolling of the application interface when the electronic device receives a user operation, the present application can realize the synchronous scrolling of the application interface and the canvas when the application interface scrolls by injecting the control corresponding to the canvas used to display the annotation handwriting into the parent control of the scroll control of the application interface. In this way, after the user annotates the first content in the application interface, and then slides the annotated application interface, the synchronous scrolling of the annotation handwriting and the annotated content can be realized, thereby improving the user experience of using the annotation function.

[0136] Below Figure 4 Based on the embodiment shown, combined with Figure 5 A schematic diagram of an interface for displaying annotations provided in an embodiment of the present application is introduced.

[0137] Figure 5 is a schematic diagram of an interface for displaying annotations provided in an embodiment of the present application. Figure 5 The electronic device shown in (a) is in a screen sharing scenario, and the current interface includes content 1, content 2, and an annotation toolbar. In this scenario, the electronic device responds to the user's annotation operation on content 1 and displays the following Figure 5 The interface shown in (b) includes annotation handwriting 1 made on content 1; in this scenario, the electronic device responds to the user's upward sliding operation and displays the following Figure 5 The interface shown in (c) in FIG. 1 includes content 1, content 2, content 3, an annotation toolbar, and annotation handwriting 1 made for content 1. Compared with Figure 5 In the interface shown in (b), the relative positions of content 1 and annotation handwriting 1 have not changed. Figure 5 In the interface shown in (b), all contents except the annotation toolbar are moved upward.

[0138] in, Figure 5 The function entries of the annotation toolbar in the interface shown are: Close, Laser Pen, Paint Brush, Screenshot, Clear, Minimize. Among them, Laser Pen is the default entry and does not support style switching.

[0139] For example, if the user clicks "Brush", Figure 5The interface shown shows the handwriting kit, where the user can select the type of brush and the color of the handwriting. The user clicks "Close" to exit the annotation function; the user clicks "Clear" to clear all handwriting on the canvas. The user clicks "Minimize" to minimize the toolbar. The user clicks "Screenshot", which takes a screenshot of the current content and saves it to the gallery by default.

[0140] It should be understood that Figure 5 The annotation toolbar shown is only an example, and the annotation toolbar may include more or fewer function entries, which is not limited in the present application.

[0141] Figure 5 The application interface shown may be the first application interface in this application, content 1 may be the first content in this application, and annotation note 1 may be the first batch of annotation notes in this application.

[0142] Still taking the scenario of electronic devices entering screen sharing as an example, combined with Figure 6 This section describes how to start the annotation function after an electronic device enters a screen sharing scenario.

[0143] Figure 6 is a schematic diagram of an interface for enabling the annotation function provided in an embodiment of the present application. The electronic device detects that a screen sharing scene has been entered and displays the following Figure 6 The interface shown in (a) above; users can invoke the annotation function provided by the speech annotation service in the following two ways.

[0144] The first possible implementation method is that the user Figure 6 In the upper right corner of the interface shown in (a), the “Speech Annotation” pops up, showing Figure 6 In response to the user clicking the "speech annotation" control, the following is displayed: Figure 6 The interface shown in (c) in .

[0145] The second possible implementation method is that the user double-clicks the stylus Figure 6 The interface shown in (a) in the figure invokes the annotation function and displays the following Figure 6 The interface shown in (c) in .

[0146] Optionally, before S401, the method 400 further includes: when entering a preset scene is detected, starting a speech annotation service; and injecting a control corresponding to the canvas into a parent control of a target scrolling control of the first application interface.

[0147] Optionally, the method 400 further includes: closing the speech annotation service when it is detected that the preset scene is exited.

[0148] Optionally, the method 400 further includes: when it is detected that the preset scene is exited, deleting a control corresponding to the canvas injected under the parent control of the scroll control.

[0149] Optionally, the operation of deleting the control corresponding to the canvas under the parent control may be performed before or after closing the presentation service, but it can only be performed when it is detected that the preset scene is exited.

[0150] In some possible implementations, the electronic device may be in more than two preset scenarios at the same time. For example, the electronic device is in the screen sharing and screen recording scenarios at the same time, and the electronic device will start the speech annotation service in both scenarios. When the electronic device is in two preset scenarios at the same time, if the speech annotation service needs to be turned off, it is necessary to exit all preset scenarios, that is, exit the screen recording and screen sharing scenarios.

[0151] The changes of preset scenes in this application can be monitored by monitoring events. For example, when the electronic device is monitored to enter at least one of the screen sharing scene, the same-source screen projection scene or the screen recording scene, the speech annotation service is started to provide annotation function services. The method of starting the speech annotation service provided by this application can pull up the speech annotation service after any software is started, without relying on the third-party conference software for adaptation, or relying on the system preset software whitelist, etc., with good adaptability and more stable functions.

[0152] This application monitors the screen sharing and screen recording scenarios by adding listener registration and deregistration interfaces in the media projection manager service, allowing applications with MANAGE_MEDIA_PROJECTION permission to monitor the registration status, so as to start the speech annotation service when it is started (start) and exit the speech annotation service when it is stopped (stop).

[0153] This application monitors the same-source same-screen scenario by adding a status description field: cast_plus_info in the multicast source discovery protocol (MDSP) of the wireless screen projection service, so that after the screen projection status changes, the change result is written into the status database. If the speech annotation service detects that the database field has changed, it can determine whether to enable same-source screen projection by reading the database.

[0154] The following takes the screen sharing scenario as an example. Figure 7 and Figure 8 Describes in detail the process of starting the speech annotation service.

[0155] When an electronic device enters a screen sharing scenario and starts a speech annotation service, the process mainly involves: a three-party conference APP at the application layer, a system user interface (UI), and a speech annotation service, as well as a media projection manager service, a media projection manager service, and a media projection management extension at the application framework layer.

[0156] Figure 7 FIG. 7 is a schematic flow chart of a method 700 for starting a speech annotation service provided in an embodiment of the present application. Figure 7 As shown, the method 700 may include S701 to S707. The various steps shown in the method 700 are described in detail below.

[0157] S701: When the three-party conference APP receives an operation to start screen sharing, it sends a request to start screen sharing to the media projection management.

[0158] S702: The media projection management sends a request to the system UI to start screen sharing.

[0159] Correspondingly, the system UI receives the screen sharing request and displays the following information on the interface of the electronic device: Figure 8 If the electronic device receives a user click Figure 8 When the operation of the control is “allowed” in the dialog box shown, the following S703 to S707 are continued.

[0160] S703, the system UI sends a message to the media projection management to start screen sharing.

[0161] S704: The media projection management sends a message to start screen sharing to the media projection management service.

[0162] Among them, the media projection management service is a system service of the application framework layer.

[0163] S705: The media projection management service sends a screen sharing start message to the media projection management extension.

[0164] Among them, the media projection management extension is a newly added module in this application, which is used to provide an API interface for the speech annotation service.

[0165] S706: The media projection management extension sends a screen sharing start message to the speech annotation service.

[0166] S707, speech annotation service starts.

[0167] Fig. 9 This is a schematic diagram of a same-source screen projection provided by an embodiment of the present application. Fig. 9 As shown, electronic device 1 (source device) encodes the video to be projected to obtain an encoded video stream; and the encoded video stream is transmitted to electronic device 2 (destination device) via wired or wireless transmission. Electronic device 2 decodes the received encoded video stream to obtain a decoded video stream, and displays the decoded video stream on the upper screen.

[0168] For example, when the electronic device 1 starts the same-source screen projection, the following is displayed: Fig.10 When the electronic device 2 receives the confirmation operation of screen casting, Fig.10 The waiting interface shown disappears, the electronic device enters the same-source screen projection scene, and the speech annotation service is started at the same time.

[0169] Combine the following Fig.11 , describes in detail the process of injecting the control corresponding to the canvas into the parent control of the target scroll control of the application interface. This process mainly involves the first APP, content sensing APP, speech annotation service of the application layer, and the activity management extension, activity task management service, target activity task management service, activity thread (ActivityThread), software development kit, and target system service of the application framework layer.

[0170] Fig.11 Schematic diagram of a process flow of injecting a canvas provided in an embodiment of the present application. Fig.11 As shown, the speech annotation service includes a main service module, a scroll control detection module, an agent module and a decoration control (décor view), and the content-aware APP includes content-aware management.

[0171] Injecting the control corresponding to the canvas under the scroll control in the application interface can be achieved through the following two processes: 1. Obtaining the target scroll control of the application interface, 2. Adding the control corresponding to the canvas to the parent control of the target scroll control.

[0172] 1. Obtaining the target scroll control of the application interface may include the following steps 1 to 6:

[0173] Step 1: After the third-party APP is started, an Activity instance is generated in ActivityThread;

[0174] Step 2: After the speech annotation service is started, obtain the package name and interface name of the third-party APP currently on the top through the system interface;

[0175] Step 3, the main service module in the speech annotation service transmits the obtained package name and interface name of the third-party APP to ActivityThread through the activity management extension, activity task management service, and target activity task management service extension in sequence;

[0176] Step 4, ActivityThread compares the package name and interface name transmitted in step 3 with the package name and page name recorded in the generated Activity instance;

[0177] If the two are the same, proceed to step 5 and step 6:

[0178] Step 5, ActivityThread receives the activity instance of the third-party application through the content-aware management in the content-aware APP and sends the activity instance corresponding to the current application interface to the proxy module in the speech annotation service.

[0179] Step 6: The proxy module receives the activity instance, and searches for the target scroll control in the current application interface based on the activity instance.

[0180] The detailed process of the speech annotation service obtaining the activity instance of the third-party application can be referred to the description of method 1000, which will not be repeated here.

[0181] 2. Adding the control corresponding to the canvas to the parent control of the scroll control can include the following steps 7 and 8:

[0182] Step 7, the system service creates a canvas control through a software toolkit;

[0183] Step 8: Add the control corresponding to the created canvas to the parent control of the target scroll control.

[0184] Optionally, before injecting the control corresponding to the canvas into the parent control of the target scroll control of the first application interface, the method 400 further includes: acquiring the target scroll control in the first application interface.

[0185] Exemplarily, the process of acquiring the target scroll control in the application interface can be implemented through the following steps 1 to 3.

[0186] Step 1, obtain the package name and interface name of the APP to which the first application interface belongs.

[0187] Among them, the package name of the APP is used to identify different applications. The interface name is used to identify different interfaces in the same application. According to the package name and interface name, a unique application interface can be determined. In other words, an application interface corresponds to a unique package name and interface name.

[0188] Step 2: Get the activity instance corresponding to the first application interface according to the package name and the interface name.

[0189] The activity instance is used to search for all controls in the first application interface, including the target scroll control.

[0190] Step 3: Determine a target scrolling control of the first application interface according to the activity instance.

[0191] Combine the following Fig.12 , describes in detail the process of an electronic device acquiring an activity instance. Exemplarily, the process involves the first APP and speech annotation service of the application layer, as well as the activity management extension module, speech annotation service activity task management, speech annotation service activity task management service extension, activity management, and activity thread of the application framework layer.

[0192] Fig.12 1 is a schematic flow chart of a method 1200 for obtaining an activity instance provided in an embodiment of the present application. Fig.12 As shown, the method 1200 may include S1201 to S1207. Fig.12 The steps shown.

[0193] S1201, the speech annotation service sends first information to the activity management extension. The first information is the package name and interface name of the APP obtained by the speech annotation service through the system interface.

[0194] S1202: The activity management extension sends first information to the target activity task management.

[0195] S1203: The target activity task management sends first information to the activity task management service.

[0196] S1204: The activity task management service sends first information to the target activity task management service extension.

[0197] S1205: The target activity task management service extension sends first information to the activity thread module.

[0198] S1206, the activity thread compares whether the APP package name and interface name in the first information are consistent with the package name and interface name in the saved activity instance.

[0199] In the case of consistency, S1207 is executed, and the activity thread sends the activity instance to the content-aware service; and S1208 is executed, and the content-aware service sends the activity instance to the speech annotation APP.

[0200] In order to further ensure the synchronous scrolling of the annotation pen and the annotated content, this application obtains the scrolling distance of the scroll control and controls the scrolling distance of the canvas control, so as to better achieve the synchronous scrolling of the two.

[0201] Exemplarily, in response to the second user operation, the electronic device controls the canvas and the first application interface to scroll synchronously according to the sliding distance of the maximum scroll control, so that the first batch of annotations slide synchronously with the first content.

[0202] The ratio of the sliding distance of the canvas to the sliding distance of the scroll control is equal to the ratio of the sliding distance of the first content to the sliding distance of the scroll control.

[0203] Optionally, when it is detected that the first application interface is switched to the second application interface, the control corresponding to the canvas is injected into the parent control of the target scroll control of the second application interface.

[0204] The process of injecting the control corresponding to the canvas into the parent control of the target scroll control of the second application interface can refer to the relevant description above, which will not be repeated here. The description of the target scroll control can also refer to the description above, which will not be repeated here.

[0205] In other words, in the preset scenario, every time the electronic device detects a change in the interface running in the foreground, the speech annotation service needs to re-traverse all controls in the switched application interface to obtain the target scrolling control, and add the control corresponding to the canvas provided by the speech annotation service to the parent control of the obtained target scrolling control.

[0206] The first application interface and the second application interface can be different interfaces of the same application, or different interfaces of different applications. The first application interface and the second application interface correspond to different activities, and each activity corresponds to an activity instance, so the first application interface and the second application interface correspond to different activity instances.

[0207] Optionally, when it is detected that the first application interface is switched to the second application interface, the control corresponding to the canvas under the parent control of the scrolling control of the first application interface is also deleted.

[0208] The canvas in this application is located above the application interface, so the canvas will receive the user's operation first. If the user's operation is an annotation operation, there is no need to pass the operation through to the application interface, and it can be displayed on the canvas; if the user's operation is a sliding operation, and the canvas cannot pass the sliding operation through to the application interface, then the current application interface cannot be slid, zoomed or rotated. In order to ensure that the electronic device can handle different user operations according to the user's needs, this application designs a canvas with special properties that can pass the user's sliding operation, zooming operation or rotation operation on the application interface to the application interface without passing the user's annotation operation. Among them, the application interface refers to the APP interface displayed by the electronic device when the user is screen sharing, screen recording, or same-source screen projection.

[0209] Exemplarily, the present application determines how to respond to the operation by determining the type of device used to perform the user operation and based on the determination result. The type of device used to perform the user operation may be a finger, a stylus, a mouse, a scroll wheel, a game controller, or a keyboard.

[0210] Optionally, after S401, the method 400 further includes: when the first user operation is performed by a stylus, determining that the first user operation is an annotation operation; or when the first user operation is performed by a finger, determining that the first user operation is not an annotation operation.

[0211] It should be understood that the fingers here may also refer to knuckles or elbow joints, etc.

[0212] For example, in the case where the first user operation is performed by a stylus pen, the electronic device may Fig.13 In the manner shown, the annotation handwriting formed by the received stylus touch operation is displayed on the annotation canvas.

[0213] Fig.13 Schematic diagram of the method of displaying annotation handwriting provided by the embodiment of the present application. Fig.13 As shown, the canvas is located above the application interface, and the APP layer includes text 1.

[0214] like Fig.13 As shown, when the canvas receives the annotation operation of the stylus, it transmits the stylus touch point to the stylus engine through the speech annotation service. The stylus engine draws the handwriting point image through off-screen drawing, and transmits the drawn handwriting point image back to the canvas through the speech annotation service to display the annotation handwriting.

[0215] It should be understood that off-screen drawing means that the stylus engine draws the received stylus touch points through a frame buffer object (framebuffer object, FBO).

[0216] Optionally, after S403, the method 400 further includes: when the second user operation is performed by a finger, determining that the second user operation is a sliding operation; or when the second user operation is performed by a stylus, determining that the second user operation is not a sliding operation.

[0217] For example, in the case where the second user operation is performed by a finger, the electronic device may Fig.14 The sliding of the application interface is achieved in the manner shown.

[0218] Fig.14 Schematic diagram of the interface sliding process provided by the embodiment of the present application. Fig.14 As shown in (a) of FIG. 1 , the canvas is located above the application interface. The APP layer includes text 1, and the canvas displays the annotations of text 1. When the canvas receives an upward sliding operation by the user, the canvas can transmit the sliding operation to the application interface, thereby obtaining the following: Fig.14 In the image shown in (b), the text 1 on the APP layer under the canvas is switched to text 2, realizing scrolling and page turning. Since the canvas scrolls synchronously with the APP layer, the annotation handwriting for text 1 on the same layer of the canvas also scrolls.

[0219] Combine the following Fig.15 , describes in detail the process of the above-mentioned electronic device determining the type of device used to perform user operations.

[0220] Fig.15 1 is a schematic flow chart of a method 1500 for determining a device type provided in an embodiment of the present application. Fig.15 As shown, the method 1500 may include S1500 to S1505. Fig.15 The steps shown.

[0221] S1501: Obtain the type of device used to perform the user operation.

[0222] S1502, determine the device type.

[0223] In case the user operation is performed by the stylus, the canvas intercepts the user operation.

[0224] In the case where the user operation is performed by a finger, S1503 to S1507 are continuously executed.

[0225] S1503, determine whether there is a finger mark.

[0226] If there is no finger identifier, the canvas intercepts the operation.

[0227] If there is a non-stylus pen identifier, continue to execute S1504 and S1505.

[0228] S1504, determine whether a window identifier exists.

[0229] If the window flag is not present, the canvas intercepts the operation.

[0230] If the window flag exists, continue to execute S1505.

[0231] S1505, determine whether there is an authority identifier.

[0232] If the permission token does not exist, the canvas intercepts the operation.

[0233] If there is a permission flag, the operation will be passed through to the APP layer.

[0234] Exemplarily, after receiving the user operation, the electronic device can determine the device type used to execute the user operation through the AMOTION_EVENT_TOOL_TYPE_STYLUS function; when the device type is a stylus, the user operation is intercepted; when the device type is not a stylus, it is the skipflag variable or the upper flag in the calculateskipflag function: SKIP_FLAG_MOTION_EVENT_WITHOUT_STYLUS, which is used to identify that the current device type is a non-stylus device: then the SKIP_FLAG_MOTION_EVENT_WITHOUT_STYLUS function is used to detect whether the skipflag variable has a flag for a non-stylus device; if not, the operation is intercepted; if so, the PRIVATE_FLAG_FLAG_STYLUS_WINDOW function is further used to detect whether the window information has window attributes; if not, the operation is intercepted; if so, the MONTOR_ROINTER_EVE function is further used to detect whether the operation is authorized; if not, the operation is intercepted; if so, the operation is transmitted to the application interface.

[0235] Fig.16 Schematic diagram of a method for transmitting annotation handwriting provided by an embodiment of the present application. Fig.16 As shown, the electronic device is in a screen sharing, same-source screen projection or screen recording scenario, starts the speech annotation service, injects the canvas into the application interface through the injection graph operation (implemented by injecting the control corresponding to the canvas into the parent control of the scroll control of the application interface), and completes the merger of the canvas and the application interface; then encodes the merged interface, and transmits the encoded interface through the server and the air interface. After receiving the encoded interface, the receiving device decodes it and merges the decoded application interface and the canvas again, and finally displays them through the display screen of the receiving device.

[0236] The present application also provides an electronic device, which may include: a processor coupled to a memory, wherein the memory may be used to store a computer program; and the processor may be used to call the computer program in the memory so that the electronic device executes each step shown in the above method embodiment.

[0237] It should be understood that the processor in the embodiment of the present application can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by the hardware integrated logic circuit or software instructions in the processor. The above processor can be a general processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor can be combined to perform. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0238] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0239] The present application also provides a computer program product, which includes: a computer program (also referred to as code, or instruction), which enables an electronic device to execute each step in the method embodiment when the computer program is executed.

[0240] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program (also referred to as code or instruction). When the computer program is executed, the electronic device executes each step in the method embodiment.

[0241] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0242] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, terminal devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0243] In the several embodiments provided in the present application, it should be understood that the disclosed systems, terminal devices and methods can be implemented in other ways. For example, the terminal device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the terminal device or unit can be electrical, mechanical or other forms.

[0244] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0245] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0246] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0247] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for displaying annotations, characterized in that: Applied to electronic equipment, the method comprises: In a preset scenario, a first user operation is received, where the first user operation is an annotation operation on first content on a first application interface, and the preset scenario includes screen sharing, same-source screen projection, or screen recording; In response to the first user operation, a first batch of annotation traces is displayed, the first batch of annotation traces are located on a canvas, and a control corresponding to the canvas is injected into a parent control of a target scroll control of the first application interface; In the preset scenario, a second user operation is received, where the second user operation is a sliding operation on the first application interface; In response to the second user operation, the canvas is controlled to scroll synchronously with the first application interface, so that the first batch of annotation traces scroll synchronously with the first content.

2. The method according to claim 1, characterized in that The method further comprises: The first batch of annotation traces are displayed after sliding synchronously with the first content.

3. The method according to claim 1, characterized in that The method further comprises: In the preset scenario, a third user operation is received, where the third user operation is a zoom operation on the first application interface; In response to the third user operation, the canvas is controlled to be scaled synchronously with the first application interface, so that the first batch of annotation traces are scaled synchronously with the first content.

4. The method according to any one of claims 1 to 3, characterized in that The canvas is provided by a speech annotation service, which is a service provided by the operating system of the electronic device and used to provide an annotation function; The method further comprises: In the case of detecting entering the preset scene, starting the speech annotation service; The control corresponding to the canvas is injected into the parent control of the target scroll control of the first application interface.

5. The method according to claim 4, characterized in that There are multiple scroll controls in the first application interface, and the target scroll control is the largest scroll control in the first application interface; The step of injecting the control corresponding to the canvas into the parent control of the target scroll control of the first application interface includes: The control corresponding to the canvas is injected into the parent control of the largest scroll control of the first application interface.

6. The method according to claim 4, characterized in that The target scrolling control includes each scrolling control of the first application interface; The step of injecting the control corresponding to the canvas into the parent control of the target scroll control in the first application interface includes: The control corresponding to the canvas is injected into the parent control of each scroll control in the first application interface.

7. The method according to claim 5, characterized in that In response to the second user operation, the canvas and the first application interface are scrolled synchronously so that the first batch of annotation traces slide synchronously with the first content, including: In response to the second user operation, the canvas and the first application interface are controlled to scroll synchronously according to the sliding distance of the maximum scroll control, so that the first batch of annotations slide synchronously with the first content, wherein the ratio of the sliding distance of the canvas to the sliding distance of the scroll control is equal to the ratio of the sliding distance of the first content to the sliding distance of the scroll control.

8. The method according to any one of claims 4 to 7, characterized in that Before injecting the control corresponding to the canvas into the parent control of the target scroll control of the first application interface, the method further includes: Obtain the package name and interface name of the application APP to which the first application interface belongs; According to the package name and the interface name, acquiring an activity instance corresponding to the first application interface, wherein the activity instance is used to search for all controls in the first application interface, wherein all controls include the target scroll control; The target scroll control of the first application interface is determined according to the activity instance.

9. The method according to any one of claims 1 to 8, characterized in that The method further comprises: When it is detected that the first application interface is switched to a second application interface, the canvas is injected into a scroll control of the second application interface.

10. The method according to claim 9, characterized in that The method further comprises: The control corresponding to the canvas is deleted from the parent control of the target scroll control of the first application interface.

11. The method according to any one of claims 1 to 10, characterized in that After receiving the first user operation, the method further includes: In the case where the first user operation is performed by a stylus pen, determining that the first user operation is the annotation operation; or, In a case where the first user operation is performed by a finger, it is determined that the first user operation is not the annotation operation.

12. The method according to any one of claims 1 to 11, characterized in that After receiving the second user operation, the method includes: In a case where the second user operation is performed by a finger, determining that the second user operation is the sliding operation; or, In a case where the second user operation is performed by a stylus pen, it is determined that the second user operation is not the slide operation.

13. The method according to claim 4, characterized in that The method further comprises: When it is detected that the preset scene is exited, the speech annotation service is closed.

14. An electronic device, characterized in that: include: A processor, wherein the processor is coupled to a memory, wherein the memory is used to store a computer program, and when the processor calls the computer program, the electronic device executes the method as claimed in any one of claims 1 to 13.

15. A computer-readable storage medium, characterized in that: Used to store a computer program, the computer program comprising instructions for implementing the method according to any one of claims 1 to 13.

Images