Method and electronic device for recording content

Abstract

The embodiment of the invention provides a content recording method and electronic equipment, the electronic equipment comprises a first application program APP, the first APP is used for recording and recording content input by a user, and the method comprises the following steps: starting the first APP; a starting instruction is received through the first APP, the starting instruction is used for indicating to start recording operation in the system, and the recording operation in the system refers to recording audio in the system of the electronic equipment; and in response to the starting instruction, executing an in-system recording operation. According to the method provided by the embodiment of the invention, the in-system recording function of the first APP can be realized, and the user experience is improved.

Description

[0001] This application is a divisional application of the invention patent application with application number 202211350637.4 submitted to the China Patent Office, application date October 31, 2022, and invention name “Method and electronic device for recording content”. All contents of the original application are incorporated into this application by reference. Technical Field

[0002] The present application relates to the field of electronic technology, and in particular to a method for recording content and an electronic device. Background Art

[0003] Note-taking applications (applications, APPs) are commonly used by users. At present, many note-taking APPs can not only record text content, but also record audio, which can meet the needs of users in classroom learning, attending press conferences and other usage scenarios.

[0004] However, in some scenarios, the recording effect of the note-taking app is not ideal, affecting the user experience. Summary of the invention

[0005] The present application provides a method and electronic device for recording content, which can realize the recording function within the system of the note APP, improve the recording effect of the note APP, and thus improve the user experience.

[0006] In a first aspect, the present application provides a method for recording content, which is executed by an electronic device, wherein the electronic device includes a first application APP, and the first APP is used to record audio and content input by a user. The method includes: starting the first APP; receiving a start instruction through the first APP, the start instruction is used to indicate the start of a recording operation within the system, and the recording operation within the system refers to recording audio within the system of the electronic device; and executing the recording operation within the system in response to the start instruction.

[0007] Optionally, the first APP may be a note APP, a memo APP or other APP that can record audio and user input content. This application does not impose any limitation on the specific type of the first APP.

[0008] The electronic device may start the first APP in response to the user's operation of starting the first APP, or in response to the user's related operations in other APPs (such as sharing content to the first APP), etc. Depending on the actual application scenario, after starting the first APP, the first APP may run in the background without displaying the interface, or may display the interface of the first APP in a floating window, a floating ball, or a full-screen display, a split-screen display, etc., and this application does not make any limitation on this.

[0009] The first APP can receive a start-up instruction for instructing to start a recording operation in the system. Optionally, the first APP can receive a start-up instruction input by the user through its interface, for example, receiving a start-up instruction from the user through a control in the interface for starting recording in the system. Optionally, the first APP can also receive a start-up instruction sent by other modules or other APPs in the electronic device. Optionally, in some scenarios, the electronic device can also automatically generate a start-up instruction when starting the note APP according to a preset start-up logic.

[0010] In the method for recording content provided by the first aspect, after starting the first APP, a startup instruction can be received through the first APP, and the electronic device can respond to the startup instruction and perform a recording operation in the system. In this way, the first APP can realize the recording function in the system. In this way, in a scenario where the user plays audio through other APPs while recording through the first APP, the system recording function of the first APP can eliminate the surrounding ambient sound and the noise and noise generated by the speaker during playback, etc., thereby improving the recording effect. At the same time, it also solves the problem of not being able to record when listening to audio through headphones, thereby improving the user experience.

[0011] In a possible implementation, performing the in-system recording operation includes: when it is determined that there is audio in a playing state in the system of the electronic device, performing the in-system recording operation.

[0012] That is to say, when the electronic device plays audio in the system, the recording operation in the system is performed; when the audio is not played in the system, the recording operation in the system is not performed. This can reduce the blank recording in the recorded audio. On the one hand, the recorded audio will not be too large, saving the storage space of the electronic device; on the other hand, when playing the recorded audio, there will be no blank recording, and the user does not need to adjust the playback progress, which improves the user experience.

[0013] In a possible implementation, the electronic device further includes a media player, and the method further includes: if it is determined that an instantiated first object exists in the media player and an audio stream exists in the media player, determining that audio in a playing state exists in the system of the electronic device.

[0014] When a media player plays audio or video, it needs to instantiate an object. In addition, when playing audio or video, there is an audio stream in the media player. Therefore, by determining whether there is an audio stream and an instantiated object in the media player, it is possible to quickly and accurately determine whether there is audio in the playing state in the system, and thus accurately remove blank recordings.

[0015] In one possible implementation, before executing the recording operation within the system, the method also includes: performing a permission check on the first APP, the permission check is used to determine whether there is user authorization information in the electronic device, and whether the first APP meets the preset conditions, and the user authorization information is used to indicate that the user agrees that the electronic device performs the recording operation within the system; if the permission check passes, the recording operation within the system is executed; if the permission check fails and it is determined that there is no user authorization information, the application authorization information is displayed, and the application authorization information is used to apply for user authorization information.

[0016] That is to say, before starting the recording operation within the system, it is necessary to first determine whether the first APP has the authority to obtain the recording within the system. Specifically, on the one hand, determine whether there is user authorization information in the electronic device; on the other hand, determine whether the APP (first APP) requesting to perform the recording operation within the system meets the preset conditions. If there is user authorization information in the electronic device and the first APP meets the preset conditions, the permission check passes, otherwise the permission check fails. Optionally, the preset requirements may, for example, be: the capture policy of the APP is AudioAttributes.ALLOW_CAPTURE_BY_ALL.

[0017] When the permission verification is caused by the absence of user authorization information, the electronic device displays the application authorization information. Specifically, the electronic device can display the application authorization information in the form of a pop-up window to apply for authorization from the user.

[0018] In a possible implementation, after displaying the authorization information, the method further includes: receiving user authorization information input by the user, and returning to the execution step to perform permission verification on the first APP.

[0019] In this implementation, by performing permission verification on the first APP, the security and reliability of recording operations in the system are improved, while also protecting user privacy and improving user experience.

[0020] Optionally, after receiving the user authorization information input by the user, the electronic device can also save the user authorization information. In this way, when performing permission verification later, there is no need to repeatedly apply for authorization from the user, which reduces disturbance to the user and improves the user experience.

[0021] In one possible implementation, the method also includes: displaying an interface of a first APP, the interface of the first APP including a content input area; during execution of a recording operation within the system, when the recording duration is a first duration, in response to the user inputting the first content in the content input area, saving the first content, and generating mapping relationship information based on the first duration and the input position of the first content.

[0022] Optionally, the electronic device may directly generate mapping relationship information based on the first duration and the input position of the first content, or may generate other information based on the first duration and / or the input position of the first content, and then generate mapping relationship information based on the generated information. This application does not impose any restrictions on the specific parameters in the mapping relationship information, as long as it can characterize the mapping relationship between the recording duration and the input position of the content.

[0023] In this implementation, by saving the first content and generating mapping relationship information, when the recorded audio is subsequently played, when the audio is played to the first duration, the first content corresponding to the first duration can be determined based on the mapping relationship information, and then the first content can be displayed. Moreover, when the user selects the first content, the first duration corresponding to the input position of the first content can also be determined based on the mapping relationship information, and the audio can be jumped to the first duration. In this way, two-way positioning of recording playback and content display can be achieved, further improving the user experience.

[0024] In a specific embodiment, the first content is saved, and mapping relationship information is generated according to the first duration and the input position of the first content, including: generating a first HTML tag corresponding to the first content based on the hypertext markup language HTML; saving the first content and the first HTML tag; determining first position information according to the first HTML tag, the first position information representing the position of the first content in the HTML page; and generating mapping relationship information according to the first duration and the first position information.

[0025] In this implementation, a first HTML tag corresponding to the first content is generated based on the HTML language, and the first content and the first HTML tag are saved. First location information is generated based on the first HTML tag, and then mapping relationship information is generated based on the first location information. The HTML language follows the same parsing protocol in electronic devices of different systems, and cross-system data interoperability can be achieved. Therefore, in the method provided in this embodiment, the note content and mapping relationship information recorded in the HTML language can also be recognized in electronic devices of other systems, so the first content and the first HTML tag and mapping relationship information saved by the method can also be recognized in other systems, thereby realizing cross-system bidirectional positioning of recording playback and note content display, thereby improving user experience.

[0026] In one possible implementation, the method also includes: in response to a user's pause recording operation, stopping the recording operation within the system to obtain recorded audio; determining information of a second APP displayed simultaneously with the first APP in the interface of the electronic device; and displaying prompt information based on the information of the second APP, the prompt information being used to prompt the user to save the information of the second APP.

[0027] Optionally, the second APP is an APP for playing audio or video.

[0028] The first APP and the second APP are displayed on the same interface at the same time, indicating that the audio in the system recorded by the first APP may be played by the second APP, that is, the audio played by the second APP may be the original audio corresponding to the recorded audio. Therefore, when the recording is paused, the electronic device displays a prompt message to prompt the user to save the relevant information of the second APP, that is, to save the information of the original audio.

[0029] Optionally, the prompt information can be specifically used to remind the user to save the page information currently displayed by the second APP, the title of the audio or video being played, etc. The page information can be, for example, the link information of the page (such as a uniform resource locator (URL)).

[0030] In this implementation, prompt information is displayed to increase interaction with the user and further improve the user experience.

[0031] In a possible implementation, the method further includes: the interface of the electronic device includes a first window and a second window, the first window displays the interface of the first APP, and the second window displays the interface of the second APP.

[0032] In other words, the first APP runs in the first window, and the second APP runs in the second window. In other words, the first APP and the second APP are displayed in multiple windows. In this way, it is convenient for users to use the second APP while recording and recording content through the second APP. For example, a user plays a video through a video playback APP while recording the sound and content of the video through a note APP.

[0033] In a possible implementation, the method further includes: one of the first window and the second window is displayed in suspension; or the first window and the second window are displayed side by side.

[0034] That is, the first window and the second window can be displayed in any multi-window format. Optionally, the parallel display can be displayed in split screen.

[0035] In this implementation, displaying the first window and the second window in a multi-window format can reduce occlusion and improve display effects.

[0036] In one possible implementation, launching the first APP includes: in response to a sharing operation performed by a user on a first page, launching the first APP and displaying the interface of the first APP; the sharing operation is used to indicate that the first page is shared to the first APP, and the interface of the first APP includes a content input area; and writing the link information of the first page into the content input area.

[0037] Optionally, the first page can be a page in a browser or a page in another APP. The link information of the first page can be the URL of the first page. Optionally, when the user clicks the sharing control in the first page, an object selection card can be displayed in the first page. The object selection card is used for the user to select the sharing object. When the user clicks the option of the note APP in the object selection card, the electronic device displays the start of the first APP and displays the interface of the first APP.

[0038] In this implementation, when a user performs a sharing operation on a page, the electronic device directly obtains the link information of the page and writes the link information into the content input area. This allows the user to know the source of the recorded content when viewing the recorded content later, thereby facilitating the user to trace back to the first page and improving the user experience.

[0039] In one possible implementation, launching the first APP includes: in response to the user's copy link operation on the second page, copying the link information of the second page and displaying a split-screen control; in response to the user clicking the split-screen control, launching the first APP, and displaying the second page and the first APP in a split-screen, wherein the interface of the first APP includes a content input area; and writing the link information of the second page into the content input area.

[0040] Optionally, the second page may be a page in a browser or a page in another APP. Optionally, the split-screen control may be displayed suspended above the second page. The split-screen control may be, for example, a split-screen suspended ball.

[0041] In general, when a user copies information, there is a high probability that the copied information will be pasted and recorded. In this implementation, when the user performs a copy link operation, it is considered that the user is likely to need to record the content through the first APP, so the split-screen control is directly displayed. The user can simply and quickly realize the split-screen display of the first APP interface and the second APP interface through the split-screen control. This makes it convenient for users to open the first APP to record content, thereby improving the user experience. At the same time, after opening the first APP, the electronic device automatically writes the copied link information of the second page into the content area without the user having to perform a paste operation, and it is convenient for users to know the source of the recorded content, further improving the user experience.

[0042] In a possible implementation, after starting the first APP, the method further includes: generating a startup instruction.

[0043] In this implementation, when the user performs the operation of copying the link, the electronic device automatically generates a start instruction. In this way, after starting the first APP, the recording operation in the system can be automatically started without manual operation by the user, thereby improving the user experience.

[0044] In one possible implementation, the system of the electronic device includes an application layer and an application framework layer, the first APP is located in the application layer, and the electronic device also includes a recording module, the recording module is located in the application framework layer, and executes recording operations within the system, including: the recording module executes recording operations within the system.

[0045] In one possible implementation, the recording module executes a recording operation within the system, including: the recording module constructs a second object and adds configuration parameters to the second object, the configuration parameters indicating that the recording operation corresponding to the second object is a recording operation within the system; the recording module executes the recording operation within the system based on the second object.

[0046] In one possible implementation, the electronic device also includes a media player, which is located in the application framework layer. Before the recording module performs an in-system recording operation based on the second object, the method also includes: the first APP sends a first mark to the recording module, and the first mark is used to indicate that when it is determined that there is audio in a playing state in the system of the electronic device, the in-system recording operation is performed; the recording module performs the in-system recording operation based on the second object, including: the recording module performs the in-system recording operation based on the second object according to the first mark when it is determined that there is an instantiated first object in the media player and there is an audio stream in the media player.

[0047] In a second aspect, the present application provides a device, which is included in an electronic device, and the device has the function of implementing the electronic device behavior in the above-mentioned first aspect and the possible implementation of the above-mentioned first aspect. The function can be implemented by hardware, or it can be implemented by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above-mentioned functions. For example, a receiving module or unit, a processing module or unit, etc.

[0048] In a third aspect, the present application provides an electronic device, comprising: a processor, a memory, and an interface; the processor, the memory, and the interface cooperate with each other so that the electronic device executes any one of the methods in the technical solution of the first aspect.

[0049] In a fourth aspect, the present application provides a chip, including a processor. The processor is used to read and execute a computer program stored in a memory to perform the method in the first aspect and any possible implementation thereof.

[0050] Optionally, the chip also includes a memory, and the memory is connected to the processor via a circuit or wire.

[0051] Further optionally, the chip also includes a communication interface.

[0052] In a fifth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the processor executes any one of the methods in the technical solution of the first aspect.

[0053] In a sixth aspect, the present application provides a computer program product, the computer program product comprising: a computer program code, when the computer program code runs on an electronic device, the electronic device executes any one of the methods in the technical solution of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a schematic diagram of the structure of an electronic device 100 provided in an embodiment of the present application;

[0055] Figure 2 is a software structure block diagram of an example of an electronic device 100 provided in an embodiment of the present application;

[0056] Figure 3 This is a schematic diagram of interface changes in a process of starting a recording function in a system provided by an embodiment of the present application;

[0057] Figure 4 This is another example of a schematic diagram of interface changes during the process of starting a recording function in a system provided by an embodiment of the present application;

[0058] Figure 5 This is another example of a schematic diagram of interface changes during the process of starting a recording function in a system provided by an embodiment of the present application;

[0059] Figure 6 This is another example of a schematic diagram of interface changes during the process of starting a recording function in a system provided by an embodiment of the present application;

[0060] Figure 7 This is another example of a schematic diagram of interface changes during the process of starting a recording function in a system provided by an embodiment of the present application;

[0061] Figure 8 This is another example of an application authorization interface for a recording function in a system provided in an embodiment of the present application;

[0062] Fig. 9 It is a timing flow diagram of an audio recording process provided in an embodiment of the present application;

[0063] Fig.10 This is a schematic diagram of interface changes in a note generation process in a text input mode provided in an embodiment of the present application;

[0064] Fig.11 This is a timing flow chart of a note content writing process provided by an embodiment of the present application;

[0065] Fig.12 This is a schematic diagram of an interface change for inserting a picture through a note interface provided in an embodiment of the present application;

[0066] Fig.13 This is a schematic diagram of interface changes during a note stopping process in a text input mode provided by an embodiment of the present application;

[0067] Fig.14 This is a timing flow chart of a note stopping process provided in an embodiment of the present application;

[0068] Fig.15 This is a schematic diagram of an application scenario of a shared note provided in an embodiment of the present application;

[0069] Fig.16 This is a schematic diagram of interface changes in a process of sequentially presenting notes on a tablet computer provided by an embodiment of the present application;

[0070] Fig.17 It is a timing flow chart of an example of a note sequential presentation process provided by an embodiment of the present application;

[0071] Fig.18 This is a schematic diagram of an interface change of a playback progress jump provided in an embodiment of the present application;

[0072] Fig.19 This is a timing flow diagram of an example of a playback progress jump process provided by an embodiment of the present application;

[0073] Fig. 20 This is a schematic diagram of an interface change of a note content jump provided in an embodiment of the present application;

[0074] Fig.21 This is a timing flow diagram of an example of a playback progress jump process provided by an embodiment of the present application;

[0075] Fig. 22 This is a schematic diagram of interface changes in a note generation process in handwriting input mode provided in an embodiment of the present application;

[0076] Fig.23 This is a schematic diagram of a timing flow in a handwritten content input process provided by an embodiment of the present application;

[0077] Fig.24 is a timing flow chart of another example of a note sequential presentation process provided by an embodiment of the present application;

[0078] Fig.25 is a timing flow chart of another example of a playback progress jump process provided by an embodiment of the present application;

[0079] Fig.26is a timing flow chart of another example of a note content jump process provided by an embodiment of the present application;

[0080] Fig. 27 This is a schematic diagram of interface changes in a note sharing process provided in an embodiment of the present application;

[0081] Fig.28 This is a flowchart of a note format conversion example provided in an embodiment of the present application. DETAILED DESCRIPTION

[0082] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0083] In the following, the terms "first", "second", and "third" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first", "second", and "third" may explicitly or implicitly include one or more of the features.

[0084] References to "one embodiment" or "some embodiments" etc. described in the specification of the present application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in the specification of the present application do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0085] To facilitate understanding, the terms and concepts involved in the application embodiments are first explained.

[0086] 1. Recording in the system

[0087] In-system recording is also called system recording, which means recording the sound (audio) inside the electronic device system. In-system recording is the opposite of out-system recording. Out-system recording is also called microphone recording, which means recording the sound (audio) outside the electronic device through a microphone.

[0088] 2. Recording files

[0089] Recording files refer to audio files recorded through the note-taking app.

[0090] 3. Input mode

[0091] In an embodiment of the present application, the interface of the note APP (hereinafter referred to as the note interface) may include a content input area, and the content input area is used for users to input note content. When the user enters the note content in the content input area, the input mode is mainly divided into handwriting input mode and non-handwriting input mode. The handwriting input mode refers to an input mode in which the user writes in the content input area with a finger or a stylus. The non-handwriting input mode refers to an input mode other than the handwriting input mode. Optionally, the non-handwriting input mode may include a text typing mode, a picture input mode, an audio input mode, a video input mode, a table input mode, etc. Among them, the text typing mode refers to a mode for inputting text content through a soft keyboard.

[0092] 4. Note content

[0093] Note content refers to the content input by the user through the interface of the note APP, also known as input content or recorded content. In the embodiment of the present application, according to the different input modes of the note APP during input, the note content can be divided into hypertext markup language (HTML) elements and handwritten content, that is, the content types of the note content include HTML elements and handwritten content. HTML elements refer to content that is input in a non-handwriting input mode and can be recognized and read by the HTML language. Corresponding to the type of non-handwriting input mode, the types of HTML elements may include text elements, picture elements, audio elements, video elements, table elements, etc. Among them, the picture element can be a static picture element or a graphic interchange format (GIF) dynamic picture element. In the following embodiments of the present application, HTML is mainly explained by taking the fifth generation of HTML (also known as HTML5.0, hereinafter referred to as H5) as an example, so the following HTML elements are all explained by taking the H5 element (or simply referred to as the element) as an example.

[0094] In an embodiment of the present application, the information contained in the H5 element may include H5 element content, the input position of the H5 element content, the attributes of the H5 element content, the format of the H5 element content, etc. The H5 element content refers to the content or information input by the user, such as the text "abc" itself, or the picture itself, etc. The input position of the H5 element content refers to the position where the user inputs the H5 element content. The attributes of the H5 element content refer to the category to which the H5 element content belongs, such as title, text, etc. The format of the H5 element content refers to the format in which the H5 element content is presented when it is input or displayed, such as font style, font size, display style (color, background color, etc.), etc.

[0095] Handwritten content refers to the content of handwriting input in the handwriting input mode. The information contained in the handwritten content may include handwritten strokes, the input position of the handwritten strokes, the attributes of the handwritten strokes, the format of the handwritten strokes, etc. The meaning of the information contained in the handwritten content is similar to the meaning of the information contained in the above-mentioned H5 element, and will not be repeated. In an embodiment of the present application, a handwritten stroke corresponds to a group of touch events, and a group of touch events includes a press (down) event, a move (move) event and a lift (up) event. In addition, in an embodiment of the present application, the information contained in the handwritten content may also include the time when the handwritten stroke is put down, etc. Among them, the time when the handwritten stroke is put down can be the time when the down event corresponding to the handwritten stroke occurs.

[0096] 5. Content files

[0097] Content files are files containing note content, and are also called note content files or record files.

[0098] In application scenarios such as students studying in class and journalists attending press conferences, note-taking apps enable users to quickly record key content while recording, meeting the user's usage needs. However, in some scenarios, such as students taking online courses or video conferencing, if the user uses the speaker of an electronic device to play audio and then records it through a note-taking app, it is difficult to eliminate playback noise, etc., resulting in poor recording effects. Moreover, if the user listens to audio through headphones, recording cannot be performed through the note-taking app. In short, in some usage scenarios, the note-taking app cannot meet the user's usage needs and affects the user experience.

[0099] In view of this, in the method for recording content provided in the embodiment of the present application, the note APP has an in-system recording function, which can eliminate the surrounding environmental sound and the noise and noise generated by the speaker during playback, thereby improving the recording effect. At the same time, it also solves the problem of being unable to record when listening to audio through headphones, thereby improving the user experience.

[0100] In addition, in the method for recording content provided by the embodiment of the present application, when the user inputs the note content during the recording process, the note content is recorded, and mapping relationship information between the note content and the recording duration when the note content is input is generated. In this way, when the recording is played later, the note content corresponding to the playback duration can be synchronously displayed according to the mapping relationship information; and when the user clicks on the note content at a certain position, the recording can jump to the corresponding playback duration. In other words, bidirectional positioning of recording playback and note content display can be achieved later, which is convenient for users to view notes and further improve user experience.

[0101] Moreover, the method for recording content provided in the embodiment of the present application is edited based on the HTML language (H5 is taken as an example later), and the note content entered by the user is displayed and saved in the form of an H5 page, and the mapping relationship information between the relative time and the H5 location information is generated. Among them, the relative time is determined according to the recording duration when the note content is entered, and the H5 location information represents the position of the note content in the H5 page. The H5 language follows the same parsing protocol in electronic devices of different systems, and cross-system data intercommunication can be achieved. Therefore, the note content and mapping relationship information recorded based on the H5 language can also be recognized in electronic devices of other systems, so a two-way positioning function can be achieved across systems.

[0102] The method for recording content provided in the embodiments of the present application can be applied to mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), and other electronic devices on which application programs can be installed. The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

[0103] For example, Figure 11 is a schematic diagram of the structure of an electronic device 100 provided in an embodiment of the present application. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0104] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0105] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0106] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0107] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0108] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0109] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0110] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.

[0111] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0112] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0113] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0114] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0115] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0116] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0117] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0118] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0119] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to microphone 170C to input the sound signal into microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0120] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0121] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the touch operation intensity according to the pressure sensor 180A. The electronic device 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0122] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0123] In the embodiment of the present application, the software system of the electronic device 100 can be an Android system, a Windows system, an IOS system, etc., and the present application does not make any limitation on this. The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.

[0124] Figure 2 1 is a software structure diagram of the electronic device 100 of the embodiment of the present application. The layered architecture divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime (Android runtime) and the system library, and the kernel layer. The application layer can include a series of application packages.

[0125] like Figure 2As shown, the application package may include note APP, browser APP, audio APP, video APP and online course APP, etc. Of course, the application package may also include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, short message and other applications ( Figure 2 not shown).

[0126] In this embodiment, the note APP may include a user interface (UI) layer, a logic control layer, and an application kernel layer. Among them, the UI layer may include an interface module. The interface module is used to display the interface during the operation of the note APP, and to receive instructions or information input by the user through the note interface. Optionally, the interface module can implement its functions by calling relevant modules of the application architecture layer. Optionally, the note interface may include one or more of a recording control, a handwriting control, a picture control, a content input area, a recording progress control, a recording playback control, etc. Among them, the content input area is used to input H5 elements or handwritten content, etc. The specific content of the note interface will be described in conjunction with the accompanying drawings in subsequent embodiments.

[0127] The logic control layer may include a recording and playback control module. During the note generation process, the recording and playback control module is mainly used to control the recording module in the application architecture layer to obtain the audio stream and generate a recording file. During the note presentation process (i.e., playing the recording and synchronously displaying the note content), the recording and playback control module is also used to play the recording file.

[0128] The application kernel layer may include an H5 editor and a storage module. When the note content is an H5 element, during the note generation process, the H5 editor is used to edit, display and save the H5 element input by the user to a content file based on the H5 language. Specifically, the H5 editor records the H5 element content through an H5 tag (hereinafter referred to as a tag), and marks the information of the H5 element content, for example, anchoring the position of the H5 element content in the H5 page through the tag. In addition, the H5 editor is also used to determine the H5 relative time based on the recording duration when the H5 element is input, and determine the H5 position information based on the tag corresponding to the H5 element. The H5 position information represents the position of the H5 element content recorded in the tag in the H5 page. The H5 editor generates mapping relationship information between the H5 relative time and the H5 position information. During the note presentation process, the H5 editor can be used to read the content file and pre-display all the note contents in the content file, and determine whether there is an H5 relative time (called the target H5 relative time) that matches the current playback duration based on the mapping relationship information; if so, determine the target H5 position information corresponding to the target H5 relative time, and highlight the element content in the content file that matches the target H5 position information to achieve a two-way phase between recording playback and H5 element display. Optionally, the H5 editor can poll and search based on a javascript (JS) polling mechanism to determine whether there is a target H5 relative time that matches the current playback duration.

[0129] In addition, the H5 editor in the embodiment of the present application can also embed a handwriting editor (not shown), so that the H5 editor can also realize the input and display of handwritten content. Specifically, when the note content is handwritten content, during the note generation process, the H5 editor is used to edit, display, and save the handwritten content input by the user to the content file. In addition, the H5 editor is also used to record the start time of recording and the time when each handwritten stroke is put down. During the note presentation process, the H5 editor is used to determine the handwriting relative time of each handwritten stroke according to the start time of recording and the time when each handwritten stroke is put down. When the current playback duration matches a certain handwriting relative time (called the target handwriting relative time), the handwriting strokes corresponding to the target handwriting relative time are highlighted to realize the two-way phase of recording playback and handwritten content display.

[0130] The storage module is used to store files or data generated by other modules in the note APP, for example, storing recording files generated by the recording and playback control module, or storing content files and mapping relationship information generated by the H5 editor.

[0131] It should be noted that the H5 editor in the application kernel layer may also be an editor of other versions of HTML, for example, HTML4.0, HTML3.0, and the present application embodiment does not limit the version of the HTML editor. For ease of description, the following description is based on the H5 editor as an example.

[0132] Audio APP refers to an APP that plays audio. Video APP refers to an APP that plays videos. Online course APP refers to an APP used for online course learning. The audio APP, video APP and online course APP described in this embodiment all refer to APPs that can realize corresponding functions, and no limitation is made on the specific APP products. For example, the online course APP can be It can also be You can also And other APPs.

[0133] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0134] like Figure 2 As shown, the application framework layer may include a media player, an activity manager service (AMS), a package management service (PMS), a window manager service (WMS), a recording module, a sharing module, and the like.

[0135] The media player is used to play video files or audio files. When playing a video file or an audio file, the media player first needs to instantiate the object (referred to as instantiation), then obtain the audio stream and play it.

[0136] AMS, also known as activity management service, is used to manage activities to display, close, and switch pages.

[0137] PMS is used to manage the installation packages (such as jar packages or apk packages) of applications in the system, is responsible for system permissions, and is responsible for the installation, uninstallation, update, and analysis of applications or services. When managing installation packages, PMS can manage the information of installation packages by scanning the installation packages in the target folder of the Android system and establishing a mapping data structure of the installation packages, such as managing the names of the installation packages. The name of the corresponding application can be determined by the name of the installation package.

[0138] WMS is also called a window manager, which is used to manage window programs. WMS can obtain the size of the display screen, determine whether there is a status bar, lock the screen, capture the screen, etc. In an embodiment of the present application, multi-window display can be implemented on the display screen through WMS. Multi-window display may include multiple windows displayed side by side, and multiple different applications may run in multiple windows, that is, split-screen display of multiple applications is implemented. In addition, multi-window display may also include at least one window in the multiple windows being suspended. An application may run in the suspended window, that is, the application is displayed in the form of a suspended window.

[0139] The recording module is also called an audio recorder, and is used to record audio. Optionally, an application at the application layer can record audio by calling the recording module through a function.

[0140] The sharing module is used to share web pages or content to corresponding applications based on the user's selection on the page.

[0141] Of course, in addition to the above modules, the application framework layer can also include content providers, view systems, phone managers, resource managers, notification managers, etc.

[0142] Content providers are used to store and retrieve data and make it accessible to applications. The data can include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0143] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0144] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including connecting, hanging up, etc.).

[0145] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0146] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0147] Android runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0148] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0149] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0150] The system library may include multiple functional modules, such as surface manager, media libraries, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0151] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0152] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0153] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis and layer processing.

[0154] A 2D graphics engine is a drawing engine for 2D drawings.

[0155] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0156] For ease of understanding, the following embodiments of the present application will be described in detail with Figure 1 and Figure 2 Taking the electronic device with the structure shown as an example, the method for recording content provided in the embodiment of the present application is specifically described in combination with the accompanying drawings and application scenarios.

[0157] The method for recording content provided in the embodiment of the present application can start the note APP and its in-system recording function when playing audio or video in a browser APP, audio APP, video APP or online course APP, etc., to obtain the audio stream in the system, thereby improving the recording effect. The following describes several methods for starting the in-system recording function of the note APP in combination with specific application scenarios.

[0158] 1. Use the operation window to start the note APP and its recording function in the system.

[0159] Taking the scenario of learning through online course APP as an example, users can display the online course APP and note APP in multi-window mode by operating the window, so that they can watch videos through the online course APP while entering notes through the note APP. In addition, users can start the recording function in the system to obtain the audio stream in the system, realize the recording of the played audio, and improve the recording effect.

[0160] The multi-window forms realized by operating windows mainly include split-screen display and floating windows:

[0161] (1) Split screen display

[0162] For example, Figure 3 This is a schematic diagram of the interface changes in the process of starting the recording function in the system provided in an embodiment of the present application. Taking the electronic device as a mobile phone as an example, Figure 3 As shown in Figure (a), the user has started the online course APP, and the online course APP is displayed in full screen. The top of the full-screen window may include a top bar 301. Optionally, the user can switch the full-screen window to a split screen by dragging the top bar 301 and moving it to the left or right. If the user drags the top bar 301 to the left, the online course APP is displayed in the left window after switching to the split screen, and if the user drags the top bar 301 to the right, the online course APP is displayed in the right window after switching to the split screen.

[0163] For example, when the user drags the top bar 301 to the left, in response to the user's dragging operation, the online course APP displayed in full screen is split into the left window, and then the user can open the note APP in the right window. In this way, the split screen display of the online course APP and the note APP is realized. Figure 3 As shown in Figure (b).

[0164] Of course, if the electronic device supports it, the user can also achieve split screen in other ways, and the embodiments of the present application do not impose any limitations on this.

[0165] The interface of the note-taking APP may include a recording control 302. The user clicks the recording control 302, and the interface jumps to Figure 3The interface shown in FIG. (c) in FIG. 3 includes a recording type tab sheet 303. The recording type tab sheet 303 includes a system recording option 304.

[0166] In response to the user clicking the system recording option 304, the note APP starts the system recording function. At this time, the interface is as follows: Figure 3 As shown in Figure (d).

[0167] Figure 3 In the example of starting the online course app first and the note app later, the process of starting the note app and its recording function in the system when the screen is split is explained. It can be understood that the process of starting the note app first and the online course app later is similar to this, and will not be repeated.

[0168] (2) Floating window

[0169] For example, Figure 4 This is another example of the interface change diagram of the process of starting the recording function in the system provided by the embodiment of the present application. Figure 4 As shown in Figure (a), the user has started the online course APP and the online course APP is displayed in full screen. Optionally, the user can open the floating window of the note APP through the side application bar. Specifically, the user can slide from the right edge of the screen to the left to pull up the side application bar 401, as shown in Figure 402. Figure 4 As shown in Figure (b) in FIG. 4 , the user clicks the icon 402 of the note APP in the side application bar 401 to open the floating window 403 of the note APP. Figure 4 As shown in Figure (c).

[0170] Of course, if the electronic device supports it, the user can also open the floating window of the note APP in other ways, and the embodiment of the present application does not impose any limitation on this.

[0171] The floating window 403 of the note APP includes a new control 404, and the user can click the new control 404 to create a new note. Figure 4 In this interface, users can start the system recording function by clicking the recording control, recording type tab, and system recording option. Figure 3 As shown in Figures (b) to (d) in the figure, no further details are given.

[0172] Figure 4 In the example of the online course APP being displayed in full screen and the note APP being displayed in a floating window, the process of starting the note APP and its recording function in the system is explained. It can be understood that the process of the note APP being displayed in full screen and the online course APP being displayed in a floating window is similar to this and will not be repeated.

[0173] In other embodiments, the activated note APP can also be displayed in the form of a floating ball. The quick function card in the floating ball can be used to open the note APP interface and start the recording function in the system.

[0174] For example, Figure 5 This is another example of the interface change diagram of the process of starting the recording function in the system provided by the embodiment of the present application. Figure 5 As shown in Figure (a), the user has started the online course APP, and the online course APP is displayed in full screen, and the note APP is displayed in the interface in the form of a floating ball 501. The user clicks the floating ball 501, and the interface displays the shortcut function card 502 of the note APP, as shown in Figure 5a. Figure 5 As shown in Figure (b) in the figure. The quick function card 502 includes a new note (recording in the system) option 503. The user clicks the new note (recording in the system) option 503, and the note APP is displayed in the interface in the form of a floating window, and the recording function in the system is activated, as shown in Figure 504. Figure 5 As shown in Figure (c).

[0175] 2. Share or copy the link to start the note-taking app and its recording function.

[0176] Taking the video playback scenario on a web page through a browser APP as an example, the user can open the note APP and its recording function in the system by sharing or copying the link. The following is an explanation of the two methods of sharing and copying the link in combination with the interface diagram.

[0177] (1) Share

[0178] For example, Figure 6 This is another example of the interface change diagram of the process of starting the recording function in the system provided by the embodiment of the present application. Figure 6 As shown in Figure (a) in FIG. 1 , a user is watching a video on a web page through a browser. The web page includes a more function control 601. The user clicks the more function control 601, and a function tab 602 pops up in the interface, as shown in FIG. Figure 6 As shown in Figure (b) of FIG. 602, the function tab 602 includes a system sharing option 603. When the user clicks the system sharing option 603, the mobile phone enters Figure 6 The interface shown in Figure (c) of FIG. The interface includes a sharing object selection card 604, and the sharing object selection card 604 includes a note APP option 605. The user clicks the note APP option 605, and the note APP is displayed in the interface in the form of a card, automatically creates a new note with the same title as the web page currently opened by the browser APP, saves the link of the web page, and automatically starts the recording function in the system, such as Figure 6 As shown in Figure (d).

[0179] Of course, in other embodiments, when the user clicks on the option 605 of the note APP in the sharing object selection card 604, the recording function in the system may not be automatically started. Instead, some options may be displayed for the user to choose whether to start the recording function in the system. The specific design can be based on actual needs, and the embodiments of the present application do not impose any limitations on this.

[0180] (2) Copy the link

[0181] For example, Figure 7 This is another example of the interface change diagram of the process of starting the recording function in the system provided by the embodiment of the present application. Figure 7 As shown in Figure (a) in FIG. 1 , a user is watching a video on a web page through a browser APP. The web page includes a more function control 601. The user clicks the more function control 601, and a function tab 602 pops up in the interface, as shown in FIG. Figure 7 As shown in Figure (b) of FIG. 602, the function tab 602 includes a copy link option 701. When the user clicks the copy link option 701, the interface displays a split screen prompt 703 and a split screen floating ball 704, prompting the user to quickly split the screen through the split screen floating ball 704. Figure 7 As shown in Figure (c) in the figure. The split screen floating ball 704 includes an icon 705 of the APP (browser APP) currently opened by the user and an icon 706 of the note APP, which is used to prompt the user to display the browser APP and the note APP in split screen. When the user clicks the split screen floating ball 704, the mobile phone displays the browser APP and the note APP in split screen, and automatically writes the link of the web page opened by the current browsing APP into the content input area 702, as shown in Figure 703. Figure 7 As shown in Figure (d) in the figure. After that, the user can click Figure 7 The recording control 302 in (d) of FIG. 1 and the recording in the system are started by clicking the recording in the system option in the pop-up recording type tab sheet. For details, see Figure 3 Figures (c) and (d) in the figure are not repeated here.

[0182] When the user does not click the split screen floating ball 704, the link of the current web page is copied to the clipboard. The user can open the note APP, create a new note, and manually click the paste control in the note interface to paste the link of the web page into the content input area, which will not be repeated here.

[0183] It should be noted that the controls in the interface and the process of interface changes in the above embodiments are only examples and do not impose any limitation on the content of this application.

[0184] In addition, when the user starts the recording function in the system for the first time, in order to protect user privacy and improve security, the note APP can also ask the user for authorization in the form of a pop-up window. Figure 8This is a schematic diagram of an application authorization interface for the recording function in the system provided in the embodiment of the present application. Figure 3 In Figure (c), if the user uses the recording function in the system for the first time, in response to the user clicking Figure 3 In the system recording option 304 in Figure (c), an authorization application window 801 pops up in the note interface, such as Figure 8 The authorization application window 801 includes an allow option 802, a reject option 803, and a reject and do not ask again option 804. If the user clicks the allow option 802, the recording function in the system is activated and recording begins. The interface jumps to Figure 3 If the user clicks on the reject option 803, the recording function in the system will not be activated. If the user clicks on the reject and do not ask option 804, the recording function in the system will not be activated, and when the user clicks on the recording option 304 in the system later, the authorization application window 801 will not pop up.

[0185] After the user starts the note APP and its in-system recording function according to any of the above methods, the note APP calls the relevant modules in the electronic device to start performing the in-system recording operation. At the same time, if the user enters the note content in the content input area in the note interface, the note APP records the content entered by the user, establishes mapping relationship information, and generates notes. Afterwards, the note APP can also present the generated notes and realize two-way positioning. The following will take the scenario of learning through the online course APP, and the split-screen display of the online course APP and the note APP as an example to illustrate the specific process of the note APP realizing the above functions. For the sake of convenience, the realization of the note APP function is divided into the following processes:

[0186] (1) Note generation process;

[0187] (2) Note stopping process;

[0188] (3) Note sequence presentation process;

[0189] (4) Playback progress jump process;

[0190] (5) Note content jump process.

[0191] As mentioned above, the note content is divided into H5 elements and handwritten content. Therefore, the following combines the interface change diagram and the timing flow chart to explain the implementation method of the above process when the note content is H5 elements and handwritten content.

[0192] 1. The note content is an H5 element

[0193] (1) Note generation process

[0194] The note generation process includes the audio recording process and the note content writing process. First, take the first time a user starts the recording function in the system as an example to explain the audio recording process.

[0195] Corresponding to the above Figure 3 , Figure 8 as well as Fig. 9 In the case where the online course APP and the note APP are displayed in split screen, the timing flow diagram of the note generation process provided in this embodiment can be as follows: Fig. 9 As shown, the audio recording process includes steps S101 to S122:

[0196] S101. In response to a user opening a video in an online course APP, the online course APP instructs a media player of an application framework layer to play the video by calling a function.

[0197] Among them, the operation of opening a video can be an operation of opening a video file, or an operation of entering a live broadcast, etc. This application does not impose any limitation on this.

[0198] S102: After receiving an instruction to play a video, the media player instantiates a video object to play the video.

[0199] S103: In response to the user clicking the recording option in the system for the first time, the interface module in the note APP sends a command to start recording in the system to the recording and playback control module.

[0200] The command to start recording in the system is used to instruct to start recording in the system.

[0201] S104: The recording and playback control module responds to the recording instruction in the start system, and after the recording module permission check of the application framework layer, determines that the permission check fails.

[0202] Specifically, each time the recording and playback control module receives a command to start recording in the system, it initiates a permission verification request to the recording module, and the recording module performs the permission verification. The recording module mainly performs two aspects of verification. On the one hand, it determines whether there is user authorization information, which indicates that the user agrees that the recording module captures the audio in the system. The second aspect is to determine whether the APP requesting to capture the audio in the system meets the preset requirements. Optionally, in this embodiment, the preset requirements can be: the capture policy of the APP is AudioAttributes.ALLOW_CAPTURE_BY_ALL.

[0203] The recording module verifies the two pieces of information. If the verification is successful, the recording module returns verification success information to the recording and playback control module; if the verification is unsuccessful, the recording module returns verification failure information to the recording and playback control module. In this embodiment, the user starts the recording function in the system for the first time, so there is no user authorization information, the authority verification fails, and step S105 is executed.

[0204] S105: The recording and playing control module calls a function to instruct the AMS of the application framework to display an authorization application window.

[0205] In a specific embodiment, the recording and playback control module can instruct the AMS to display the authorization application window by calling the MediaProjectionManager.createScreenCaptureIntent() function.

[0206] S106. The AMS displays an authorization application window in response to the call of the recording and playback control module. The authorization application window is used to prompt the user to authorize the system recording.

[0207] Optionally, the authorization application window can be as above Figure 8 As shown in 801.

[0208] S107. In response to the user clicking the Allow option in the authorization application window, the interface module of the note-taking APP sends user authorization information to the recording and playback control module.

[0209] S108. After receiving the user authorization information, the recording and playback control module verifies the permission of the recording module of the application framework layer and determines that the permission verification has passed.

[0210] It is understood that after obtaining the user authorization information, the recording and playback control module can save the information. In this way, the next time the user starts the recording function in the system, the recording module can directly obtain the authorization information during the permission verification without having to apply for authorization from the user.

[0211] S109: After the permission check is passed, the recording and playback control module calls a function to instruct the recording module to construct an object, and instructs the configuration parameters of the recording in the system to be added to the object.

[0212] Optionally, the recording and playback control module can construct an object by calling newAudioRecord(). In addition, it can indicate to add an AudioPlaybackCaptureConfiguration configuration parameter to the constructed object. The configuration parameter indicates recording in the recording system, that is, indicates capturing audio in the system.

[0213] S110. The recording module constructs an object in response to an instruction of the recording and playing control module and adds configuration parameters of recording in the system to the object.

[0214] S111, the recording and playing control module calls a function to instruct the recording module to start recording in the system, and sends a blank removal mark to the recording module. The blank removal mark is used to indicate that when there is no audio playing in the system, the recording in the system is paused.

[0215] Optionally, the recording and playback control module can start recording in the system by calling the Audio_start() function.

[0216] Optionally, the recording and playback control module can send a blanking mark through a custom interface opened by the software system to the note APP. The custom interface is an interface customized according to the functional requirements of the note APP during system design. In a specific embodiment, the blanking mark can be a value of the blanking mark bit, and the blanking mark can be true, for example.

[0217] S112: The recording and playing control module sends a recording progress display instruction to the interface module.

[0218] The recording progress display instruction is used to instruct the interface module to display the recording progress control.

[0219] S113. The interface module displays a recording progress control in response to the recording progress display instruction.

[0220] At this point, the interface of the electronic device can jump to Figure 3 Figure (d) in .

[0221] S114, after receiving the instruction to start recording in the system and the blank removal mark, the recording module executes the blank removal recording process, that is, when it is determined that the media player is instantiated and the audio stream exists, the audio stream in the system is captured.

[0222] The media player is instantiated, or there is an instance in the media player, which means that there is an instantiated object in the media player.

[0223] Specifically, when a media player plays audio or video, on the one hand, the object needs to be instantiated, and on the other hand, an audio stream will exist. Therefore, the recording module can determine whether the media player is playing audio or video by determining whether the media player is instantiated and whether there is an audio stream. If the media player is instantiated and there is an audio stream in the media player, it is determined that the media player is playing audio or video, and there is an audio stream in the system, and the audio stream in the system is captured.

[0224] In actual applications, there are some scenarios where there is no audio stream in the system, such as when the user pauses the video playback in the online course APP, or the microphones of the participants are muted before the live broadcast starts, or the user only opens the note APP and does not open other APPs that can play audio. In these scenarios, the recording module determines that the media player is not instantiated and / or there is no audio stream in the media player, and thus determines that there is no audio stream in the system and does not capture the audio stream, that is, the recording is paused. This can reduce the blank recordings in the recorded audio files. On the one hand, the recorded audio files will not be too large, saving storage space for electronic devices. On the other hand, when playing the recorded audio files, there are no blank recordings, and the user does not need to adjust the playback progress, thereby improving the user experience.

[0225] S115. The recording module returns the captured audio stream and recording duration in the system to the recording and playback control module.

[0226] The recording duration refers to the total duration of recording by the recording module, excluding the duration when recording is paused.

[0227] S116: The recording and playback control module records the recording start time.

[0228] Specifically, the recording and playback control module records the time when the audio stream is first received as the recording start time. The recording start time represents the time when the recording starts, which is an absolute time, for example, 8:00:00 on October 18, 2022.

[0229] S117, the recording and playback control module sends the recording start time to the H5 editor.

[0230] S118. The H5 editor records the recording start time in the content file in the storage module.

[0231] The main use of the recording start time is: when the note content is handwritten content, the H5 editor can determine the handwriting relative time of the stroke according to the recording start time recorded in the content file and the time when each stroke is put down when displaying the note content later (further explained in the subsequent embodiments). In this embodiment, the note content is an H5 element, and the H5 editor may not record the recording start time. However, in order to simplify the operation logic, regardless of the input mode, the recording and playback control module sends the recording start time to the H5 editor, and the H5 editor saves the recording start time for subsequent use when displaying the handwritten content. Of course, in other embodiments, the H5 editor may also obtain the recording start time from the recording and playback control module when it determines that the input note content is handwritten content, and the present application does not impose any limitation on this.

[0232] It can be understood that when the note APP creates a new note, a group of files such as content files, recording files and mapping files can be created in the storage module. The content of these files is empty in the initial state. When the recording starts or the note content starts, the H5 editor writes some information (such as the start time of recording) or the note content into the content file. Similarly, when the recording starts, the recording and playback control module writes the audio stream into the recording file. In addition, after generating the mapping relationship information, the H5 editor can write the mapping relationship information into the mapping file. For details, please refer to the description of the subsequent embodiments.

[0233] S119: After receiving the audio stream, the recording and playing control module writes the audio stream into the recording file in the storage module.

[0234] S120: The recording and playback control module sends the recording duration to the interface module.

[0235] S121. The interface module refreshes the recording duration information in the recording progress control according to the received recording duration.

[0236] Optionally, the interface module may immediately refresh the recording duration information in the recording progress control upon receiving the recording duration, or may periodically refresh the recording duration information, for example, refreshing the recording duration information once every 1 second.

[0237] S122, the recording and playback control module sends the recording duration to the H5 editor in real time.

[0238] The above process is the audio recording process. The following is an explanation of the note content writing process.

[0239] For example, Fig.10 This is a schematic diagram of the interface changes in the note generation process in the text input mode provided in an embodiment of the present application. The user can enter the note content during the recording process:

[0240] Continue Figure 3 As shown in Figure (d) Fig.10 As shown in Figure (a), after the recording function in the system is started, the note interface includes a recording progress control 1001, and the recording progress control 1001 includes a pause recording control 1002 and recording duration information 1003. Optionally, if the mobile phone supports the voice-to-text function and the current network connection of the mobile phone is normal, the note APP can also start the voice-to-text of the recording content when starting the recording, and can display the relevant information of the voice-to-text (such as the voice-to-text control, the text content obtained by the voice-to-text, etc.) in the area 1004 in the recording progress control 1001.

[0241] At this time, the user Fig.10In the content input area 702 of FIG. (a), the user types the four characters "modernism" through the soft keyboard, and when the user completely types the four characters "modernism", the recording duration is 00:00:02, i.e., 2 seconds, and the mobile phone displays the following: Fig.10 In the embodiment of the present application, for text input, the time of complete input refers to the time when the electronic device receives all the information of the text content, rather than the time of other states (such as style change, typesetting, list, etc.) during the input process.

[0242] exist Fig.10 In the interface shown in Figure (a), the user further enters some text content, such as Fig.10 As shown in Figure (b).

[0243] Corresponds to Fig.10 The interface change process, the timing flow diagram of the note generation process provided in this embodiment can be as follows Fig.11 As shown, the audio recording process includes steps S123 to S129:

[0244] S123. In response to the user typing an H5 element (taking element a as an example) in the content input area, the interface module sends element a to the H5 editor.

[0245] As described above, the information contained in element a includes element content a, input position of element content a, attributes of element content a, format of element content a, and the like.

[0246] For details, please refer to Fig.10 In Figure (a), the user types the four words "Modernism" at the cursor position in the content input area 702. The interface module responds to the user's typing operation and sends the four words "Modernism" and the input position, attributes and format of the four words "Modernism" to the H5 editor.

[0247] S124. The H5 editor determines that the content type of element a is an H5 element according to the current input mode of the note APP.

[0248] It can be understood that H5 Editor, as an editing module of the note APP, can recognize the switching of the input mode of the note APP and the current input mode of the note APP. Specifically, when the user clicks the handwriting control in the note interface (such as Fig.10 In the handwriting control 305 in (a), the H5 editor can recognize that the note APP has entered the handwriting input mode. At the same time, each non-handwriting input mode can have a corresponding control in the note interface. When the user clicks a certain control, the H5 editor can recognize that the note APP has entered the corresponding non-handwriting input mode. For example, when the user clicks Fig.10In the picture control 1006 in Figure (a), the H5 editor recognizes that the note APP has entered the picture input mode.

[0249] The H5 editor determines the content type of the note content according to the input mode of the note APP when a certain note content is input. When the input mode of the note APP is handwriting input mode, the H5 editor determines that the content type of the note content is handwriting content. When the input mode of the note APP is non-handwriting input mode, that is, any one of text typing mode, picture input mode, audio input mode, video input mode, table input mode, etc., the H5 editor determines that the content type of the note content is an H5 element. In a specific embodiment, when the value of the content type parameter (Type) is 1, it indicates an H5 element, and when the value of the content type parameter (Type) is 2, it indicates handwriting content.

[0250] Of course, in other embodiments, the H5 editor may also specifically distinguish each type of H5 element and set a unique identifier for each type of H5 element. For example, when the value of the content type parameter (Type) is 1-1, it indicates a text element, and when the value of the content type parameter (Type) is 1-2, it indicates a picture element.

[0251] In this embodiment, the note APP is currently in text input mode, so the H5 editor can determine that the content type of element a is an H5 element.

[0252] S125. The H5 editor displays element a in the content input area in the form of an H5 page based on the H5 language.

[0253] Specifically, the editing language of the H5 editor is the H5 language, and the note content is recorded in the form of tags, and the display form of the note content is an H5 page (web page). For an H5 page edited in the H5 language, each set of user input can be called a set of H5 elements, and each content in a set of H5 elements is called an H5 sub-element (referred to as a sub-element). And each set of H5 elements corresponds to a set of tags (which can be one, a pair, or multiple). The H5 editor uses this set of tags as a marker to record the element content of the H5 element, as well as the input position, attributes, format and other information of the element content. Among them, each sub-content contained in the element content is called a sub-element content, the first sub-element content in a set of element content is called the starting sub-element content (hereinafter referred to as the starting sub-content), and the last sub-element content is called the ending sub-element content (hereinafter referred to as the ending sub-content, or the last sub-content). In addition, when recording the input position, a set of element content can be anchored to a certain line in the H5 page through a paragraph tag.

[0254] by Fig.10Take the group of elements "Modernism" in Figure (a) as an example. This group of elements includes four sub-elements "Modern", "Generation", "Main" and "Justice". Therefore, the element content recorded in the content file is "Modernism", and this element content includes four sub-element contents "Modern", "Generation", "Main" and "Justice". Among them, "Modern" is the starting sub-content and "Justice" is the ending sub-content. This element corresponds to a group of tags, which records the element content "Modernism" and marks and limits the input position, attributes, format and other information of "Modernism". Among them, regarding the input position of the element content, the starting position of the element content "Modernism" corresponds to a starting paragraph tag . This paragraph tag is the first paragraph tag in the H5 page, so it can be anchored that the element content "Modernism" is located in the first paragraph of the H5 page.

[0255] S126. The H5 editor writes element a into the content file in the storage module through tag a based on the H5 language. Tag a is used to mark element content a of element a and the position of element content a.

[0256] That is, the content file includes a group of tags a corresponding to element a, and the group of tags a represents the element content a of element a and the input position, attributes, format and other information of the element content a.

[0257] After the H5 editor writes element a into the content file based on the H5 language, the element a can be displayed in the form of an H5 page by reading and parsing tag a in the content file.

[0258] S127. The H5 editor determines the H5 relative time a corresponding to the element a according to the current recording duration.

[0259] Here, the current recording duration refers to the recording duration corresponding to the moment when element a is completely typed. The moment when element a is completely typed is also the moment when the electronic device receives all the information contained in element a, rather than the moment in other states during the input process (such as the moment of style change, layout, list, etc.). It is similar for other types of H5 elements. When determining the H5 relative time, it is determined based on the recording duration corresponding to the moment when the H5 element is completely input. For example, when inserting an image element, the H5 relative time is determined based on the recording duration corresponding to the moment when the image is completely inserted, rather than determining the H5 relative time based on the moment when the user operates the controls in the interface, or the moment when the user takes or selects a picture.

[0260] Optionally, the H5 editor can generate an H5 relative time according to preset rules based on the current recording duration sent by the recording and playback control module. In a specific embodiment, the H5 editor can round the current recording duration backward to obtain the H5 relative time. For example, when the user enters an element in the content input area, the recording duration sent to the H5 editor by the recording and playback control module is 1 second and 42 milliseconds (00:00:01:42), then the H5 editor rounds the recording duration backward to obtain an H5 relative time of 2 seconds (00:00:01). Of course, the H5 editor can also round the current recording duration forward to obtain the H5 relative time, or calculate according to other rules. In another specific embodiment, the H5 editor can also directly determine the current recording duration as the H5 relative time.

[0261] For ease of explanation, in the following embodiments, "H5 relative time is the time obtained by rounding the current recording time backward by whole seconds" is used as an example for explanation.

[0262] S128. The H5 editor generates H5 position information a according to tag a. The H5 position information a represents the position of element content a recorded in the content file in the H5 page.

[0263] As described above, tag a can represent element content a and the input position of element content a. Therefore, the position information of element content a in the H5 page can be parsed and extracted through tag a, which is referred to as H5 position information a in the embodiment of the present application. It is worth noting that although H5 position information a and the input position of the above-mentioned element content a both represent the position of element content a, H5 position information a is the position recorded in the content file by the H5 editor, which is the information used when the H5 page is displayed, and represents the position of element content a in the H5 page. The input position of element content a is the position information sent by the interface module to the H5 editor, which is a parameter collected when the electronic device receives the user's note content, and represents the position of element content a on the screen or in the interface when it is input. H5 position information a is represented by the relevant information in tag a, and the input position of element content a is generally represented in the form of coordinate points, etc.

[0264] In simple terms, the H5 editor first converts the input position of the element content a sent by the interface module into the tag a, and then extracts the H5 position information a based on the tag a and other content recorded in the content file. The H5 editor can display element a at the corresponding position in the H5 page by reading the content file based on the H5 position information a, but the H5 editor cannot display element a at the corresponding position in the H5 page by reading the content file based on the position of the element content a.

[0265] Optionally, the H5 position information a may include the absolute position and relative position of the starting sub-content a0 in the element content a, and the absolute position and relative position of the ending sub-content an. Among them, the absolute position of the starting sub-content a0 refers to the position of the starting sub-content a0 in the H5 page. Optionally, the absolute position of the starting sub-content a0 can be represented by the paragraph where the starting sub-content a0 is located and the position in the paragraph. The paragraph where the starting sub-content a0 is located can be represented by the paragraph order number of the paragraph tag a in the tag a in the content file. Specifically, the H5 editor can parse all paragraph tags in the content file, and determine which group of paragraph tags the paragraph tag a is among all paragraph tags, that is, the paragraph order number of the paragraph tag a in the content file. The position of the starting sub-content a0 in the paragraph can be represented by the order of the starting sub-content a0 in the element content defined by the paragraph tag a. Specifically, the H5 editor can parse all element contents defined by the paragraph tag a, and determine the order number of the starting sub-content a0 in these element contents (that is, which one it is in the paragraph). The relative position of the starting subcontent a0 refers to the position of the starting subcontent a0 in the element content a. The relative position of the starting subcontent a0 can be represented by the ordinal number of the starting subcontent a0 in the element content a. As described above, the starting subcontent is the first subelement content in a group of element contents, so the ordinal number of the starting subcontent a0 in the element content is the first, which can be represented as 0.

[0266] The representation and determination method of the absolute position and relative position of the ending sub-content an are the same as those of the starting sub-content a0, and will not be described in detail.

[0267] That is to say, the H5 position information actually anchors a position range in a paragraph in the H5 page through the positions of the starting sub-content and the ending sub-content. The position range is the position of element a recorded in the content file in the H5 page.

[0268] S129. The H5 editor generates mapping relationship information a between H5 relative time a and content type of element a (H5 element) and H5 location information a.

[0269] That is, mapping relationship information a of element a is generated, and mapping relationship information a is used to represent the correspondence between H5 relative time a, content type of element a and H5 position information a.

[0270] In a specific embodiment, the mapping relationship information can be represented by a map with the following structure:

[0271]

[0272] Combination Fig.10 In Figure (a), assume that the element content input by the user when the recording duration is 1 second and 10 milliseconds is "Modernism", and the input mode of the note APP during input is the text typing mode. Therefore, the H5 editor determines that the content type of "Modernism" is an H5 element according to the input mode. The H5 editor rounds up the recording duration of 1 second and 10 milliseconds to the nearest whole second to obtain an H5 relative time of 2 seconds. And according to the label corresponding to this element, it is determined that the paragraph where the starting sub - content "现" in "Modernism" is located is the first paragraph (denoted as p1), and the starting sub - content "现" is the first sub - content in the first paragraph (denoted as 0). Therefore, the absolute position of the starting sub - content "现" can be expressed as [p1, 0]. The starting sub - content "现" is the first sub - element content in the element content "Modernism". Therefore, the relative position of the starting sub - content "现" can be expressed as 0. The paragraph where the ending sub - content "义" is located is also the first paragraph (denoted as p1), and the ending sub - content "义" is the fourth sub - element content in the first paragraph (denoted as 3). Therefore, the absolute position of the ending sub - content "义" can be expressed as [p1, 3]. The ending sub - content "义" is the fourth sub - element content in the element content "Modernism". Therefore, the relative position of the ending sub - content "义" can be expressed as 3.

[0273] Therefore, the map of the "Modernism" element established by the H5 editor can be:

[0274]

[0275]

[0276] Following the above process, each time the user makes an input, a set of H5 position information and content type will be generated. The number of groups of H5 elements input by the user within 1 second is random. Therefore, the H5 position information and content type corresponding to the same H5 relative time may be one group or multiple groups. The H5 editor can integrate multiple groups of H5 position information and content type corresponding to the same H5 relative time into the same map. In this way, each map corresponds to an H5 relative time, and the H5 position information and content type in each map may be one group or multiple groups. For example, assume that after the user inputs "Modernism" at 1 second and 10 milliseconds, the user also inputs the symbol "X" at 1 second and 59 milliseconds. Then, in the map with an H5 relative time of 2 seconds (second: 2) generated by the H5 editor, there are two groups of H5 position information and content type, that is, two groups of "noteUnites".

[0277] The above steps S123 to S129 illustrate the process of the H5 editor writing note content to the content file and generating mapping relationship information when the user enters the note content in the content input area. When there is no input in the content input area, the H5 editor does not perform any operation. That is to say, the recording and playback control module sends the recording duration to the H5 editor, and the H5 editor does not perform any operation when there is no input in the content input area (the note content has not changed). In the case of an input H5 element in the content input area, the H5 editor generates mapping relationship information based on the latest recording duration (called the current recording duration) sent by the recording and playback control module and the label corresponding to the H5 element. The mapping relationship information represents the correspondence between the H5 relative time and the H5 position information and the content type. Among them, the H5 position information represents the position of the content of the input H5 element recorded in the content file in the H5 page, and the H5 relative time is determined according to the current recording duration.

[0278] Of course, in some embodiments, when generating mapping relationship information, the content type may be ignored, and only the mapping relationship between the H5 relative time and the H5 location information may be established.

[0279] The above description is made by taking the note APP in text input mode as an example in combination with the accompanying drawings. The following description is made of the note generation process in the picture input mode in combination with the interface diagram.

[0280] While the recording is continuing, the user can click the image control in the note interface (e.g. Fig.10 Optionally, the user can insert a photo taken by the camera APP or a photo saved in the electronic device. Fig.10 In Figure (b), the interface after the user inserts an image into the note interface can be as follows Fig.12 shown.

[0281] In the picture input mode, the pictures inserted by the user at one time are a group of H5 elements (specifically picture elements). If the user inserts multiple pictures at one time, the group of H5 elements includes multiple sub-elements (specifically picture sub-elements), and each picture is a sub-element. A group of picture elements corresponds to a group of tags, and the element content in this group of tags is the group of pictures itself. Optionally, for picture elements, the element content defined in the tag may only include information such as the name of the picture, and the pictures can be stored separately in a preset folder. However, the information such as the picture name in the tag has a unique correspondence with the pictures in the folder. In the picture input mode, the process of generating notes in the note APP is the same as above. Fig.11 The process shown in is similar and will not be described in detail.

[0282] As for other non-handwriting input modes, they are similar to the text input mode and the picture input mode and will not be described in detail.

[0283] It should be noted that, while recording is in progress and the H5 editor has generated some mapping relationship information, if the user modifies the input H5 element, the H5 editor can update the mapping relationship information according to the user's modified content and modification method in addition to the content recorded in the corresponding modified content file. Specifically, the following steps are included:

[0284] 1) In response to the user's modification operation on the input H5 element, the interface module sends a modification element to the H5 editor, which includes information such as modification type, modification content and modification position; the modification type may include deletion, addition, replacement, etc., the modification position refers to the position of the modified content, and the modification content may include deleted content, added content and replaced content (including deleted content and content added after deletion), etc.

[0285] 2) The H5 editor displays the modified elements in the content input area in the form of an H5 page.

[0286] 3) The H5 editor modifies the content file according to the modification type, modification content and modification position. Specifically, if the modification type is addition, the modified content is added to the content file according to the modification position; if the modification type is deletion, the modified content is deleted from the content file according to the modification position; if the modification type is replacement, the deleted content is deleted from the content file according to the modification position, and the added content after the deletion is written.

[0287] 4) The H5 editor determines the absolute position of the modified content in the modified element in the H5 page (called modified absolute position) according to the tag corresponding to the modified element, and updates the mapping relationship information according to the modified absolute position and the modified content. Specifically, when updating the mapping relationship information, it can be divided into the following situations:

[0288] a. Delete the H5 element in the entire group

[0289] That is, the modification type is deletion, and the deleted content is all sub-elements in a group of H5 elements. Take the case where a user deletes a group of H5 elements x as an example. In this case, the positions of all elements displayed after the H5 element x in the H5 page are changed. The H5 editor determines the H5 position information x in the mapping relationship information that matches the modified absolute position. In other words, it is determined which H5 position information anchors the modified absolute position, and the determined H5 position information is called H5 position information x. For ease of description, the H5 relative time corresponding to the H5 position information x is called H5 relative time x. Afterwards, the H5 editor deletes the H5 position information x, and modifies the mapping relationship information of each group of elements entered after the H5 element x according to the number of sub-elements x1 contained in the H5 element x. Specifically, the H5 editor can determine the H5 position information corresponding to the H5 relative time greater than the H5 relative time x, and subtract x1 from the order numbers of the starting sub-content and the last sub-content in the paragraph in these H5 position information.

[0290] For example, if a user deletes Fig.10 In the figure (b), the H5 editor deletes the "noteUnites" of "Modernism" in the 2s map, and reduces the order of the first sub-content and the last sub-content in the paragraph by 4 in all maps with a relative time greater than 2s. Of course, if the order of the paragraphs changes after the modification, the paragraph order number also needs to be modified accordingly. The same is true for the following cases, which will not be repeated here.

[0291] b. Delete some of the H5 elements in a group

[0292] That is to say, the modification type is deletion, and the deleted content is some sub-elements in a certain group of H5. Take the case where the user deletes some sub-elements in H5 element x as an example. In this case, the positions of all elements displayed after the deleted sub-elements in the H5 page are changed. The H5 editor determines the H5 position information x that matches the modified absolute position in the mapping relationship information. In other words, it is determined which H5 position information anchors the modified absolute position, and the determined H5 position information is called H5 position information x. The H5 relative time corresponding to the H5 position information x is called H5 relative time x. Afterwards, the H5 editor modifies the H5 position information x and the mapping relationship information of each group of elements entered after the H5 element x according to the number of deleted sub-elements x2. Specifically, the H5 editor resets the absolute position and relative position of the starting sub-content in the H5 position information x, as well as the absolute position and relative position of the ending sub-content. Afterwards, the H5 editor will subtract the number of deleted sub-elements x2 from the order numbers of the starting sub-content and the last sub-content in the paragraph in the mapping relationship information corresponding to all H5 relative times that are greater than the H5 relative time x.

[0293] For example, if a user deletes Fig.10 In the figure (b), the H5 editor modifies the information in the "noteUnites" of "Modernism" in the 2smap, modifies the absolute position of its end sub-content to "P1, 1", and modifies the relative position of the end sub-element to 1. At the same time, the H5 editor adjusts all maps with H5 relative time greater than 2s, and reduces the order number of the start sub-content and the order number of the end sub-content in these maps in the paragraph by 2.

[0294] c. Replace part or all of the content in a group of H5 elements

[0295] That is to say, the modification type is replacement, and the object of replacement is part or all of a group of H5 elements. Take the example of a user replacing part or all of the sub-elements in the H5 element x. In this case, the number of sub-elements in this group of H5 elements does not change, and therefore, the positions of other elements in the H5 page except the H5 element do not change. The H5 editor determines the H5 position information x in the mapping relationship information that matches the modified absolute position. The H5 relative time corresponding to the H5 position information x is called the H5 relative time x. Afterwards, the H5 editor modifies the H5 position information x according to the number and relative position of the deleted sub-elements. At the same time, the H5 editor determines the H5 relative time p according to the recording duration when the content added after the deletion is fully input, and determines the H5 position information p of the content added after the deletion in the H5 page recorded in the content file, and generates the mapping relationship information between the H5 relative time p and the H5 position information p. In short, when the modification type is replacement, the content added after the deletion needs to create new mapping relationship information.

[0296] The process of modifying H5 position information x includes two cases: In the first case, the deleted content contains the starting sub-content or the ending sub-content, that is, the deleted content is the sub-element at both ends of the H5 element x, then the H5 editor resets the absolute position and relative position of the starting sub-content in the H5 position information x, as well as the absolute position and relative position of the ending sub-content. In the second case, the deleted content does not contain the starting sub-content or the ending sub-content, that is, the deleted content is the sub-element in the middle of the H5 element x, then the H5 editor regenerates an H5 position information for the content before and after the deleted content in the element content of the H5 element x, and both of these H5 position information are mapped to the H5 relative time x. In other words, the H5 position information x is split to form two H5 position information, thereby generating two groups of "noteUnites" to exclude the position of the deleted content in the H5 position information corresponding to the H5 relative time x.

[0297] d. Add content in the middle of a group of H5 elements

[0298] That is to say, the modification type is addition, and the modification position is in the middle of a group of H5 elements. Take the case where the user adds content in the middle of H5 element x as an example. In this case, the positions of all elements after the position where the content is added are changed. The H5 editor regenerates an H5 position information for the content before and after the added content in the element content of H5 element x, and both H5 position information are mapped to the H5 relative time x. In other words, the H5 position information x is split to form two H5 position information, thereby generating two groups of "noteUnites" to exclude the position of the added content in the H5 position information corresponding to the H5 relative time x. At the same time, the H5 editor determines the H5 relative time p based on the recording duration when the added content is fully input, and determines the H5 position information p of the added content recorded in the content file in the H5 page, and generates the mapping relationship information between the H5 relative time p and the H5 position information p. In short, the added content needs to create a new mapping relationship information. Afterwards, the H5 editor modifies the mapping relationship information corresponding to each group of elements input after the H5 element x based on the number of added child elements x3. Specifically, the H5 editor may determine the H5 position information corresponding to the H5 relative time greater than the H5 relative time x, and increase the order numbers of the starting sub-content and the last sub-content in the paragraph in the H5 position information by x3 respectively.

[0299] e. Add content before or after a group of H5 elements

[0300] That is to say, the modification type is addition, and the modification position is before or after a group of H5 elements, not in the middle of the H5 element. For example, if a user adds content before H5 element x, in this case, the added content creates new mapping relationship information, and the positions of all elements after the added content are changed. For details, please refer to the above case d, which will not be described in detail.

[0301] In this embodiment, during the recording process, when the user modifies the input H5 element, not only the content recorded in the content file is modified, but also the mapping relationship information is updated, so that the accuracy of the mapping relationship information can be guaranteed, so that in the subsequent note presentation process, the two-way positioning between the recording playback and the note content is more accurate, further improving the user experience. In addition, in this embodiment, for the added content and the content added after deletion (collectively referred to as the added content), the mapping relationship information is re-established, so that in the subsequent note presentation process, the highlighted display of the added content and the two-way positioning of the recording playback can be achieved, which is convenient for the user to know the note recording process more clearly, further improving the user experience.

[0302] (2) Note stopping process

[0303] For example, Fig.13 This is a schematic diagram of interface changes in a note-taking stop process in a text input mode provided in an embodiment of the present application. Taking the electronic device as a mobile phone as an example, the user can stop taking notes by the following operations:

[0304] Continue Fig.10 As shown in Figure (b) Fig.13 As shown in Figure (a), when the recording duration is 5 minutes and 7 seconds (00:05:07), the user clicks the pause recording control 1002 in the note interface and defines the title of the note as "Modernism". In response to the user's operation, the note APP stops recording and jumps to Fig.13 The interface shown in FIG. (b) in FIG. The interface displays a first recording playback control 1301 and a prompt message 1308 pops up in the interface.

[0305] The prompt information 1308 includes the source of the audio recorded in the note (online course APP), which is used to prompt the user to save the original audio information (or original video) corresponding to the audio, such as the name of the original audio, the network link of the original audio, the author of the original audio, the recording time of the original audio, etc., so that the user can find the original audio (or original video) according to the note later, increase the interaction between the electronic device and the user, and improve the user experience. Optionally, the prompt information 1308 can be displayed for a preset time and then disappear.

[0306] like Fig.13 As shown in Figure (b), the first recording playback control 1301 includes a continue recording control 1302, a first playback control 1303, and a segmented recording viewing control 1309. After the user clicks the first playback control 1303, the note APP can play the currently recorded recording file, and synchronously display the note content, and can achieve two-way positioning. The recording can be restarted after the user clicks the continue recording control 1302. Optionally, after the recording is restarted, you can continue recording based on the original recording, or you can re-establish a recording to achieve segmented continuation. The user clicks the segmented recording viewing control 1309 to view multiple recorded recordings, and can delete, share, and perform other operations on each recording.

[0307] Optionally, when recording is paused, the user can continue to edit the note content in the content input area, or modify the entered note content. Fig.13 In the recording pause state shown in Figure (b) of FIG. , the user can type the letters "ABC" at the end of the text. For another example, in Fig.13 In the recording pause state shown in Figure (b), the user can also type the letter "D" between "Machine" and "Art". Of course, in the recording pause state, the user can continue to make other modifications, such as changing the note content, deleting the note content, etc., which will not be repeated here.

[0308] After the recording is paused, the user can click Fig.13 The save control 1305 in the interface shown in FIG. (b) completes the saving of the note, and the interface jumps to Fig.13 The interface shown in FIG. (c) includes a second recording and playing control 1306. The second recording and playing control 1306 includes a second playing control 1307.

[0309] Corresponding to the above interface change diagram, the timing flow diagram of the note stopping process provided in this embodiment can be as follows: Fig.14 As shown, the process includes:

[0310] S201. In response to a user clicking a pause recording control, the interface module of the note-taking APP displays a first recording playback control.

[0311] The interface changes corresponding to this step are as described above Fig.13 Figure (a) to Fig.13 The changes in Figure (b) will not be repeated here.

[0312] S202: The interface module sends an end recording instruction to the recording and playback control module.

[0313] The interface module displays the pause recording control and sends an end recording instruction to the recording and playback control module. The end recording instruction is used to indicate the end of recording.

[0314] S203: The recording and playing control module responds to the end recording instruction and calls a function to instruct the recording module of the application framework layer to end recording.

[0315] Optionally, the recording and playback control module can instruct the recording module to end recording by calling audio_end().

[0316] S204: After receiving the instruction, the recording module stops recording.

[0317] S205. The recording and playback control module displays the name of the APP together with the note APP in the PMS acquisition interface.

[0318] Specifically, the recording and playback control module may send a request message to the PMS, where the request message is used to request the PMS to determine the name of the APP that is currently displayed on the interface simultaneously with the note APP.

[0319] S206. The PMS determines that the name of the APP displayed in the interface simultaneously with the note APP is the online course APP, and sends the name of the online course APP to the recording and playback control module.

[0320] Optionally, the PMS can determine the window displayed in the current interface through WMA, and determine the information of the activity running in the window through AMS, and then determine the name of the APP running in the window based on the information of the activity. In this embodiment, the note APP and the online course APP are displayed in the interface in split screens, so the APP displayed on the interface at the same time as the note APP is determined to be the online course APP. The PMS sends the name of the online course APP to the recording and playback control module.

[0321] S207, the recording and playing control module sends the name of the online course APP to the interface module.

[0322] S208. The interface module displays prompt information in the interface according to the name of the online course APP.

[0323] Optionally, the prompt message can be Fig.13 As shown in 1308 in Figure (b) .

[0324] In addition, as an optional implementation, the recording and playback control module can also send the name of the online course APP to the H5 editor, and the H5 editor records the name of the online course APP in the content file. In this way, when the user opens the note later, the name of the online course APP is displayed on the web page, which is convenient for the user to know the source of the original audio and improve the user experience.

[0325] S209, the recording and playback control module sends an end recording instruction to the H5 editor.

[0326] S210. In response to the end recording instruction, the H5 editor writes all generated mapping relationship information into the mapping file in the storage module.

[0327] S211. The H5 editor sends mapping file information to the recording and playback control module.

[0328] Optionally, the information of the mapping file includes but is not limited to the file name and storage path of the mapping file.

[0329] S212: The recording and playback control module generates corresponding relationship information among the recording file, the content file and the mapping file.

[0330] Optionally, the recording and playback control module can establish a corresponding relationship table of the note titles according to the note titles input by the user. Fig.13 Taking the note title "Modernism" input by the user in Figure (b) as an example, the corresponding relationship table established by the recording and playback control module may be as follows:

[0331] Table 1

[0332]

[0333] It can be understood that Table 1 is only an example used to illustrate the content included in the correspondence information and does not represent actual data. Table 1 does not impose any limitations on this application. The actual correspondence information may also include more or less content than Table 1.

[0334] S213: The recording and playing control module persists the corresponding relationship to the storage module.

[0335] It should be noted that in the embodiment of the present application, for "pause recording" and "stop recording", for the note-taking APP, the above process is executed in response to the user clicking the pause recording control. Therefore, there is no need to make a clear distinction between "pause recording" and "stop recording".

[0336] The above process is Fig.13 The execution process of pausing recording corresponding to (a) and (b) in FIG. Fig.14 As shown, in the method provided in the embodiment of the present application, the process of continuing editing after recording is paused includes:

[0337] S214. In response to the user modifying an element in the content input area, the interface module sends the modified element to the H5 editor.

[0338] Similar to the above embodiment, modifying the H5 element during recording is similar to that in the above embodiment. Here, the modified element sent by the interface module to the H5 editor also includes information such as the modification position, modification content, and modification type.

[0339] S215. After receiving the modified element, the H5 editor displays the modified element in the content input area in the form of an H5 page.

[0340] S216. The H5 editor modifies the corresponding content in the content file according to the modification element.

[0341] S217. The H5 editor determines the absolute position of the modified content in the modified element in the H5 page according to the tag corresponding to the modified element (modifies the absolute position).

[0342] The method for determining the absolute position of the modification is similar to the method for determining the absolute positions of the starting sub-content and the ending sub-content during the note generation process. The paragraph where the modified content is located and the position in the paragraph can be determined by reading the content file, which will not be repeated here.

[0343] S218. The H5 editor updates the mapping relationship information in the mapping file according to the modified content and the modified absolute position.

[0344] The specific method of this step is similar to the above-mentioned embodiment of modifying the mapping relationship information when modifying the H5 element during recording. The difference is that in this step, when the modification type is addition or replacement, the added content does not need to create new mapping relationship information. Instead, the added content is used as a sub-element in a group of H5 elements before or after the modification position, and the mapping relationship information corresponding to the group of H5 elements is updated, and the mapping relationship information corresponding to each group of elements input after the group of H5 elements is modified.

[0345] Specifically, for case c in the above embodiment, unlike the method described in the above embodiment, in this embodiment, there is no need to create a new mapping relationship for the content added after deletion, and there is no need to split the H5 location information x. In this way, for case c, if the number of sub-contents in the content added after deletion is the same as the number of sub-contents deleted, the mapping relationship information in the mapping file does not need to be changed; if they are different, and the number of deleted sub-contents is large, then the processing method of case b is used; if they are different, and the number of sub-contents in the content added after deletion is large, then the processing method of case d is used.

[0346] Regarding the situation d in the above embodiment, different from the method described in the above embodiment, in this embodiment, there is no need to create a new mapping relationship for the added content, and there is no need to split the H5 location information x, but directly reset the absolute position and relative position of the starting sub-content of the H5 location information x, as well as the absolute position and relative position of the ending sub-content. The same parts of the process will not be repeated.

[0347] Regarding situation e in the above embodiment, different from the method described in the above embodiment, in this embodiment, there is no need to create a new mapping relationship for the added content. Instead, the mapping relationship information of a group of H5 elements before or after the modified position is modified directly according to preset rules, and the position corresponding to the newly added content is included in the position range in the mapping relationship information.

[0348] In addition, for the case where the modification type is addition, and the added content is after all the H5 elements that have been entered, that is, the content is added at the very end of the document: Optionally, in one implementation, the added content can be processed as the content of the last group of elements in the H5 elements that have been entered. In another implementation, the saved mapping file may not be changed. In other words, the mapping relationship information between the element added at the very end of the document when the recording is paused may not be established. When the subsequent notes are presented, if the user clicks on the modified element, the recording progress will not be jumped. This implementation can simplify the calculation logic of the H5 editor and improve the algorithm running speed and accuracy.

[0349] In addition, if the note APP supports segmented continuation, if the user clicks the play pause control, the recording and playback control module will save the recording file. When the user clicks the continue recording control, the recording and playback control module re-records a new recording file in the above steps S101 to S122. However, the content file does not need to be re-established, and the note content and mapping relationship information can be recorded in accordance with steps S123 to S129 during the process of establishing a new recording file. In this way, the final notes include multiple recording files, a content file and a mapping file.

[0350] The segmented recording function of the note APP makes it easy for users to delete one or several recording files later, meeting the needs of users in more scenarios. When a user deletes a recording file, the H5 editor can delete the corresponding mapping relationship information in the mapping file according to the H5 relative time corresponding to the recording file.

[0351] The method provided in the embodiment of the present application, in the process of generating notes, edits and saves the note content based on the H5 language through the H5 editor, and generates mapping relationship information of the information of the tag corresponding to the relative time and the note content. The H5 language follows the same parsing protocol in electronic devices of different systems, and can achieve cross-system data interoperability. In this way, when using the notes subsequently, there is no need to read the note content and mapping relationship information through the native module of the system in the electronic device, and the note content and mapping relationship information can be read through the H5 editor in the note APP installed in the electronic device, thereby realizing cross-system use of two-way positioning of the note content and the recording, thereby improving the user experience.

[0352] The above describes the process of generating note files by the note APP, taking the split-screen display of the online course APP and the note APP in the online course APP scenario as an example. For several other note APPs and their system recording startup methods, the note generation process is similar to the above process and will not be repeated. However, it should be noted that in the two methods of sharing and copying links, when the note APP is started, a new note is automatically created and the web link is recorded. The recorded web link can be displayed in the form of note content and the content input area, and written to the content file. In this way, it is not only convenient for users to know the source of the recording or note content, but also simplifies the user's operation and further improves the user experience.

[0353] The following describes the process of reading the notes generated according to the above process in combination with application scenarios and accompanying drawings.

[0354] For example, Fig.15 This is a schematic diagram of an application scenario of sharing notes provided in an embodiment of the present application. Users can share notes through a data sharing system. Specifically, Fig.15 As shown, the data sharing system may include a mobile phone, a cloud platform, and a tablet computer. Specifically, the software system of the mobile phone is an Android system, and the software system of the tablet computer is a Windows system. The note APP provided in the embodiment of the present application is installed in both the mobile phone and the tablet computer. Both the mobile phone and the tablet computer log in to the account of the cloud platform, so that the mobile phone and the tablet computer can synchronize data through the cloud platform. After the user generates a note through the note APP installed in the mobile phone, the file of the note (including recording files, content files, mapping files, etc.) is synchronized with the cloud through the cloud platform, so that the tablet computer can also obtain the file of the note.

[0355] The following is in Fig.15 It is understood that in this embodiment, presenting notes refers to playing the recording file and displaying the note content.

[0356] (3) Note sequence presentation process

[0357] The note sequence presentation process means that the recording files in the notes are played in the default order, and the note content displays the content of the corresponding position according to the playback progress of the recording files.

[0358] For example, Fig.16 This is a schematic diagram of an interface change in a process of sequentially presenting notes in a tablet computer provided in an embodiment of the present application. Fig.16 As shown in Figure (a) in FIG. 1 , the desktop of the tablet computer includes an icon of the note APP. When the user clicks the icon of the note APP to open the note APP, the user enters the following Fig.16 The interface shown in FIG. (b) in FIG. The left half of the interface displays the titles of multiple notes, and the right half of the interface displays the interface of the first note (i.e., "Modernism") by default. The note interface includes a second recording playback control 1306, and the second recording playback control 1306 includes a second playback control 1307. Moreover, all the note contents in the note interface are grayed out.

[0359] When the user clicks Fig.16 When the second play control 1307 of the note "Modernism" in FIG. (b) is clicked, the recording file in the note starts to play. Fig.16 As shown in Figure (c) in the figure, the note interface displays a playback progress control 1601, which includes a playback progress bar 1602, playback duration information 1603, and a playback pause control 1604. The playback duration information 1603 includes the current playback duration and the total recording duration. At the same time, when the recording is played to the H5 relative time in each mapping relationship information, the corresponding element content in the H5 page is highlighted. For example, Fig.16 As shown in Figure (c) in Figure 1, when the recording is played to 2 seconds (00:00:02), the four words "modernism" are highlighted. As the recording continues to play, other notes are gradually highlighted. For another example, Fig.16 As shown in Figure (d), when the recording plays to 2 minutes (00:02:00), all the content before "Machine Art" is highlighted.

[0360] Corresponding to the above interface change diagram, the timing flow diagram of the note sequence presentation process provided in this embodiment can be as follows: Fig.17 As shown, the process includes:

[0361] S401. In response to the user opening the note, the interface module sends a pre-display instruction to the H5 editor.

[0362] The pre-display instruction is used to instruct to pre-display the content in the content file. The pre-display method can be pre-set according to the requirements, and the gray display is used as an example for explanation in the embodiment of the present application.

[0363] Optionally, the pre-display instruction may include a note title or other information that can distinguish the note.

[0364] S402. The H5 editor reads the content file of the note from the storage module.

[0365] Optionally, the H5 editor can search Table 1 according to the note title in the pre-display instruction, determine the name of the content file corresponding to the note title, the storage path of the content file, etc., and then read the content file according to the name and storage path of the content file.

[0366] S403. The H5 editor grays out all the content in the content file.

[0367] The H5 editor grays out the text, pictures, tables, video play icons, audio play icons, and handwritten content in the content file. The display effect can be as follows: Fig.16 As shown in Figure (b).

[0368] Specifically, the H5 editor can modify the display style of all note contents in the content file to a grayed-out style, thereby making all element contents grayed-out.

[0369] S404: In response to the user clicking the second play control, the interface module displays a play progress control.

[0370] Corresponds to Fig.16 In Figure (b), the user clicks the second play control 1307, and the interface module displays the following Fig.16 The play progress control 1601 in Figure (c) .

[0371] S405: The interface module sends a play instruction to the recording control module.

[0372] Optionally, the play instruction may include information of the note opened by the user, such as the note title.

[0373] S406: In response to the play instruction, the recording and play control module reads the recording file and the mapping file from the storage module.

[0374] The recording and playback control module can search Table 1 according to the note title in the playback instruction, determine the name of the recording file corresponding to the note title, the storage path of the recording file, the name of the mapping file, the storage path of the mapping file, etc., and then read the recording file and mapping file from the storage module.

[0375] S407: The recording and playing control module parses the mapping file to obtain all mapping relationship information in the mapping file.

[0376] S408. The recording and playback control module sends all mapping relationship information to the H5 editor.

[0377] S409, after reading the recording file, the recording and playing control module starts playing the recording file from the 0th second and determines the playing time in real time.

[0378] S410, the recording and playing control module sends the playing time to the interface module in real time.

[0379] S411: After receiving the playback time sent by the recording and playback control module, the interface module refreshes the playback time information in the playback progress control.

[0380] S412, the recording and playback control module sends the playback duration to the H5 editor in real time.

[0381] S413, the H5 editor determines the current H5 relative time according to the current playback duration.

[0382] Specifically, the rule that the H5 editor uses to determine the current H5 relative time based on the current playback duration is the same as the rule that the H5 relative time is determined based on the recording duration during the note generation process. For example, if the H5 editor rounds the recording duration backwards to the nearest integer to obtain the H5 relative time during the note generation process, then the H5 editor here rounds the current playback duration backwards to obtain the current H5 relative time.

[0383] S414. The H5 editor determines whether the current H5 relative time exists in the mapping relationship information based on the JS polling mechanism; if so, execute step S415; if not, return to execute step S413.

[0384] Specifically, the H5 editor can poll and search each map in the mapping file based on the JS polling mechanism to determine whether there is a map that matches the current H5 relative time.

[0385] Optionally, the H5 editor can poll in real time, that is, each time the playback duration is received, a poll is performed. Optionally, the H5 editor can also preset a polling cycle and poll periodically. Specifically, after the H5 editor presets the polling cycle, it starts a polling search for the mapping file at the start of each polling cycle. In other words, the H5 editor starts a polling search for the mapping file once every preset polling cycle duration. Periodic polling can control the number of polling times of the H5 editor, reduce resource consumption, and save power consumption of electronic devices. Optionally, step S413 can also be started and executed at the start of each polling cycle to further reduce resource consumption and save power consumption of electronic devices.

[0386] In a specific embodiment, the preset polling cycle duration is less than the time difference between two adjacent H5 relative times in the mapping file. For example, according to the above embodiment, the H5 relative time in the mapping file is obtained by taking the whole second of the recording time, and thus, the time difference between two adjacent H5 relative times is 1 second. The preset polling cycle duration can be, for example, 100 milliseconds (ms), which reduces the number of polling times while ensuring the accuracy of the polling results, thereby improving the accuracy of the display of the note content, ensuring the smoothness of the display progress, and improving the user experience.

[0387] S415. The H5 editor determines the H5 position information (referred to as the current H5 position information) corresponding to the current H5 relative time according to the mapping relationship information, determines the element content (referred to as the current element content) matching the current H5 position information in the content file, and highlights the current element content.

[0388] Specifically, when it is determined that the current H5 relative time exists in the mapping relationship information, the H5 editor further determines the current H5 position information corresponding to the current H5 relative time. Afterwards, the H5 editor determines the corresponding current element content in the content file according to the current H5 position information, and modifies the display style of the current element content to a highlight style, so that the element content is highlighted.

[0389] When it is determined that the current H5 relative time does not exist in the mapping relationship information, the H5 editor returns to execute step S413. This cycle is executed until the playback stops (the recording file is played completely or the user clicks the play pause control). During the recording playback and the gradual highlighting of the H5 element, if the H5 element has a lot of content and cannot be fully displayed in the current H5 page, the H5 element can be scrolled by line to improve the display effect and further improve the user experience.

[0390] In other implementations, step S415 may also be replaced by the following step S415': the H5 editor determines the H5 position information corresponding to each H5 relative time that is less than or equal to the current H5 relative time based on the mapping relationship information, and based on these H5 position information, modifies the display style of the corresponding element content in the content file to a highlight style to highlight all of these element contents. The implementation process of step S415' is different from the above-mentioned step S415, but the display effect achieved by the two is the same. The advantage of this implementation method is that it can be consistent with the subsequent playback progress jump process, or the polling display method during the note content jump process, thereby reducing the algorithm's distinction between scenes and improving algorithm consistency.

[0391] It should be noted that in the embodiment of the present application, "grayed out display" and "highlighted display" are only used as an example, and are only used to illustrate that the display styles of unlocated note content and located note content are different. In other embodiments, other display methods may also be used, and the present application does not impose any limitations on this.

[0392] The method provided in this embodiment, during the sequential presentation process of notes, reads the content files and mapping relationship information edited in the H5 language through the H5 editor, thereby realizing the synchronization of the recording playback progress and the content file highlighting progress. There is no need to read the note content and mapping relationship information through the native module of the system in the electronic device, so it can be used across systems to improve the user experience.

[0393] (4) Playback progress jump process

[0394] The playback progress jump process is to locate the note content display progress through the recording playback progress. Fig.18 This is a schematic diagram of an interface change of a playback progress jump provided in an embodiment of the present application. Fig.18 As shown in Figure (a) in FIG. 1 , when the recording of the note “Modernism” is played to the 2nd minute and 30th second (00:02:30), the user drags the play progress bar 1602 in the play progress control 1601 to the 3rd minute and 28th second (00:03:28). Fig.18 As shown in Figure (b) in the figure. In response to the user's dragging operation, in addition to the change in the recording playback progress in the note interface, the highlighted content in the H5 page also changes, realizing the positioning of the note content display progress through the playback recording progress.

[0395] Corresponding to the above interface change diagram, the timing flow diagram of the playback progress jump process provided by this embodiment can be as follows: Fig.19 As shown, the process includes:

[0396] S501: In response to the user dragging the progress bar to the position of the playing time Tx, the interface module refreshes the current playing time in the playing progress control to Tx.

[0397] S502: The interface module sends a play jump instruction to the recording and playing control module, where the play jump instruction is used to instruct to adjust the playing time to Tx.

[0398] S503: The recording and playing control module responds to the play jump instruction and starts playing the recording file from the play time Tx.

[0399] S504, the recording and playing control module sends the play jump mark information and the play duration Tx to the H5 editor.

[0400] The play jump flag information is used to indicate that the play time Tx is obtained through the play jump instruction, and is used to distinguish the play time Tx from the play time sent in real time by the recording and playing control module.

[0401] Optional, in Fig.17 When step S415' is adopted in the illustrated embodiment, the play jump mark information may not be sent in step S504.

[0402] S505 , after receiving the play jump mark information and the play duration Tx, the H5 editor determines the H5 relative time Tx according to the play duration x.

[0403] S506. The H5 editor determines the H5 position information Tx corresponding to the H5 relative time that is less than or equal to the H5 relative time Tx according to the mapping relationship information and based on the JS polling mechanism, determines the element content Tx matching the H5 position information Tx in the content file, highlights the element content Tx, and grays out the other element contents except the element content Tx in the content file.

[0404] In this step, H5 location information Tx is a general term for one or more H5 location information. Correspondingly, element content Tx is also one or more groups of element content, that is, element content in one tag or multiple tags.

[0405] This process is similar to the process of step S415' in the above embodiment and will not be described in detail.

[0406] After executing step S506, the note APP continues to execute the above steps S410 to S415.

[0407] In this embodiment, when the user changes the recording playback progress, the H5 editor determines the H5 relative time Tx through the playback duration Tx, and searches for the H5 relative time corresponding to the H5 position information Tx that is less than or equal to the H5 relative time Tx based on the JS polling mechanism according to the mapping relationship information, and performs position matching and highlighting based on the H5 position information Tx. In this way, after the playback progress jumps, the H5 element corresponding to the playback progress after the jump and the H5 element input before are highlighted, and the H5 element after the H5 element corresponding to the playback progress after the jump is grayed out, realizing the positioning of the H5 element display progress through the playback recording progress.

[0408] (5) Note content jump process

[0409] The process of jumping to the note content is to locate the corresponding playback progress by clicking the note content at a certain position. Fig. 20 This is a schematic diagram of an interface change of a note content jump provided in an embodiment of the present application. Fig. 20 As shown in Figure (a) in the figure, the recording of the note "Modernism" is played to the 3rd minute and 28th second (00:03:28), and the note content is highlighted to "mid-twentieth century," and the user clicks on "Machine Art" in the note content. In response to the user's click operation, the note APP highlights "Machine Art" and the content entered before, grays out the content entered after "Machine Art", and jumps the recording playback progress to the 2nd minute and 30th second (00:02:30). Fig. 20 As shown in Figure (b), it is possible to locate the recording progress by clicking on the note content.

[0410] Corresponding to the above interface change diagram, the timing flow diagram of the playback progress jump process provided by this embodiment can be as follows: Fig.21 As shown, the process includes:

[0411] S601. In response to a user clicking on element m in the H5 page, the interface module sends a display jump instruction to the H5 editor, where the display jump instruction is used to instruct to jump the highlighted display progress to element m.

[0412] S602: In response to displaying the jump instruction, the H5 editor determines that the content type of the element m is an H5 element.

[0413] Specifically, the H5 editor parses the information contained in the element m, i.e., the element content of the element m (referred to as the element content m), and the content type, position, and other information of the element content m, according to the displayed jump instruction. When the H5 editor determines that the content type of the element content m is an H5 element, the H5 editor executes step S603; if the H5 editor determines that the content type of the element content m is handwritten content, the jump process corresponding to the handwritten content is executed, which will be further described in detail in the subsequent embodiments.

[0414] S603 . The H5 editor determines, according to the mapping relationship information and based on the JS polling mechanism, the H5 position information m that matches the position of the element content m of the element m.

[0415] The H5 editor determines the position of the element content m according to the H5 position information that matches the position of the element content m, that is, determines which H5 position information anchors the position of the element content m. The H5 position information that matches the position of the element content m determined by the H5 editor is defined as H5 position information m.

[0416] S604. The H5 editor determines the H5 relative time m corresponding to the H5 position information m according to the mapping relationship information.

[0417] S605. The H5 editor determines the H5 position information n corresponding to the H5 relative time that is less than or equal to the H5 relative time m according to the mapping relationship information and based on the JS polling mechanism, determines the element content n matching the H5 position information n in the content file, highlights the element content n, and grays out the other element contents except the element content n in the content file.

[0418] This process is similar to the process of step S506 in the above embodiment and will not be described in detail.

[0419] S606. The H5 editor sends the H5 relative time m to the recording and playback control module.

[0420] S607, the recording and playing control module starts playing the recording file from the relative time m of H5.

[0421] S608. The recording and playing control module sends the H5 relative time m to the interface module.

[0422] S609: The interface module refreshes the current playing time in the recording progress control to m.

[0423] After executing step S609, the note APP continues to execute the above steps S410 to S415.

[0424] In this embodiment, when the user clicks the H5 element in the H5 page, the H5 editor determines the H5 position information m that matches the element content m based on the JS polling mechanism, and then determines the H5 relative time m corresponding to the H5 position information m according to the mapping relationship information. After obtaining the H5 relative time m, on the one hand, the H5 position information n corresponding to the H5 relative time less than or equal to the H5 relative time m is determined, and the position matching and highlighting are performed based on the H5 position information n. In this way, the H5 element clicked by the user and the H5 element input before are highlighted, and the H5 element after the H5 element clicked by the user is grayed out. On the other hand, the recording and control module jumps the playback progress to m according to the H5 relative time m, so that the playback progress is reversely positioned by clicking the H5 element. In addition, in this embodiment, step S603 is performed only when it is determined that the content type is an H5 element, and the polling of the mapping relationship information is started, and when the content type is not an H5 element, the polling is not started. This can reduce the number of times the H5 editor polls to find the mapping relationship information, reduce resource consumption, and save power consumption of electronic equipment.

[0425] Above Figures 16 to 21 The method provided by the illustrated embodiment realizes bidirectional positioning in note playback, and the entire process is completed based on the H5 editor in the note APP, without the need for native modules in the electronic device system, so it can be used across systems to improve user experience.

[0426] 2. Notes are handwritten

[0427] The above describes the note generation process, note stop process, note sequence presentation process, playback progress jump process and note content jump process when the note content is an H5 element. Below, in conjunction with the accompanying drawings, the implementation method of the relevant process when the note content is handwritten content is described. For ease of understanding and viewing, the following embodiments will no longer be described using the multi-window display of two APPs as an example, but only the relevant processes when the note APP is displayed in full screen and the note content is handwritten are described. It can be understood that when displayed in multiple windows, the relevant interface, method operation process and implementation principle of the note APP are the same as those when displayed in full screen, and will not be repeated.

[0428] (1) Note generation process

[0429] In the note generation process provided in this embodiment, see the part of starting recording. Figure 3 and Fig. 9 The embodiments shown are not described in detail. Fig. 22 This is a schematic diagram of the interface changes during the note generation process in handwriting input mode provided in an embodiment of the present application. Figure 3 As shown in Figure (d) Fig. 22 As shown in Figure (a) in FIG. 1 , after the user clicks the recording control to start recording, the user further clicks the handwriting control 305 in the interface, and the input mode of the content input area 702 is switched from the text input mode to the handwriting input mode, and the mobile phone displays the following Fig. 22 The user can handwrite the note content in the content input area 702. For example, the user handwrites the four words "modernism" and the note interface is as follows: Fig. 22 Similarly, during the recording process, the user continued to handwrite some content, such as Fig. 22 As shown in Figure (d) in the figure. In addition, Fig. 22 As shown in FIG. 2(d), the interface in the handwriting input mode may include a control 2201 for switching to the typing mode. If the user wants to end the handwriting input and use the soft keyboard to input again, the user may click the control 2201. After clicking the control 2201, the interface may jump to a similar Fig.10 The note interface shown in the right window of Figure (b) is different in that the content displayed in the content input area is the handwritten input content.

[0430] Corresponding to the above interface change diagram, the timing flow diagram during the handwriting content input process can be as follows: Fig.23 As shown, the process includes:

[0431] S701. In response to the user inputting content b by handwriting in the content input area, the interface module sends the content b to the H5 editor.

[0432] As described above, the information included in the content b may include the handwritten stroke b, and the input position, attribute, format, etc. of the handwritten stroke b.

[0433] S702. The H5 editor determines that the content type of content b is handwritten content according to the current input mode of the note APP.

[0434] S703. The H5 editor displays content b in the content input area.

[0435] S704, H5 editor writes the content type of content b and content b into the content file in the storage module, and establishes a corresponding relationship between the content type and content b; wherein content b includes handwritten strokes b, the position of handwritten strokes b, and the time when handwritten strokes b are put down.

[0436] It should be noted that the time when the pen is put down is the same as the time when the recording starts, which is an absolute time.

[0437] It can be seen from the above process that in the note generation process, when inputting handwritten content, there is no need to establish mapping relationship information, but the time when each handwritten stroke is put down needs to be recorded.

[0438] (2) Note stopping process

[0439] The note stopping process in this embodiment is the same as that described above. Fig.14 The difference of the process shown is that: in this embodiment, there is no need to execute step S210 and step S211, and in step S212, the corresponding relationship information generated by the recording and playback control module only includes the corresponding relationship between the recording file and the content file, and does not include the mapping file.

[0440] (3) Note sequence presentation process

[0441] Schematic diagram of interface changes in the note sequence presentation process in this embodiment Fig.16 Similar, the difference is that in this embodiment, the handwritten content is displayed in the H5 page, and the handwritten content is gradually highlighted according to the strokes.

[0442] The following is an explanation of the sequential presentation process of notes containing handwritten content with reference to the accompanying drawings. Fig.24 Another example of a sequential presentation process of notes provided in an embodiment of the present application is shown in the following timing flow chart: Fig.24 As shown, the process includes:

[0443] S801. In response to the user opening the note, the interface module sends a pre-display instruction to the H5 editor.

[0444] S802. The H5 editor reads the content file of the note from the storage module.

[0445] S803. The H5 editor grays out all the content in the content file.

[0446] S804: In response to the user clicking the second play control, the interface module displays a play progress control.

[0447] S805: The interface module sends a play instruction to the recording control module.

[0448] S806. The recording and playing control module sends a playing instruction to the H5 editor.

[0449] S807. In response to the play instruction, the H5 editor determines the time difference between the writing time of each handwritten stroke in the content file and the recording start time, and obtains the handwriting relative time of each handwritten stroke.

[0450] As mentioned above Fig. 9 In step S118, during the note generation process, the H5 editor records the recording start time sent by the recording and playback control module in the content file. Fig.23 In step S704, the H5 editor writes the time when the handwritten stroke is put down into the content file. Based on this, in step S807, the handwriting relative time of the handwritten stroke is obtained by calculating the time difference between the time when the handwritten stroke is put down and the time when the recording starts. The handwriting relative time can represent the recording duration corresponding to the time when the handwritten stroke is put down during the note generation process.

[0451] S808. In response to the play instruction, the recording and play control module reads the recording file from the storage module.

[0452] S809, after reading the recording file, the recording and playing control module starts playing the recording file from the 0th second and determines the playing time in real time.

[0453] S810. The recording and playing control module sends the playing time to the interface module in real time.

[0454] S811. After receiving the playback time sent by the recording and playback control module, the interface module refreshes the playback time information in the playback progress control.

[0455] S812. The recording and playback control module sends the playback duration to the H5 editor in real time.

[0456] S813. The H5 editor determines, based on the JS polling mechanism, whether there is a current handwriting relative time equal to the current playback duration among all handwriting relative times; if so, execute step S814; if not, return to execute step S813.

[0457] S814. The H5 editor highlights the handwritten strokes corresponding to the current second playback time.

[0458] In the above process Fig.17 The same steps are shown in Fig.17 The corresponding instructions will not be repeated here.

[0459] The method provided in this embodiment determines the handwriting relative time of each handwritten stroke through the recording start time and the writing time of each handwritten stroke pre-stored in the H5 editor during the sequential presentation of notes. The handwriting relative time can characterize the recording duration corresponding to the handwritten stroke when it is input. Therefore, during the note presentation process, by determining whether there is a current handwriting relative time that is consistent with the current playback duration, it is determined whether there are handwritten strokes corresponding to the current playback duration. When the current handwriting relative time exists, the corresponding handwritten strokes are highlighted. In this way, the synchronization of the handwriting stroke display progress and the recording playback progress is achieved, and there is no need to read the handwritten content and the corresponding stroke writing time through the native module of the system in the electronic device, so it can be used across systems to improve the user experience.

[0460] (4) Playback progress jump process

[0461] The interface change diagram of the playback progress jump process in this embodiment is similar to Fig.18 Similar, the difference is that in this embodiment, the handwritten content is displayed in the H5 page, and when the playback progress jumps to a certain position, the corresponding handwritten strokes in the H5 page are highlighted.

[0462] The following is an explanation of the playback progress jump process of a note containing handwritten content with reference to the accompanying drawings. Fig.25 Another example of a timing flow chart of a playback progress jump process provided in an embodiment of the present application is as follows: Fig.25 As shown, the process includes:

[0463] S901: In response to the user dragging the progress bar to the position of the playing time Tx, the interface module refreshes the current playing time in the playing progress control to Tx.

[0464] S902: The interface module sends a play jump instruction to the recording and playing control module. The play jump instruction is used to instruct to adjust the playing time to x.

[0465] S903: The recording and playing control module responds to the play jump instruction and starts playing the recording file from the play time Tx.

[0466] S904, the recording and playing control module sends the play jump mark information and the play duration Tx to the H5 editor.

[0467] S905. The H5 editor determines the handwriting relative time Tx that is less than or equal to the playback time Tx in the content file based on the JS polling mechanism, highlights the handwriting strokes corresponding to the handwriting relative time Tx, and grays out the handwriting strokes corresponding to the handwriting relative time other than the handwriting relative time Tx.

[0468] In this step, the handwriting relative time Tx is a general term for one or more handwriting relative times.

[0469] After executing step S905, the note APP continues to execute the above steps S812 to S814.

[0470] In this embodiment, when the user changes the playback progress of the recording, the H5 editor searches for the handwriting relative time Tx that is less than or equal to the playback time Tx based on the JS polling mechanism, highlights the handwriting strokes corresponding to the second phase time Tx, and grays out other handwriting strokes, that is, grays out the handwriting strokes corresponding to the handwriting relative time greater than the playback time Tx. In this way, after the playback progress jumps, the handwriting content corresponding to the playback progress after the jump and the handwriting content input before are highlighted, and the handwriting content after the handwriting content corresponding to the playback progress after the jump is grayed out, realizing the positioning of the handwriting content display progress through the playback recording progress.

[0471] (5) Note content jump process

[0472] The interface change diagram of the note content jump process in this embodiment is similar to Fig. 20 Similar, the difference is that in this embodiment, the handwritten content is displayed in the H5 page, and when the user clicks a certain position, the corresponding handwritten strokes in the H5 page and the handwritten strokes before the handwritten strokes are highlighted.

[0473] The following is an explanation of the process of jumping the content of a note containing handwritten content with reference to the accompanying drawings. Fig.26 Another example of a timing flow diagram of a note content jump process provided in an embodiment of the present application is as follows: Fig.26 As shown, the process includes:

[0474] S1001. In response to a user clicking on content m in the H5 page, the interface module sends a display jump instruction to the H5 editor, where the display jump instruction is used to instruct to jump the highlighted display progress to content m.

[0475] S1002. In response to the display jump instruction, the H5 editor determines that the content type of the content m is handwritten content according to the established mapping relationship between handwritten content and content type.

[0476] S1003. The H5 editor determines the handwriting relative time m corresponding to the handwriting strokes contained in the content m.

[0477] S1004. The H5 editor determines the handwriting relative time n in the content file that is less than or equal to the handwriting relative time m based on the JS polling mechanism, highlights the handwriting stroke n corresponding to the handwriting relative time n, and grays out the other handwriting strokes in the content file except the handwriting stroke n.

[0478] S1005. The H5 editor sends the handwriting relative time m to the recording and playback control module.

[0479] S1006: The recording and playing control module starts playing the recording file from the handwriting relative time m.

[0480] S1007: The recording and playing control module sends the handwriting relative time m to the interface module.

[0481] S1008. The interface module refreshes the current playing time in the recording progress control to m.

[0482] After executing step S1008, the note APP continues to execute the above steps S812 to S814.

[0483] In this embodiment, when the user clicks on the handwritten content in the H5 page, the H5 editor determines the handwriting relative time m corresponding to the handwriting stroke m contained in the content m. After obtaining the handwriting relative time m, on the one hand, the handwriting relative time n that is less than or equal to the handwriting relative time m is determined, and the handwriting strokes corresponding to the handwriting relative time n are highlighted, and other handwriting strokes are grayed out. In this way, the handwriting content clicked by the user and the handwriting content input before are highlighted, and the handwriting content after the handwriting content clicked by the user is grayed out. On the other hand, the recording and control module jumps the playback progress to m according to the handwriting relative time m, so that the reverse positioning of the playback progress by clicking on the handwriting content is realized. In addition, in this embodiment, step S1003 is executed only when it is determined that the content type is handwriting content, and the polling search for the handwriting relative time n is started, and when the content type is not handwriting content, the polling is not started. In this way, the number of polling times of the handwriting content in the content file by the H5 editor can be reduced, resource consumption can be reduced, and the power consumption of electronic equipment can be saved.

[0484] In the above embodiment, the example of presenting notes in an electronic device with a note APP installed is used for explanation. The embodiment of the present application also provides a method for converting files generated by a note APP so that the converted files can also be presented in an electronic device that does not have the note APP installed. The following is an explanation with reference to the accompanying drawings.

[0485] For example, Fig. 27 This is a schematic diagram of the interface changes of an example note sharing process provided in an embodiment of the present application. Fig. 27 As shown in Figure (a) of FIG. 1 , after the user pauses the recording and saves the note, the note interface includes a sharing control 2701. When the user clicks the sharing control 2701, the mobile phone displays the following Fig. 27 The interface shown in FIG. (b) in FIG. 2 includes a sharing method selection card 2702, and the sharing method selection card 2702 includes an export as a document option 2703. The user clicks the export as a document option 2703, and the mobile phone displays the following Fig. 27 The interface shown in FIG. (c) in FIG. 2 includes a document type selection card 2704, and the document type selection card 2704 includes an export to HTML option 2705. The user clicks the export to HTML option 2705, and the note "Modernism" is exported to HTML format and saved to a preset storage path. The mobile phone displays the following Fig. 27 The interface shown in Figure (d) in the figure. The exported HTML file can be opened through a browser in an electronic device that does not have the note APP installed, and two-way positioning of the recording playback progress and the note content display can be achieved.

[0486] Corresponding to the above interface change diagram, the flowchart of note format conversion can be as follows: Fig.28 As shown, the process includes:

[0487] S2101. In response to the user clicking the export to HTML option, the interface module sends a format conversion instruction to the H5 editor.

[0488] Optionally, the format conversion instruction may include information of the note, such as the note title, etc. The format conversion instruction is used to instruct to export the note into HTML format.

[0489] S2102, the H5 editor determines whether the content file includes handwritten content in response to the format conversion instruction;

[0490] If yes, a handwriting conversion JS file is generated. The handwriting conversion JS file is a code file in JS format, which is used to read the handwriting strokes in the content file and convert the handwriting strokes into a picture format, and then execute step S2103;

[0491] If not, directly execute step S2103.

[0492] Optionally, the H5 editor may determine whether the content file includes handwritten content based on the content type in the mapping relationship information recorded during the note generation phase and the content type corresponding to the handwritten strokes.

[0493] S2103. The H5 editor generates a positioning logic JS file. The positioning logic JS file is a code file in JS format, which is used to implement the two-way positioning logic in the note APP, that is, to implement the two-way positioning logic of the recording playback progress and the note content display. The two-way positioning logic in the note APP is also the logic content of the above-mentioned note sequential playback process, playback progress jump, and note content jump, which will not be repeated here.

[0494] S2104. The H5 editor creates a resource folder, which includes a reading relationship JS file, a handwriting conversion JS file, a recording file, a content file, and a mapping file.

[0495] S2105. H5 Editor saves the H5 page of the note as a file in HTML format.

[0496] For example, save the H5 page of "Modern Notes" as a file "Modern Notes.html". In this way, the elements in "Modern Notes.html" correspond one-to-one with the tags in the content file.

[0497] After that, a correspondence between the resource folder and the HTML format file can be established.

[0498] Later, if the user opens the note in an electronic device that does not have the note APP installed, the HTML format file can be read through the browser, and the handwriting conversion JS file in the resource folder can read the handwritten strokes when the H5 page contains handwritten content, and convert the handwritten strokes into a picture format without affecting the display of the handwritten strokes. At the same time, the positioning logic JS file in the resource folder can read the mapping relationship information according to the playback duration, and poll and search based on the JS polling mechanism to achieve the synchronous display of the note content corresponding to the playback duration. At the same time, the positioning logic JS file can also locate the corresponding note content when the playback progress jumps, and when a certain note content is clicked, the playback progress will jump to the corresponding playback duration. In this way, even if the note APP is not installed in the electronic device, the note can be opened, and two-way positioning can be achieved, further improving the user experience.

[0499] The above describes in detail an example of a method for recording content provided by an embodiment of the present application. It is understandable that, in order to implement the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. Those skilled in the art should easily appreciate that, in conjunction with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered to exceed the scope of the present application.

[0500] The embodiment of the present application can divide the functional modules of the electronic device according to the above method example. For example, each function can be divided into various functional modules, such as a detection unit, a processing unit, a display unit, etc., or two or more functions can be integrated into one module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0501] It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0502] The electronic device provided in this embodiment is used to execute the above-mentioned method of recording content, and thus can achieve the same effect as the above-mentioned implementation method.

[0503] In the case of an integrated unit, the electronic device may further include a processing module, a storage module and a communication module. The processing module may be used to control and manage the actions of the electronic device. The storage module may be used to support the electronic device to execute stored program codes and data, etc. The communication module may be used to support the electronic device to communicate with other devices.

[0504] Among them, the processing module can be a processor or a controller. It can implement or execute various exemplary logic boxes, modules and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc. The storage module can be a memory. The communication module can specifically be a device that interacts with other electronic devices, such as a radio frequency circuit, a Bluetooth chip, a Wi-Fi chip, etc.

[0505] In one embodiment, when the processing module is a processor and the storage module is a memory, the electronic device involved in this embodiment may be a Figure 1 Device of the structure shown.

[0506] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the processor executes the method for recording content of any of the above embodiments.

[0507] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer is caused to execute the above-mentioned related steps to implement the method for recording content in the above-mentioned embodiment.

[0508] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer execution instructions, and when the device is running, the processor can execute the computer execution instructions stored in the memory so that the chip executes the method of recording content in the above-mentioned method embodiments.

[0509] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here.

[0510] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0511] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0512] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0513] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0514] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0515] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for recording content, the method being performed by an electronic device, characterized in that: The electronic device includes a first application APP, the first APP is used to record audio and content input by a user, and the method includes: Launch the first APP; Receiving a start instruction through the first APP, the start instruction is used to instruct to start a recording operation within the system, and the recording operation within the system refers to recording audio within the system of the electronic device; In response to the start instruction, the recording operation in the system is executed.

2. The method according to claim 1, characterized in that The performing of the recording operation in the system includes: When it is determined that there is audio in a playing state in the system of the electronic device, the recording operation in the system is performed.

3. The method according to claim 2, characterized in that The electronic device further includes a media player, and the method further includes: If it is determined that there is an instantiated first object in the media player and there is an audio stream in the media player, it is determined that there is audio in a playing state in the system of the electronic device.

4. The method according to any one of claims 1 to 3, characterized in that Before performing the recording operation in the system, the method further includes: Performing a permission check on the first APP, the permission check is used to determine whether there is user authorization information in the electronic device and whether the first APP meets a preset condition, the user authorization information is used to indicate that the user agrees that the electronic device performs the recording operation in the system; If the permission check is passed, the recording operation in the system is executed; If the authority check fails and it is determined that the user authorization information does not exist, the application authorization information is displayed, and the application authorization information is used to apply for the user authorization information.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Displaying the interface of the first APP, where the interface of the first APP includes a content input area; During the process of executing the recording operation within the system, in response to the user inputting the first content in the content input area, the first content is saved, and mapping relationship information is generated based on the first duration and the input position of the first content, wherein the first duration is the recording duration when the user inputs the first content, and the first content is HTML content.

6. The method according to claim 5, characterized in that The saving of the first content and generating mapping relationship information according to the first duration and the input position of the first content includes: Based on the hypertext markup language HTML, generating a first HTML tag corresponding to the first content; Saving the first content and the first HTML tag; Determine first position information according to the first HTML tag, where the first position information represents a position of the first content in the HTML page; The mapping relationship information is generated according to the first duration and the first location information.

7. The method according to claim 6, characterized in that The method further comprises: In response to a recording pause operation by a user, stopping the recording operation in the system to obtain recorded audio; In response to a play operation by the user, playing the recorded audio; During the playback of the recorded audio, when it is determined that the current playback duration matches the first duration, determining the first position information corresponding to the first duration according to the mapping relationship information; The first content is displayed in a first manner in the HTML page according to the first position information.

8. The method according to claim 7, characterized in that The method further comprises: During the process of playing the recorded audio, in response to the user selecting the first content in the HTML page, the recorded audio starts to be played from the playback duration corresponding to the first duration.

9. The method according to any one of claims 7 or 8, characterized in that The method further comprises: When the recording operation in the system starts, recording the recording start time; During the process of executing the recording operation within the system, in response to the user inputting the second content in the content input area, the second content and the moment when the pen is put down are recorded, wherein the second content is handwritten content, and the second content is the content input from the time the pen is put down to the time it is lifted.

10. The method according to claim 9, characterized in that The method further comprises: In response to the playing operation, determining a time difference between a writing time of the second content and a recording start time, and obtaining a second duration corresponding to the second content; During the process of playing the recorded audio, when it is determined that the current playing duration matches the second duration, the second content is displayed in a second manner.

11. The method according to claim 10, characterized in that The method further comprises: During the process of playing the recorded audio, in response to the user selecting the second content, the recorded audio is played starting from the playback duration corresponding to the second duration.

12. The method according to any one of claims 7 to 11, characterized in that The current playback duration is the playback duration of the recorded audio played sequentially, or the playback duration selected by the user.

13. The method according to any one of claims 7 to 12, characterized in that After stopping the recording operation in the system and obtaining the recorded audio, the method further includes: Determine information of a second APP displayed simultaneously with the first APP in the interface of the electronic device; According to the information of the second APP, a prompt message is displayed, where the prompt message is used to prompt the user that the source of the recorded audio is the second APP, and prompt the user to save the recorded audio.

14. The method according to claim 13, characterized in that The method further comprises: The interface of the electronic device includes a first window and a second window, wherein the first window displays the interface of the first APP, and the second window displays the interface of the second APP.

15. The method according to claim 14, characterized in that The method further comprises: One of the first window and the second window is displayed in a floating manner; or, The first window and the second window are displayed side by side.

16. The method according to any one of claims 1 to 15, characterized in that The starting the first APP includes: In response to a sharing operation performed by a user on a first page, the first APP is started and an interface of the first APP is displayed; the sharing operation is used to instruct to share the first page to the first APP, and the interface of the first APP includes a content input area; The link information of the first page is written into the content input area.

17. The method according to any one of claims 1 to 15, characterized in that The starting the first APP includes: In response to a copy link operation performed by a user on a second page, copying link information of the second page, where the second page includes a video or an audio; Display split-screen controls; In response to the user clicking the split-screen control, the first APP is started, and the second page and the first APP are displayed in a split-screen, wherein the interface of the first APP includes a content input area; The link information of the second page is written into the content input area.

18. The method according to claim 17, characterized in that After starting the first APP, the method further includes: The startup instruction is generated.

19. The method according to any one of claims 1 to 18, characterized in that The system of the electronic device includes an application layer and an application framework layer, the first APP is located in the application layer, the electronic device also includes a recording module, the recording module is located in the application framework layer, and the performing of the recording operation in the system includes: The recording module performs recording operations within the system.

20. The method according to claim 19, characterized in that The recording module performs recording operations in the system, including: The recording module constructs a second object and adds configuration parameters to the second object, wherein the configuration parameters indicate that the recording operation corresponding to the second object is a recording operation within the system; The recording module performs the recording operation within the system based on the second object.

21. The method according to claim 20, characterized in that The electronic device further includes a media player, and the media player is located in the application framework layer. Before the recording module performs the recording operation in the system based on the second object, the method further includes: The first APP sends a first mark to the recording module, where the first mark is used to indicate that, when it is determined that there is audio in a playing state in the system of the electronic device, a recording operation in the system is performed; The recording module performs the recording operation in the system based on the second object, including: The recording module performs the in-system recording operation based on the second object when it is determined, according to the first mark, that there is an instantiated first object in the media player and there is an audio stream in the media player.

22. An electronic device, characterized in that: include: Processors, memory, and interfaces; The processor, the memory and the interface cooperate with each other so that the electronic device executes the method as claimed in any one of claims 1 to 21.

23. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 21.

Images