Video processing method, storage medium and electronic equipment

Abstract

The invention relates to the field of terminals, and discloses a video processing method, a storage medium and electronic equipment, the method is applied to the electronic equipment, the method comprises the steps that a first interface is displayed, the first interface comprises a first display area, the first interface is an interface of a first application, and a first cursor is displayed in the first display area; in response to a first operation of a user, the first application starts to record a video; after the first application starts to record the video, character content is displayed in a second display area, the character content is obtained by converting voice in the recorded video, a second cursor is displayed in the second display area, and the position of the second cursor is at the tail of the character content; in response to a second operation of the user, the position of the first cursor is moved down by one line, and the position of the second display area is moved down by one line. Through the method, the user can perform other editing in the note text area in the process of converting the voice into the character in the note application, so that normal use of the overall function of the note application is ensured, and the user experience is improved.

Description

Technical Field

[0001] The present application relates to the field of terminals, and in particular, to a video processing method, a storage medium and an electronic device. Background Art

[0002] Users have an increasing demand for extracting and organizing video content. Users can record part of the video content from a video to obtain the recorded video. In addition, users can also use an electronic device with a voice-to-text function to convert the voice information of the video in the electronic device into text information to form a video note. The video note can be recorded in the voice-to-text area of ​​the note application in the electronic device. Users can also calibrate, annotate, and edit the video note.

[0003] However, when the user uses the cursor in the note body editing area, it will conflict with the cursor in the speech-to-text area, resulting in a poor user experience. Based on this, how to solve the problem of cursor conflict to improve the user experience has become an urgent problem to be solved. Summary of the invention

[0004] The present application provides a video processing method, a storage medium and an electronic device, which solve the problem of conflict between the cursor in the speech-to-text area and the cursor in the note text editing area in the electronic device in the prior art.

[0005] In order to achieve the above objectives, this application adopts the following technical solutions:

[0006] In a first aspect, a video processing method is provided, which is applied to an electronic device, wherein the electronic device includes a first application, and the method includes: displaying a first interface, wherein the first interface includes a first display area, the first interface is the interface of the first application, and a first cursor is displayed in the first display area; in response to a first operation of a user, the first application starts recording a video; after the first application starts recording the video, text content is displayed in a second display area, the text content is obtained by converting voice in the recorded video, a second cursor is displayed in the second display area, and the position of the second cursor is at the end of the text content; in response to a second operation of the user, the position of the first cursor moves down by one line, and the position of the second display area moves down by one line.

[0007] In an embodiment of the present application, during the process of video recording and voice-to-text conversion, two non-conflicting cursors can be displayed simultaneously, and each can change accordingly in response to user operations. It can be understood that the position of the second cursor at the end of the text content can be represented by the change in the position of the second cursor to indicate the progress of the voice conversion, and the position of the second cursor can change with the progress of the voice conversion or the output text. The position of the second cursor at the end of the text content can be after the last word of the text content, before the last word, before the last word, before the last symbol, etc. In addition, the two cursors each have their own functions and produce corresponding effects: the first cursor can be used by the user to make other inputs in the note text editing area; the second cursor can be used to indicate the progress of voice-to-text conversion or help voice-to-text conversion. Therefore, this video processing method that ensures the existence and non-conflict of the two cursors can fully improve the display effect of voice-to-text conversion and ensure the realization of the original note application editing function, thereby improving the user experience of the note application.

[0008] In an implementation manner of the first aspect, the method further includes: after the first operation, creating a first view; the position of the second display area is moved down by one line, including: the position of the first view is moved down by one line.

[0009] In the embodiment of the present application, since it is impossible to separate the two cursors, the two cursors can exist independently through the above-mentioned method of creating the first view, which is convenient for users to use, thereby improving the user experience.

[0010] In an implementation of the first aspect, the method further includes: in response to a second operation of the user, the first application ends recording the video; after the second operation, the first view is removed; and the text content is displayed in the second display area.

[0011] In an implementation of the first aspect, the method further includes: creating a second view, the second view corresponding to the first display area; during the video recording process, the text content is rendered to the display area corresponding to the first view; after removing the first view, the text content is rendered to the display area corresponding to the second view. In this embodiment, the first display area is the display area corresponding to the second view.

[0012] In the embodiment of the present application, since the first view is removed but the previous text information may still be carried therein, the text information is needed by the user. Therefore, when the text information is removed, it is necessary to display the text information in the original position to avoid confusion among the user, thereby improving the user experience.

[0013] In an implementation manner of the first aspect, the method further includes: the first view is EditText, and the second view is Webview.

[0014] In the embodiment of the present application, the Webview already carries the first cursor, so in order to display an independent and unaffected second cursor, a new view is required. Moreover, the information output by speech-to-text conversion is in text format, and EditText has the function of displaying a text page to the user and can carry a second cursor.

[0015] In an implementation of the first aspect, the method also includes: the first application is a note application, the first application includes a note editing module, a JS interface rendering control module and a video control module; the first display area is a note text editing area, the first display area includes a note text editing area, a video note card area, a video recording area and a speech-to-text area.

[0016] In an embodiment of the present application, the note application may include modules such as an interface display module, a video control module, a note editing module, a speech-to-text editing module, a video excerpt editing module, and a javascript (JS) interface rendering module. The JS interface rendering control module is used to render data to Webview to display the web page content in Webview. The video control module is the overall control module in JS, which can be used to distribute specific requests of JS to each module of JS. The video control module is a function transfer module that is responsible for processing some common functions. The note editing module can be a fragment of an activity in an Android native note application, and the fragment can be a loaded object.

[0017] In the embodiment of the present application, dividing the first display area into different areas helps the user understand the functional differences between different partitions, and also helps to better display more information, thereby improving the user experience.

[0018] In an implementation of the first aspect, in response to a first operation of a user, the first application starts recording a video, and also includes: the note editing module sends relevant information of the video recording to the JS interface rendering control module by calling a first insertion function, the JS interface rendering control module encapsulates the relevant information of the video recording, and after completing the encapsulation, sends the encapsulated relevant information of the video recording to the video control module by calling a second insertion function.

[0019] In the embodiment of the present application, the information related to the video recording includes necessary data related to the video note card, for example, data such as the video title on the video note card when recording the video. The information related to the video recording may include the element to be inserted in the video note card, and the element to be inserted may be the title of the video note card in the card title area and other information. The information related to the video recording may also include the element type of the element to be inserted. By passing this information to the video control module, the video control module can perform subsequent rendering steps.

[0020] In an implementation of the first aspect, before the creation of the first view, it also includes: the video control module renders the html area, including: the video control module performs DOM rendering and first placeholder area rendering; the JS interface rendering control module sends the first position information of the first placeholder area to the video control module, and the video control module sends the first position information of the first placeholder area to the note editing module.

[0021] In the embodiment of the present application, the effect of selecting to render the first placeholder when performing DOM rendering is that the placeholder rendering can be performed more quickly and efficiently. DOM rendering refers to the process of generating a DOM document presented to the user in the html area. It can also be understood that when the electronic device creates a video note card, it creates a placeholder area in the video note card with the same size and position as the speech-to-text area.

[0022] In an implementation of the first aspect, the creation of the first view includes: the note editing module creates the EditText and calculates the attribute parameters of the window of the first placeholder area; the note editing module creates the EditText according to the attribute parameters of the window of the first placeholder area through a first creation function.

[0023] In an implementation of the first aspect, removing the first view includes: removing the EditText through a first removal function based on the note editing module, thereby removing the first view.

[0024] In an implementation of the first aspect, after the first view is removed, the text content is rendered to a display area corresponding to a second view, including: the note editing module sends relevant information about the content in the EditText to the video control module; and the video control module updates the html area.

[0025] In an implementation manner of the first aspect, the method further includes: after the video control module performs DOM rendering and the first placeholder rendering, the video control module registers a first listener.

[0026] In an implementation of the first aspect, in response to the user's second operation, the position of the first cursor moves down one line, and the position of the second display area moves down one line, including: based on the first listener, the video control module detects that the content of the note body editing area has changed; the JS interface rendering control module sends the second position information of the first placeholder area to the video control module, and the video control module sends the second position information to the note editing module; based on the second position information, the note editing module updates the EditText.

[0027] In the embodiment of the present application, the user's operation may cause the information of the speech-to-text placeholder area to change. For example, a carriage return or typing operation is performed above the video note card area, causing the display position of the video note card to change. In this case, the first listener can monitor the change in the content of the note body editing area. Optionally, in response to the user's second operation, the position of the first cursor moves down one line, and the position of the second display area moves down one line. The meaning of the position moving down one line means that the upper edge of the corresponding area moves down one line (because the size of different areas themselves may change as the text of speech-to-text increases, that is, the lower edge of the area may itself move down based on other factors).

[0028] In a second aspect, an electronic device is provided, comprising a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to execute the computer program in the memory so that the electronic device implements a method as described in any one of the first aspects.

[0029] According to a third aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by one or more processors, the method according to any one of the first aspects is implemented.

[0030] In a fourth aspect, a chip system is provided, which is applied to an electronic device including a display screen and a storage medium; the chip system includes one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected through a line; the interface circuit is used to receive a signal from the memory of the electronic device and send the signal to the processor, the signal including a computer program stored in the memory; when the processor executes the computer program, the method described in any one of the first aspects is implemented.

[0031] In a fifth aspect, a computer program product is provided, and when the computer program product is run on a computer, the computer is caused to execute the method as described in any one of the first aspects. It can be understood that the beneficial effects that can be achieved by the electronic device described in the second aspect, the computer-readable storage medium described in the third aspect, the chip system described in the fourth aspect, and the computer program product described in the fifth aspect provided above can refer to the beneficial effects in the first aspect and any possible design thereof, and will not be repeated here.

[0032] Through the above method, the user can perform other editing in the note text area during the process of converting speech to text in the note application, thereby ensuring the normal use of the overall functions of the note application and improving the user experience.

[0033] Therefore, the embodiment of the present application provides a solution to the problem of conflict between the editing cursors in the speech-to-text area and the text editing area in the process of forming video notes based on videos. It ensures that the cursor in the speech-to-text area does not conflict with the note text editing area. During the continuous output and editing of speech-to-text in the speech-to-text area, the note text area can also perform other editing including but not limited to adding text, pictures, etc., thereby ensuring the normal use of the overall function of the note application. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figures 1A to 1L According to some embodiments of the present application, some interface schematic diagrams are shown;

[0035] FIG. 2A to FIG. 2C According to some embodiments of the present application, some other interface schematic diagrams are shown;

[0036] FIG. 3A to FIG. 3C According to some embodiments of the present application, some other interface schematic diagrams are shown;

[0037] Figure 4A and Figure 4B According to some embodiments of the present application, some other interface schematic diagrams are shown;

[0038] Figure 5 According to some embodiments of the present application, a hardware structure diagram of a terminal device 500 is shown;

[0039] Figure 6 According to some embodiments of the present application, a software structure diagram of a terminal device 500 is shown;

[0040] Figure 7 According to some embodiments of the present application, a software architecture block diagram of a note-taking application is shown;

[0041] Fig. 8AAccording to some embodiments of the present application, a schematic diagram of an interface display structure is shown;

[0042] FIG. 8B to FIG. 8D According to some embodiments of the present application, some other interface schematic diagrams are shown;

[0043] Fig.9A and Fig. 9B According to some embodiments of the present application, some other interface schematic diagrams are shown;

[0044] Fig.10 According to some embodiments of the present application, a software module interaction diagram of a video processing method is shown;

[0045] Fig.11 According to some embodiments of the present application, a flow chart of a video processing method is shown;

[0046] Fig.12 According to some embodiments of the present application, another flow chart of a video processing method is shown;

[0047] Fig.13 According to some embodiments of the present application, some other interface display structure schematic diagrams are shown;

[0048] Fig.14 According to some embodiments of the present application, a schematic diagram of a chip system provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0049] The illustrative embodiments of the present application include but are not limited to a video processing method, a storage medium, and an electronic device.

[0050] In the following description, specific details such as specific system structures and technologies are provided for illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details.

[0051] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0052] It should also be understood that in the embodiments of the present application, "one or more" refers to one, two or more than two; "and / or" describes the association relationship of associated objects, indicating that three relationships may exist; for example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B may be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0053] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", "fourth", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0054] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0055] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are only examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0056] First, for ease of understanding, the terms involved in the embodiments of the present application are introduced here:

[0057] 1) Note application: It is a note application in an electronic device, and the note application can also be called a memo application. The note application can be an application in a terminal device with a rich text editor function that allows users to record text or pictures. The display interface of the note application (also referred to as a note interface or a note window) may include a note text editing area (also referred to as a first display area), a video note card area, a video recording area, and a voice-to-text area. In an embodiment of the present application, the display interface of the note application may be the display area 8011 of the note application in Figure 8, which may include a note text editing area 8012, a video note card area 8013, a video recording area 8014, and a voice-to-text area 8015. The information in the note application includes, but is not limited to, voice information, text information, picture information, and video information.

[0058] 2) Video recording function: When playing a video, the electronic device can use the note application to record the played video, and display the video note card corresponding to the recorded video in the display interface of the note application. In response to the user's operation (including clicks, touches, etc.), the electronic device can play the video displayed in the video note card. The user can also enter other information in the card or in the display interface of the note application where the card is located. It should be noted that the other information mentioned here includes information other than the video. For example, text information (which may be referred to as text in the following content), voice information (which may be referred to as voice in the following content), picture information (which may be referred to as picture in the following content), etc. In the embodiments of the present application, Figure 1A-1L Introduce how to enable the video recording function, and combine Figure 2A-2C The specific process of recording video is introduced. The video recording function can be Figure 1E The video recording control 10411 shown in is turned on.

[0059] It should be noted that recording can be understood as the action of an electronic device capturing or recording the video being played in the current display interface. In the embodiment of the present application, recording can also be referred to as excerpting, and the video recording function can also be referred to as the video excerpting function, which is only a difference in name. For ease of description, the present application records the recorded video as the recorded video, and the card generated in the note window as the video note card.

[0060] 3) Recorded video: It is a video recorded by the electronic device through the video recording function. The recorded video can be displayed in the video note card area. In the embodiment of the present application, the recorded video can be as follows: Figure 8B The video in the video note card shown in Figure 8BFor ease of understanding, the various areas and related contents that may be included in the video note card are shown in an exemplary manner, which may be different from the actual presentation method in use. For example, when recording a video, the video recording area 8014 may display text content such as "Recording" instead of displaying the recorded video in real time.

[0061] It should be noted that, in the embodiment of the present application, the recorded video may also be referred to as recorded video.

[0062] 4) Playing video: During the video recording process, the video played in the interface of the video playback application. The electronic device can record the played video through the video recording function to obtain the recorded video (also referred to as the recorded video). The video playback application can be a third-party application or other application, which is not limited in the embodiment of the present application. In the embodiment of the present application, the playing video can be such as Figure 2B The video S is played in the video playback area 20211 shown in .

[0063] 5) Voice-to-text function: This is a function of an electronic device that converts voice information included in a played video into text information. In an embodiment of the present application, when a user selects the voice-to-text function and records a video, the recording can be converted into text information, and the converted text information is displayed in the video note card area in the note card application.

[0064] In an embodiment of the present application, the speech-to-text function can be used alone or simultaneously with the video recording function. When the speech-to-text function and the video recording function are used simultaneously (for example, the user turns on the speech-to-text function after turning on the video recording function, or when the speech-to-text function is automatically turned on when the video recording function is turned on), the electronic device can generate a video note card in the note window of the note application, and the converted text can be added to the video note card. Optionally, after turning on the speech-to-text function, the electronic device can switch between turning on the speech-to-text function and turning off the speech-to-text function.

[0065] In the present application embodiment, Figure 3A-3C The specific process of turning on the voice-to-text function of an electronic device when recording a video is introduced. The voice-to-text function can be turned on by Figure 3A The speech-to-text control 10413 shown in FIG. 1 is turned on.

[0066] 5) View: In Android TM) system, the base class of view components (or controls) used to display information (including text information, image information, etc.) in the UI (User Interface). A view (also called a view component, a view control, a view component, a view control, etc., collectively referred to as a view component in this article) can be a single control or a group of controls. TM In the system, the view component used to display information (including text information, image information, etc.) in the UI (User Interface) can include two types of controls: web view (Webview) and input box view (EditText), and each control has a corresponding type of data.

[0067] A control, also known as a component or a component, is a reusable functional module in software. In the embodiments of the present application, for ease of description, they are uniformly described as components.

[0068] 6) Webview: It is a type of Android view control. The data type corresponding to Webview is web page. The Webview is an output control for displaying information to users. The control can be a control for displaying web pages.

[0069] Webview is compatible with various types of web content, solving the problem of displaying the note editing interface. Optionally, the web view can be used to display content edited by hypertext markup language (HTML), including but not limited to text, static images, dynamic images, audio, video, tables, etc. Optionally, the HTML language can be, for example, the fifth generation HTML (also known as HTML5.0, or H5 for short).

[0070] In the embodiment of the present application, optionally, the web page content of HTML, Cascading Style Sheets (CSS), and JavaScript (JS) runs inside Webview, and the interaction between other modules and Webview needs to be transferred through Android. Optionally, the interaction between the editor of Webview and Android is realized through the JSBridge and JavascriptInterface interfaces provided by Webview and Android. Optionally, Webview controls the content and logic in Webview by using the JS interface rendering control module and the video control module.

[0071] In the embodiments of the present application, Fig. 8A, Webview can be Webview8041 (also called the second view).

[0072] 7) Input box view (EditText): It is also a type of Android view control. EditText is a subclass of TextView, which is an output control used to display information to the user. This control can be a control used to display a text page. Because EditText is a subclass of TextView, the methods and features of TextView also exist in EditText. EditText is similar to TextView, but the difference is that EditText can accept user input.

[0073] In the embodiments of the present application, Fig. 8A , EditText can be EditText8042 (also called the first view).

[0074] 8) Note text editing area: The note application display interface includes a note text editing area (also referred to as the first display area). The note text editing area refers to the area used to edit and record note content. In an embodiment of the present application, the note text editing area can be used to edit and record speech-to-text content, text content, static pictures, videos, audio, GIF dynamic pictures, and tables, etc. The note text editing interface, also referred to as the editing interface, refers to an interface that can edit note content. The interface presented after the user opens a note through the note application, the interface presented during the user editing content, and the interface presented after the user edits and clicks the save control in the interface to save the content are all referred to as note editing interfaces.

[0075] Optionally, the content of the note body editing area is displayed through Webview. Optionally, Webview carries the note body editing area. In the embodiment of the present application, Figure 8C As shown, the display interface 8011 of the note application includes a note text editing area 8012.

[0076] 9) Video note card area: It is a card area generated in the note window when using the video recording function. The video note card area is a custom area in the note body editing area. The video note card area consists of other areas such as the video recording area, the speech-to-text area, and the video note area. Optionally, the content of the video note card area is displayed through the Webview corresponding to the note body editing area. Optionally, the video note card area is a custom HTML area. Optionally, the video note card is located in the video note card area, and the video note card is implemented through HTML, CSS, and JS.

[0077] In the embodiments of the present application, Figure 8C As shown, the note text editing area 8012 (also referred to as the first display area) includes a video note card area 8013.

[0078] 10) Video recording area: The video recording area can be used to display and play the recorded video. In the embodiment of the present application, the content of the video recording area can be displayed through the Webview corresponding to the note text editing area.

[0079] In the embodiments of the present application, Figure 8C As shown, the video note card area 8013 includes a video recording area 8014.

[0080] 11) Speech-to-text area: The speech-to-text area can display the cursor and the text converted by the speech-to-text function when the video recording function is turned on. The speech-to-text area corresponds to an EditText, which is used to display the text converted by the speech. Optionally, the content of the speech-to-text area is displayed through the EditText. Optionally, the EditText carries the speech-to-text area.

[0081] In the embodiments of the present application, Figure 8C As shown, the video note card area 8013 includes a speech-to-text area 8015 .

[0082] 12) Cursor: It is an input element on the user interface and / or user interaction, which is used to indicate the location of the user's current focus, and can also be a form of expression of a focus selector. Regardless of the specific details of the focus selector, in some embodiments, the contact detected on the touch screen can be used as a focus selector, so that when input (such as a push of contact input) is detected on the touch screen display at the location of a specific user interface element (such as a button, window, slider or other user interface element), the specific user interface element is adjusted according to the detected input. For example, when a user wants to enter text in the note body editing area in the note application, the focus selector can be controlled to move to the place where the user wants to enter text by touching, moving the stylus, moving the mouse, etc., and then performing user operations such as single-clicking, double-clicking or touching the screen to complete the movement of the cursor, so that the cursor is displayed at the location where the user performed the corresponding user operation.

[0083] In the embodiments of the present application, Fig.9A As shown, the first cursor 10431 and the second cursor 10433 are both forms of cursor expression.

[0084] Combine the following Figure 1A-1LThis section describes how to enable the video recording function.

[0085] 1. Enable the video recording function (such as Figure 1A-Figure 1C )

[0086] In response to the user's operation of opening application A, the electronic device can start application A and display the main interface of application A (i.e., the interface displayed by default after startup), and application A is a video application. The main interface of application A may include thumbnails of several videos. The thumbnails of the several videos can be used to trigger the electronic device to play the several videos. The electronic device can detect the user's operation on the corresponding thumbnail (including but not limited to touch, click, voice control, gesture, etc.). In response to the user operation, the electronic device can display the playback interface of the video corresponding to the corresponding thumbnail. In one possible implementation, the electronic device can detect a three-finger sliding operation on the playback interface, and in response to the operation, the electronic device can enter the video recording function. In another possible implementation, the electronic device can also detect the user's operation on the stylus shortcut control (including but not limited to touch, click, voice control, gesture, etc.), and in response to the user operation, the electronic device can display an interface with a video recording option. The video recording option can be used to trigger the electronic device to enter the video recording function.

[0087] Optional, reference Figure 1A The second interface 101 shown includes multiple application icons. For example, a weather application icon, a calendar application icon, an email application icon, a settings application icon, an application store application icon, a note application icon 1011, an album application icon, an application A application icon 1012, etc. These application icons can be used to trigger the electronic device to start the corresponding application. Among them, the note application icon 1011 is an icon of a note application (i.e., a note application). The note application is a note application. That is, the user can record text information, picture information, etc. through the note application. The application A application icon 1012 is an icon of application A. Application A is a video application.

[0088] Optionally, in response to a user operation on the application icon 1012 of application A, the electronic device may start application A and display a Figure 1B The user interface 102 is shown. The user interface 102 is the main interface of the application A. The user interface 102 may include thumbnails of several videos, for example, thumbnail 1021. For ease of description, the video corresponding to thumbnail 1021 is recorded as video S in this application.

[0089] Optionally, the electronic device may detect a user operation on the thumbnail 1021, and in response to the user operation, the electronic device may play the video corresponding to the thumbnail 1021, that is, the video S, and display the video as shown in FIG. Figure 1CThe user interface 103 shown. The user interface 103 may include a display area 1031 and a display area 1032. The display area 1031 is the main interface of the application A displayed in a split screen of the electronic device, and the display area 1032 is the playback interface of the video S displayed in a split screen of the electronic device. The display area 1032 may include the video S, and may also include other areas (for example, an introduction area, a video recommendation area, etc.) and a download button, a share button, a favorite button, etc.

[0090] Optionally, in some embodiments, Figure 1C The display area 1032 shown can also be displayed in full screen (not shown in the figure). Figure 1C The display area 1032 shown may also be displayed on the right side of the display area 1031 (not shown in the figure).

[0091] 1. Split screen display of video applications and note applications (such as Figure 1C-1E )

[0092] In some embodiments of the present application, in response to a three-finger swipe down operation, the electronic device may display the playback interface of the video S and a newly created note of a note application (eg, a memo application and a note application, etc.) in split screen.

[0093] Optionally, the electronic device displays Figure 1C After the user interface 103 shown, refer to Figure 1D As shown, the electronic device can detect the user's three-finger sliding operation on the user interface 103. The three-finger sliding operation may refer to the user sliding three fingers on the electronic device screen from top to bottom for a certain distance. During the sliding process, the user's fingers keep in contact with the electronic device screen. In response to the user operation, the electronic device displays Figure 1E The user interface 104 shown. The user interface 104 includes area 1, area 2 and a first toolbar 1041, and the first cursor 10431 is displayed in area 2. Optionally, the first cursor 10431 is included in area 2. Among them, area 1 includes the playback interface of the video S, and area 2 includes the user interface of the note application. The user interface of the note application can be an initialization interface for creating a new note, that is, a new note interface.

[0094] Optionally, the electronic device displays Figure 1C After the user interface 103 shown, refer to Figure 1D As shown, the electronic device can detect the user's three-finger sliding operation on the user interface 103. The three-finger sliding operation may refer to the user sliding three fingers on the electronic device screen from top to bottom for a certain distance. During the sliding process, the user's fingers keep in contact with the electronic device screen. In response to the user operation, the electronic device displays Figure 1EThe user interface 104 shown. The user interface 104 includes area 1, area 2 and a first toolbar 1041, where area 1 includes a playback interface of the video S, and area 2 includes a user interface of a note application, which may be an initialization interface for creating a new note. The first toolbar 1041 may be suspended on area 1 and area 2. The user may drag the first toolbar 1041 to change its position. The electronic device may detect the user's drag operation on the first toolbar 1041. In response to the operation, the electronic device may move the display position of the first toolbar 1041 on the user interface 104. Among them, the first toolbar 1041 may include a video recording control 10411, a screenshot control 10412, a speech-to-text control 10413, a text extraction control 10414, and an exit control 10415. The video recording control 10411 may be used to enable a video recording function. The screenshot control 10412 is used to capture a video screen. The speech-to-text control 10413 is used to enable a speech-to-text function and may be used for speech-to-text. The text extraction control 10414 is used to extract text from the video. The exit control 10415 is used to exit the video recording function.

[0095] Optionally, the electronic device may also respond to other user operations and display Figure 1E In the user interface 104 shown, the user operation may be any one of the following: the user pressing a physical button (for example, pressing the power button and the volume up button at the same time), the user performing a predefined gesture (for example, double-tapping the screen, tapping the screen with knuckles, shaking the electronic device), the user operating a corresponding button on the screen, the user inputting a voice command, etc.

[0096] Optionally, the electronic device can detect a three-finger swipe operation of the user on the user interface of a non-video application. In response to this operation, the electronic device can still display the first toolbar 1041. However, at this time, the first toolbar 1041 is in a state of not responding to user clicks, and the first toolbar 1041 can appear gray, that is, although the user interface displays the first toolbar 1041, the user cannot turn on the video recording function. The non-video application may be an application that is not in the video application whitelist preset by the electronic device, and the video application whitelist may include common video applications, such as: Bilibili, Tencent Video, YouTube, etc. When the user clicks in the non-video application to enter an interface with a playable video, the first toolbar 1041 returns to normal, that is, it becomes a state that can respond to user click operations.

[0097] Optionally, in the case where the application to which the application interface corresponding to the three-finger swipe operation belongs does not support the split-screen function, in response to the three-finger swipe operation, the electronic device may also display the following Figure 1G The floating window 1042 is shown.

[0098] It should be noted that the following descriptions of other contents are based on the split screen display of the playback interface of the video S and the new notes of the note application (for example, the memo application and the note application, etc.) for exemplary description. The following descriptions of the floating window state display note application, toolbar and controls therein can refer to this example.

[0099] 2. Display the note application in floating window state (such as Figure 1F-1H )

[0100] In some embodiments of the present application, after the electronic device displays the user interface of the video application (i.e., the video playback interface), a floating window may be displayed. The floating window displays the user interface of a note application (e.g., a memo application and a note application, etc.). The user interface of the note application may be an initialization interface for creating a new note.

[0101] like Figure 1F As shown, the user interface 103 may also include a note application floating control 1033. The electronic device may detect a user operation on the note application floating control 1033, which may be a drag, a click, or a touch. In response to the user operation, the electronic device may display Figure 1E The user interface 104 shown, or Figure 1G In response to the user operation, the electronic device displays the user interface 105 shown in FIG. Figure 1G The user interface 105 shown. Similar to the user interface 103, the user interface 105 may still include the main interface of the application A and the playback interface of the video S. The difference is that the user interface 105 may also include a first toolbar 1041 and a floating window 1042. The third cursor 10432 is displayed in the floating window 1042. Optionally, the interface of the floating window 1042 includes the third cursor 10432.

[0102] Optionally, the user can drag the floating window 1042 to change its position. The electronic device can detect the drag operation on the floating window 1042. In response to the operation, the electronic device can move the position of the floating window 1042. The floating window 1042 displays a new note created by the electronic device in a note application (e.g., a memo application and a note application, etc.).

[0103] Optional, such as Figure 1H As shown, the electronic device displays a user interface 106 including a video playback interface of a video application and a first toolbar 1041. The first toolbar 1041 may include a video recording control 10411. In response to the user's operation on the video recording control 10411, as shown in FIG. Figure 1G As shown, the electronic device displays Figure 1G User interface 105 is shown.

[0104] 2. Trigger the video recording function in the note application ( Figure 1I-1L )

[0105] An electronic device may detect a user operation on a note application (e.g., a memo application and a note application, etc.). In response to the user operation, the electronic device may start the note application and display the main interface of the note application. The main interface of the note application may include a new control. The electronic device may detect a user operation on the new control. In response to the user operation, the electronic device may create a new note and display an initialization interface for the new note. The electronic device may enter a video recording function based on the initialization interface. In one possible implementation, the electronic device may detect a user operation on the initialization interface of the new note, and in response to the user operation, the electronic device may display an interface with a video recording option. The video recording option may be used to trigger the electronic device to enter the video recording function. In another possible implementation, the initialization interface for the new note may include a video recording option.

[0106] like Fig. 1I As shown, the electronic device can detect the user's operation on the note application icon 1011. In response to the user operation, the electronic device can start the note application. In response to the user operation, the electronic device can create a new note, and the note can be displayed as shown in FIG. Figure 1J The user interface 108 shown in FIG. 1 is an initialization interface involved after creating a new note. The user interface 108 may include a control 1081. The electronic device may detect a user operation on the control 1081. In response to the user operation, the electronic device may display a Figure 1K The user interface 109 shown in FIG. 109 may include a display area 1091. The display area 1091 may include several tool options (e.g., a video recording option 10911, a photo taking option, a document scanning option, a card collection option, and a selection from gallery option). The electronic device may detect the user's operation on the video recording option 10911. Figure 1L As shown, in response to the user operation, the electronic device may display icons of several video applications (e.g., icon 11011). Icon 11011 is an icon of application A. Optionally, the several video applications may be applications in the video application whitelist and installed in the electronic device. The electronic device may detect the user operation on icon 11011. In response to the user operation, the electronic device may display icons of several video applications (e.g., icon 11011). Icon 11011 is an icon of application A. Optionally, the several video applications may be applications in the video application whitelist and installed in the electronic device. Figure 1G The user interface 105 shown or Figure 1E User interface 104 is shown.

[0107] Optionally, the electronic device can detect other user operations (for example, other gesture operations, clicking an application icon, clicking a floating ball, clicking a specific entrance on the main interface of the electronic device, etc.) to trigger the electronic device to enter the video recording function, and this application does not impose any restrictions on this.

[0108] Optionally, the electronic device can detect a user operation (for example, a three-finger swipe down operation), and in response to the user operation, simultaneously display the note application (for example, in a split-screen format or a floating window format) and the first toolbar 1041; the electronic device can also first detect the user's operation of opening the note application, and then detect a user operation such as a three-finger swipe down operation, and in response to the user operation, enter the video recording function.

[0109] Optionally, in some embodiments, the electronic device can enter the video recording function in the video excerpt application instead of entering the video recording function in the note application. The video excerpt application can have similar functions to the note application, and the specific functions can refer to the description of the note application in the embodiments of the present application. The video excerpt application can be an independent application with similar but different functions from the note application, and the video excerpt application can be named a video excerpt application or a video recording application, etc.

[0110] Combine the following Figure 2A-2C An exemplary electronic device may record video.

[0111] like Figure 2A As shown, the electronic device can detect the user's operation on the video recording control 10411 included in the first toolbar 1041. In response to the video recording control 10411, the electronic device can start recording a video, generate a video note card in the note, and display the video note card as shown in FIG. Figure 2B User interface 202 is shown. User interface 202 may include area 1 and area 2. Area 1 is the playback interface of video S. Area 1 may include video playback area 20211. In addition to video playback area 20211, area 1 may also include other areas such as introduction area and video recommendation area. Among them, the introduction area may include the content introduction of the video currently being played, and the video recommendation area may include some video thumbnails or video links. In response to the user's operation on these video thumbnails or video links, the corresponding video may be displayed. Area 2 is a note interface. The notes displayed on the note interface may be a newly created note interface of the electronic device (such as Figure 1E and Figure 1G ), you can also trigger the note interface that was originally displayed before entering the video recording function (such as Figure 1J ). Area 2 may include a video note card generated by an electronic device. For ease of description, this application refers to the video note card as card 1. Figure 2BAs shown, "Recording" may be displayed in card 1, which indicates that video recording is currently in progress. Optionally, card 1 may include a card title area 20221. Optionally, card 1 may include an original video link 20222. Optionally, card title area 20221 is an editable area. The user can change the title of the video note card displayed in the card title area 20221. The original video link 20222 can be used to jump to the original video (e.g., video S) corresponding to the video note card. The electronic device can detect the user's operation on the original video link 20222. In response to the user operation, the electronic device can start application A and display the playback interface of video S in application A.

[0112] like Figure 2B As shown, the user interface 202 may further include a second toolbar 2023. In response to a user operation on the video recording control 10411 of the first toolbar 1041, the electronic device does not display the first toolbar 1041, but displays the second toolbar 2023. Optionally, from the perspective of code implementation, the second toolbar 2023 may be the next form of the first toolbar 1041. Optionally, from the perspective of interface display, the second toolbar 2023 is the same as the first toolbar 1041 in that it includes a screenshot control 10412, a speech-to-text control 10413, a text extraction control 10414, and an exit control 10415. Different from the first toolbar 1041, the second toolbar 2023 may further include an end recording control 20231, a doodle control 20232, and a tag control 20233. Among them, the end recording control 20231 may be used to end video recording. The end recording control 20231 may also include the remaining maximum recording time. Optionally, the maximum time for recording a video can be preset according to actual needs (e.g., 5 minutes), and this application does not limit this. Graffiti control 20232 can be used to graffiti on the video screen. Label control 20233 can be used to add a time label to the recorded video.

[0113] Optionally, the electronic device may display Figure 2C The user interface 204 shown. The user interface 204 may include a keyboard 2041. The user may input text after the cursor based on the keyboard 2041. In some embodiments of the present application, the number of texts that can be added does not exceed the preset number of texts. Optionally, the preset number of texts may be pre-set, and the present application does not limit this.

[0114] It should be noted that the keyboard 2041 can be Figure 2CAs shown, it is displayed in the form of a floating window at the lower left of the user interface 204 and only covers the video application, it can also be displayed at the bottom of the user interface 204 and cover the video application and the note application, and it can also be displayed at the lower right of the user interface 204 and only cover the note application. Other descriptions related to the keyboard below can be referred to here.

[0115] Combine the following Figure 3A-3C An exemplary electronic device can enable the speech-to-text function when recording a video.

[0116] like Figure 3B After starting to record a video, the electronic device can start the speech-to-text function in response to the user's operation on the speech-to-text control 10413. Figure 1E As shown, the electronic device can detect the user operation of the video recording control 10411 in the user interface 104. In response to the user operation, the electronic device can start the video recording function. The electronic device can detect the user operation of the speech-to-text control 10413 in the user interface 302. Optionally, the electronic device can display the following Figure 2B Optionally, the electronic device detects the user operation of the speech-to-text control 10413 and displays the following Figure 3B The user interface 302 is shown. For specific details, please refer to the following introduction of automatically turning on the speech-to-text function of an electronic device when recording a video.

[0117] like Figure 3A The electronic device can also automatically start the speech-to-text function when starting to record a video in response to the user's operation on the video recording control 10411. The following will introduce how the electronic device automatically starts the speech-to-text function when recording a video.

[0118] Combination Figure 1E , the electronic device displays Figure 1E The user interface 104 shown includes area 1, area 2 (also referred to as the first interface) and a first toolbar 1041. Area 1 includes the playback interface of video S, and area 2 is the interface of the note application, or in other words, area 2 includes the user interface of the note application, which can be the initialization interface for creating a new note. Among them, the first toolbar 1041 may include a video recording control 10411 and a speech-to-text control 10413. The video recording control 10411 can be used to turn on the video recording function. The first cursor 10431 is displayed in area 2. Optionally, area 2 includes the first cursor 10431.

[0119] Optional, such as Figure 1EAs shown, the electronic device can detect the user's operation on the video recording control 10411 in the user interface 104.

[0120] In response to the user operation (also referred to as the first operation), the electronic device may enable the video recording function and the voice-to-text function. Figure 2B The user interface 202 shown in FIG. Figure 3A User interface 301 shown. User interface 301 may include area 2. Area 2 is a note interface. Area 2 may include card 1. Card 1 may include a speech-to-text area 30111. Optionally, after the electronic device turns on the speech-to-text function, it may collect the sound signal of the video playing in area 1, and convert the sound signal into text and display it in the speech-to-text area 30111 (also referred to as the second display area). Optionally, after the note application (also referred to as the first application) starts recording the video, text content is displayed in the speech-to-text area 30111 (also referred to as the second display area), and the text content is obtained by converting the voice in the recorded video. Optionally, in some embodiments of the present application, the position of the speech-to-text area 30111 in card 1 may be below the text "Recording..."

[0121] like Figure 3C As shown, since the speech-to-text function needs to display new text information, a cursor is needed. Therefore, under the video recording function, when the electronic device turns on the speech-to-text function, a second cursor 10433 will appear in the user interface, and the position of the second cursor 10433 changes with the position of the latest one or more converted texts obtained by the speech-to-text function.

[0122] Optional, such as Figure 3C As shown, the electronic device can display the text converted from the sound signal of the video played in area 1 in the speech-to-text area 30111, and the text appears as the second cursor 10433 moves, or the second cursor 10433 moves as the text appears. The second cursor 10433 is displayed in the speech-to-text area 30111 (also referred to as the second display area), and the position of the second cursor 10433 is at the end of the text content (for example, it can be after the last word of the text content, or before the last word, or near the last few words of the text content.)

[0123] Optionally, after the electronic device detects a user operation on the speech-to-text control 10413, the speech-to-text control 10413 may be in a selected state. In some embodiments of the present application, when the speech-to-text control 10413 is in a selected state, the background of the speech-to-text control 10413 may be highlighted. In some other embodiments of the present application, when the speech-to-text control 10413 is in a selected state, the color of the speech-to-text control 10413 is blue (not shown in the color change diagram). When the speech-to-text control 10413 is in an unselected state, the color of the speech-to-text control 10413 is black (not shown in the color change diagram).

[0124] Optionally, the electronic device may detect a user operation on the speech-to-text control 10413 in the user interface 301. In response to the user operation, the electronic device may turn off the speech-to-text function.

[0125] Optionally, the electronic device may detect a downward sliding operation of the user in area 2. In response to this operation, the electronic device may display other contents of the note corresponding to area 2 (eg, recorded video, text, screenshots, etc., not shown in the figure).

[0126] Combine the following Figure 4A-4B The following is an example of how to add text when recording a video by an electronic device.

[0127] like Figure 4A As shown, the electronic device can detect the user's operation on the text adding area 40111. In response to the operation, the electronic device can display Figure 4A The user interface 4011 shown in FIG. The user interface 4011 may include a keyboard 40112. A cursor 4012 may be displayed in a text adding area 40111 in the user interface 4011. The user may input in the text adding area 40111 based on the keyboard 40112. In response to the user inputting in the text adding area 40111 based on the keyboard 40112, the electronic device may display the text content added by the user in the text adding area 40111. Figure 4B As shown, the cursor 402 can move backward as the user adds text, or move downward as the user changes lines (not shown in the figure).

[0128] It should be noted that if Figure 4BAs shown, when the electronic device is recording a video, adding text requires two different cursors to be displayed in the note application, and the two cursors are independent of each other and do not interfere with each other. If both cursors are displayed in one view, there may be a conflict problem. For example, a cursor is required to add text or pictures in the note text editing area of ​​the note application. If the user moves the cursor of the speech-to-text area to the note text editing area and adds text, since the speech-to-text function is not stopped, the text converted by the speech-to-text function will not only be displayed in the speech-to-text area, but the same text will also be added to the note text editing area corresponding to the cursor. For another example, the user does not move the cursor in the speech-to-text area, but tries to add a picture directly in the speech-to-text area where the cursor is located. Since the speech-to-text area is a pure text area, the electronic device may not respond to the user's operation of adding a picture in the speech-to-text area, or it may also display the option to add a picture, but the picture cannot be displayed after the user chooses to add it. When the user uses the video recording function and the speech-to-text function, the picture cannot be inserted normally or the text converted from the voice in the recorded video is added unnecessarily, which seriously affects the user experience.

[0129] In view of this, in order to allow the electronic device to continuously output the converted text in the speech-to-text area, the note body editing area can also respond to the user's operation and perform other edits (including but not limited to adding text, pictures, etc.), the present application provides a video processing method. This method can be used to ensure that in the process of turning on the speech-to-text function under the video recording function, the cursor of the speech-to-text area can exist in the display interface of the note application at the same time as the cursor of the note body editing area, and there is no conflict between them. Some embodiments of the present application adopt the following technical solutions:

[0130] The following first introduces the device involved in the embodiments of the present application.

[0131] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0132] The video processing method provided in the embodiments of the present application can be applied to electronic devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), etc. The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

[0133] By way of example, the hardware structure of an electronic device is described. Figure 5 As shown, Figure 5 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0134] The electronic device may include a processor 510 , an external memory interface 520 , an internal memory 221 , an audio module 530 , a speaker 530A, a receiver 530B, a microphone 530C, an earphone interface 530D, a display screen 540 , a camera 550 , and a touch sensor 560 .

[0135] The structure illustrated in the embodiments of the present invention does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware. The interface connection relationship between the modules illustrated in the embodiments of the present invention is only a schematic illustration and does not constitute a structural limitation on the electronic device.

[0136] The processor 510 may include one or more processing units, for example, the processor 510 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. A memory may also be provided in the processor 510 for storing instructions and data.

[0137] The electronic device implements the display function through the GPU, the display screen 540, and the application processor.

[0138] The GPU is a microprocessor for image processing, which is connected to the display screen 540 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 510 may include one or more GPUs, which execute program instructions to generate or change display information. The display screen 540 is used to display images, videos, etc. In some embodiments, the electronic device may include 1 or N display screens 540, where N is a positive integer greater than 1.

[0139] In the embodiment of the present application, the ability of the electronic device to display a user interface depends on the display function provided by the above-mentioned GPU, display screen 540, and application processor.

[0140] The camera 550 is used to capture still images or videos. The ISP is used to process the data fed back by the camera 550. Light is transmitted to the camera photosensitive element through the lens, and the light signal is converted into an electrical signal. The camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The electronic device may include 1 or N cameras 550, where N is a positive integer greater than 1. The video codec is used to compress or decompress digital video. The electronic device may support one or more video codecs. In this way, the electronic device can play or record videos in multiple encoding formats, such as: Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0141] The internal memory 221 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM). The random access memory can be directly read and written by the processor 510, and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, and can also be used to store user and application data. The non-volatile memory can also store executable programs and store user and application data, etc., and can be loaded into the random access memory in advance for direct reading and writing by the processor 510.

[0142] In the embodiment of the present application, the code for implementing the video processing method described in the embodiment of the present application may be stored in a non-volatile memory. When running the camera application, the electronic device may load the executable code stored in the non-volatile memory into the random access memory.

[0143] The external memory interface 520 can be used to connect to an external non-volatile memory to expand the storage capacity of the electronic device.

[0144] The electronic device can implement audio functions through the audio module 530, the speaker 530A, the receiver 530B, the microphone 530C, the headphone jack 530D, and the application processor.

[0145] The audio module 530 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The speaker 530A, also called "speaker", is used to convert audio electrical signals into sound signals. The receiver 530B, also called "earpiece", is used to convert audio electrical signals into sound signals. The microphone 530C, also called "microphone", "microphone", is used to convert sound signals into electrical signals. The headphone interface 530D is used to connect wired headphones.

[0146] In an embodiment of the present application, when the electronic device turns on the speech-to-text function but does not turn on the video recording function, the microphone 530C can be enabled to collect sound signals and convert the sound signals into corresponding text.

[0147] The touch sensor 560 is also called a "touch control device". The touch sensor 560 can be set on the display screen 540, and the touch sensor 560 and the display screen 540 form a touch screen, also called a "touch control screen". The touch sensor 560 is used to detect a touch (also called a touch screen) operation acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 540. In other embodiments, the touch sensor 560 can also be set on the surface of the electronic device, which is different from the position of the display screen 540.

[0148] In the embodiment of the present application, the electronic device can use the touch sensor 560 to detect the user's click, slide and other operations on the display screen 540 to implement the video processing method in the embodiment of the present application.

[0149] The following introduces the software architecture of the electronic device involved in the embodiments of the present application.

[0150] Exemplarily, the software architecture of an electronic device is described. The electronic device may be Figure 5 The electronic device 500 shown, the software system of the electronic device can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes a software system of a layered architecture as an example to exemplify the software structure of the electronic device 500. Figure 6 This is a software architecture block diagram of an electronic device provided in an embodiment of the present application.

[0151] The layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, Android TM The system is divided into four layers, from top to bottom: application layer, application framework layer, Android runtime (Android TM runtime) and system libraries, as well as the kernel layer.

[0152] The application layer can include a series of application packages. Figure 6 As shown, the application package may include applications such as camera, gallery, calendar, call, map, WLAN, music, SMS, video, note, etc. It should be noted that, in some embodiments, the application package may also include a video excerpt application.

[0153] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0154] like Figure 6 As shown, the application framework layer may include a Java module and a native module.

[0155] The java module includes the view system, window manager, content provider, resource manager, and notification manager.

[0156] The view system may include a view component, which represents a basic component element in the application interface, occupies a rectangular area on the screen, and is responsible for interface drawing and event response to user operations. In some embodiments, the view system can be used to build an application. The application interface can be composed of one or more views. For example, a display interface including a text notification icon may include a view for displaying text and a view for displaying pictures.

[0157] The window manager is used to manage window programs, including the creation, deletion and modification of windows, as well as the size, level, focus position, etc. of windows. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, capture the screen, etc. The application can add and delete windows in the Activity by calling the addview() and removeview() functions of the WindowManager class. The content provider is used to store and obtain data and make the data accessible to the application. The data may include video, images, audio, calls made and received, browsing history and bookmarks, phone books, etc. The phone manager is used to provide communication functions for the electronic device 100. For example, the management of call status (including connected, hung up, etc.). The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, etc. The notification manager enables the application to display notification information in the status bar, which can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify the completion of downloads, message reminders, etc. The notification manager can also be a notification that appears in the top status bar of the system in the form of an icon or scroll bar text, such as a notification of an application running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is prompted in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0158] The native module can be implemented in C++ language, including input event reading (inputreader) module and input event distribution (inputdispatcher) module. The inputreader module is responsible for reading input events generated by input device driver. The inputdispatcher module is responsible for distribution of input events.

[0159] Android TM The runtime includes the core library and the virtual machine. Android TM The runtime is responsible for scheduling and management of the Android system.

[0160] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0161] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0162] The system library can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing library (for example: OpenGL ES), 2D graphics engine (for example: SGL), etc. The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis and layer processing, etc. The 2D graphics engine is a drawing engine for 2D drawing.

[0163] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, sensor driver, input / output device driver (for example, keyboard, touch screen, mouse, speaker, microphone, etc.).

[0164] Among them, the user performs an input operation (for example, a touch screen operation) on the input device, and the kernel layer of the mobile terminal 100 can generate a corresponding input event (such as a touch screen event) according to the input operation, and report the input event to the upper layer. In some embodiments, the mobile terminal 500 also includes an input event management service (inputmanagerservice), and the inputmanagerservice includes the above-mentioned inputreader module and inputdispatcher module, and the inputmanagerservice is responsible for the service of input event management corresponding to the external input device.

[0165] Understandably, Figure 6 The software structure of the electronic device 500 shown is only an example. In other embodiments, the software structure of the electronic device 500 may include more or fewer layers, each layer may include more or fewer modules, or may be other operating system structures, such as iOS. TM , Windows TM Etc., no limitation is made here.

[0166] The following introduces the software architecture of the note-taking application involved in the embodiment of the present application.

[0167] Figure 7 This is a software architecture block diagram of a note-taking application provided in an embodiment of the present application.

[0168] In this embodiment, the note application may include modules such as an interface display module, a video control module, a note editing module, a speech-to-text editing module, a video excerpt editing module, and a javascript (JS) interface rendering module.

[0169] The interface display module is a module used to display the interface during the running of the note application and receive instructions or information input by the user through the note interface. Optionally, the interface module can implement its functions by calling relevant modules of the application kernel layer and / or the application architecture layer.

[0170] The JS interface rendering control module may be EditorControl(js). The JS interface rendering control module is used to render data to the web page view to realize the display of web page content in the web page view.

[0171] The video control module can be video(js). The video control module is the overall control module in JS, which can be used to distribute specific requests of JS to each module of JS. The video control module is a function transfer module and is responsible for processing some common functions.

[0172] The note editing module can be EditWebFragment (java). The note editing module can be a fragment of an activity in an Android native note application, and the fragment can be a loaded object. The note editing module includes a web view component and an HTML editor. The web view component is used to display the web view. The HTML editor is an editor that performs editing based on the HTML language. Optionally, the HTML editor can be an H5 editor. The HTML editor is used to edit the content entered by the user in a non-speech-to-text state and generate web page content. The data and content generated by the HTML editor can be rendered to the web view through the video control module.

[0173] The speech-to-text editing module includes a speech-to-text view component and a speech-to-text editor. The speech-to-text view component is used to realize the display of the speech-to-text view. The speech-to-text editor is used to edit the content input by the user under the speech-to-text function and generate speech-to-text content.

[0174] The video excerpt editing module includes a video excerpt view component and a video excerpt editor. The video excerpt view component is used to realize the display of the video excerpt view. The speech-to-text editor is used to edit the content input by the user under the video recording function and generate the content of the recorded video.

[0175] The JS interface rendering control module is a module used to implement note-related functions through logical control, such as recording function, recording playback function, two-way positioning function of recording playback and content display, video recording function, function of playing recorded videos, two-way positioning function of video playback and tag display, document insertion function, etc. The video control module can be used to control the relevant modules in the application architecture layer to record, and also to realize the playback of recordings, etc. The video control module can be used to control other modules or APPs to record videos, and also to realize the playback of recorded videos, etc. In a specific embodiment, the video control module can achieve the effect of recording videos by controlling the video recording function.

[0176] Optionally, the note-taking application may also include other logical control modules in addition to the video control module, such as a recording and playback control module, an excerpt and playback control module, a speech-to-text control module, etc. The recording and playback control module, the excerpt and playback control module, the speech-to-text control module, etc. may also be included in the video control module, which are not listed one by one in this application.

[0177] It should be noted that the above-mentioned video excerpt view component, speech-to-text view component, web page view component, video control module, etc. may also be partially or completely set in the application framework layer or system library. The present application embodiment does not make any limitation to this.

[0178] The following is a schematic diagram of an interface of a note-taking application provided in an embodiment of the present application.

[0179] Please refer to Figure 8. Fig. 8A A structural diagram of an interface of a note-taking application provided in an embodiment of the present application. Figure 8B The interface display corresponding to the structural diagram of the video note card area 8013 is shown. Figure 8C for Fig. 8A The structural schematic diagram shown corresponds to the interface schematic diagram and each area schematic diagram. Fig.8D for Figure 8C Schematic diagram of the view and cursor position corresponding to the interface.

[0180] like Fig. 8A and Figure 8C As shown, in the embodiment of the present application, the display area 8011 of the note application includes but is not limited to the note text editing area 8012 (also referred to as the first display area), the video note card area 8013, the video recording area 8014, the speech-to-text area 8015, and the video note area 8016. Among them, the note text editing area 8012 includes the video note card area 8013, and the video note card area 8013 includes the video recording area 8014, the speech-to-text area 8015, and the video note area 8016. It should be noted that although the note text editing area 8012 and the video note area 8016 can carry the function of displaying text, pictures and other contents, the two are different areas, and the video note area 8016 is included in the video note card area 8013.

[0181] For example, Figure 8B The interface display corresponding to the structural diagram of the video note card area 8013. In some embodiments, Figure 8B The video note card area 8013 shown in the figure only occupies part of the control of the area when the display area 8011 of the note application is displayed, and a sliding bar ( Figure 8B Slide bar not shown). Figure 8B The video recording area 8014 shown in FIG. 8 can be used to display the recorded video. Figure 8B For ease of understanding, the various areas and related contents that may be included in the video note card are shown in an exemplary manner, which may be different from the actual presentation method in use. For example, when recording a video, the video recording area 8014 may display text content such as "Recording" instead of displaying the recorded video in real time. Figure 8B The speech-to-text area 8015 shown in the figure can be used to display text converted from the speech information in the video recorded in the video recording area 8014. Figure 8B The video note area 8016 shown in can be used to display screenshots and user-entered text.

[0182] like Fig. 8A , Figure 8C and Fig.8D As shown, in the note application, the content of the note text editing area 8012 (also referred to as the first display area) is carried by a view, which can be Webview8041 (also referred to as the second view). In some embodiments, the size of Webview8041 is equal to the size of the note text editing area 8012 and completely covers the note text editing area 8012. In other embodiments, the size of Webview8041 may also be slightly smaller than the size of the note text editing area 8012, which is not limited in this embodiment of the present application. Optionally, the note text editing area 8012 is the display area corresponding to Webview8041.

[0183] like Fig. 8A , Figure 8C and Fig.8D As shown, in some embodiments, the content of the speech-to-text area 8015 (also referred to as the second display area) is carried by a view, which may be an EditText 8042 (also referred to as the first view). The speech-to-text area 8015 (also referred to as the second display area) is covered with an EditText 8042 (also referred to as the first view), and the size of EditText 8042 is equal to the size of the speech-to-text area 8015 and completely covers the speech-to-text area 8015. In other embodiments, the size of EditText 8042 may also be slightly smaller than the size of the speech-to-text area 8015, which is not limited in the present embodiment of the application.

[0184] Optionally, EditText8042 can be overlaid on Webview8041, and a view group (view group) is formed by EditText8042 and Webview8041. It should be noted that the view group composed of EditText8042 and Webview8041 can include multiple views. It should be noted that in this application, the names of each view are only used as examples and are not intended to be limiting.

[0185] As shown Figure 8C and Fig.8DAs shown, after the electronic device detects that the note application is turned on, a Webview8041 is covered on the note text editing area 8012, and the Webview8041 has a first cursor, and the content of the note text editing area 8012 is displayed by Webview8041. After the electronic device detects that the video recording function and the voice-to-text function are turned on, the electronic device creates EditText8042 to display the text content of the voice-to-text area, and EditText8042 has a second cursor, so that the voice-to-text area 8015 has a second cursor that is different from the first cursor. Optionally, EditText8042 can be the first view, and Webview8041 can be the second view.

[0186] The following is a flow chart of a video processing method provided by an embodiment of the present application.

[0187] Based on the execution subjects and application scenarios provided in the above embodiments, combined with the accompanying drawings and application scenarios, the video processing method provided in the embodiments of the present application is introduced below. Through the video processing method, for example: Fig.9A As shown, the electronic device displays a first cursor 10431 in the note text editing area 8012, and after detecting the user's enter operation, as shown in FIG. Fig. 9B As shown, the electronic device displays a blank line 9025 in the note text editing area 8012. The position of the speech-to-text area 9024 is also moved down one line.

[0188] Please refer to Fig.10 , Fig.10 The figure is a flowchart of a video processing method according to an exemplary embodiment.

[0189] Optionally, in some embodiments, Figure 2A As shown, in response to the user clicking the video recording control 10411, after the hardware sensor of the electronic device receives the user's operation, the driver or sensor driver of the electronic device reports the user's input event to the input event reading module after receiving the user's operation for starting the video recording function. The input event reading module sends the event to the input event distribution module, and the electronic device distributes the input event to the corresponding application (i.e., the note application) through the input event distribution module. After the note application receives the event, the electronic device starts to execute the following steps 1101.

[0190] Please refer to Fig.11 , Fig.11 FIG. 1 is a flow chart of a video processing method according to an exemplary embodiment. As an example but not a limitation, the method is applied to an electronic device installed with a note application and a video application. The electronic device uses Fig.11Taking the interaction of multiple modules shown in the figure as an example, the method may include some or all of the following contents:

[0191] Step 1101: In response to a user starting a video recording operation, the electronic device starts recording the video and performs speech-to-text conversion on the played video.

[0192] Optionally, in some embodiments, Figure 2A As shown, turning on the video recording function will automatically turn on the voice-to-text function, or both the turning on of the video recording function and the turning on of the voice-to-text function correspond to the video recording control 10411. Figure 2A As shown, in response to the user clicking the video recording control 10411, the electronic device starts recording the video and voice-to-text conversion. Optionally, in some embodiments, the note editing module starts recording the video and voice-to-text conversion of the played video.

[0193] Optionally, in other embodiments, such as Figure 2A As shown, in response to the user clicking the video recording control 10411, the electronic device starts recording the video. Figure 2B As shown, in response to the user clicking on the speech-to-text control 10413, the electronic device starts to perform speech-to-text conversion.

[0194] refer to Fig.13 13A, before the user turns on the video recording function, the display area 8011 of the note application includes a note text editing area 8012, and the note text editing area 8012 is hosted by Webview8041.

[0195] Step 1102: The note editing module sends relevant information of the video recording to the video control module.

[0196] Optionally, in some embodiments, in response to the electronic device starting to record a video and performing voice-to-text conversion on the played video, the note editing module sends the relevant information of the video recording to the video control module. Optionally, in other embodiments, after the electronic device starts to record a video and performs voice-to-text conversion on the played video, the note editing module sends the relevant information of the video recording to the video control module.

[0197] Optionally, in some embodiments, in response to the note editing module starting to record a video and converting speech to text for the played video, the note editing module sends relevant information about the video recording to the video control module. Optionally, in other embodiments, after the note editing module starts to record a video and converts speech to text for the played video, the note editing module sends relevant information about the video recording to the video control module.

[0198] The note editing module can send relevant information of video recording to the JS interface rendering control module by calling the first insertion function. The JS interface rendering control module can encapsulate the relevant information of video recording, and after completing the encapsulation, send the encapsulated relevant information of video recording to the video control module by calling the second insertion function.

[0199] Optionally, the note editing module can call the execjavascript method through JSBridge to send relevant information of the video recording to the video control module.

[0200] Optionally, the electronic device displays the relevant information of the video recording sent by the note editing module to the video control module as follows: Figure 2B Card 1 in user interface 202 is shown.

[0201] Optionally, step 1102 includes step 11021 and step 11022.

[0202] Step 11021: The note editing module may send relevant information of the video recording to the JS interface rendering control module by calling the first insertion function.

[0203] Optionally, the first insertion function may be an insert function. Calling the insert function may insert the relevant information of the video recording into the JS interface rendering control module. The relevant information of the video recording includes necessary data related to the video note card, for example, the video title on the video note card when recording the video. The relevant information of the video recording may include the elements to be inserted in the video note card, combined with Figure 2B The element to be inserted may be the title of the video note card in the card title area 20221. The relevant information of the video recording may also include the element type of the element to be inserted.

[0204] Optionally, the element type of the element to be inserted into the video note card is video. The note editing module determines the element type of the element to be inserted according to the current input mode of the note application. For example, when the input mode of the note application is text typing mode, the note editing module determines that the element type of the element to be inserted is an H5 element. For another example, when the input mode of the note application is video recording mode, the note editing module determines that the element type of the element to be inserted is a video element.

[0205] Optionally, the element type of the element to be inserted can be represented by a type parameter (Type). When the value of Type is 1, it indicates an H5 element, and when the value of Type is 2, it indicates video content. Of course, in other embodiments, the H5 editor can also specifically distinguish each type of H5 element and set a unique identifier for each H5 element. For example, when the value of Type is 1-1, it indicates a text element, and when the value of Type is 1-2, it indicates a video element.

[0206] In this embodiment, based on the fact that the video recording function has been enabled in the note application, the note editing module determines that the type of element to be inserted into the video note card is a video element.

[0207] The content to be inserted is data corresponding to the inserted element type, and the video note card corresponds to the necessary data for creating a video note card, including: video title, location to be inserted, attributes, format, etc. In this embodiment, the note application is currently in video recording mode, so the note editing module can determine that the type of element to be inserted in the video note card is a video element.

[0208] Optionally, in this embodiment, the element to be inserted may include a video title. Figure 2B As shown, in response to the user clicking on the video recording function control 20231, the note editing module sends relevant information of the video recording to the JS interface rendering control module by calling the insert function. The relevant information of the video recording may include the video title of video 20211.

[0209] Step 11022: The JS interface rendering control module may encapsulate the relevant information of the video recording, and after completing the encapsulation, send the encapsulated relevant information of the video recording to the video control module by calling the second insertion function.

[0210] Optionally, the second insertion function may be an insertNodes function. Calling the insertNodes function may insert relevant information of the video recording into the video control module.

[0211] Step 1103: The video control module renders the HTML area.

[0212] Optionally, the video control module renders the HTML area to be displayed as the electronic device creates the following Figure 2B The card 1 shown is a video note card. The html area can be a custom html area. Optionally, refer to Fig.13In 13B, based on the video control module rendering html area, the display area 8011 of the note application includes a note text editing area 8012, and the note text editing area 8012 is carried by Webview 8041. The note text editing area 8012 includes a video note card area 8013.

[0213] Optionally, step 1103 includes step 11031 and step 11032.

[0214] Step 11031: The video control module performs document object model (DOM) rendering.

[0215] Optionally, DOM rendering can be used to display the corresponding window or card. DOM rendering refers to the process of generating a DOM document that is displayed to the user in the HTML area. For detailed instructions on DOM rendering, please refer to the relevant technical documents, which will not be repeated here.

[0216] Step 11032: The video control module performs speech-to-text placeholder rendering.

[0217] It should be noted that the rendered speech-to-text placeholder area is an area that users cannot see. Figure 8C As shown, the size and position of the speech-to-text placeholder area are the same as those of the speech-to-text area 8015. For ease of understanding, the speech-to-text placeholder area is referred to as the placeholder area (also referred to as the first placeholder area) hereinafter.

[0218] Optionally, when creating a video note card, the electronic device creates a placeholder area in the video note card that has the same size and position as the speech-to-text area.

[0219] Step 1104: The JS interface rendering control module sends the position information of the placeholder area to the video control module.

[0220] Optionally, the video control module receives the position information of the placeholder area (also referred to as the first position information of the first placeholder area) by calling a function interface. For ease of understanding, the step of the video control module requesting the JS interface rendering control module for the position information of the placeholder area by calling a function is not shown in the figure.

[0221] Optionally, the video control module calls the getElementAttributeInfo interface in the JS interface rendering control module. In response to the video control module calling the getElementAttributeInfo interface in the JS interface rendering control module, the JS interface rendering control module returns the position information of the placeholder area to the video control module.

[0222] Optionally, the position information of the placeholder area may include the position information of the speech-to-text placeholder area: the length (height), width (width), top margin (top), bottom margin (bottom), left margin (left), and right margin (right) of the placeholder area.

[0223] Among them, optionally, left can be the pixel value of the distance between the placeholder area and the left border of the note text editing area. top can be the pixel value of the distance between the placeholder area and the top border of the note text editing area. bottom can be the pixel value of the distance between the placeholder area and the bottom border of the note text editing area. right can be the pixel value of the distance between the placeholder area and the right border of the note text editing area. width can be the pixel value of the width of the placeholder area. height can be the pixel value of the height of the placeholder area.

[0224] Optionally, left can be equal to the distance in pixels between the speech-to-text area and the left border of the note text editing area. top can be equal to the distance in pixels between the speech-to-text area and the top border of the note text editing area. bottom can be equal to the distance in pixels between the speech-to-text area and the bottom border of the note text editing area. right can be equal to the distance in pixels between the speech-to-text area and the right border of the note text editing area. width can be equal to the width of the speech-to-text area in pixels. height can be equal to the height of the speech-to-text area in pixels.

[0225] In addition, optionally, the position information of the placeholder area may also include placeholder area information of other areas, for example, placeholder area information of the video note card and placeholder area information of the title, so as to support functions such as dragging and dropping of the video note card in the later stage.

[0226] It should be noted that, in some embodiments, the border of the note text editing area is the border used to display the note text editing area Webview. Optionally, in some embodiments, the border of the note text editing area is consistent with the border of the note application.

[0227] Step 1105: The video control module sends the placeholder area location information to the note editing module.

[0228] Optionally, the video control module uploads the position information of the placeholder area to the note editing module by calling a function interface. For ease of understanding, this step is not shown in the figure.

[0229] The video control module calls the updateElementAttributeInfo interface in the note editing module. By calling the updateElementAttributeInfo interface, the JS interface rendering control module reports the position information of the placeholder area of ​​the speech-to-text area to the note editing module. The updateElementAttributeInfo interface is an interface for updating the position area, and can be used to report the position information of the placeholder area.

[0230] Step 1106: The note editing module creates a text editing control (EditText).

[0231] EditText is a subclass of Textview, which can be equivalent to another implementation of view. Among them, the role of EditText includes displaying text content to the user, so that the user can selectively edit the text. The role of EditText is basically the same as that of Textview. The difference is that Textview does not allow users to edit it, while EditText allows users to edit it. Optionally, EditText can also be set with a listener, which can be used to detect whether the user's input is legal. Optionally, the electronic device creates EditText, and the size and position of EditText and the placeholder area are consistent.

[0232] refer to Fig.13 In 13C, EditText is created based on the note editing module, and the display area 8011 of the note application includes a note text editing area 8012, and the note text editing area 8012 is carried by Webview8041. The note text editing area 8012 includes a video note card area 8013, and the video note card area 8013 includes a speech-to-text area 8015. Webview8041 includes EditText8042.

[0233] Optionally, step 1106 includes step 11061 and step 11062.

[0234] Step 11061, the note editing module calculates the attribute parameters (layoutparams) of the speech-to-text placeholder window.

[0235] The layoutparams may include information such as the size and position of the placeholder. The calculation may be to create a new layoutparams based on the position information of the uploaded speech-to-text placeholder. Based on the layoutparams, the electronic device may determine an area that can be used to carry the display of the view.

[0236] Step 11062: By calling the addview() function, the note editing module creates EditText according to layoutparams.

[0237] Optionally, the addview() function can also be called the first creation function.

[0238] Optionally, the electronic device creates an EditText based on the information included in layoutparams.

[0239] Optionally, by calling the addview() function of the WindowManager class, the electronic device creates an EditText in the area corresponding to the speech-to-text placeholder area.

[0240] Step 1107: In response to the user turning off the video recording function, the electronic device ends recording.

[0241] Optionally, in response to the user's second operation, the note-taking application ends recording the video.

[0242] Step 1108, the note editing module removes EditText.

[0243] Optionally, the electronic device removes the first view after the second operation.

[0244] Optionally, the note editing module removes EditText by calling the removeview(EditText) function (also referred to as the first removal function).

[0245] refer to Fig.13 13E in the code, EditText is removed based on the note editing module, and EditText8042 in Webview8041 is removed.

[0246] Step 1109: The note editing module sends relevant information about the content in EditText to the video control module.

[0247] Optionally, step 1109 includes step 11091 and step 11092.

[0248] Step 11091, the note editing module sends relevant information about the content in EditText to the JS interface rendering control module.

[0249] Optionally, the note editing module sends the relevant information of the content in EditText to the JS interface rendering control module by calling the updateVideoByld interface (also referred to as the first interface) from the JS interface rendering control module. The updateVideoByld interface can be used to instruct the JS interface rendering control module to end the video recording function.

[0250] The relevant information of the content in EditText is the information carried by EditText, including but not limited to the text information obtained by voice-to-text conversion. Fig. 9B As shown, the relevant information may include text information in video note card area 8014, "This tree is also found in Australia and Mexico. The tropical rain forest is their home."

[0251] Step 11092, the JS interface rendering control module sends relevant information about the content in EditText to the video control module.

[0252] Optionally, the JS interface rendering control module sends relevant information about the content in EditText to the video control module by calling the setNode interface (also referred to as the second interface).

[0253] Step 1110: The video control module updates the HTML area.

[0254] Optionally, the video control module updates the html area based on the relevant information of the content in EditText, and the html area can be a custom html area. Optionally, the video control module updates the html area, which can be manifested as the electronic device inserting the relevant information of the content in EditText into the video note card. It should be noted that since the video control module updates the custom html area based on the relevant information of the content in EditText, even if EditText is removed, the speech-to-text information is still displayed in the video note card area. Optionally, the electronic device displays a video note card with a speech-to-text area.

[0255] Optionally, step 1110 includes step 11101 and step 11102.

[0256] Step 11101: The video control module performs DOM rendering.

[0257] For a specific description, please refer to the DOM rendering in 1103, which will not be repeated here.

[0258] Step 11102: The video control module removes the speech-to-text placeholder area.

[0259] Please refer to Fig.12 , Fig.12 is a flow chart of a video processing method according to another exemplary embodiment. Fig.11 The difference between the corresponding embodiments is that in the embodiment of the present application, the electronic device detects whether the user has operated the note text editing area by registering a listener, and adjusts the position of the EditText according to the specific circumstances of the user's operation. This ensures that during the continuous output and / or editing of the speech-to-text area, the note text editing area can also be edited in other ways (including but not limited to adding text, pictures, etc.).

[0260] As an example but not a limitation, the method is applied to an electronic device installed with a note application and a video application. Fig.12 Taking the interaction of multiple modules shown in the figure as an example, the method may include some or all of the following contents:

[0261] Step 1201: In response to a user's operation of starting a video recording function, the electronic device starts recording the obtained video and performs speech-to-text conversion on the played video.

[0262] For a specific description, please refer to step 1101, which will not be repeated here.

[0263] Step 1202: The note editing module sends the video recording related information to the video control module.

[0264] For a specific description, please refer to step 1102, which will not be repeated here.

[0265] Step 12021: The note editing module can send relevant information of the video recording to the JS interface rendering control module by calling the first insertion function.

[0266] For a specific description, please refer to step 11021, which will not be repeated here.

[0267] Step 12022, the JS interface rendering control module can encapsulate the relevant information of the video recording, and after completing the encapsulation, send the encapsulated relevant information of the video recording to the video control module by calling the second insertion function.

[0268] For a specific description, please refer to step 11022, which will not be repeated here.

[0269] Step 1203: The video control module renders the HTML area.

[0270] For a specific description, please refer to step 1103, which will not be repeated here.

[0271] Step 12031: The video control module performs document object model (DOM) rendering.

[0272] For a specific description, please refer to step 11031, which will not be repeated here.

[0273] Step 12032: The video control module performs speech-to-text placeholder rendering.

[0274] For a specific description, please refer to step 11032, which will not be repeated here.

[0275] Step 1204: The video control module registers a listener for content changes in the note body editing area.

[0276] The note body editing area content change listener can be used to detect whether the content in the note body editing area has changed. Optionally, the video control module can be informed of the change of the content in the note body editing area through a callback function.

[0277] Optionally, the content change listener of the note body editing area can also be called the first listener.

[0278] Step 1205: The JS interface rendering control module sends the placeholder area position information to the video control module.

[0279] For a specific description, please refer to step 1104, which will not be repeated here.

[0280] Step 1206: The video control module sends the placeholder area location information to the note editing module.

[0281] For a specific description, please refer to step 1105, which will not be repeated here.

[0282] Step 1207: The note editing module creates a text editing control (EditText).

[0283] For a specific description, please refer to step 1106, which will not be repeated here.

[0284] Step 12071, the note editing module calculates the attribute parameters (layoutparams) of the speech-to-text placeholder window.

[0285] For a specific description, please refer to step 11061, which will not be repeated here.

[0286] Step 12072, by calling the addview() function, the note editing module creates EditText according to layoutparams.

[0287] For a specific description, please refer to step 11062, which will not be repeated here.

[0288] Step 1208: The video control module detects that the content of the note body editing area has changed.

[0289] User operations may cause changes in the speech-to-text placeholder information, for example, Fig.9A and Fig. 9B As shown, the electronic device detects the user's operation of editing the note content (for example, a carriage return or typing operation is performed at the first cursor 10431 above the video note card area 8013), and changes the display position of the video note card. In this case, the text content change listener (first listener) registered by the video control module can monitor the change of the content in the note text editing area. Optionally, based on the first listener, the video control module detects that the content in the note text editing area has changed. Optionally, in response to the user's second operation, the position of the first cursor moves down by one line, and the position of the second display area moves down by one line. The meaning of the position moving down by one line means that the upper edge of the corresponding area moves down by one line.

[0290] Optionally, in response to the user's operation of editing the note content, the text content change listener registered by the video control module detects that the content of the note text editing area has changed. The user's operation of editing the note content may be operations such as entering, spacing, and typing in the note text editing area.

[0291] Step 1209: The JS interface rendering control module sends the second position information of the first placeholder area to the video control module.

[0292] Optionally, after the electronic device detects that the content of the note body editing area has changed, the JS interface rendering control module sends the placeholder area position information (also referred to as the second position information of the first placeholder area) to the video control module.

[0293] Optionally, in response to a text content change listener registered by the video control module detecting a change in the content of the note text editing area, the JS interface rendering control module sends placeholder area position information to the video control module.

[0294] It should be noted that step 1209 is consistent with step 1205. For a specific description, please refer to step 1205, which will not be repeated here.

[0295] Step 1210: The video control module sends the second position information of the first placeholder area to the note editing module.

[0296] Optionally, after the JS interface rendering control module sends the placeholder area position information to the video control module, the JS interface rendering control module sends the placeholder area position information to the video control module.

[0297] Optionally, in response to the JS interface rendering control module sending the placeholder area position information to the video control module, the JS interface rendering control module sends the placeholder area position information to the video control module.

[0298] It should be noted that step 1210 is consistent with step 1206. For a specific description, please refer to step 1206, which will not be repeated here.

[0299] Step 1211, the note editing module updates EditText.

[0300] Optionally, based on the second position information of the first placeholder area, the note editing module updates the EditText.

[0301] Since the corresponding EditText has been created before, it is only necessary to update the parameters in the view according to the new information. For a specific description, please refer to step 1207, which will not be repeated here.

[0302] refer to Fig.13 In 13D, based on the update of EditText by the note editing module, the position of EditText8042 in Webview8041 changes.

[0303] Step 1212: In response to the user clicking to close the video recording function, the recording is ended.

[0304] Step 1213, note editing module removeview(EditText).

[0305] Step 1214: The note editing module sends relevant information about the content in EditText to the video control module.

[0306] For a specific description, please refer to step 1109, which will not be repeated here.

[0307] Step 12141, the note editing module sends relevant information about the content in EditText to the JS interface rendering control module.

[0308] For the specific description, please refer to step 11091, which will not be repeated here.

[0309] Step 12142, the JS interface rendering control module sends relevant information about the content in EditText to the video control module.

[0310] For a specific description, please refer to step 11092, which will not be repeated here.

[0311] Step 1215: The video control module updates the HTML area.

[0312] For a specific description, please refer to step 1110, which will not be repeated here.

[0313] Step 12151, the video control module performs DOM rendering.

[0314] Step 12152: The video control module removes the speech-to-text placeholder area.

[0315] After the electronic device detects that the video recording function is turned on, this solution creates an input box view for displaying the contents of the speech-to-text area and records the placeholder information of the speech-to-text area; when the information in the note body editing area changes, the position of the placeholder is adjusted; after the video recording function is turned off, the content in the input box view is moved to the web page view for displaying the contents of the note body editing area.

[0316] The present application also provides a chip system, such as Fig.14 As shown, the chip system 1400 includes at least one processor 1401 and at least one interface circuit 1402. The processor 1401 and the interface circuit 1402 can be interconnected via lines. For example, the interface circuit 1402 can be used to receive signals from other devices (such as a memory of an electronic device). For another example, the interface circuit 1402 can be used to send signals to other devices (such as processor 1401). Exemplarily, the interface circuit 1402 can read instructions stored in the memory and send the instructions to the processor 1401. When the instructions are executed by the processor 1401, the electronic device can execute the various steps in the above embodiments. Of course, the chip system can also include other discrete devices, which are not specifically limited in the embodiments of the present application.

[0317] An embodiment of the present application also provides a computer storage medium, which includes a computer program. When the computer program runs on the above-mentioned electronic device, the electronic device executes each function or step executed by the mobile phone in the above-mentioned method embodiment.

[0318] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute each function or step executed by the mobile phone in the above method embodiment.

[0319] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0320] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0321] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0322] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0323] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

Claims

1. A video processing method, characterized in that: Applied to an electronic device, the electronic device includes a first application, and the method includes: Displaying a first interface, wherein the first interface includes a first display area, the first interface is an interface of the first application, and a first cursor is displayed in the first display area; In response to a first operation of the user, the first application starts recording a video; After the first application starts recording the video, text content is displayed in the second display area, the text content is converted from the voice in the recorded video, and a second cursor is displayed in the second display area, and the second cursor is located at the end of the text content; In response to a second operation by the user, the first cursor moves down by one line, and the second display area moves down by one line.

2. The method according to claim 1, characterized in that The method further comprises: After the first operation, creating a first view; The position of the second display area is moved down by one line, comprising: The position of the first view is shifted down by one row.

3. The method according to claim 2, characterized in that The method further comprises: In response to a second operation of the user, the first application ends recording the video; After the second operation, removing the first view; The text content is displayed in the second display area.

4. The method according to claim 2 or 3, characterized in that: The method further comprises: Creating a second view, where the second view corresponds to the first display area; During the video recording process, the text content is rendered to a display area corresponding to the first view; After the first view is removed, the text content is rendered to a display area corresponding to the second view.

5. The method according to claim 4, characterized in that The method further comprises: The first view is EditText, and the second view is Webview.

6. The method according to claim 5, characterized in that The method further comprises: The first application is a note application, and the first application includes a note editing module, a JS interface rendering control module, and a video control module; The first display area is a note text editing area, and the first display area includes a note text editing area, a video note card area, a video recording area, and a speech-to-text area.

7. The method according to claim 6, characterized in that In response to the first operation of the user, the first application starts recording the video, further comprising: The note editing module sends the relevant information of the video recording to the JS interface rendering control module by calling the first insertion function. The JS interface rendering control module encapsulates the relevant information of the video recording, and after completing the encapsulation, sends the encapsulated relevant information of the video recording to the video control module by calling the second insertion function.

8. The method according to claim 7, characterized in that Before creating the first view, the method further includes: The video control module renders the HTML area, including: The video control module performs DOM rendering and first placeholder rendering; The JS interface rendering control module sends the first position information of the first placeholder area to the video control module, and the video control module sends the first position information of the first placeholder area to the note editing module.

9. The method according to claim 8, characterized in that The step of creating the first view includes: The note editing module creates the EditText and calculates the attribute parameters of the window of the first placeholder area; The note editing module creates the EditText according to the attribute parameters of the window of the first placeholder area through a first creation function.

10. The method according to claim 9, characterized in that The removing the first view comprises: Based on the note editing module, the EditText is removed through the first removal function, and the first view is removed.

11. The method according to claim 10, characterized in that After removing the first view, the text content is rendered to a display area corresponding to the second view, including: The note editing module sends the relevant information of the content in the EditText to the video control module; The video control module updates the html area.

12. The method according to claim 11, characterized in that The method further comprises: After the video control module performs DOM rendering and the first placeholder rendering, the video control module registers a first listener.

13. The method according to claim 12, characterized in that In response to the second operation of the user, the first cursor moves down by one line, and the second display area moves down by one line, comprising: Based on the first listener, the video control module detects that the content of the note body editing area changes; The JS interface rendering control module sends the second position information of the first placeholder area to the video control module, and the video control module sends the second position information to the note editing module; Based on the second location information, the note editing module updates the EditText.

14. An electronic device, characterized in that: The electronic device comprises a memory and one or more processors, the memory being used to store a computer program, and the processor being used to execute the computer program in the memory, so that the electronic device implements the method according to any one of claims 1 to 13.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by one or more processors, the method according to any one of claims 1 to 13 is implemented.

Images