Multimedia data processing method and device and related product

Abstract

The embodiment of the invention provides a multimedia data processing method and device and a related product, and the method comprises the steps: responding to a first triggering operation of a user for first multimedia data, and determining a first target multimedia segment in the first multimedia data; generating first note data of the first multimedia data based on the content of the first target multimedia segment; the first note data is used for representing first information of the first target multimedia fragment; and performing associated display on the first target multimedia fragment and the first note data.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a multimedia data processing method, device and related products. Background Art

[0002] Currently, users can browse multimedia data, such as watching video data or listening to audio data, through various electronic devices such as mobile phones or computers. Taking the browsing of video data as an example, when browsing video data, if the user needs to study and record the interesting segments in the video data, the user needs to manually capture the video segment and manually generate note data, which has the defects of cumbersome operation steps and low operation efficiency, thereby reducing the browsing efficiency of video data and the convenience of processing video data. Summary of the invention

[0003] The embodiments of the present disclosure provide a multimedia data processing method, device and related products, which can obtain multimedia clips of interest to the user and automatically generate associated note data when the user browses the multimedia data, thereby simplifying user operations and improving the browsing efficiency of multimedia data and the convenience of processing multimedia data.

[0004] In a first aspect, an embodiment of the present disclosure provides a multimedia data processing method, including:

[0005] In response to a first trigger operation of a user on first multimedia data, determining a first target multimedia segment in the first multimedia data;

[0006] Based on the content of the first target multimedia segment, generating first note data of the first multimedia data; the first note data is used to represent first information of the first target multimedia segment;

[0007] The first target multimedia segment and the first note data are associated and displayed.

[0008] In a second aspect, an embodiment of the present disclosure provides a multimedia data processing device, including:

[0009] a determining unit, configured to determine a first target multimedia segment in the first multimedia data in response to a first trigger operation of a user on the first multimedia data;

[0010] A generating unit, configured to generate first note data of the first multimedia data based on the content of the first target multimedia segment; the first note data is used to represent first information of the first target multimedia segment;

[0011] A display unit is used to display the first target multimedia segment and the first note data in association with each other.

[0012] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: a processor; and a memory configured to store computer-executable instructions, wherein the computer-executable instructions, when executed, enable the processor to implement the steps of the method described in the first aspect above.

[0013] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and the computer-executable instructions implement the steps of the method described in the first aspect when executed by a processor.

[0014] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0015] In one or more embodiments of the present disclosure, first, in response to a user's first trigger operation on first multimedia data, a first target multimedia segment is determined in the first multimedia data, and then, based on the content of the first target multimedia segment, first note data of the first multimedia data is generated, and the first note data is used to characterize the first information of the first target multimedia segment. Finally, the first target multimedia segment and the first note data are associated and displayed. It can be seen that through this embodiment, when a user browses multimedia data, the first target multimedia segment of interest to the user can be obtained and the associated first note data can be automatically generated, and the first target multimedia segment and the first note data can be associated and displayed, thereby simplifying user operations and improving the browsing efficiency of multimedia data and the convenience of processing multimedia data. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate one or more embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative labor.

[0017] Figure 1 A flowchart of a multimedia data processing method provided by an embodiment of the present disclosure;

[0018] Figure 2a A schematic diagram of browsing first multimedia data through a browser provided by an embodiment of the present disclosure;

[0019] Figure 2b A schematic diagram of recording notes on first multimedia data provided by an embodiment of the present disclosure;

[0020] Figure 2c A schematic diagram of displaying a first target multimedia segment and first note data in association with each other provided by an embodiment of the present disclosure;

[0021] Figure 3a A schematic diagram of a user editing first note data provided by an embodiment of the present disclosure;

[0022] Figure 3b A schematic diagram of displaying a first target multimedia segment and edited first note data in association with each other provided by another embodiment of the present disclosure;

[0023] Figure 4a A schematic diagram of a user editing first note data provided by another embodiment of the present disclosure;

[0024] Figure 4b A schematic diagram of associating and displaying a first target multimedia segment and edited first note data provided by another embodiment of the present disclosure;

[0025] Figure 5a A schematic diagram of generating representation data of a first target multimedia segment provided by an embodiment of the present disclosure;

[0026] Figure 5b A schematic diagram of representation data of a first target multimedia segment provided by an embodiment of the present disclosure;

[0027] Figure 6a A schematic diagram of sharing a first note card provided by an embodiment of the present disclosure;

[0028] Figure 6b A schematic diagram of a first sharing card provided in an embodiment of the present disclosure;

[0029] Figure 7a A schematic diagram of downloading a first note card provided by an embodiment of the present disclosure;

[0030] Figure 7b A schematic diagram of a first card image provided by an embodiment of the present disclosure;

[0031] Figure 8 A schematic diagram of the structure of a multimedia data processing device provided by an embodiment of the present disclosure;

[0032] Fig. 9 A schematic diagram of the structure of an electronic device provided in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of the present disclosure, the technical solutions in one or more embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in one or more embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on one or more embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the protection scope of the present disclosure.

[0034] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0035] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0036] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0037] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0038] The disclosed embodiment provides a multimedia data processing method, which can obtain multimedia segments of interest to the user and automatically generate associated note data when the user browses multimedia data, simplifying user operations, improving the browsing efficiency of multimedia data and the convenience of processing multimedia data. The multimedia data processing method can be applied to and executed by a terminal device, which includes but is not limited to electronic devices such as mobile phones, computers, tablet computers, laptops, car computers, wearable devices, etc.

[0039] Figure 1 A flowchart of a multimedia data processing method provided by an embodiment of the present disclosure is shown in FIG. Figure 1 As shown, the process includes:

[0040] Step S102, in response to a first trigger operation of a user on first multimedia data, determining a first target multimedia segment in the first multimedia data;

[0041] Step S104, generating first note data of the first multimedia data based on the content of the first target multimedia segment; the first note data is used to represent first information of the first target multimedia segment;

[0042] Step S106: associate the first target multimedia segment with the first note data and display them.

[0043] In this embodiment, first, in response to the user's first trigger operation on the first multimedia data, the first target multimedia segment is determined in the first multimedia data, and then, based on the content of the first target multimedia segment, the first note data of the first multimedia data is generated, and the first note data is used to characterize the first information of the first target multimedia segment. Finally, the first target multimedia segment and the first note data are associated and displayed. It can be seen that through this embodiment, when the user browses the multimedia data, the first target multimedia segment of interest to the user can be obtained and the associated first note data can be automatically generated, and the first target multimedia segment and the first note data can be associated and displayed, which simplifies the user operation and improves the browsing efficiency of multimedia data and the convenience of processing multimedia data.

[0044] Below Figure 1 The method flow is described in detail.

[0045] In the above step S102, the terminal device determines a first target multimedia segment in the first multimedia data in response to a first trigger operation of the user on the first multimedia data.

[0046] In this embodiment, the user can browse the first multimedia data through the browser. The first multimedia data may include video data and audio data. Therefore, in this embodiment, the user can browse the video data through the browser, or listen to the audio data through the browser. When browsing the first multimedia data, the user can perform a first trigger operation on the first multimedia data. In one example, the first trigger operation includes an operation of taking notes on the first multimedia data. Furthermore, the terminal device determines a first target multimedia segment in the first multimedia data in response to the first trigger operation.

[0047] In one embodiment, the terminal device determines a first target multimedia segment in the first multimedia data in response to a first trigger operation of a user on the first multimedia data, including:

[0048] Determine and obtain a first timestamp at which the first multimedia data is played when the first trigger operation is performed;

[0049] Based on the first timestamp and the multimedia segment selection rule, a first target multimedia segment is determined in the first multimedia data.

[0050] In this embodiment, the terminal device determines the first timestamp to which the first multimedia data is played when the user's first trigger operation is obtained. For example, if it is determined that the first multimedia data is played to 5 minutes and 30 seconds when the user's first trigger operation is obtained, the first timestamp is 5 minutes and 30 seconds.

[0051] Next, the terminal device determines the first target multimedia segment in the first multimedia data based on the first timestamp and the multimedia segment selection rule. In one example, the multimedia segment selection rule may include selecting a combination of multimedia segments of first durations before and after the first timestamp as the first target multimedia segment. Then, based on the first timestamp and the multimedia segment selection rule, a combination of multimedia segments of first durations before and after the first timestamp may be selected as the first target multimedia segment. For example, if the first timestamp is 5 minutes and 30 seconds and the first duration is 5 seconds, a multimedia segment with a playback timestamp between 5 minutes and 25 seconds and 5 minutes and 35 seconds is selected as the first target multimedia segment.

[0052] It can be seen that through this embodiment, the first timestamp to which the first multimedia data is played when the first trigger operation is obtained can be determined, and based on the first timestamp and the multimedia segment selection rule, the first target multimedia segment is determined in the first multimedia data, thereby achieving the effect of accurately determining the first target multimedia segment according to the user's first trigger operation.

[0053] In one embodiment, the user can also manually adjust the time range of the first target multimedia segment. For example, in response to the user's first trigger operation on the first multimedia data, the terminal device displays the first timestamp to which the first multimedia data is played when the first trigger operation is obtained, and displays the start timestamp and end timestamp of the first target multimedia segment, and displays controls for adjusting the start timestamp of the first target multimedia segment, such as "+" and "-" controls, and displays controls for adjusting the end timestamp of the first target multimedia segment, such as "+" and "-" controls. By triggering the above controls, the user can adjust the start timestamp and end timestamp of the first target multimedia segment, thereby adjusting the time range of the first target multimedia segment.

[0054] In the above step S104, the terminal device generates first note data of the first multimedia data based on the content of the first target multimedia segment, and the first note data is used to represent the first information of the first target multimedia segment.

[0055] In this embodiment, the first note data is used to characterize the first information of the first target multimedia segment, and the first information may be the core information or key information of the first target multimedia segment. In one example, the terminal device may use each subtitle in the first target multimedia segment as the first note data of the first multimedia data. When the first target multimedia segment is video data, the subtitles may be subtitles displayed in the video data, and when the first target multimedia segment is audio data, the subtitles may be subtitles displayed in the audio data such as lyrics. In another example, the terminal device may summarize the content of the first target multimedia segment and use the summarized content as the first note data of the first multimedia data.

[0056] In the above step S106, the terminal device associates and displays the first target multimedia segment and the first note data.

[0057] In this embodiment, after determining the first target multimedia segment and the first note data, the terminal device may associate and display the first target multimedia segment and the first note data through a first note card, where a screenshot of the first target multimedia segment and the first note data are displayed.

[0058] Figure 2a A schematic diagram of browsing first multimedia data through a browser provided by an embodiment of the present disclosure, such as Figure 2a As shown, taking the first multimedia data as video data as an example, the user can browse the first multimedia data in the browser. The browser is pre-installed with a plug-in, which can be created based on a generative model such as LLM (Language Model with Lattice, large language model), and the plug-in provides a function of summarizing the content of the multimedia data, such as Figure 2a In the "Summary" control, if the user triggers the "Summary" control, such as Figure 2a As shown, the browser can display the content summary data obtained by summarizing the content of the first multimedia data by the generative model. In addition, the plug-in also provides a function of recording notes on the content of the multimedia data, such as Figure 2a In the "Note" control, the user can trigger the "Note" control, and the trigger operation can be the first trigger operation mentioned above, so that the terminal device obtains the user's first trigger operation for the first multimedia data. It is worth noting that the method flow in this embodiment can be executed by the above plug-in.

[0059] Figure 2b A schematic diagram of recording notes on first multimedia data provided by an embodiment of the present disclosure, such as Figure 2a and Figure 2bAs shown, when the first multimedia data is played in the browser, the user can trigger the "note" control provided by the above-mentioned plug-in, and the trigger operation can be the above-mentioned first trigger operation, so that the terminal device responds to the user's first trigger operation on the first multimedia data, determines the first target multimedia segment in the first multimedia data, and generates the first note data of the first multimedia data based on the content of the first target multimedia segment, and displays the note recording interface, and displays the first note data in the note recording interface.

[0060] like Figure 2b As shown, the terminal device can determine the first timestamp to which the first multimedia data is played when the first trigger operation is obtained, determine the first target multimedia segment in the first multimedia data based on the first timestamp and the multimedia segment selection rule, and display each subtitle in the first target multimedia segment as the first note data of the first multimedia data. Figure 2b The following three subtitles are used as an example: "Attention should be paid here", "Drawing details are particularly important", and "Also pay attention to the matching of drawing tools". Figure 2b As shown, the note recording interface also displays a note input area for the user to input notes, and the user can input customized notes through the note input area.

[0061] like Figure 2b As shown, the note recording interface also displays a "Save" button and a "Cancel" button. The user can trigger the "Save" button, and the trigger operation can be a confirmation operation for the first note data. In response to the confirmation operation, the terminal device displays the first target multimedia clip and the first note data in association.

[0062] Figure 2c A schematic diagram of an embodiment of the present disclosure providing a first target multimedia segment and a first note data for association display, such as Figure 2c As shown, the terminal device can associate and display the first target multimedia segment and the first note data through the first note card, and the first note card displays a screenshot of the first target multimedia segment and the first note data, wherein the first note data includes each subtitle in the first target multimedia segment, and the screenshot of the first target multimedia segment can be a screenshot of the first multimedia data played when the terminal device obtains the user's first trigger operation.

[0063] like Figure 2c As shown, the terminal device may also display a note card list, which may display each note card of the first multimedia data in chronological order, so when the first note card is the latest generated note card, the first note card may be located at the top of the list.

[0064] It can be seen that through this embodiment, when the user browses multimedia data, the first target multimedia segment that the user is interested in can be obtained and the associated first note data can be automatically generated, and the first target multimedia segment and the first note data can be associated and displayed, thereby simplifying user operations and improving the browsing efficiency of multimedia data and the convenience of processing multimedia data.

[0065] In one embodiment, the terminal device generates first note data of the first multimedia data based on the content of the first target multimedia segment, including:

[0066] For each object in the first target multimedia segment, generating sub-note data of the object according to the relevant content of the object in the first target multimedia segment;

[0067] Based on the sub-note data of each object, first note data of the first multimedia data is generated.

[0068] In this embodiment, there may be multiple objects in the first target multimedia segment. For example, when the first target multimedia segment is video data, there may be multiple characters in the video data, and each of the characters is an object. When the first target multimedia segment is audio data, there may be multiple timbres in the audio data, and each of the timbres is an object.

[0069] In this embodiment, for each object in the first target multimedia segment, the object has related content in the first target multimedia segment. For example, when the first target multimedia segment is video data, the object can be a character, and the related content of the object in the first target multimedia segment can be the plot performed by the object in the first target multimedia segment; when the first target multimedia segment is audio data, the object can be a timbre, and the related content of the object in the first target multimedia segment can be the speech made or the song sung by the timbre in the first target multimedia segment. In this embodiment, for each object, the related content of the object in the first target multimedia segment can be summarized to obtain the sub-note data of the object.

[0070] In this embodiment, the first note data of the first multimedia data is generated based on the sub-note data of each object. For example, the sub-note data of each object is combined to obtain the first note data of the first multimedia data.

[0071] It can be seen that through this embodiment, when multiple objects are included in the first target multimedia segment, sub-note data of the object can be generated according to the relevant content of the object in the first target multimedia segment, and the first note data of the first multimedia data can be generated based on the sub-note data of each object. In this way, the first note data can reflect the core information of the first target multimedia segment from the dimension of each object in the first target multimedia segment, thereby improving the accuracy of the first note data.

[0072] In one embodiment, for each object in the first target multimedia segment, the terminal device generates sub-note data of the object according to the relevant content of the object in the first target multimedia segment, including:

[0073] Summarizing the relevant content of the object in the first target multimedia segment through a generative model to obtain content summary data of the object;

[0074] Obtaining extended content of the object associated with content summary data of the object;

[0075] Generate sub-note data of the object based on the content summary data of the object and the extended content of the object.

[0076] In this embodiment, as mentioned above, a plug-in is installed in the browser, and the plug-in can be created based on a generative model. In this embodiment, first, the plug-in, that is, the generative model, is used to summarize the relevant content of the object in the first target multimedia segment to obtain the content summary data of the object. In this embodiment, when the relevant content of the object in the first target multimedia segment is summarized by the generative model to obtain the content summary data, the content summary data may include at least one of text and image. For example, the content summary data of the object is expressed in at least one form of text and image.

[0077] In one example, when the first target multimedia segment is video data, the object is a person, and the above-mentioned content summary data can be summary data of the plot performed by the object; when the first target multimedia segment is audio data, the object is a timbre, and the above-mentioned content summary data can be summary data of the remarks made by the person corresponding to the object.

[0078] Next, the extended content of the object associated with the content summary data of the object is obtained. For example, a keyword in the content summary data of the object is obtained, the keyword is searched through a search engine, and the search result is used as the extended content of the object associated with the content summary data of the object. Alternatively, a keyword in the content summary data of the object is obtained, the keyword is input into a generative model, and the generative model outputs the extended content of the object associated with the content summary data of the object. Alternatively, the content summary data of the object is input into a generative model, and the generative model outputs the extended content of the object associated with the content summary data of the object. In this embodiment, the extended content of the object associated with the content summary data of the object can be exemplified by the background knowledge of the object related to the content summary data of the object. In this embodiment, the content related to the object that can be associated with the content summary data of the object can be classified as the extended content of the object associated with the content summary data of the object. The extended content may include at least one of text and image. For example, the extended content is expressed in at least one form of text and image.

[0079] In one example, when the first target multimedia clip is video data, the object is a person, and the above-mentioned extended content can be the introduction data of the film and television drama in which the object has participated. When the first target multimedia clip is audio data, the object is a tone, and the above-mentioned content summary data can be the introduction data of the music albums released by the singer corresponding to the object.

[0080] Finally, the sub-note data of the object is generated according to the content summary data of the object and the extended content of the object. For example, the content summary data of the object and the extended content of the object are combined into the sub-note data of the object. The sub-note data of the object may include at least one of text and image. For example, the sub-note data of the object is expressed in at least one form of text and image.

[0081] It can be seen that through this embodiment, the relevant content of the object in the first target multimedia segment can be summarized through a generative model to obtain the content summary data of the object, and based on the content summary data of the object, the extended content of the object can be obtained, and according to the content summary data of the object and the extended content of the object, the sub-note data of the object can be generated, thereby comprehensively generating the sub-note data of the object and improving the comprehensiveness of the sub-note data.

[0082] In one embodiment, the terminal device generates first note data of the first multimedia data based on the sub-note data of each object, including:

[0083] Acquire a note template corresponding to the content type of the first multimedia data;

[0084] The note template is edited using the sub-note data of each object to generate first note data of the first multimedia data.

[0085] In this embodiment, the content type of the first multimedia data is determined. The content type can indicate whether the first multimedia data is video data or audio data, and the content of the multimedia data. Examples of the content type include video data - film and television drama, video data - game recording, video data - conference recording, audio data - song, audio data - conference recording, etc.

[0086] Next, a note template corresponding to the content type of the first multimedia data is obtained. In this embodiment, various note templates are preset, and different note templates correspond to different content types. The note template corresponding to the content type of the first multimedia data is obtained.

[0087] Finally, the note template is edited using the sub-note data of each object to generate the first note data of the first multimedia data. In this embodiment, the sub-note data of each object can be filled into the note template, and the filled note template is the first note data of the first multimedia data. For example, the sub-note data of each object includes at least one of text and / or image, and the note template specifies the first filling position of the text in the sub-note data of each object, and specifies the second filling position of the image in the sub-note data of each object. Based on the first filling position and the second filling position, the sub-note data of each object are filled into the note template, and the filled note template is the first note data of the first multimedia data.

[0088] It can be seen that through this embodiment, it is possible to obtain a note template corresponding to the content type of the first multimedia data, edit the note template using the sub-note data of each object, generate the first note data of the first multimedia data, generate structured first note data based on the note template, and improve the standardization and readability of the first note data.

[0089] In one embodiment, the terminal device generates first note data of the first multimedia data based on the content of the first target multimedia segment, including:

[0090] Obtaining each subtitle of the first target multimedia segment;

[0091] Based on the subtitle selection rule, a first subtitle is selected from each subtitle of the first target multimedia segment;

[0092] Based on the first subtitle, first note data of the first multimedia data is generated.

[0093] In this embodiment, when the first target multimedia segment has subtitles, for example, when the first target multimedia segment is video data and has subtitles, or when the first target multimedia segment is audio data and has subtitles, each subtitle of the first target multimedia segment is obtained, and based on a subtitle selection rule, a first subtitle is selected from each subtitle of the first target multimedia segment, and the number of first subtitles is at least one. The first subtitle may be all subtitles in each subtitle, or may be part of the subtitles in each subtitle. For example, the first subtitle is the subtitle with actual content remaining after removing subtitles without actual meaning in each subtitle. Then, based on each first subtitle, first note data of the first multimedia data is generated, for example, each first subtitle is used as the first note data of the first multimedia data.

[0094] It can be seen that through this embodiment, when the first target multimedia segment has subtitles, each subtitle of the first target multimedia segment can be obtained, and based on the subtitle selection rule, at least one first subtitle is selected from the subtitles of the first target multimedia segment, and based on the first subtitles, the first note data of the first multimedia data is generated, thereby efficiently and quickly generating the first note data based on the subtitles of the first target multimedia segment.

[0095] In one embodiment, before the terminal device associates and displays the first target multimedia segment with the first note data, the terminal device further includes:

[0096] In response to the user's editing operation on the first note data, displaying the edited first note data;

[0097] The first target multimedia segment and the first note data are displayed in association with each other, including:

[0098] The first target multimedia segment and the edited first note data are associated and displayed.

[0099] In this embodiment, after the terminal device generates the first note data, the terminal device can also display the first note data, for example, by Figure 2b The first note data is displayed in the note recording interface in . Then, the user can also edit the first note data, for example, modify, delete or add the first note data, so that the terminal device displays the edited first note data in response to the user's editing operation on the first note data, and associates and displays the first target multimedia segment with the edited first note data. For example, after the user confirms the edited first note data, the first target multimedia segment and the edited first note data are associated and displayed.

[0100] It can be seen that through this embodiment, the user can also edit the first note data, so that the user can record the required note data according to his own needs.

[0101] In one embodiment, in response to the user's editing operation on the first note data, the terminal device displays the edited first note data, including:

[0102] If the first note data includes a first subtitle in the first target multimedia segment, in response to a user's selection operation on the first subtitle, the first subtitle selected by the user is used as the edited first note data, and the edited first note data is displayed;

[0103] and / or,

[0104] In response to the user's note input operation, the first note data and the user note input by the user are taken together as edited first note data, and the edited first note data is displayed.

[0105] In this embodiment, in one case, as described above, the first note data may include the first subtitles in the first target multimedia segment, the number of the first subtitles is at least one, and the first subtitles may be all subtitles in the subtitles of the first target multimedia segment, or may be part of the subtitles in the subtitles of the first target multimedia segment. In this case, when the terminal device displays each first subtitle, each first subtitle may be in a fully selected state or in a fully unselected state, and the user may select or deselect each first subtitle by triggering the first subtitle. Then, in response to the user's selection operation on the first subtitle, the terminal device marks the first subtitle selected by the user, for example, marks the first subtitle selected by the user with a "√" symbol, and uses the first subtitle selected by the user as the edited first note data, and displays the edited first note data. In this case, the terminal device may also associate and display the first target multimedia segment and the edited first note data in the form of a first note card.

[0106] Figure 3a A schematic diagram of a user editing first note data provided by an embodiment of the present disclosure, such as Figure 3a As shown, the terminal device responds to the user's first trigger operation on the first multimedia data, determines the first target multimedia segment in the first multimedia data, uses each subtitle in the first target multimedia segment as the first subtitle, uses the first subtitle as the first note data of the first multimedia data, and displays the note recording interface, and displays the first subtitle in the note recording interface. Figure 3a The first subtitle has two subtitles, which are "Attention should be paid here" and "Details of the drawing are particularly important". Figure 3a As shown, by default, all first subtitles may be in a fully selected state, and the user may select or deselect each first subtitle by triggering each first subtitle.

[0107] like Figure 3a As shown, the user selects a first subtitle from among the first subtitles by clicking on the first subtitle, and the terminal device responds to the selection operation by marking the selected first subtitle and marking the first subtitle selected by the user with a "√" symbol, thereby achieving the effect of displaying the edited first note data. Then, when the user triggers Figure 3a After clicking the "Save" button in the dialog box, the terminal device displays the first target multimedia clip and the edited first note data in association with each other through the first note card in response to the user's confirmation operation on the edited first note data.

[0108] Figure 3b A schematic diagram of another embodiment of the present disclosure providing a first target multimedia segment and an edited first note data in association with each other is shown as follows: Figure 3b As shown, the first target multimedia segment and the edited first note data are associated and displayed through the first note card. In the first note card, the first note data edited by the user can be displayed, for example, "Drawing details are particularly important" is displayed, and a screenshot of the first target multimedia segment is displayed. The screenshot can be a screenshot of the first multimedia data played when the terminal device obtains the user's first trigger operation.

[0109] In this embodiment, in another case, the terminal device also provides a note input area for the user to input notes, and the user can perform a note input operation based on the note input area, so that the terminal device responds to the user's note input operation, takes the first note data and the user notes input by the user as the edited first note data, and displays the edited first note data. The user notes are in the form of text or text plus pictures. In this case, the terminal device can also associate and display the first target multimedia segment and the edited first note data in the form of a first note card.

[0110] Of course, it can be understood that when the first note data includes the first subtitles in the first target multimedia segment, and the terminal device also provides a note input area, the user can select the required subtitles from the first subtitles, and can also input user notes based on the note input area. The terminal device responds to the user's selection operation on the first subtitles and the user's note input operation, and uses the first subtitles selected by the user and the user notes input by the user as the edited first note data, and displays the edited first note data. In this case, the terminal device can also display the first target multimedia segment and the edited first note data in an associated manner in the form of a first note card.

[0111] Figure 4a A schematic diagram of a user editing first note data provided by another embodiment of the present disclosure, such as Figure 4aAs shown, in response to the user's first trigger operation on the first multimedia data, the terminal device determines the first target multimedia segment in the first multimedia data, uses each subtitle in the first target multimedia segment as the first subtitle, uses the first subtitle as the first note data of the first multimedia data, and displays the note recording interface, and displays the first note data in the note recording interface. Figure 4a As shown, a note input area is also displayed in the note recording interface for the user to input notes. The user can input customized user notes through the note input area. Thus, the terminal device responds to the note input operation performed by the user based on the note input area, and uses the first note data and the user notes input by the user as the edited first note data, and displays the edited first note data. Figure 4a In the example, the user can also select all or part of the first subtitles. Here, the example of the user selecting all the first subtitles is used for illustration.

[0112] Next, the user triggers Figure 4a , the terminal device displays the first note card in response to the user's confirmation operation on the selected first subtitle and the input user notes. Figure 4b A schematic diagram of associating and displaying a first target multimedia segment and edited first note data is provided as another embodiment of the present disclosure, such as Figure 4b As shown, the first target multimedia segment and the edited first note data are associated and displayed through the first note card, and the first note card displays the first note data of the first multimedia data, and displays a screenshot of the first target multimedia segment, wherein the first note data includes a first subtitle selected and confirmed by the user, and also includes a user note input by the user, and the screenshot can be a screenshot of the first multimedia data played when the terminal device obtains the user's first trigger operation.

[0113] It can be seen that through this embodiment, the user can also edit the first note data by inputting the required user notes and / or selecting each first subtitle, thereby improving the convenience of the user in editing the first note data, increasing the diversity of the user's note-taking methods, and improving the user experience.

[0114] To summarize, when the first note data includes the first subtitle in the first target multimedia segment, this embodiment includes at least three situations:

[0115] 1. The user can directly confirm each first subtitle to generate a first note card. The first note card is used to display the first note data. The first note data includes all the first subtitles. Figure 2cAs shown, in this case, the first note card also displays the timestamp and screenshot of the first target multimedia segment. The timestamp may be the first timestamp when the first multimedia data is played when the terminal device obtains the first trigger operation of the user, for example, 5 minutes and 30 seconds. The screenshot may be a screenshot of the first target multimedia segment at the first timestamp.

[0116] 2. The user selects the desired first subtitle from each first subtitle and confirms it, thereby generating a first note card. The first note card is used to display the first note data. The first note data includes the first subtitle selected by the user. Figure 3b As shown, in this case, the first note card also displays the timestamp and screenshot of the first target multimedia segment. The timestamp may be the first timestamp when the first multimedia data is played when the terminal device obtains the first trigger operation of the user, for example, 5 minutes and 30 seconds. The screenshot may be a screenshot of the first target multimedia segment at the first timestamp.

[0117] 3. The user selects the desired first subtitle from each first subtitle, and inputs the user's notes based on the note input area, thereby generating a first note card, which is used to display the first note data. The first note data includes the first subtitle selected by the user and the user's notes. Figure 4b As shown, in this case, the first note card also displays the timestamp and screenshot of the first target multimedia segment. The timestamp can be the first timestamp when the first multimedia data is played when the terminal device obtains the user's first trigger operation, for example, 5 minutes and 30 seconds, and the screenshot can be a screenshot of the first target multimedia segment at the first timestamp.

[0118] Through the above three situations, the user's note editing needs in different scenarios can be met, and the browsing efficiency of multimedia data and the convenience of multimedia data processing can be improved. In addition, in this embodiment, it can be set in the first note card that the first subtitle is displayed above the user's note, thereby highlighting the first subtitle.

[0119] In one embodiment, the above method flow further includes:

[0120] In response to a second trigger operation of the user on the first target multimedia segment, extracting a plurality of target frames from the first target multimedia segment;

[0121] Based on the target frames, representation data of the first target multimedia segment is obtained by combination, and the representation data is displayed.

[0122] As mentioned above, the terminal device can associate the first note data and the first target multimedia segment through the first note card for display, and the first note card displays a screenshot of the first target multimedia segment and the first note data. The screenshot of the first target multimedia segment can be a screenshot of the first multimedia data played when the terminal device obtains the user's first trigger operation.

[0123] Based on this, in this embodiment, the user can also perform a second trigger operation on the first target multimedia segment, for example, perform a trigger operation on the first note card, or perform a trigger operation on a screenshot of the first target multimedia segment in the first note card. In response to the second trigger operation, the terminal device extracts multiple target frames from the first target multimedia segment, and can use all frames of the first target multimedia segment as target frames, or intermittently extract some frames from all frames of the first target multimedia segment as target frames. Based on the target frames, representation data of the first target multimedia segment is obtained by combining them, and the representation data can be GIF (Graphics Interchange Format) data, and the representation data is displayed.

[0124] Figure 5a A schematic diagram of generating representation data of a first target multimedia segment provided by an embodiment of the present disclosure, such as Figure 5a As shown, when the user moves the cursor to the first note card, the word "record" will be displayed on the first note card, and the user can trigger the word "record". In response to the recording operation of the first target multimedia segment triggered by the user based on the first note card, the terminal device intermittently extracts multiple target frames from the first target multimedia segment. Then, the terminal device groups each target frame into representation data of the first target multimedia segment, and displays the representation data. The representation data can be animated GIF data.

[0125] Figure 5b A schematic diagram of representation data of a first target multimedia segment provided by an embodiment of the present disclosure is as follows: Figure 5b As shown, the representation data of the first target multimedia segment may be animated picture GIF data.

[0126] It can be seen that through this embodiment, the terminal device can also respond to the user's second trigger operation on the first target multimedia segment, extract multiple target frames in the first target multimedia segment, combine and obtain representation data of the first target multimedia segment based on each target frame, and display the representation data, thereby achieving the effect of recording the first target multimedia segment.

[0127] In one embodiment, the above method flow further includes:

[0128] In response to a user's downloading operation on the representation data of the first target multimedia segment, the representation data of the first target multimedia segment is downloaded.

[0129] In this embodiment, the user can download the representation data of the first target multimedia segment. In response to the user's download operation on the representation data of the first target multimedia segment, the representation data of the first target multimedia segment is downloaded, and the downloaded representation data can be stored locally.

[0130] It can be seen that through this embodiment, the user can also download the representation data of the first target multimedia segment, thereby improving the convenience of the user browsing the representation data.

[0131] In one embodiment, the first target multimedia clip and the first note data are displayed in association with each other through a first note card; the above method flow further includes:

[0132] Acquire multiple second note cards of the user; the second note cards are used to associate and display the second note data with the second target multimedia segment; the second target multimedia segment comes from the second multimedia data;

[0133] Clustering the plurality of second note cards of the user according to the second multimedia data to obtain at least one type of note card set;

[0134] Note summary data corresponding to each type of note card set is generated according to the second note data and / or the second target multimedia segment in each type of note card set.

[0135] As mentioned above, the first target multimedia segment and the first note data can be associated and displayed through the first note card. Based on this, in this embodiment, multiple second note cards of the user are obtained. The second note card may include the above-mentioned first note card or not. Each second note card may be generated based on the same multimedia data or based on different multimedia data. For each second note card, the second note card is generated in the same way as the first card. The second note card is used to associate and display the second note data with the second target multimedia segment, and the second target multimedia segment comes from the second multimedia data.

[0136] In this embodiment, based on the second multimedia data, multiple second note cards of the user are clustered to obtain at least one type of note card set. In one case, the second note cards generated based on the same second multimedia data can be clustered into one type to obtain a type of note card set. For example, if the second note cards A and B are both generated based on the second multimedia data C, the second note cards A and B are clustered into one type to obtain a type of note card set. In another case, the second note cards generated based on the second multimedia data of the same subject can be clustered into one type to obtain a type of note card set. For example, the second note card D is generated based on the second multimedia data E, and the second note card F is generated based on the second multimedia data G. The second multimedia data E and G are video data introducing the same or similar content. If they are both video data introducing the same event or the same type of event, the second note cards C and D are clustered into one type to obtain a type of note card set.

[0137] Next, for each type of note card set, the note summary data corresponding to the note card set is generated according to the second note data and / or the second target multimedia segment in the note card set. For example, the note summary data corresponding to the note card set is generated according to the second note data in the note card set, or the note summary data corresponding to the note card set is generated according to the second target multimedia segment in the note card set, or the note summary data corresponding to each type of note card set is generated according to the second note data and the second target multimedia segment in each type of note card set.

[0138] According to the second note data in the note card set of this type, the note summary data corresponding to the note card set of this type is generated, which can be exemplified as follows: the content of the second note data in the note card set of this type is summarized and generalized by the generative model, and the summarized data is used as the note summary data corresponding to the note card set of this type. The summarized data can be displayed in the form of a knowledge point list, a mind map, etc.

[0139] According to the second target multimedia segment in the note card set of this type, the note summary data corresponding to the note card set of this type is generated, which can be exemplified as follows: the content of the second target multimedia segment in the note card set of this type is summarized and generalized by the generative model, and the summarized data is used as the note summary data corresponding to the note card set of this type. The summarized data can be represented by animated images, short videos, texts, etc.

[0140] According to the second note data and the second target multimedia segment in each note card set, the note summary data corresponding to each note card set is generated. For example, the content of the second note data and the content of the second target multimedia segment in the note card set of this type is summarized and generalized by the generative model, and the summarized data is used as the note summary data corresponding to the note card set of this type. The summarized data may include text data and video data. The text data may be displayed in the form of text, knowledge point list, mind map, etc., and the video data may be displayed in the form of animated image, short video, etc.

[0141] It can be seen that through this embodiment, multiple second note cards of a user can be obtained, and the multiple second note cards can be clustered to obtain at least one type of note card set, and note summary data corresponding to each type of note card set can be generated, so that for the same user, the note cards of the user are classified and summary data of each type of card is generated, so as to achieve the effect of sorting out the various note cards of the same user and assisting the user to deeply understand the browsed multimedia data.

[0142] In this embodiment, the note summary data corresponding to the generated collection of various note cards can be input into the generative model as a model learning sample, for example, when training the generative model, the data can be input into the generative model, or when the user uses the generative model, the data can be input into the generative model as background knowledge of the generative model, so as to facilitate the generative model to learn the content of multimedia data introducing the same event or the same type of event, and improve the generative model's ability to answer questions about the event. Of course, similarly, the collection of various note cards can also be directly input into the generative model to achieve the above effect.

[0143] In one embodiment, the first target multimedia clip and the first note data are displayed in association with each other through a first note card; the above method flow further includes:

[0144] In response to a user triggering operation on a first note card, a first sharing card corresponding to the first note card is displayed; the first sharing card is used to share the first note data and the first target multimedia clip;

[0145] In response to a sharing operation of the user on the first sharing card, the first sharing card is shared to a target user.

[0146] In this embodiment, the user can share the first note card. For example, when the user moves the cursor to the first note card, the word "Share" will be displayed on the first note card, and the user can trigger the word "Share". Thus, in response to the user's triggering operation on the first note card, the terminal device displays the first sharing card corresponding to the first note card. The first sharing card can be in the form of a picture. The first sharing card is used to share the first note data and the first target multimedia clip. The first sharing card displays the screenshots of the first note data and the first target multimedia clip, and can also display the identification information of the plug-in installed in the browser, such as the name of the plug-in.

[0147] Next, the user can perform a sharing operation on the first sharing card, and the terminal device shares the first sharing card to the target user in response to the user's sharing operation on the first sharing card. For example, the user downloads the first sharing card as a picture to the local computer, and sends the downloaded first sharing card to the target user. The sharing operation includes a download operation and a send operation. The terminal device sends the downloaded first sharing card to the target user in response to the download operation and the send operation. For another example, the user copies the first sharing card in the same way as copying a picture, and sends the copied first sharing card to the target user. The sharing operation includes a copy operation and a send operation. The terminal device sends the copied first sharing card to the target user in response to the copy operation and the send operation.

[0148] Figure 6a A schematic diagram of sharing a first note card provided by an embodiment of the present disclosure, such as Figure 6a As shown, when the user moves the cursor to the first note card, the word "Share" will be displayed on the first note card, and the user can trigger the word "Share". In response to the user's triggering operation on the first note card, the terminal device displays the first sharing card corresponding to the first note card. Figure 6b A schematic diagram of a first sharing card provided in an embodiment of the present disclosure is shown as follows: Figure 6b As shown, the first sharing card can be in the form of a picture, and the first sharing card is used to share the first note data and the first target multimedia clip. The first sharing card displays the first note data and the first target multimedia clip screenshots, and can also display the identification information of the plug-in installed in the browser, such as the name of the plug-in. The screenshot can be a screenshot of the first multimedia data played when the terminal device obtains the user's first trigger operation.

[0149] like Figure 6bAs shown, the first sharing card has a download control and a copy control. The user can download or copy the first sharing card and send the downloaded or copied first sharing card to the target user. The terminal device then shares the first sharing card to the target user in response to the user's sharing operation on the first sharing card.

[0150] It can be seen that through this embodiment, the first note card can also be shared with other users in the form of a first sharing card, so as to facilitate sharing of the first note card among various users.

[0151] In one embodiment, the first sharing card carries first link information corresponding to the first target multimedia segment, and the first link information is used for the target user to browse the first target multimedia segment.

[0152] like Figure 6b As shown, in the first sharing card, in addition to displaying the first note data and the screenshot of the first target multimedia segment, the first link information corresponding to the first target multimedia segment is also displayed. The link information may be the link information corresponding to the first multimedia data. The first link information may be in the form of a QR code, and the target user may browse the first target multimedia segment through the first link information. For example, the target user scans the QR code using a terminal device, and the terminal device starts a browser after scanning and loads the first multimedia data in the browser, and directly locates to the starting position of the first target multimedia segment to play the first target multimedia segment. Of course, in other embodiments, after scanning the first link information, the terminal device may also start playing the first multimedia data from the starting position of the first multimedia data.

[0153] In this embodiment, by directly locating the starting position of the first target multimedia segment to play the first target multimedia segment, the target user can directly browse the content of the first target multimedia segment associated with the first note card, thereby improving the multimedia data browsing experience of the target user.

[0154] It can be seen that through this embodiment, by carrying the first link information corresponding to the first target multimedia clip in the first sharing card, it is possible to facilitate the target user to browse the first target multimedia clip based on the first link information, thereby improving the target user's browsing experience of the first target multimedia clip.

[0155] In one example, the user may also trigger the first note card, for example, trigger a screenshot of the first target multimedia segment in the first note card, and the terminal device may respond to the trigger operation by playing the first multimedia data in the browser, for example, playing the first target multimedia segment from the start timestamp of the first target multimedia segment in the browser, or playing the first multimedia data from the start position of the first multimedia data.

[0156] Furthermore, the above embodiment also includes:

[0157] In response to the download operation for the first note card, a first card image corresponding to the first note card is downloaded; the first card image is used to display the first note data and the first target multimedia segment.

[0158] In this embodiment, the user can download the first note card. For example, when the user moves the cursor to the first note card, the word "download" will be displayed on the first note card, and the user can trigger the word "download". Thus, in response to the download operation on the first note card, the terminal device downloads the first card image corresponding to the first note card. The first card image can be in the form of a picture. The first card image displays the first note data and the screenshot of the first target multimedia segment. It can also display the identification information of the plug-in installed in the browser, such as the name of the plug-in. The screenshot can be a screenshot of the first multimedia data played when the terminal device obtains the first trigger operation of the user.

[0159] Figure 7a A schematic diagram of downloading a first note card provided by an embodiment of the present disclosure is shown in FIG. Figure 7a As shown, when the user moves the cursor to the first note card, the word "Download" will be displayed on the first note card. The user can trigger the word "Download", and the terminal device downloads the first card image corresponding to the first note card in response to the user's download operation on the first note card. Figure 7b A schematic diagram of a first card image provided by an embodiment of the present disclosure, such as Figure 7b As shown, the first card image can be in the form of a picture, and the first note data and the screenshot of the first target multimedia clip are displayed on the first card image, and the identification information of the plug-in installed in the browser can also be displayed, such as the name of the plug-in. The screenshot can be a screenshot of the first multimedia data played when the terminal device obtains the user's first trigger operation.

[0160] It can be seen that, through this embodiment, it is also possible for the user to conveniently download the first note card in the form of the first card image to the local computer, thereby conveniently allowing the user to store the first note card.

[0161] In one example, the first card image may be the same as the data displayed on the first sharing card. The first card image and the first sharing card both display the first note data and a screenshot of the first target multimedia clip, which may be a screenshot of the first multimedia data played when the terminal device obtains the user's first trigger operation. In one example, a timestamp may also be displayed on the first card image, the first sharing card, and the first note card, which may be the first timestamp of the first multimedia data played when the terminal device obtains the user's first trigger operation. In addition, the first card image and the first sharing card may also display the identification information of the plug-in installed in the browser, such as the plug-in name, and display the name of the first multimedia data, such as the name of the video data.

[0162] In summary, through this embodiment, when a user browses multimedia data, the first target multimedia segment that the user is interested in can be obtained and the associated first note data can be automatically generated, and the first target multimedia segment and the first note data can be displayed in association with each other, thereby simplifying user operations and improving the browsing efficiency of multimedia data and the convenience of processing multimedia data.

[0163] Figure 8 A schematic diagram of the structure of a multimedia data processing device provided by an embodiment of the present disclosure is shown in FIG. Figure 8 As shown, the device comprises:

[0164] A determining unit 81, configured to determine a first target multimedia segment in the first multimedia data in response to a first trigger operation of a user on the first multimedia data;

[0165] A generating unit 82, configured to generate first note data of the first multimedia data based on the content of the first target multimedia segment; the first note data is used to represent first information of the first target multimedia segment;

[0166] The display unit 83 is used to display the first target multimedia segment and the first note data in association with each other.

[0167] Optionally, the generating unit 82 is specifically configured to:

[0168] For each object in the first target multimedia segment, generating sub-note data of the object according to the relevant content of the object in the first target multimedia segment;

[0169] Based on the sub-note data of each of the objects, first note data of the first multimedia data is generated.

[0170] Optionally, the generating unit 82 is further specifically configured to:

[0171] Summarizing the relevant content of the object in the first target multimedia segment through a generative model to obtain content summary data of the object;

[0172] Obtaining extended content of the object associated with content summary data of the object;

[0173] Sub-note data of the object is generated based on the content summary data of the object and the extended content of the object.

[0174] Optionally, the generating unit 82 is further specifically configured to:

[0175] Acquire a note template corresponding to the content type of the first multimedia data;

[0176] The note template is edited using the sub-note data of each of the objects to generate first note data of the first multimedia data.

[0177] Optionally, the generating unit 82 is specifically configured to:

[0178] Obtaining each subtitle of the first target multimedia segment;

[0179] Based on a subtitle selection rule, selecting a first subtitle from each subtitle of the first target multimedia segment;

[0180] Based on the first subtitles, first note data of the first multimedia data is generated.

[0181] Optionally, the system further comprises an editing unit, configured to display the edited first note data in response to the user's editing operation on the first note data before the first target multimedia segment and the first note data are displayed in association with each other;

[0182] The display unit 83 is specifically used to display the first target multimedia segment and the edited first note data in association with each other.

[0183] Optionally, the editing unit is specifically used for:

[0184] If the first note data includes a first subtitle in the first target multimedia segment, in response to the user's selection operation on the first subtitle, the first subtitle selected by the user is used as the edited first note data, and the edited first note data is displayed;

[0185] and / or,

[0186] In response to the note input operation of the user, the first note data and the user note input by the user are taken together as the edited first note data, and the edited first note data is displayed.

[0187] Optionally, an extraction unit is further included, for:

[0188] In response to a second trigger operation of the user on the first target multimedia segment, extracting a plurality of target frames from the first target multimedia segment;

[0189] Based on the target frames, representation data of the first target multimedia segment is obtained by combination, and the representation data is displayed.

[0190] Optionally, the first target multimedia segment and the first note data are displayed in association with each other through a first note card; and further comprising an aggregation unit, configured to:

[0191] Acquire a plurality of second note cards of the user; the second note cards are used to associate and display second note data and a second target multimedia segment; the second target multimedia segment comes from the second multimedia data;

[0192] clustering the plurality of second note cards of the user according to the second multimedia data to obtain at least one type of note card set;

[0193] Note summary data corresponding to each type of note card set is generated according to the second note data and / or the second target multimedia segment in each type of note card set.

[0194] Optionally, the first target multimedia clip and the first note data are displayed in association with each other through a first note card; and further comprising a sharing unit, configured to:

[0195] In response to the user's triggering operation on the first note card, displaying a first sharing card corresponding to the first note card; the first sharing card is used to share the first note data and the first target multimedia clip;

[0196] In response to the user's sharing operation on the first sharing card, the first sharing card is shared to a target user.

[0197] Optionally, the first sharing card carries first link information corresponding to the first target multimedia segment; the first link information is used for the target user to browse the first target multimedia segment.

[0198] The multimedia data processing device in the embodiment of the present disclosure can implement each process of the above-mentioned multimedia data processing method embodiment and achieve the same effect and function, which will not be repeated here.

[0199] An embodiment of the present disclosure further provides an electronic device, Fig. 9A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure, such as Fig. 9 As shown, the electronic device may have relatively large differences due to different configurations or performances, and may include one or more processors 901 and memory 902, and one or more applications or data may be stored in the memory 902. Among them, the memory 902 may be a temporary storage or a permanent storage. The application stored in the memory 902 may include one or more modules (not shown in the figure), and each module may include a series of computer executable instructions in the electronic device. Furthermore, the processor 901 can be configured to communicate with the memory 902 to execute a series of computer executable instructions in the memory 902 on the electronic device. The electronic device may also include one or more power supplies 903, one or more wired or wireless network interfaces 904, one or more input or output interfaces 905, one or more keyboards 906, etc.

[0200] In a specific embodiment, the electronic device includes a processor; and a memory configured to store computer executable instructions, wherein when the computer executable instructions are executed, the processor implements the following process:

[0201] In response to a first trigger operation of a user on first multimedia data, determining a first target multimedia segment in the first multimedia data;

[0202] Based on the content of the first target multimedia segment, generating first note data of the first multimedia data; the first note data is used to represent first information of the first target multimedia segment;

[0203] The first target multimedia segment and the first note data are associated and displayed.

[0204] The electronic device in the embodiment of the present disclosure can implement each process of the above-mentioned multimedia data processing method embodiment and achieve the same effect and function, which will not be repeated here.

[0205] Another embodiment of the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, the following process is implemented:

[0206] In response to a first trigger operation of a user on first multimedia data, determining a first target multimedia segment in the first multimedia data;

[0207] Based on the content of the first target multimedia segment, generating first note data of the first multimedia data; the first note data is used to represent first information of the first target multimedia segment;

[0208] The first target multimedia segment and the first note data are associated and displayed.

[0209] The storage medium in the embodiment of the present disclosure can implement each process of the above-mentioned multimedia data processing method embodiment and achieve the same effect and function, which will not be repeated here.

[0210] Another embodiment of the present disclosure further provides a computer program product, the computer program product comprising a computer program, and when the computer program is executed by a processor, the following process is implemented:

[0211] In response to a first trigger operation of a user on first multimedia data, determining a first target multimedia segment in the first multimedia data;

[0212] Based on the content of the first target multimedia segment, generating first note data of the first multimedia data; the first note data is used to represent first information of the first target multimedia segment;

[0213] The first target multimedia segment and the first note data are associated and displayed.

[0214] The computer program product in the embodiment of the present disclosure can implement each process of the above-mentioned multimedia data processing method embodiment and achieve the same effect and function, which will not be repeated here.

[0215] In various embodiments of the present disclosure, the computer-readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0216] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0217] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0218] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0219] For the convenience of description, the above devices are described in terms of functions and are divided into various units. Of course, when implementing the embodiments of the present disclosure, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0220] Those skilled in the art will appreciate that one or more embodiments of the present disclosure may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, one or more embodiments of the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0221] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0222] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0223] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0224] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0225] One or more embodiments of the present disclosure may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0226] Each embodiment in the present disclosure is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0227] The above description is only an embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the scope of the claims of the present disclosure.

Claims

1. A multimedia data processing method, characterized in that: include: In response to a first trigger operation of a user on first multimedia data, determining a first target multimedia segment in the first multimedia data; generating first note data of the first multimedia data based on the content of the first target multimedia segment; The first note data is used to represent first information of the first target multimedia segment; The first target multimedia segment and the first note data are associated and displayed.

2. The method according to claim 1, characterized in that The step of generating first note data of the first multimedia data based on the content of the first target multimedia segment includes: For each object in the first target multimedia segment, generating sub-note data of the object according to the relevant content of the object in the first target multimedia segment; Based on the sub-note data of each of the objects, first note data of the first multimedia data is generated.

3. The method according to claim 2, characterized in that The step of generating sub-note data of the object according to the relevant content of the object in the first target multimedia segment includes: Summarizing the relevant content of the object in the first target multimedia segment through a generative model to obtain content summary data of the object; Obtaining extended content of the object associated with content summary data of the object; Sub-note data of the object is generated based on the content summary data of the object and the extended content of the object.

4. The method according to claim 2, characterized in that: The generating the first note data of the first multimedia data based on the sub-note data of each of the objects comprises: Acquire a note template corresponding to the content type of the first multimedia data; The note template is edited using the sub-note data of each of the objects to generate first note data of the first multimedia data.

5. The method according to claim 1, characterized in that The step of generating first note data of the first multimedia data based on the content of the first target multimedia segment includes: Obtaining each subtitle of the first target multimedia segment; Based on a subtitle selection rule, selecting a first subtitle from each subtitle of the first target multimedia segment; Based on the first subtitles, first note data of the first multimedia data is generated.

6. The method according to claim 1, characterized in that Before the first target multimedia segment and the first note data are displayed in association with each other, the method further includes: In response to the user's editing operation on the first note data, displaying the edited first note data; The associating and displaying the first target multimedia segment and the first note data includes: The first target multimedia segment and the edited first note data are associated and displayed.

7. The method according to claim 6, characterized in that In response to the user's editing operation on the first note data, displaying the edited first note data includes: If the first note data includes a first subtitle in the first target multimedia segment, in response to the user's selection operation on the first subtitle, the first subtitle selected by the user is used as the edited first note data, and the edited first note data is displayed; and / or, In response to the note input operation of the user, the first note data and the user note input by the user are taken together as the edited first note data, and the edited first note data is displayed.

8. The method according to claim 1, characterized in that The method further comprises: In response to a second trigger operation of the user on the first target multimedia segment, extracting a plurality of target frames from the first target multimedia segment; Based on the target frames, representation data of the first target multimedia segment is obtained by combination, and the representation data is displayed.

9. The method according to claim 1, characterized in that: The first target multimedia segment and the first note data are displayed in association with each other through a first note card; the method further includes: Acquire a plurality of second note cards of the user; the second note cards are used to associate and display second note data and a second target multimedia segment; the second target multimedia segment comes from the second multimedia data; clustering the plurality of second note cards of the user according to the second multimedia data to obtain at least one type of note card set; Note summary data corresponding to each type of note card set is generated according to the second note data and / or the second target multimedia segment in each type of note card set.

10. The method according to claim 1, characterized in that The first target multimedia segment and the first note data are displayed in association with each other through a first note card; the method further includes: In response to the user's triggering operation on the first note card, displaying a first sharing card corresponding to the first note card; the first sharing card is used to share the first note data and the first target multimedia clip; In response to the user's sharing operation on the first sharing card, the first sharing card is shared to a target user.

11. The method according to claim 10, characterized in that The first sharing card carries first link information corresponding to the first target multimedia segment; the first link information is used for the target user to browse the first target multimedia segment.

12. A multimedia data processing device, characterized in that: include: a determining unit, configured to determine a first target multimedia segment in the first multimedia data in response to a first trigger operation of a user on the first multimedia data; A generating unit, configured to generate first note data of the first multimedia data based on the content of the first target multimedia segment; The first note data is used to represent first information of the first target multimedia segment; A display unit is used to display the first target multimedia segment and the first note data in association with each other.

13. An electronic device, characterized in that: include: processor; as well as, A memory configured to store computer executable instructions, which, when executed, cause the processor to implement the steps of the method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store computer-executable instructions, and the computer-executable instructions, when executed by a processor, implement the steps of the method described in any one of claims 1 to 11.

15. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images