Interaction method and device based on multimedia content, equipment, medium and product

Abstract

The invention discloses an interaction method, device and equipment based on multimedia content, a medium and a product, and relates to the technical field of computers. The method comprises the following steps: displaying a playing interface of first multimedia content; receiving an interaction operation for the first interaction information in the interaction area; and based on the interaction operation, adding a first work corresponding to the first multimedia content to a work storage list. On one hand, a user can identify the multimedia content through the interactive operation triggering system to determine the corresponding first work, and the tedious process that the user determines the first work in the modes of manual search, inquiry and the like is avoided; and on the other hand, after the system identifies the first work, the first work can be automatically added to the storage list, and the operation step of manually adding the first work to the storage list by the user is also omitted, so that the user can more conveniently obtain and store the interested work when browsing the multimedia content, and the man-machine interaction efficiency is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to an interactive method, device, equipment, medium and product based on multimedia content. Background Art

[0002] Online video platforms, also known as video websites or streaming platforms, refer to platforms that provide video playback functions for users on the Internet. With the vigorous development of online video platforms, the video content in online video platforms is becoming more and more abundant, and these video contents usually have background music.

[0003] In the related art, if a user wants to obtain the background music in the video content played on an online video platform, he or she needs to first obtain the music name of the background music in the video content through channels such as the comment area or information provided by the creator, then open the music playback platform to search for the music name, and finally add the background music to the playlist collection from the search results.

[0004] However, the above process of adding background music to the playlist collection is too cumbersome and the human-computer interaction efficiency is low. Summary of the invention

[0005] The embodiments of the present application provide a method, device, equipment, medium and product for interaction based on multimedia content. The technical solution is as follows:

[0006] In one aspect, a multimedia content-based interactive method is provided, the method comprising:

[0007] Displaying a playback interface of the first multimedia content, wherein the playback interface includes an interactive area, and the interactive area is used to display interactive information published for the first multimedia content;

[0008] receiving an interactive operation for first interactive information in the interactive area, wherein the first interactive information includes a first interactive identifier, and the first interactive identifier is used to identify the content of the first multimedia content;

[0009] Based on the interactive operation, a first work corresponding to the first multimedia content is added to a work storage list, where the first work is obtained after content recognition is performed on the first multimedia content.

[0010] In another aspect, there is provided an interactive device based on multimedia content, the device comprising:

[0011] A display module, configured to display a playback interface of the first multimedia content, wherein the playback interface includes an interactive area, and the interactive area is configured to display interactive information published for the first multimedia content;

[0012] A receiving module, configured to receive an interactive operation for first interactive information in the interactive area, wherein the first interactive information includes a first interactive identifier, and the first interactive identifier is used to identify the content of the first multimedia content;

[0013] The processing module is used to add a first work corresponding to the first multimedia content to a work storage list based on the interactive operation, where the first work is obtained after content recognition is performed on the first multimedia content.

[0014] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement any of the above-mentioned multimedia content-based interactive methods.

[0015] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, a code set or an instruction set is stored in the storage medium, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement any of the above-mentioned multimedia content-based interactive methods.

[0016] On the other hand, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs any of the above-mentioned interactive methods based on multimedia content.

[0017] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0018] In the playback interface of multimedia content, the user only needs to trigger an interactive operation for the first interactive information (including the first interactive identifier), and the system can automatically complete the recognition process of the multimedia content and add the recognized first work to the work storage list. On the one hand, the user can trigger the system to recognize the multimedia content through the interactive operation to determine the corresponding first work, avoiding the tedious process of users determining the first work through manual search, inquiry, etc.; on the other hand, after recognizing the first work, the system can automatically add the first work to the storage list, which also saves the user the step of manually adding the first work to the storage list, so that users can more conveniently obtain and save works of interest when browsing multimedia content, thereby improving the efficiency of human-computer interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 is a schematic diagram of a computer system provided by an exemplary embodiment of the present application;

[0021] Figure 2 is a schematic diagram of an interface of an interactive method based on multimedia content provided by an exemplary embodiment of the present application;

[0022] Figure 3 is a flow chart of an interactive method based on multimedia content provided by an exemplary embodiment of the present application;

[0023] Figure 4 is a flow chart of an interactive method based on multimedia content provided by another exemplary embodiment of the present application;

[0024] Figure 5 is a flowchart of an interactive method based on multimedia content provided by another exemplary embodiment of the present application;

[0025] Figure 6 is a structural block diagram of an interactive device based on multimedia content provided by an exemplary embodiment of the present application;

[0026] Figure 7 It is a structural block diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present application clearer, the implementation mode of the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0028] In this application, the terms "first", "second", etc. are used to distinguish identical or similar items with substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first" and "second", nor is there any limitation on quantity and execution order.

[0029] It should be noted that before collecting relevant data of users and during the process of collecting relevant data of users, this application can display a prompt interface, pop-up window or output voice prompt information, and the prompt interface, pop-up window or voice prompt information is used to prompt the user that its relevant data is currently being collected, so that this application only starts to execute the relevant steps of obtaining relevant data of users after obtaining the confirmation operation issued by the user to the prompt interface or pop-up window, otherwise (that is, when the confirmation operation issued by the user to the prompt interface or pop-up window is not obtained), the relevant steps of obtaining relevant data of users are terminated, that is, the relevant data of users are not obtained. In other words, all user data collected by this application are collected with the consent and authorization of the user, and the collection, use and processing of relevant user data need to comply with relevant laws, regulations and standards.

[0030] Online video platforms, also known as video websites or streaming media platforms, refer to platforms that provide users with video playback functions on the Internet. With the vigorous development of online video platforms, the video content in online video platforms is becoming more and more abundant, and these video contents usually have background music. In the related art, if a user wants to obtain the background music in the video content played on the online video platform, he needs to first obtain the music name of the background music in the video content through channels such as the comment area or information provided by the creator, and then open the music playback platform to search for the music name, and finally add the background music to the playlist collection from the search results. However, the above process of adding background music to the playlist collection is too cumbersome, and the human-computer interaction efficiency is low.

[0031] Based on this, the present application provides an interactive method based on multimedia content. In the multimedia content playback interface, the user only needs to trigger an interactive operation for the first interactive information (including the first interactive identifier), and the system can automatically complete the recognition process of the multimedia content and add the recognized first work to the work storage list.

[0032] Next, a computer system for implementing the multimedia content-based interactive method of the present application is introduced.

[0033] Figure 1 A schematic diagram of a computer system provided by an exemplary embodiment of the present application is as follows: Figure 1 As shown, the computer system includes a terminal 110 and a server 120. The terminal 110 and the server 120 are connected via a communication network 130. In some optional embodiments, the communication network 130 may be a wired network or a wireless network, which is not limited in this embodiment.

[0034] In some embodiments, the terminal 110 is a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart home appliance, a smart car terminal, etc., but is not limited thereto. The server 120 provides background services for the client installed in the terminal. The server 120 can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services. The server 120 can also be implemented as a node in a blockchain system.

[0035] In some embodiments, a client of a first application is installed in the terminal 110, and the first application may be a traditional application, a cloud application, a mini-program or an application module in a host application, or a web platform, which is not limited in this embodiment. Optionally, the first application may be implemented as an application that provides a multimedia content playback function, such as a video application, a short video application, an audio playback application, an instant messaging application, etc., which is not limited here.

[0036] Illustratively, a playback interface of a first multimedia content is displayed in a client of a first application of a terminal 110, a first account is logged in to the client of the first application, and the playback interface includes an interactive area, which is used to provide account interaction functions, and a user can publish interactive information for the first multimedia content in the interactive area, such as comments, bullet screen messages, etc. When a user publishes a first interactive information including a first interactive identifier (e.g., @music playback platform A official account) in the interactive area, the client of the first application will forward the first interactive information to the server 120 through the communication network 130. After receiving the first interactive information, the server 120 triggers a content recognition process for the first multimedia content when the first interactive identifier is recognized, for example, recognizing the background music of the first multimedia content.

[0037] After triggering the content identification process, if the server 120 identifies the first work corresponding to the first multimedia content, it will obtain the work information of the first work and send the work information of the first work to the client of the first application of the terminal 110. After receiving the work information, the client of the first application will display reply information to the first interactive information in the interactive area of ​​the playback interface, and the reply information includes the work information of the first work; or, the client of the first application will display the work information of the first work on the conversation interface, where the conversation interface refers to the interface for conducting a conversation between the first account and the target account, and the target account is associated with the first interactive identifier. For example, if the first interactive identifier is @music playback platform A official account, then the target account is the music playback platform A official account.

[0038] At the same time, after identifying the first work, the server 120 will also trigger the process of adding the first work to the work storage list. Take the work storage list implemented as a storage list provided by the second application as an example for explanation. Optionally, the terminal 110 is installed with a client of the second application, and a second account is logged in to the client of the second application. The second application can be a traditional application, a cloud application, a mini-program or application module in the host application, or a web page platform, which is not limited in this embodiment. Optionally, the second application can be implemented as an application that provides a work storage function, such as: an audio playback application, an instant messaging application, a novel reading application, a video playback application, a picture browsing application, etc., which is not limited here.

[0039] Illustratively, after the server 120 identifies the first work, if there is an association between the second account logged in to the client of the second application and the first account logged in to the client of the first application (for example, the user authorizes the binding of the first account and the second account), the server 120 will directly send a work addition request to the client of the second application of the terminal 110. After receiving the work addition request, the client of the second application will add the first work to the work storage list. For example: taking the first work as song 1 as an example, the client of the second application creates a new playlist a for the second account, and then adds song 1 to playlist a.

[0040] If the second account logged in the client of the second application has not established an association with the first account logged in the client of the first application, the server 120 will send the work extraction information to the client of the first application of the terminal 110. After receiving the work extraction information, the client of the first application will display the reply information to the first interactive information in the interactive area of ​​the playback interface, and the reply information includes the work extraction information; or the client of the first application will display the work extraction information in the session interface. The user can start the client of the second application on the terminal 110. When the user pastes the work extraction information on the application interface of the client of the second application, or the user searches for the work extraction information in the search bar of the client of the second application, the client of the second application will add the first work to the work storage list.

[0041] The following is an example of implementing the first application as a video playback platform and the second application as a music playback platform A. Figure 2, which shows an interface schematic diagram of an interactive method based on multimedia content, in which a playback interface is displayed in the client of the video playback platform running on the terminal 110, and the video playback area 211 of the playback interface plays video content; the playback interface also includes an interactive area 212, in which users can comment on the video content or send bullet comments. Assuming that the user posts a comment "@music playback platform A official account" in the interactive area 212, on the one hand, the client of the video playback platform will display the reply information 213 of the official account of the music playback platform A to the comment in the interactive area 212, that is, "the background music of the video content is as follows: song name: aaaaa singer: bbb", and the reply information is obtained after identifying the background music of the video content; on the other hand, after identifying the song "aaaaa", the song "aaaaa" will be added to "My Playlist 1" 221 in the client of the music playback platform A running on the terminal 110.

[0042] Next, the process of the multimedia content-based interactive method provided in the embodiment of the present application is introduced.

[0043] Combined with the above introduction, Figure 3 is a flowchart of an interactive method based on multimedia content provided by an embodiment of the present application, and the method is applied to Figure 1 Taking the terminal 110 as an example, the method is as follows: step 310 to step 330.

[0044] Step 310: display a playback interface of the first multimedia content.

[0045] Optionally, the first multimedia content includes at least one of video content, audio content, picture content, text content, etc., which is not limited here.

[0046] In some embodiments, the playback interface includes a content playback area, and the playback interface of the first multimedia content is displayed in the content playback area.

[0047] The play interface includes an interactive area, and the interactive area is used to display interactive information published for the first multimedia content. Schematically, the interactive information includes at least one of comments, bullet comments, and the like.

[0048] In some embodiments, the playback interface of the first multimedia content is an interface provided by the first application. Optionally, the playback interface of the first multimedia content is displayed on a client of the first application, and the first account is logged in to the client of the first application.

[0049] Optionally, the first application may be implemented as an application program that provides a multimedia content playback function, such as a video application, a short video application, an audio playback application, an instant messaging application, etc., which is not limited here.

[0050] Step 320: receiving an interactive operation for the first interactive information in the interactive area.

[0051] The first interactive information refers to interactive information published for the first multimedia content. Optionally, the first interactive information can be implemented as information in text form.

[0052] The first interactive information includes a first interactive identifier.

[0053] The first interaction identifier is used to perform content recognition on the first multimedia content. In other words, the first interaction identifier is used to trigger a content recognition process for the first multimedia content.

[0054] Illustratively, the first interaction identifier is a pre-set identifier. When the client of the first application recognizes the first interaction identifier, it sends the first interaction identifier to the server. After receiving the first interaction identifier, the server triggers a content recognition process for the first multimedia content.

[0055] In some embodiments, the first interactive identifier can be implemented as a reminder identifier for the target account, for example: "@target account name". Optionally, for the target account: the target account refers to the account registered by the target platform in the first application. If the work storage list described below is a storage list provided by the first application, the target platform is the first application, for example: the target account is the official account of the first application; if the work storage list described below is a storage list provided by the second application, the target platform is the second application, for example: the target account is the official account registered by the second application in the first application.

[0056] Optionally, performing content recognition on the first multimedia content refers to recognizing the artwork in the first multimedia content. The artwork includes at least one of literary works (such as novels, poems, etc.), audio works (such as radio dramas, songs, etc.), film and television works (such as TV series, movies, variety shows, etc.), and paintings. The following takes the first multimedia content as video content as an example for explanation:

[0057] (1) Identify literary works in video content.

[0058] Illustratively, the video content involves literary works, for example: the video content is an introduction to a certain literary work, or the video content is a TV series or movie based on a certain literary work, or the video content mentions a certain literary work. To identify literary works in video content, the text content corresponding to the video frames in the video content and the text content corresponding to the audio frames in the video content can be identified, and the literary works involved in the video content can be determined by analyzing these text contents, such as: obtaining a literary work keyword library, the literary work keyword library includes keywords of multiple literary works, matching the obtained text content with the literary work keyword library, and if the text content contains keywords of a certain literary work, the literary work is used as the recognition result.

[0059] (2) Identify audio works in video content.

[0060] Illustratively, the video content involves audio works, for example: the background music of the video content is a certain audio work, or the video content is an introduction to a certain audio work, or the video content mentions a certain literary work. To identify the audio work involved in the background music of the video content, the background music can be first extracted from the video content, and then the audio feature representation corresponding to the background music can be extracted, and the audio feature representation can be matched with the audio work database, wherein the audio work database includes audio feature representations corresponding to multiple audio works respectively. If the audio feature representation corresponding to the background music matches the audio feature representation corresponding to a certain audio work in the audio work database, the audio work is used as the recognition result.

[0061] (3) Identify film and television works in the video content.

[0062] Illustratively, the video content involves film and television works, for example, the video content is the commentary content for a certain film and television work, or the video content is a clip of a certain film and television work, etc. If the film and television work in the video content is to be identified, the content feature representation of the video content can be extracted; the film and television work involved in the video content can be determined by analyzing the content feature representation, such as matching the content feature representation with a film and television work database, wherein the film and television work database includes work feature representations corresponding to multiple film and television works (for example, character feature representation, scene feature representation, storyline feature representation, etc.), if the content feature representation of the video content matches the work feature representation corresponding to a certain film and television work in the film and television work database, then the film and television work is used as the identification result.

[0063] (4) Identify paintings in video content.

[0064] Illustratively, the video content involves paintings, for example, the video content is an introduction to a certain painting. If the painting in the video content is to be identified, the image feature representation of the video frame corresponding to the video content may be extracted; the painting involved in the video content may be determined by analyzing the image feature representation, such as matching the image feature representation corresponding to the video content with a painting database, wherein the painting database includes image feature representations corresponding to multiple paintings, and if the image feature representation corresponding to the video content matches the image feature representation corresponding to a painting in the painting database, the painting is taken as the identification result.

[0065] It should be noted that the above-mentioned method for identifying video content is only for illustrative purposes and is not limited to this in the embodiments of the present application.

[0066] In some embodiments, performing content recognition on the first multimedia content refers to performing content recognition on the first multimedia content and other information associated with the first multimedia content. Optionally, the other information associated with the first multimedia content includes information such as a title and introduction of the first multimedia content.

[0067] In some embodiments, the first interactive information also includes first prompt information, and the first interactive identifier is used to perform content recognition on the first multimedia content under the guidance of the first prompt information. Illustratively, if the first multimedia content is recognized through a content recognition model, the first prompt information and the first multimedia content can be input into the content recognition model, and the content recognition model will perform content recognition on the first multimedia content under the guidance of the first prompt information. For example, if the first prompt information is "recognize background music", the content recognition model will recognize the background music of the first multimedia content.

[0068] In some embodiments, the first interactive information also includes content identification prompt information, and the content identification prompt information is used to indicate content identification of a first multimedia segment in the first multimedia content; optionally, based on the interactive operation, a first work corresponding to the first multimedia segment in the first multimedia content is added to a work storage list, and the work storage list is the works obtained after content identification of the first multimedia segment.

[0069] Illustratively, taking the first multimedia content implemented as video content as an example, the identification content prompt information can be implemented as "[start time]-[end time]", and an example of the first interactive information can be "background music from 00:01:10 to 00:02:30 @target account name".

[0070] When the server receives the first interactive information including the identification content prompt information, it will cut the first multimedia content into segments according to the identification content prompt information to obtain the first multimedia segment; then perform content identification on the first multimedia segment to obtain the work corresponding to the first multimedia segment (for example: the first work).

[0071] In the above embodiment, by identifying content prompt information, the user can more accurately obtain the works of interest when browsing multimedia content. In addition, content identification is performed only on multimedia clips rather than the entire multimedia content, which also reduces the consumption of computer resources to a certain extent.

[0072] Optionally, the receiving of the interactive operation for the first interactive information in the interactive area includes at least one of the following operations:

[0073] (1) Sending operation of the first interactive information.

[0074] Optionally, receiving a sending operation of the first interactive information in the interactive area as the interactive operation. Schematically, the first account sends a bullet screen or comment containing the first interactive identifier as the first interactive information in the playback interface of the first multimedia content.

[0075] (2) Agreeing with the first interactive information.

[0076] Optionally, the interactive area displays first interactive information, which is interactive information posted in the interactive area by other accounts (except the first account); receiving an approval operation for the first interactive information in the interactive area as an interactive operation, illustratively, the approval operation includes a like operation and a "+1" operation. The interactive area displays comments containing a first interactive identifier sent by other accounts, and the first account can like the comment, or the first account can perform a "+1" operation on the comment, wherein the "+1" operation refers to expressing approval of the comment. In some embodiments, when the number of "+1"s obtained by the comment reaches a preset number, the recognition process for the first multimedia content is started and the recognized work is added to the work storage list. Optionally, the "+1" operation refers to the operation of copying and sending the comment. Schematically, the "+1" operation option is displayed under the comment containing the first interactive identifier. When the first account clicks the "+1" operation option, the comment containing the first interactive identifier is automatically sent to the comment area.

[0077] (3) Forwarding the first interactive information.

[0078] Optionally, the interactive area displays first interactive information; a forwarding operation of the first interactive information in the interactive area is received as an interactive operation. Schematically, the interactive area displays comments containing a first interactive identifier, and the first account can forward the first interactive identifier to a conversation interface with the target account.

[0079] The above examples of interactive operations are merely illustrative and are not limited to these in the embodiments of the present application.

[0080] Step 330: Based on the interactive operation, add the first work corresponding to the first multimedia content to the work storage list.

[0081] The first work is a work obtained by performing content recognition on the first multimedia content.

[0082] Illustratively, after the interactive operation is triggered, the server performs content recognition on the first multimedia content, and after identifying the first work, triggers a process of adding the first work to the work storage list.

[0083] Optionally, the first work includes at least one of a literary work, an audio work, a film and television work, a painting work, etc.

[0084] Based on the interactive operation, the case where the first work corresponding to the first multimedia content is added to the work storage list includes at least one of the following cases:

[0085] (1) The works storage list is a storage list provided by the first application.

[0086] Optionally, based on the interactive operation, the first work corresponding to the first multimedia content is added to a work storage list in the client of the first application.

[0087] Illustratively, after the server identifies the first work, it sends a work adding request to the client of the first application. After the client of the first application receives the work adding request, it adds the first work to the client of the first application.

[0088] (2) The works storage list is a storage list provided by the second application.

[0089] Optionally, based on the interactive operation, the first work corresponding to the first multimedia content is added to a work storage list in the client of the second application.

[0090] Optionally, the current terminal is installed with a client of the second application, and a second account is logged into the client of the second application. The second application can be implemented as an application program that provides a work storage function, such as an audio playback application, an instant messaging application, a novel reading application, a video playback application, a picture browsing application, etc., which is not limited here. The second application and the first application are different applications.

[0091] In the above embodiment, based on the interactive operation, the first work corresponding to the first multimedia content can be directly added to the work storage list in the client of the second application, providing users with a convenient way to organize and collect multimedia works of interest without having to re-search or find the work in the second application, saving users' time and energy and improving the efficiency of work management.

[0092] In some embodiments, when the first account and the second account are associated, the first work corresponding to the first multimedia content is automatically added to the work storage list in the client of the second application.

[0093] The association between the first account and the second account means that the user authorizes the first account to be bound to the second account. Schematically, on the client of the first application, the user can initiate a request to bind the first account to the second account, and the first account to the second account is bound through the request.

[0094] When the first account and the second account are bound, after the server recognizes the first work, it will send a work addition request to the client of the second application. After receiving the work addition request, the client of the second application adds the second work to the client of the first application.

[0095] In the above scheme, when the first account and the second account are associated, the first work corresponding to the first multimedia content can be automatically added to the work storage list in the client of the second application, which saves the user's manual operation steps, improves the operation efficiency, and allows the user to organize and store the works of interest in different application clients more quickly and conveniently.

[0096] In other embodiments, when there is no association relationship between the first account and the second account, work extraction information corresponding to the first work is displayed on the client of the first application; when the work extraction information is used on the client of the second application, the first work corresponding to the first multimedia content is automatically added to the work storage list in the client of the second application.

[0097] The non-existence of the association relationship between the first account and the second account means that the first account and the second account are not bound.

[0098] When the first account and the second account are not bound, after the server recognizes the first work, it will send work extraction information to the client of the first application. The work extraction information includes a unique extraction code corresponding to the first work, for example: "ABC12345". Each work has a different extraction code, which is used to add the corresponding work to the work storage list on the client of the second application.

[0099] In the above scheme, when there is no association between the first account and the second account, the work is not added automatically directly. Instead, the user is required to use the extracted information in the second application client to complete the addition. This ensures the security of the user account to a certain extent, avoids accidentally adding text to the wrong account, etc., and at the same time gives the user enough autonomy to decide whether to add the work to the second application.

[0100] Optionally, when the work is used to extract information on the client of the second application, the method of automatically adding the first work corresponding to the first multimedia content to the work storage list in the client of the second application includes at least one of the following methods:

[0101] (1) A copy operation of work extraction information is received at a client of a first application; in response to receiving an operation of starting a client of a second application, a first work corresponding to the first multimedia content is automatically added to a work storage list in the client of the second application.

[0102] Illustratively, the work extraction information is copied to the clipboard of the terminal, and then the client of the second application is started. The client of the second application reads the work extraction information in the clipboard. When the extraction code of the first work in the work extraction information is identified, the client of the second application automatically adds the first work to the work storage list in the client of the second application.

[0103] (2) receiving an input operation for extracting work information at the client of the second application; in response to receiving the input operation, automatically adding the first work corresponding to the first multimedia content to a work storage list in the client of the second application.

[0104] Illustratively, start the client of the second application and display the application interface of the client of the second application, enter the work extraction information in the extraction code input area or search area of ​​the application interface, and the client of the second application will automatically add the first work to the work storage list in the client of the second application.

[0105] Optionally, different types of first works correspond to different types of work storage lists.

[0106] Illustratively, if the first work is a novel, the work storage list can be implemented as a novel bookshelf; if the first work is a song, the work storage list can be implemented as a song playlist; if the first work is a film or television work, the work storage list can be implemented as a video collection, etc., which is not limited here.

[0107] Optionally, the number of works obtained after content recognition of the first multimedia content can be one or more, which is not limited here. If the number of works obtained after content recognition of the first multimedia content is more than one, the multiple works obtained by recognition are added to the work storage list. In principle, the multiple works are added to the same work storage list; alternatively, the multiple works are added to the work storage list according to the work type.

[0108] Take the example of the works obtained after content recognition of the first multimedia content, which include the first work and the second work: based on the interactive operation, the first work corresponding to the first multimedia content is added to the second work storage list, and the second work corresponding to the first multimedia content is added to the third work storage list; wherein, the first work matches the type of stored works corresponding to the second work storage list, and the second work matches the type of stored works corresponding to the third work storage list.

[0109] For example: if the first work is literary work a and the second work is audio work b, literary work a can be added to work collection list 1 for storing literary works, and audio work b can be added to work collection list 2 for storing audio works. Or if the first work is pop song c and the second work is rap song d, pop song c can be added to playlist 1 for storing pop songs, and rap song d can be added to playlist 2 for storing rap songs.

[0110] In the above embodiment, the adding operation is performed based on the matching between the works and the types of stored works, so that accurate classification and storage of different types of works can be achieved.

[0111] In some embodiments, when there are multiple candidate work storage lists, a candidate work storage list that matches the work can be determined from the multiple candidate work storage lists to store the work based on the work features and the storage list features.

[0112] Optionally, based on the interactive operation, the method of adding the first work corresponding to the first multimedia content to the second work storage list and adding the second work corresponding to the first multimedia content to the third work storage list further includes the following steps:

[0113] Step 1: Based on interactive operations, a list of multiple candidate works is obtained.

[0114] Optionally, the multiple candidate works storage lists are lists provided by the client of the second application. Schematically, the multiple candidate works storage lists refer to the works storage lists currently created by the second account.

[0115] After identifying the first work and the second work, the server sends the work information of the first work and the work information of the second work to the terminal, and the client of the first application or the client of the second application in the terminal receives the work information of the first work and the work information of the second work, and obtains the storage work information and list attribute information corresponding to the multiple candidate work storage lists in the second account.

[0116] Step 2: Extract the list feature representations corresponding to the multiple candidate work storage lists.

[0117] The list feature representation is used to characterize the types of stored works in the candidate work storage list; wherein the list feature representation is a feature representation determined based on the stored work information and list attribute information of the candidate work storage list.

[0118] The collected work information refers to the information of the works collected in the candidate work collection list. Indicatively, the collected work information includes but is not limited to specific information such as the category of the work, the style of the work (such as classical music, rap music, etc.), the source of the work, and the duration of the work.

[0119] List attribute information refers to the basic information corresponding to the candidate works collection list. Schematically, the list attribute information includes but is not limited to the name of the list, the creation time of the list, the specific theme tags of the list (such as "summer", "nostalgia", etc.), and other information.

[0120] For each acquired candidate work storage list, feature extraction is performed on the corresponding stored work information and list attribute information to obtain a list feature representation representing the types of stored works in the candidate work storage list.

[0121] Step 3: extracting a first work feature representation of a first work corresponding to the first multimedia content, and extracting a second work feature representation of a second work corresponding to the first multimedia content.

[0122] Illustratively, feature extraction is performed on first work information of a first work to obtain a first work feature representation, and feature extraction is performed on second work information of a second work to obtain a second work feature representation.

[0123] Optionally, the work information includes the work title, creator, creation time, work type, work introduction, work duration, etc.

[0124] Step 4: When the first work feature representation matches the list feature representation corresponding to the first candidate work storage list in the plurality of candidate work storage lists, the first candidate work storage list is determined as the second work storage list.

[0125] For the first work feature representation, the similarity (such as cosine similarity, Euclidean distance, etc.) between the first work feature representation and the list feature representations corresponding to the plurality of candidate work storage lists is calculated. If the similarity between the list feature representation corresponding to the first candidate work storage list and the first work feature representation is greater than or equal to a preset matching value, the first candidate work storage list is determined as the second work storage list. If there is a single candidate work storage list whose list feature representation is greater than or equal to the preset matching value and the similarity between the first work feature representation and the list feature representation is greater than or equal to the preset matching value, the single candidate work storage list is determined as the second work storage list. If there are multiple candidate work storage lists whose list feature representations are greater than or equal to the preset matching value and the similarity between the first work feature representation and the list feature representation is greater than or equal to the preset matching value, the candidate work storage list with the highest similarity is determined as the second work storage list.

[0126] Step 5: When the first work feature representation matches the list feature representation corresponding to the second candidate work storage list in the plurality of candidate work storage lists, the second candidate work storage list is determined as the third work storage list.

[0127] For the second work feature representation, the similarity (such as cosine similarity, Euclidean distance, etc.) between the second work feature representation and the list feature representations corresponding to the plurality of candidate work storage lists is calculated. If the similarity between the list feature representation corresponding to the second candidate work storage list and the second work feature representation is greater than or equal to a preset matching value, the second candidate work storage list is determined as the third work storage list. If there is a single candidate work storage list whose list feature representation corresponding to the second work feature representation has a similarity greater than or equal to the preset matching value, the single candidate work storage list is determined as the third work storage list. If there are multiple candidate work storage lists whose list feature representations corresponding to the second work feature representation have a similarity greater than or equal to the preset matching value, the candidate work storage list with the highest similarity is determined as the third work storage list.

[0128] Step 6: Add the first work to the second work storage list, and add the second work to the third work storage list.

[0129] Illustratively, if the client of the first application determines the second work storage list and the third work storage list, the client of the first application sends a work storage request to the server, and the server sends the work storage request to the client of the second application. The client of the second application calls the interface program to add the relevant information of the first work (such as the storage path of the work, the metadata of the work, etc.) to the database table or storage structure corresponding to the second work storage list, thereby storing the first work in the list; and adds the relevant information of the second work to the database table or storage structure corresponding to the third work storage list, thereby completing the storage operation of the second work.

[0130] If the second application client determines the second work storage list and the third work storage list, the second application client directly calls the interface program to add the first work to the second work storage list and the second work to the third work storage list.

[0131] It should be noted that the above process of determining the second work storage list and the third work storage list can also be implemented as a server, which is not limited in this embodiment of the present application.

[0132] In the above scheme, by extracting the feature representation of the works and the feature representation of the candidate works storage list and matching them to determine the appropriate storage list, more accurate classification and storage of the works can be achieved.

[0133] In some embodiments, the first work is added to a work storage list stored in a client of the first application or a client of the second application; or, a new work storage list is created in the client of the first application or the client of the second application, and the first work is added to the work storage list.

[0134] In some embodiments, the first interactive information further includes storage indication information, where the storage indication information indicates that the first work obtained by identifying the first multimedia content is added to the first work storage list.

[0135] Optionally, based on the interactive operation, the first work corresponding to the first multimedia content is added to a first work storage list, and the first work storage list is a list determined according to the storage indication information.

[0136] Illustratively, the storage instruction information includes the name and identification number of the first work storage list, which is not limited here. For example, if the storage instruction information is "add to work storage list a", the first work will be added to work storage list a.

[0137] In the above embodiment, by including storage instruction information in the first interactive information, it is clearly specified that the first work is added to a specific first work storage list, so that the work storage process is more accurate. The user can accurately place the work in the specified storage list according to his or her needs, meeting the personalized needs of the user.

[0138] To summarize, in the interactive method based on multimedia content provided by the embodiment of the present application, in the playback interface of the multimedia content, the user only needs to trigger the interactive operation for the first interactive information (including the first interactive identifier), and the system can automatically complete the recognition process of the multimedia content and add the recognized first work to the work storage list. On the one hand, the user can trigger the system to identify the multimedia content through the interactive operation to determine the corresponding first work, avoiding the tedious process of the user determining the first work through manual search, inquiry, etc.; on the other hand, after identifying the first work, the system can automatically add the first work to the storage list, and also saves the user's manual operation steps of adding the first work to the storage list, so that the user can more conveniently obtain and save the works of interest when browsing the multimedia content, thereby improving the efficiency of human-computer interaction.

[0139] In some embodiments, after the user triggers the interactive operation, in addition to adding the work information of the first work to the work storage list, the work information of the first work can also be displayed on the terminal. Figure 4 , the above Figure 3 The illustrated embodiment may also be implemented as the following steps 410 to 430 .

[0140] Step 410: display a playback interface of the first multimedia content.

[0141] Optionally, the first multimedia content includes at least one of video content, audio content, picture content, text content, etc., which is not limited here.

[0142] In some embodiments, the playback interface includes a content playback area and an interactive area, the content playback area is used to play the first multimedia content, and the interactive area is used to display interactive information published for the first multimedia content.

[0143] Among them, the interactive area includes at least one of a comment area, a barrage area, etc.

[0144] Optionally, the comment area displays the interactive information published for the first multimedia content in a static form. Schematically, the comment area is displayed at a specific location on the playback interface (such as below the content playback area or in the sidebar, etc.), and after the user posts a comment, the user comment is displayed in a static manner in the comment area. For the interactive information displayed in the bullet screen area, the interactive information published by users at different time points will be displayed in the comment area in a centralized manner, so that the viewing user can systematically browse the interactive information published by different users at different time points.

[0145] Optionally, the bullet screen area displays interactive information published for the first multimedia content in a dynamic form. Schematically, the bullet screen area can be implemented as an area covered on the content playback area. After the user sends a bullet screen, the user bullet screen is displayed in a dynamic manner in the bullet screen area. For the interactive information displayed in the bullet screen area, the interactive information will float across the screen of the first multimedia content being played. As the playback time of the first multimedia content advances, the interactive information will scroll from one side of the screen (such as the right side) to the other side (such as the left side). The interactive information published at different time points will float across the screen in the order of the release time, and will be synchronized with the playback of the first multimedia content.

[0146] Step 420: receiving an interactive operation for the first interactive information in the interactive area.

[0147] The first interactive information refers to interactive information published for the first multimedia content. Optionally, the first interactive information can be implemented as information in text form.

[0148] The first interactive information includes a first interactive identifier, which is used to identify the content of the first multimedia content. In an illustrative manner, the first interactive identifier is a pre-set identifier, and when the client of the first application identifies the first interactive identifier, the first interactive identifier is sent to the server, and the server triggers the content identification process of the first multimedia content after receiving the first interactive identifier.

[0149] Regarding the server's content recognition process for the first multimedia content, illustratively, if the server has not triggered the content recognition process for the first multimedia content before receiving the first interactive identifier, the server executes the content recognition process for the first multimedia content to obtain a recognition result (such as the first work described below); if the server has triggered the content recognition process for the first multimedia content before receiving the first interactive identifier, the server directly obtains the currently saved recognition result for the first multimedia content, and then feeds back the recognition result to the terminal (that is, to the client of the first application or the client of the second application).

[0150] Optionally, a sending operation on the first interactive information in the interactive area is received as an interactive operation; or, an approval operation on the first interactive information in the interactive area is received as an interactive operation; or, a forwarding operation on the first interactive information in the interactive area is received as an interactive operation.

[0151] Step 430: Based on the interactive operation, display the work information of the first work corresponding to the first multimedia content, and add the first work corresponding to the first multimedia content to the work storage list.

[0152] The first work is a work obtained after content recognition is performed on the first multimedia content. Optionally, the first work includes at least one of a literary work, an audio work, a film and television work, a painting work, etc.

[0153] Optionally, the playback interface is an interface provided by the first application, and based on the interactive operation, the work information of the first work corresponding to the first multimedia content is displayed on the client of the first application.

[0154] Illustratively, after the interactive operation is triggered, the server performs content recognition on the first multimedia content, and after identifying the first work, the server sends the work information of the first work to the client of the first application.

[0155] Optionally, the work information of the first work includes the work title, creator, creation time, work type, work introduction, work duration, etc. of the first work. Different types of works have different work information, which is not limited here. Illustratively, if the first work is a song, its corresponding work information may include the song title, song creator, song duration, etc.

[0156] Optionally, the method for displaying the work information of the first work corresponding to the first multimedia content includes at least one of the following methods:

[0157] (1) Displaying reply information to the first interactive information in the interactive area.

[0158] The reply information includes work information of the first work corresponding to the first multimedia content.

[0159] Illustratively, if a user sends a first interactive message in the comment area of ​​the playback interface, the client of the first application can display the work information of the first work to the user in the form of a reply, for example: replying to the first interactive message of the first account in the name of the target account, and the reply content of the target account includes the work information of the first work.

[0160] (2) Displaying work information of the first work corresponding to the first multimedia content on the conversation interface.

[0161] Optionally, the session interface is an interface provided by a client of the first application.

[0162] The conversation interface is used to display conversation messages between the first account and the target account, and the target account is associated with the first interaction identifier.

[0163] Illustratively, if a user sends a first interactive message in the comment area or barrage area of ​​the playback interface, the client of the first application can display the work information of the first work to the user in the form of a private message, for example: sending a private message to the first account in the name of the target account, and displaying the work information of the first work in the conversation interface between the first account and the target account.

[0164] Among them, the above-mentioned target account is associated with the first identification information. In some embodiments, the first interactive identifier can be implemented as a reminder identifier for the target account, for example: "@target account name". Optionally, for the target account: the target account refers to the account registered by the target platform in the first application. If the work storage list described below is a storage list provided by the first application, the target platform is the first application, for example: the target account is the official account of the first application; if the work storage list described below is a storage list provided by the second application, the target platform is the second application, for example: the target account is the official account registered by the second application in the first application.

[0165] Illustratively, after the server identifies the first work, it also triggers a process of adding the first work to the work storage list.

[0166] Optionally, the above-mentioned work storage list is a list provided by the second application, and based on the interactive operation, the first work corresponding to the first multimedia content is added to the work storage list in the client of the second application.

[0167] The current terminal has a client of the second application installed, and a second account is logged in to the client of the second application. In some embodiments, when there is an association relationship between the first account and the second account, the first work corresponding to the first multimedia content is automatically added to the work storage list in the client of the second application. When there is no association relationship between the first account and the second account, the work extraction information corresponding to the first work is displayed on the client of the first application; when the work extraction information is used on the client of the second application, the first work corresponding to the first multimedia content is automatically added to the work storage list in the client of the second application.

[0168] It should be noted that, in the process of automatically adding the first work corresponding to the first multimedia content to the work storage list in the client of the second application, the current terminal can jump to the client of the second application. After the addition is successful, the list interface corresponding to the work storage list is displayed on the client of the second application, and the list interface includes the first work; or, in the process of automatically adding the first work corresponding to the first multimedia content to the work storage list in the client of the second application, it is not adjusted to the client of the second application, and the playback interface of the first multimedia content is still displayed. After the addition is successful, a successful addition notification is displayed on the client of the first application, and the successful addition notification is used to notify that the first work has been successfully added to the work storage list in the client of the second application. Among them, the form of the successful addition notification includes at least one of a pop-up notification, a floating icon notification, a push message, etc., which is not limited here.

[0169] In some embodiments, when the first multimedia content is played, the client of the second application is automatically started; the first work is displayed on the client of the second application. When the first multimedia content is not played, a successful addition notification is displayed on the client of the first application, and the successful addition notification is used to notify that the first work has been successfully added to the work storage list in the client of the second application.

[0170] Illustratively, when the playback progress of the first multimedia content reaches the total duration of the multimedia content, it is determined to be in the playback end state; when the playback progress of the first multimedia content does not reach the total duration of the multimedia content, it is determined to be in the playback not end state.

[0171] Among them, displaying the first work refers to displaying the content of the first work, such as: playing a song, opening the homepage of a novel, playing a video, etc., which is not limited here.

[0172] For illustration, the first work is implemented as a song as an example. When the first multimedia content is finished playing, if the song is successfully added to the work storage list, the second application is automatically started and the successfully added song is played from the work storage list of the second application; when the first multimedia content is not finished playing, if the song is successfully added to the work storage list, a successful addition notification is displayed on the client of the first application, and the successful addition notification is used to notify that the song has been successfully added to the work storage list in the client of the second application.

[0173] In the above solution, when the first multimedia content is finished playing, the second application is automatically started and the first work is displayed, providing users with a seamless content experience. Before the first multimedia content is finished playing, if the first work is successfully added to the work storage list of the second application, a successful addition notification can be displayed in time on the client of the first application. This allows users to immediately know the completion of the adding operation, avoiding the need for users to perform additional operations to know the completion of the adding operation, and improving the efficiency of human-computer interaction.

[0174] In some embodiments, the above mentioned successful adding notification also includes an application jump entry, which is used to automatically start the client of the second application and display the first work on the client of the second application. In an exemplary manner, an application jump control is displayed in the successful adding notification, and the user clicks the application adjustment control to automatically start the client of the second application and display the first work on the client of the second application.

[0175] In summary, the interactive method based on multimedia content provided by the embodiment of the present application enables the user to trigger the system to identify the multimedia content and determine the corresponding first work through interactive operations, thereby avoiding the cumbersome process of the user determining the first work through manual search, inquiry, etc. After identifying the first work, the system can not only automatically display the work information of the first work to the user, allowing the user to more conveniently obtain the work information related to the multimedia content, but also automatically add the first work to the storage list, eliminating the user's manual operation steps of adding the first work to the storage list, thereby improving the efficiency of human-computer interaction.

[0176] In some embodiments, the first work includes a first song, which is a song obtained by identifying the background music of the first multimedia content; and the work storage list includes a first song list. Figure 5 , the above Figure 3 or Figure 4 The illustrated embodiment may also be implemented as the following steps 501 to 504 .

[0177] Step 501: Displaying a playback interface of a first multimedia content on a client of a first application.

[0178] The play interface includes an interactive area, and the interactive area is used to display interactive information published for the first multimedia content. The first multimedia content includes background music.

[0179] Optionally, the first application may be implemented as a video playback platform, and the first multimedia content may be implemented as a first video content. Schematically, a playback interface of the first video content is displayed on a client of the video playback platform, and the first video content includes background music.

[0180] Among them, a first account is logged in to the client of the first application.

[0181] Step 502: In response to receiving a sending operation for first interactive information in an interactive area, display the first interactive information in the interactive area.

[0182] The first interactive information includes a first interactive identifier, and the first interactive identifier is used to identify the background music of the first multimedia content.

[0183] Illustratively, when the first account triggers the sending operation of the first interactive information on the client of the first application, the client of the first application first sends the first interactive information to the server. On the one hand, the server publishes the first interactive information to the interactive area in the client of the first application, so that the interactive area displays the first interactive information. On the other hand, the server triggers the recognition process of the first interactive information. When recognizing that there is a first interactive identifier in the first interactive information, the server will recognize the background music of the first multimedia content. The process of the server recognizing the background music of the first multimedia content is introduced below:

[0184] (1) Obtain background music of the first multimedia content.

[0185] Illustratively, the server extracts the audio part from the first multimedia content. Taking the first multimedia content as the first video content as an example, the server separates the audio stream in the first video content by an audio extraction algorithm to obtain the audio part.

[0186] Among them, the audio part may include multiple sound elements, such as: background music, character dialogue, environmental sound effects, etc. At this time, audio separation technology is needed to separate the background music from the audio part. Audio separation technology includes spectrum subtraction based on signal processing, independent component analysis, etc., which are not limited here.

[0187] (2) Extract the audio feature representation corresponding to the background music.

[0188] Illustratively, the server performs feature extraction on the acquired background music to obtain an audio feature representation corresponding to the background music, wherein the audio feature representation includes but is not limited to a spectral feature representation of the audio (such as Mel-frequency cepstral coefficients), a rhythm feature representation (such as beats, rhythm intensity, etc.), a timbre feature representation (such as harmonic structure, resonance peaks, etc.), etc., which are not limited here.

[0189] (3) Match the audio feature representation corresponding to the background music with the song database to obtain the recognition result.

[0190] Illustratively, the song work database is a database provided by the client of the second application, and the song work database includes audio feature representations corresponding to multiple songs. The audio feature representation corresponding to the background music is matched one by one with the multiple audio feature representations in the song work database, such as: for each audio feature representation corresponding to the song in the song work database, the similarity between the audio feature representation and the audio feature representation corresponding to the background music is calculated, and the similarity includes Euclidean distance, cosine similarity, etc., which are not limited here.

[0191] After determining the similarities between multiple audio feature representations in the song work database and the audio feature representations corresponding to the background music, the song corresponding to the audio feature representation whose similarity is greater than or equal to the preset similarity is taken as the recognition result. If there are multiple audio feature representations whose similarity is greater than or equal to the preset similarity, the song corresponding to the audio feature representation with the highest similarity is taken as the recognition result.

[0192] Step 503: Based on the first interactive information, the song information of the first song corresponding to the first multimedia content is displayed on the client of the first application.

[0193] Optionally, when the server identifies that the background music of the first multimedia content is the first song, the server will send the song information of the first song to the client of the first application. After the client of the first application receives the song information of the first song, the server will display the reply information of the target account to the first interactive information in the interactive area, and the reply information will include the song information of the first song; or, the reply information of the first interactive information will be displayed on the session interface provided by the client of the first application.

[0194] The first interactive identifier can be implemented as a reminder identifier for the target account, for example: "@target account name". Optionally, the target account refers to an account registered by the second application in the first application, for example: the target account is an official account registered by the second application in the first application.

[0195] Step 504: Add the first song to the first playlist provided by the client of the second application.

[0196] Among them, a second account is logged in to the client of the second application, and the second application can be implemented as a music playback platform.

[0197] Optionally, when the server identifies that the background music of the first multimedia content is the first song, the server will also add the first song to the first playlist provided by the client of the second application.

[0198] In some embodiments, when the first account and the second account are associated, the first song is automatically added to the first song list in the client of the second application. When the first account and the second account are not associated, the work extraction information corresponding to the first song is displayed on the client of the first application; when the song extraction information is used on the client of the second application, the first song is automatically added to the first song list in the client of the second application.

[0199] In some embodiments, the first interactive information also includes storage indication information, and the storage indication information indicates that the first song obtained by identifying the first multimedia content is added to the first playlist.

[0200] Illustratively, the storage indication information includes the name, identification number, etc. of the first playlist, which is not limited here. For example, the storage indication information is: add background music to the "I like" playlist. When the background music of the first multimedia content is identified as the first song, the first song will be added to the "I like" playlist in the client of the second application, such as adding the relevant icon (such as album cover) and name of the first song to the "I like" playlist.

[0201] In some embodiments, a first song corresponding to a first multimedia content is added to a first playlist in a client of a second application; wherein a second account in the client of the second application corresponds to multiple candidate playlists; when multiple candidate playlists match the first song, the first playlist is a playlist determined from the multiple candidate playlists; when multiple candidate playlists do not match the first song, the first playlist is a playlist created based on the song information of the first song and the first multimedia content.

[0202] Illustratively, after determining that the background music of the first multimedia content includes the first song, the server will obtain multiple candidate playlists associated with the second account in the client of the second application through an interface with the second application or by querying a relevant database, for example: a playlist created in the second account.

[0203] The server will extract the song feature representation corresponding to the first song from the song information of the first song. The song information of the first song includes but is not limited to: the name of the song, singer information, song style (such as rock, pop, classical, etc.), song length, rhythm characteristics of the song (such as average number of beats, rhythm change curve, etc.), the emotional tone of the song (cheerful, sad, lyrical, etc.) and the release year of the song.

[0204] For each candidate playlist, its playlist feature representation is extracted. These features include the playlist name, playlist creator information, playlist theme tags (such as "classic old songs", "popular songs", etc.), the average style characteristics of the songs included in the playlist (such as the overall music style tendency, average duration range, etc.), the creation time and update frequency of the playlist, etc. Through comprehensive analysis and processing of this information, the playlist feature representation of each candidate playlist is constructed.

[0205] Using a suitable matching algorithm (such as a similarity calculation algorithm based on a vector space model, etc.), the song feature representation of the first song is matched with the playlist feature representations corresponding to the multiple candidate playlists one by one to obtain the matching degree between the first song and the multiple candidate playlists.

[0206] If there is a single candidate song list whose matching degree with the first song is greater than a preset matching threshold, the single candidate song list will be used as the first song list.

[0207] If there are multiple candidate playlists whose matching degree with the first song is greater than the preset matching threshold, the first playlist is determined according to the priority rule from the multiple candidate playlists whose matching degree is greater than the preset matching threshold. The priority rule can be determined based on factors such as the frequency of use of the playlist (such as the playlist with the most recent play times is given priority), the matching degree between the playlist and the first song (the highest matching degree is given priority), and the number of collections of the playlist (the one with the most collections is given priority). For example, from the multiple candidate playlists whose matching degree is greater than the preset matching threshold, the playlist with the most recent play times and the largest number of collections is selected as the first playlist.

[0208] If there is no candidate playlist whose matching degree with the first song is greater than a preset matching threshold, a new playlist can be created as the first playlist based on the song information of the first song and the first multimedia content. Schematically, key information of the first song is extracted, such as the name of the song, the name of the singer, and the main style category (such as rock, pop, classical, jazz, etc.) to which the song belongs, and key information of the first multimedia content is extracted, such as the title and type of the first multimedia content, and a playlist name is generated based on at least one of the key information of the first song and the key information of the first multimedia content, and the first playlist is created for the second account with the playlist name, and the name of the first playlist is also the playlist name.

[0209] After determining the first playlist, the server will send a work collection request to the client of the second application. After receiving the work collection request, the client of the second application will add the relevant information of the first song (such as the audio file storage path of the song, the metadata information of the song, etc.) to the database record or storage structure corresponding to the first playlist, thereby completing the operation of adding the first song.

[0210] In the above embodiment, by extracting the song feature representation of the first song and the song list feature representation of each candidate song list in detail and applying the matching algorithm to calculate, the matching degree between the first song and the candidate song list can be accurately determined. When there is a candidate song list with a high matching degree, the first song is added to a suitable existing song list, making the collection of songs more reasonable; when there is no candidate song list matching the first song, a new song list is automatically created based on the song information of the first song and the first multimedia content, without the need for the user to create a song list by himself, thereby improving the efficiency of human-computer interaction.

[0211] In summary, the interactive method based on multimedia content provided by the embodiment of the present application enables the user to trigger the system to identify the multimedia content and determine the corresponding first song through interactive operations, thereby avoiding the cumbersome process of the user determining the first song through manual search, inquiry, etc. After identifying the first song, the system can not only automatically display the song information of the first song to the user, allowing the user to more conveniently obtain the work information related to the multimedia content, but also automatically add the first song to the playlist, eliminating the user's manual operation steps of adding the first song to the playlist, thereby improving the efficiency of human-computer interaction.

[0212] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0213] Please refer to Figure 6 , which shows a block diagram of a multimedia content-based interactive device provided by an embodiment of the present application. The device has the functions of implementing the above examples, and the functions can be implemented by hardware or by hardware executing corresponding software. Figure 6 As shown, the device comprises:

[0214] A display module 610 is used to display a playback interface of the first multimedia content, wherein the playback interface includes an interactive area, and the interactive area is used to display interactive information published for the first multimedia content;

[0215] A receiving module 620, configured to receive an interactive operation for first interactive information in the interactive area, wherein the first interactive information includes a first interactive identifier, and the first interactive identifier is used to identify the content of the first multimedia content;

[0216] The processing module 630 is used to add a first work corresponding to the first multimedia content to a work storage list based on the interactive operation, where the first work is obtained after content recognition is performed on the first multimedia content.

[0217] In some embodiments, the playback interface is an interface provided by a first application, and the work storage list is a storage list provided by a second application; the display module 610 is used to display the playback interface of the first multimedia content on a client of the first application, and a first account is logged in to the client of the first application; the processing module 630 is used to add the first work corresponding to the first multimedia content to the work storage list in the client of the second application based on the interactive operation, and a second account is logged in to the client of the second application.

[0218] In some embodiments, the processing module 630 is used to automatically add the first work corresponding to the first multimedia content to the work storage list in the client of the second application when the first account and the second account are associated with each other.

[0219] In some embodiments, the processing module 630 is used to display the work extraction information corresponding to the first work on the client of the first application when there is no association relationship between the first account and the second account; when the work extraction information is used on the client of the second application, the first work corresponding to the first multimedia content is automatically added to the work storage list in the client of the second application.

[0220] In some embodiments, the first work includes a first song, which is a song obtained by identifying the background music of the first multimedia content; the work collection list includes a first song list; the processing module 630 is used to add the first song corresponding to the first multimedia content to the first song list in the client of the second application; wherein the second account in the client of the second application corresponds to a plurality of candidate song lists; when the plurality of candidate song lists match the first song, the first song list is a song list determined from the plurality of candidate song lists; when the plurality of candidate song lists do not match the first song, the first song list is a song list created based on the song information of the first song and the first multimedia content.

[0221] In some embodiments, the processing module 630 is used to automatically start the client of the second application when the playback of the first multimedia content is completed; display the first work on the client of the second application; and display a successful addition notification on the client of the first application before the playback of the first multimedia content is completed, wherein the successful addition notification is used to notify that the first work has been successfully added to the work storage list in the client of the second application.

[0222] In some embodiments, the display module 610 is used to display work information of the first work corresponding to the first multimedia content based on the interactive operation.

[0223] In some embodiments, the first interactive information also includes storage indication information, and the storage indication information indicates that the first work obtained by identifying the first multimedia content is added to a first work storage list; the processing module 630 is used to add the first work corresponding to the first multimedia content to the first work storage list based on the interactive operation, and the first work storage list is a list determined according to the storage indication information.

[0224] In some embodiments, the works obtained after content recognition of the first multimedia content include a first work and a second work; the processing module 630 is used to add the first work corresponding to the first multimedia content to the second work storage list and the second work corresponding to the first multimedia content to the third work storage list based on the interactive operation; wherein the first work matches the type of stored works corresponding to the second work storage list, and the second work matches the type of stored works corresponding to the third work storage list.

[0225] In some embodiments, the processing module 630 is used to obtain multiple candidate work storage lists based on the interactive operation; extract list feature representations corresponding to the multiple candidate work storage lists respectively, and the list feature representations are used to characterize the types of stored works in the candidate work storage lists; wherein the list feature representation is a feature representation determined according to the stored work information and list attribute information of the candidate work storage list, and the stored work information refers to the information of the works stored in the candidate work storage list; extract a first work feature representation of the first work corresponding to the first multimedia content, and extract a second work feature representation of the second work corresponding to the first multimedia content; if the first work feature representation matches the list feature representation corresponding to the first candidate work storage list in the multiple candidate work storage lists, determine the first candidate work storage list as the second work storage list; if the first work feature representation matches the list feature representation corresponding to the second candidate work storage list in the multiple candidate work storage lists, determine the second candidate work storage list as the third work storage list; add the first work to the second work storage list, and add the second work to the third work storage list.

[0226] In some embodiments, the first interactive information also includes content identification prompt information, and the content identification prompt information is used to indicate content identification of the first multimedia segment in the first multimedia content; the processing module 630 is used to add the first work corresponding to the first multimedia segment in the first multimedia content to the work storage list based on the interactive operation, and the work storage list is the works obtained after content identification of the first multimedia segment.

[0227] To summarize, in the multimedia content-based interactive device provided by the embodiment of the present application, in the multimedia content playback interface, the user only needs to trigger an interactive operation for the first interactive information (including the first interactive identifier), and the system can automatically complete the multimedia content recognition process and add the identified first work to the work storage list. On the one hand, the user can trigger the system to identify the multimedia content through the interactive operation to determine the corresponding first work, thereby avoiding the tedious process of the user determining the first work through manual search, inquiry, etc.; on the other hand, after identifying the first work, the system can automatically add the first work to the storage list, thereby eliminating the user's manual operation steps of adding the first work to the storage list, so that the user can more conveniently obtain and save works of interest when browsing multimedia content, thereby improving the efficiency of human-computer interaction.

[0228] Please refer to Figure 7 , which shows a structural block diagram of a computer device 700 provided in one embodiment of the present application. The computer device 700 may be Figure 1 The terminal shown can also be Figure 1 Specifically:

[0229] Typically, the computer device 700 includes a processor 710 and a memory 720 .

[0230] The processor 710 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 710 may be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor 710 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 710 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 710 may also include an AI processor, which is used to process computing operations related to machine learning.

[0231] The memory 720 may include one or more computer-readable storage media, which may be non-transitory. The memory 720 may also include a high-speed random access memory and a non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 720 is used to store a computer program, which is configured to be executed by one or more processors to implement the above-mentioned interactive method based on multimedia content.

[0232] Those skilled in the art will understand that Figure 7 The structure shown in the figure does not constitute a limitation on the computer device 700, and the computer device 700 may include more or less components than shown in the figure, or combine some components, or adopt a different arrangement of components.

[0233] In an exemplary embodiment, a computer-readable storage medium is also provided, in which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned interactive method based on multimedia content is implemented. Optionally, the computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a solid state drive (SSD) or an optical disk, etc. Among them, the random access memory may include a resistance random access memory (ReRAM) and a dynamic random access memory (DRAM).

[0234] In an exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program, the computer program being stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above-mentioned interactive method based on multimedia content.

[0235] It should be understood that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. In addition, the step numbers described in this article only illustrate a possible execution sequence between the steps. In some other embodiments, the above steps may not be executed in the order of the numbers, such as two steps with different numbers are executed at the same time, or two steps with different numbers are executed in the opposite order to the diagram. The embodiments of the present application are not limited to this.

[0236] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. An interactive method based on multimedia content, characterized in that: The method comprises: Displaying a playback interface of the first multimedia content, wherein the playback interface includes an interactive area, and the interactive area is used to display interactive information published for the first multimedia content; receiving an interactive operation for first interactive information in the interactive area, wherein the first interactive information includes a first interactive identifier, and the first interactive identifier is used to identify the content of the first multimedia content; Based on the interactive operation, a first work corresponding to the first multimedia content is added to a work storage list, where the first work is obtained after content recognition is performed on the first multimedia content.

2. The method according to claim 1, characterized in that The playback interface is an interface provided by the first application, and the work storage list is a storage list provided by the second application; The playing interface for displaying the first multimedia content includes: Displaying a playback interface of the first multimedia content on a client of the first application, wherein a first account is logged in to the client of the first application; The adding, based on the interactive operation, the first work corresponding to the first multimedia content to the work storage list includes: Based on the interactive operation, the first work corresponding to the first multimedia content is added to a work storage list in a client of the second application, and a second account is logged in to the client of the second application.

3. The method according to claim 2, characterized in that The step of adding the first work corresponding to the first multimedia content to a work storage list in the client of the second application includes: In the case where the first account and the second account are associated with each other, the first work corresponding to the first multimedia content is automatically added to the work storage list in the client of the second application.

4. The method according to claim 2, characterized in that: The step of adding the first work corresponding to the first multimedia content to a work storage list in the client of the second application includes: In the case where there is no association relationship between the first account and the second account, displaying work extraction information corresponding to the first work on the client of the first application; When the work is used to extract information on the client of the second application, the first work corresponding to the first multimedia content is automatically added to the work storage list in the client of the second application.

5. The method according to claim 2, characterized in that: The first work includes a first song, which is a song obtained by identifying the background music of the first multimedia content; the work storage list includes a first song list; The step of adding the first work corresponding to the first multimedia content to a work storage list in the client of the second application includes: Adding the first song corresponding to the first multimedia content to the first playlist in the client of the second application; Among them, the second account in the client of the second application corresponds to multiple candidate playlists; when the multiple candidate playlists match the first song, the first playlist is a playlist determined from the multiple candidate playlists; when the multiple candidate playlists do not match the first song, the first playlist is a playlist created based on the song information of the first song and the first multimedia content.

6. The method according to claim 2, characterized in that After adding the first work corresponding to the first multimedia content to the work storage list in the client of the second application based on the interactive operation, the method further includes: When the first multimedia content is finished playing, automatically start the client of the second application; and display the first work on the client of the second application; When the first multimedia content has not been played to completion, a successful adding notification is displayed on the client of the first application, where the successful adding notification is used to notify that the first work has been successfully added to the work storage list in the client of the second application.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Based on the interactive operation, work information of the first work corresponding to the first multimedia content is displayed.

8. The method according to any one of claims 1 to 6, characterized in that: The first interactive information further includes storage instruction information, wherein the storage instruction information indicates that the first work obtained by identifying the first multimedia content is added to a first work storage list; The adding, based on the interactive operation, the first work corresponding to the first multimedia content to the work storage list includes: Based on the interactive operation, the first work corresponding to the first multimedia content is added to the first work storage list.

9. The method according to any one of claims 1 to 6, characterized in that: The works obtained after content recognition of the first multimedia content include a first work and a second work; The adding, based on the interactive operation, the first work corresponding to the first multimedia content to the work storage list includes: Based on the interactive operation, adding the first work corresponding to the first multimedia content to the second work storage list, and adding the second work corresponding to the first multimedia content to the third work storage list; The first work matches the type of stored works corresponding to the second work storage list, and the second work matches the type of stored works corresponding to the third work storage list.

10. The method according to claim 9, characterized in that The adding, based on the interactive operation, the first work corresponding to the first multimedia content to the second work storage list and the adding the second work corresponding to the first multimedia content to the third work storage list comprises: Based on the interactive operation, a plurality of candidate works storage lists are obtained; Extracting list feature representations corresponding to a plurality of candidate work storage lists respectively, wherein the list feature representations are used to characterize the types of stored works in the candidate work storage lists; wherein the list feature representations are feature representations determined based on stored work information and list attribute information of the candidate work storage lists, and the stored work information refers to information about the works stored in the candidate work storage lists; extracting a first work feature representation of a first work corresponding to the first multimedia content, and extracting a second work feature representation of a second work corresponding to the first multimedia content; In a case where the first work feature representation matches a list feature representation corresponding to a first candidate work storage list in the plurality of candidate work storage lists, determining the first candidate work storage list as the second work storage list; In a case where the first work feature representation matches a list feature representation corresponding to a second candidate work storage list in the plurality of candidate work storage lists, determining the second candidate work storage list as the third work storage list; The first work is added to the second work storage list, and the second work is added to the third work storage list.

11. The method according to any one of claims 1 to 6, characterized in that: The first interactive information further includes content recognition prompt information, where the content recognition prompt information is used to instruct content recognition of the first multimedia segment in the first multimedia content; The adding, based on the interactive operation, the first work corresponding to the first multimedia content to the work storage list includes: Based on the interactive operation, a first work corresponding to the first multimedia segment in the first multimedia content is added to the work storage list, where the work storage list is the works obtained after content recognition of the first multimedia segment.

12. An interactive device based on multimedia content, characterized in that: The device comprises: A display module, configured to display a playback interface of the first multimedia content, wherein the playback interface includes an interactive area, and the interactive area is configured to display interactive information published for the first multimedia content; A receiving module, configured to receive an interactive operation for first interactive information in the interactive area, wherein the first interactive information includes a first interactive identifier, and the first interactive identifier is used to identify the content of the first multimedia content; The processing module is used to add a first work corresponding to the first multimedia content to a work storage list based on the interactive operation, where the first work is obtained after content recognition is performed on the first multimedia content.

13. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one program, and the at least one program is loaded and executed by the processor to implement the multimedia content-based interactive method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The storage medium stores at least one program, and the at least one program is loaded and executed by the processor to implement the multimedia content-based interactive method as described in any one of claims 1 to 11.

15. A computer program product, characterized in that It comprises a computer program, which, when executed by a processor, implements the interactive method based on multimedia content as claimed in any one of claims 1 to 11.

Images